Vehicle control system, vehicle control device, and vehicle control method

Interior sensors are used to determine environmental states and perform feedback control, addressing exterior sensor limitations and enhancing driving support functions in vehicle control systems.

WO2026239896A1PCT designated stage Publication Date: 2026-11-19SONY SEMICON SOLUTIONS CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/016414
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-04-24
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing vehicle control systems face challenges in providing sufficient driving support functions due to deteriorated performance of exterior sensors, such as those affected by attachable matters on lenses, and limited data acquisition from exterior sensors, leading to insufficient environmental information.

Method used

The system utilizes interior sensors to acquire sensing information, determines the environmental state outside the vehicle, and performs feedback control based on this information to enhance driving support functions.

Benefits of technology

Enhances driving support functions by compensating for exterior sensor deficiencies, ensuring robust environmental state recognition and control, even in adverse conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026016414_19112026_PF_FP_ABST
    Figure JP2026016414_19112026_PF_FP_ABST
Patent Text Reader

Abstract

To realize a more suitable driving support function, an in-vehicle system that controls a vehicle including an in-cabin sensor includes circuitry configured to acquire sensing information based on data acquired by the in-cabin sensor installed in a vehicle interior, determine an environmental state outside the vehicle based on the sensing information, and perform feedback control related to traveling of the vehicle according to the determined environmental state.
Need to check novelty before this filing date? Find Prior Art

Description

VEHICLE CONTROL SYSTEM, VEHICLE CONTROL DEVICE, AND VEHICLE CONTROL METHOD

[0001] The present technology relates to a vehicle control system, a vehicle control device, and a vehicle control method, and particularly relates to a vehicle control system, a vehicle control device, and a vehicle control method capable of realizing a more suitable driving support function.

[0002] <CROSS REFERENCE TO RELATED APPLICATIONS> This application claims the benefit of Japanese Priority Patent Application JP 2025-081695 filed on May 15, 2025, the entire contents of which are incorporated herein by reference.

[0003] In recent years, in order to realize automatic emergency braking (AEB), adaptive cruise control (ACC), and the like as a driving support function for a vehicle such as an automobile, a peripheral monitoring sensor may be mounted on the vehicle.

[0004] PTL 1 discloses a vehicle control device that executes one or more driving support functions on the basis of detection signals received from a plurality of types of sensors that detect a target around a vehicle.

[0005] JP 2024-49862 A

[0006] The function of a vehicle exterior sensor that senses the surroundings of the vehicle may temporarily deteriorate due to, for example, an attachable matter attached to a lens or the like. Furthermore, the data acquired by the vehicle exterior sensor is limited according to the number and positions of the vehicle exterior sensors installed in the vehicle. As described above, in a case where sufficient information cannot be acquired only by the vehicle exterior sensor, it is assumed that a sufficient driving support function cannot be provided.

[0007] The present technology is achieved in view of such circumstances, and an object thereof is to realize a more suitable driving support function.

[0008] A vehicle control system according to the present technology is a vehicle control system including circuitry configured to acquire sensing information based on data acquired by a sensor installed in a vehicle interior, determine an environmental state outside the vehicle based on the sensing information, and perform feedback control related to traveling of the vehicle according to the determined environmental state.

[0009] A vehicle control device according to the present technology is a vehicle control device including circuitry configured to acquire sensing information based on data acquired by a sensor installed in a vehicle interior, determine an environmental state outside the vehicle based on the sensing information, and perform feedback control related to traveling of the vehicle according to the determined environmental state.

[0010] A vehicle control method of the present technology is a vehicle control method including acquiring sensing information based on data acquired by a sensor installed in a vehicle interior; determining an environmental state outside the vehicle based on the sensing information; and performing feedback control related to traveling of the vehicle according to the determined environmental state.

[0011] In the present technology, sensing information is acquired on the basis of data acquired by a sensor installed in a vehicle interior, deterioration in recognition performance due to an environmental state outside the vehicle is determined on the basis of the sensing information, and feedback control related to traveling of the vehicle is performed according to a determination result of deterioration in the recognition performance.

[0012] Fig. 1 is a block diagram illustrating a configuration example of a vehicle control system.Fig. 2 is a diagram illustrating an example of a sensing area of an external recognition sensor of the vehicle control system of Fig. 1.Fig. 3 is a diagram for explaining sensing information that can be acquired from inside the vehicle.Fig. 4 is a block diagram illustrating a functional configuration example of a vehicle control system to which the present technology is applied.Fig. 5 is a flowchart for explaining an outline of an operation of the vehicle control system.Fig. 6 is a diagram illustrating an example of an environmental state estimated on the basis of sensing information.Fig. 7 is a diagram for explaining determination of intense light.Fig. 8 is a diagram for explaining estimation of a rainfall amount.Fig. 9 is a diagram illustrating an example of a correspondence relationship between a recognition situation and an environment type for each feature.Fig. 10 is a diagram illustrating an example of feedback control for deterioration in recognition performance.Fig. 11 is a block diagram illustrating a configuration example of a computer.

[0013] Hereinafter, a mode for carrying out the present technology (hereinafter referred to as an embodiment) will be described. Note that the description will be given in the following order.

[0014] 1. Configuration example of vehicle control system 2. Overview of present technology 3. Sensing information that can be acquired from inside the vehicle 4. Configuration and operation outline of vehicle control system according to present technology 5. Example of environmental state 6. Example of feedback control 7. Configuration example of computer

[0015] <1. Configuration example of vehicle control system> Fig. 1 is a block diagram illustrating a configuration example of a vehicle control system 11 that is a non-limiting example of a mobile device control system to which the present technology is applied.

[0016] The vehicle control system 11 is installed in a vehicle 1, and performs processing related to driving automation of the vehicle 1. This driving automation includes driving automation of levels 1 to 5, and remote driving and / or remote support of the vehicle 1 by a remote driver. The level of driving automation may refer to the Society of Automotive Engineers (SAE) J3016TM APL2021 Levels of Driving Automation, where SAE Level 0 refers to the lowest level of driving automation and SAE Level 5 refers to the highest level of driving automation. For example, the driving automation of SAE Level 1 may consist of a driver support feature that provides steering or brake / acceleration support to the driver, and the driving automation of SAE Level 5 may consist of an autonomous driving feature that allows the vehicle to be driven under all conditions.

[0017] The vehicle control system 11 includes a vehicle control electronic control unit (ECU) 21, a communication unit 22, a map information accumulation unit 23, a position information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a storage unit 28, a driving automation control unit 29, a driver monitoring system (DMS) 30, a human machine interface (HMI) 31, and a vehicle control unit 32.

[0018] Two or more (or in some cases all) of the vehicle control ECU 21, the communication unit 22, the map information accumulation unit 23, the position information acquisition unit 24, the external recognition sensor 25, the in-vehicle sensor 26, the vehicle sensor 27, the storage unit 28, the driving automation control unit 29, the DMS 30, the HMI 31, and the vehicle control unit 32 are communicably connected to each other via a communication network 41. The communication network 41 includes, for example, an in-vehicle communication network, a bus, or the like conforming to a digital bidirectional communication standard such as a controller area network (CAN), a local interconnect network (LIN), a local area network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark). In some embodiments, the communication network 41 may have two or more types of communication network, and different types of communication networks may be used depending on the type of data to be transmitted. For example, the CAN may be applied to data related to vehicle control, and the Ethernet may be applied to large-volume data. In some embodiments, two or more (or in some cases all) units of the vehicle control system 11 may be directly connected by using wireless communication (for example, communication at a relatively short distance) without going through the communication network 41. In some embodiments, the wireless communication may use a near field communication technology. Non-limiting examples of near field communication technologies include near field communication (NFC) and Bluetooth (registered trademark). In some embodiments, two or more (or in some cases all) units of the vehicle control system 11 may be connected using the communication network 41 and a wireless communication technology (for example, near field communication technology).

[0019] Hereinafter, in the embodiment where two or more components of the vehicle control system 11 perform communication via the communication network 41, the description of the communication network 41 will be omitted. For example, in the embodiment where the vehicle control ECU 21 and the communication unit 22 perform communication via the communication network 41, it will be simply described that the vehicle control ECU 21 and the communication unit 22 perform communication.

[0020] For example, the vehicle control ECU 21 includes various processors such as a central processing unit (CPU) and a micro processing unit (MPU). The vehicle control ECU 21 controls all or some of the functions of the vehicle control system 11.

[0021] The communication unit 22 communicates with various devices (hereinafter, referred to as in-vehicle devices) inside the vehicle 1, various devices (hereinafter, referred to as external devices) outside the vehicle 1, other vehicles, base stations, and the like, and transmits and receives various data. In some embodiments, the communication unit 22 may perform communication using a plurality of communication technologies.

[0022] A non-limiting example of communication between the communication unit 22 and an external device will be schematically described. In some embodiments, the communication unit 22 may communicate with a server (hereinafter, referred to as an external server) or the like existing on an external network via a base station or an access point by a wireless communication technology. Examples of non-limiting wireless communication technologies include 5G (5th generation mobile communication system), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), and the like. Examples of the external network with which the communication unit 22 can communicate include the Internet, a cloud network, a company-specific network, or the like. The communication technology used by the communication unit 22 that communicates with the external network is not particularly limited as long as it is a wireless communication technology capable of performing digital bidirectional communication at a communication speed equal to or higher than a predetermined speed and at a distance equal to or longer than a predetermined distance.

[0023] In some embodiments, the communication unit 22 may communicate with a terminal present in the vicinity of the own vehicle by using a peer to peer (P2P) technology. Examples of the terminal present in the vicinity of the own vehicle include a terminal attached to a mobile object moving at a relatively low speed such as a pedestrian or a bicycle, a terminal installed with a fixed position in a store or the like, and / or a machine type communication (MTC) terminal. In some embodiments, the communication unit 22 may perform vehicle to everything (V2X) communication. The V2X communication generally refers to communication between an own vehicle and another entity. Non-limiting examples of the V2X communication include vehicle to vehicle communication with another vehicle, vehicle to infrastructure communication with a roadside device or the like, vehicle to home communication with home, and vehicle to pedestrian communication with a terminal or the like carried by a pedestrian or mounted thereon, for example.

[0024] In some embodiments, the communication unit 22 may receive a program for updating software for controlling the operation of the vehicle control system 11 from the outside of the vehicle 1 (for example, Over The Air). In some embodiments, the communication unit 22 may receive map information, traffic information, information around the vehicle 1, and the like from the outside of the vehicle 1. In some embodiments, the communication unit 22 may transmit information regarding the vehicle 1, information around the vehicle 1, and the like to an external device or an external network. Non-limiting examples of the information on the vehicle 1 transmitted by the communication unit 22 to the external device or the external network include data indicating the state of the vehicle 1, a recognition result by a recognition unit 73, and the like. In some embodiments, the communication unit 22 may communicate with a vehicle emergency call system. A non-limiting example of a vehicle emergency call system is an eCall.

[0025] In some embodiments, the communication unit 22 may receive an electromagnetic wave transmitted by a road traffic information communication system. In some embodiments, the electromagnetic wave may be transmitted using a radio wave beacon, an optical beacon, FM multiplex broadcasting, or the like.

[0026] A non-limiting example of communication with the in-vehicle device executable by the communication unit 22 will be schematically described. In some embodiments, the communication unit 22 may communicate with an in-vehicle device by using wireless communication. For example, in some embodiments, the communication unit 22 may perform wireless communication with an in-vehicle device by a wireless communication technology capable of performing digital bidirectional communication at a communication speed equal to or higher than a predetermined speed by wireless communication. Non-limiting examples of wireless communication technologies include wireless LAN, Bluetooth, NFC, and wireless USB (WUSB). The present invention is not limited thereto, and the communication unit 22 may communicate with in-vehicle devices by using wired communication (in addition to or as an alternative to wireless communication). For example, in some embodiments, the communication unit 22 may communicate with in-vehicle devices by wired communication via a cable connected to a connection terminal (not illustrated). In some embodiments, the communication unit 22 may communicate with an in-vehicle device by a wired communication technology capable of performing digital bidirectional communication at a communication speed equal to or higher than a predetermined speed. Non-limiting examples of the wired communication technology include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI) (registered trademark), and Mobile High-definition Link (MHL).

[0027] Here, the in-vehicle device refers to, for example, a device that is not connected to the communication network 41 inside the vehicle 1. The in-vehicle devices are divided into devices constituting the vehicle control system 11 and devices not constituting the vehicle control system. Non-limiting examples of in-vehicle devices not constituting the vehicle control system 11 include a mobile device or a wearable device possessed by a user (for example, a driver or a passenger) of the vehicle 1, an information device temporarily installed inside the vehicle 1, and the like. These devices can move to the outside of the vehicle 1, for example, to be external devices.

[0028] The map information accumulation unit 23 accumulates a map acquired from an external device or an external network and / or a map created by the vehicle 1. For example, the map information accumulation unit 23 may accumulate a three-dimensional high-precision map, a global map that is lower in precision but wider in covering area than the high-precision map, and the like.

[0029] Examples of the high-precision map include a dynamic map, a point cloud map, a vector map, and the like. The dynamic map may be a map including four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and may be provided to the vehicle 1 from the external server or the like, for example. The point cloud map may be a map including a point cloud (point cloud data). The vector map may be, for example, a map in which traffic information such as a lane and a position of a traffic light is associated with a point cloud map and adapted to driving automation.

[0030] The point cloud map and the vector map may be provided from, for example, the external server or the like, or may be created by the vehicle 1 as a map for matching with a local map to be described later on the basis of a sensing result from a camera 51, a radar 52, a LiDAR 53, or the like, and may be accumulated in the map information accumulation unit 23. Furthermore, in a case where the high-precision map is provided from the external server or the like, to reduce the communication volume, map data covering an area of several hundred meters square regarding a planned route that the vehicle 1 will follow may be acquired from the external server or the like, for example.

[0031] The position information acquisition unit 24 acquires the position information of the vehicle 1. The acquired position information may be supplied to the driving automation control unit 29. In some embodiments, the position information acquisition unit 24 may receive a global navigation satellite system (GNSS) signal from a GNSS satellite. In some embodiments, the position information acquisition unit 24 may receive a signal from a beacon or the like.

[0032] The external recognition sensor 25 includes various sensors used for recognizing a situation outside the vehicle 1, and supplies sensor data from one or more (or all in some cases) sensors to one or more (or all in some cases) units of the vehicle control system 11. The external recognition sensor 25 may include any type and any number of sensors.

[0033] In some embodiments, the external recognition sensor 25 includes the camera 51, the radar 52, the light detection and ranging or laser imaging detection and ranging (LiDAR) 53, and an ultrasonic sensor 54. Alternatively, the external recognition sensor 25 may be configured by at least one of the camera 51, the radar 52, the LiDAR 53, or the ultrasonic sensor 54. The numbers of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 are not particularly limited as long as they can be practically installed in the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include sensors of other types. An example of a sensing area of each sensor included in the external recognition sensor 25 will be described later.

[0034] Any suitable imaging method can be used for the camera 51. In some embodiments, the camera 51 may use an imaging method capable of measuring a distance. Non-limiting examples of the camera using an imaging method capable of distance measurement include a time of flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. The present disclosure is not limited thereto, and the camera 51 may be a camera for simply acquiring a captured image without distance measurement.

[0035] In some embodiments, the external recognition sensor 25 may include an environmental sensor for detecting characteristics of the environment around the vehicle 1. Non-limiting examples of environmental features that may be detected include weather, climate, brightness, and the like. In some embodiments, the environmental sensor may include various sensors such as a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and an illuminance sensor.

[0036] In some embodiments, the external recognition sensor 25 may include a microphone used for detecting a sound around the vehicle 1, a position of a sound source, or the like.

[0037] The in-vehicle sensor 26 includes various sensors for detecting information inside the vehicle 1, and supplies sensor data from one or more (or all in some cases) sensors to one or more (or all in some cases) units of the vehicle control system 11. The type and number of the various sensors included in the in-vehicle sensor 26 are not particularly limited as long as they can be practically installed in the vehicle 1.

[0038] In some embodiments, the in-vehicle sensor 26 may include one or more sensors of a camera, a radar, a seating sensor, a microphone, and a biometric sensor. In some embodiments, the camera included in the in-vehicle sensor 26 may use a distance-measurable imaging method. Non-limiting examples of the camera using a distance-measurable imaging method include a ToF camera, a stereo camera, a monocular camera, and an infrared camera. The present disclosure is not limited thereto, and the camera included in the in-vehicle sensor 26 may be a camera for simply acquiring a captured image without distance measurement. For example, the biometric sensor included in the in-vehicle sensor 26 may be provided to a seat, a steering wheel, or the like, and detect various types of biometric information of a user.

[0039] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from one or more (or all in some cases) sensors to one or more (or all in some cases) units of the vehicle control system 11. The type and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they can be practically installed in the vehicle 1.

[0040] In some embodiments, the vehicle sensor 27 may include a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and / or an inertial measurement unit (IMU) obtained by integrating these sensors. In some embodiments, the vehicle sensor 27 includes a steering angle sensor that detects a steering angle of a steering wheel, a yaw rate sensor, an accelerator sensor that detects an operation amount (for example, pedal force, pedal stroke) of an accelerator pedal, and / or a brake sensor that detects an operation amount (for example, pedal force, pedal stroke) of a brake pedal. In some embodiments, the vehicle sensor 27 may include a rotation sensor that detects the number of rotations of an engine or a motor, an air pressure sensor that detects an air pressure of a tire, a slip rate sensor that detects a slip rate of the tire, and / or a wheel speed sensor that detects a rotation speed of a wheel. In some embodiments, the vehicle sensor 27 may include a battery sensor that detects the remaining amount or temperature of the battery and / or an impact sensor capable of detecting an external impact.

[0041] The storage unit 28 includes at least one of a nonvolatile storage medium or a volatile storage medium, and stores data and a program. Non-limiting examples of the storage medium include an electrically erasable programmable read-only (EEPROM) and a random access memory (RAM), and / or a magnetic storage device such as a hard disc drive (HDD), a semiconductor storage device, an optical storage device, and a magneto-optical storage device. The storage unit 28 stores various programs and data used by one or more (or all in some cases) units of the vehicle control system 11. In some embodiments, the storage unit 28 may include an event data recorder (EDR) and a data storage system for automated driving (DSSAD), and stores information regarding the vehicle 1 before and after an event such as an accident and information acquired by the in-vehicle sensor 26.

[0042] The driving automation control unit 29 controls a driving automation function of the vehicle 1. In some embodiments, the driving automation control unit 29 may include an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0043] The analysis unit 61 performs analysis processing on the vehicle 1 and / or conditions around the vehicle 1. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and the recognition unit 73.

[0044] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 on the basis of sensor data from the external recognition sensor 25 and the high-precision map accumulated in the map information accumulation unit 23. For example, the self-position estimation unit 71 may generate a local map on the basis of sensor data from the external recognition sensor 25, and perform matching between the local map and the high-precision map to estimate the self-position of the vehicle 1. The position of the vehicle 1 may be based on, for example, the center of an axle between rear wheels.

[0045] In some embodiments, the local map may include a three-dimensional high-precision map created by using a technology such as simultaneous localization and mapping (SLAM), an occupancy grid map, and the like. Examples of the three-dimensional high-precision map may include the above-described point cloud map and the like. The occupancy grid map may be a map in which a three-dimensional or two-dimensional space around the vehicle 1 is divided into grids of a predetermined size, and an occupancy state of an object is indicated in units of grids. The occupancy state of the object may be indicated by the presence or absence, or existence probability of the object, for example. In some embodiments, the local map may be also used for detection processing and / or recognition processing performed on the conditions outside the vehicle 1 by the recognition unit 73, for example.

[0046] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 on the basis of the position information acquired by the position information acquisition unit 24 and / or the sensor data from the vehicle sensor 27.

[0047] The sensor fusion unit 72 performs sensor fusion processing of combining a plurality of different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52), to obtain information. Methods for combining different types of sensor data include, but are not limited to, composite, integration, fusion, association, and the like.

[0048] The recognition unit 73 performs the detection processing on the conditions outside the vehicle 1 and / or the recognition processing on the conditions outside the vehicle 1.

[0049] For example, the recognition unit 73 may perform the detection processing and / or recognition processing on the conditions outside the vehicle 1 on the basis of information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, and the like.

[0050] Specifically, for example, the recognition unit 73 may perform detection processing, recognition processing, and / or the like on an object around the vehicle 1. As the object detection processing, for example, processing of detecting the presence or absence, size, shape, position, motion, and the like of an object may be adopted. As the object recognition processing, for example, processing of recognizing an attribute such as a type of the object or identifying a specific object may be adopted. The detection processing and the recognition processing are not necessarily clearly divided, and at least a part thereof may overlap.

[0051] In some embodiments, the recognition unit 73 may detect an object around the vehicle 1 by performing clustering to classify point clouds based on sensor data from the radar 52, the LiDAR 53, and / or the like into clusters of point clouds. Therefore, the presence or absence, the size, the shape, and the position of the object around the vehicle 1 may be detected.

[0052] In some embodiments, the recognition unit 73 may detect the motion of the object around the vehicle 1 by following the motion of the cluster of point clouds classified by clustering. Therefore, the speed and / or the advancing direction (movement vector) of the object present around the vehicle 1 may be detected.

[0053] In some embodiments, the recognition unit 73 may detect and / or recognize a vehicle (including a bicycle), a person, an obstacle, a structure, a road, a traffic light, a traffic sign, a road mark, and the like on the basis of the image data supplied from the camera 51. In some embodiments, the recognition unit 73 may recognize the type of the object around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0054] In some embodiments, the recognition unit 73 may perform recognition processing on traffic rules around the vehicle 1 on the basis of the map accumulated in the map information accumulation unit 23, the result of estimating the self-position from the self-position estimation unit 71, and / or the result of recognizing the object around the vehicle 1 from the recognition unit 73. Through this process, the recognition unit 73 may recognize the position and / or the state of the traffic light, the contents of the traffic sign and / or the road mark, the contents of the traffic regulation, and / or the travelable lane, and the like.

[0055] In some embodiments, the recognition unit 73 may perform recognition processing of the environment around the vehicle 1. In some embodiments, the recognition unit 73 may recognize weather characteristics (temperature, humidity, brightness), a road surface condition, and / or the like.

[0056] The action planning unit 62 creates an action plan of the vehicle 1. For example, the action planning unit 62 may create the action plan by performing path planning and path following.

[0057] In some embodiments, path planning may include global path planning and local path planning. The global path planning may include a process of planning a rough path from the start to the goal. The local path planning is also referred to as trajectory planning, and may include trajectory generation capable of safely and smoothly traveling in the vicinity of the vehicle 1 along a planned route in consideration of motion characteristics of the vehicle 1, presence of an arbitrary obstacle, and the like.

[0058] In some embodiments, the path following may be planning operations for safe and accurate travelling along the path planned by the path planning within a planned time. For example, the action planning unit 62 may calculate a target speed and / or a target angular velocity of the vehicle 1, on the basis of the result of the path following processing.

[0059] The operation control unit 63 controls operations of the vehicle 1 to achieve the action plan created by the action planning unit 62.

[0060] For example, in some embodiments, the operation control unit 63 may control a steering control unit 81, a brake control unit 82, and / or a drive control unit 83 included in the vehicle control unit 32 described later to perform the lateral vehicle movement control and / or the longitudinal vehicle movement control such that the vehicle 1 travels on the trajectory calculated by the trajectory planning. For example, the operation control unit 63 may perform control (for example, lateral vehicle movement control, longitudinal vehicle movement control) for the purpose of one or more driver support functions and / or driving automation. Non-limiting examples of driver support functions include collision avoidance or shock mitigation, inter-vehicle distance control (for example, control for maintaining a specific distance from a vehicle traveling in front of the vehicle 1), vehicle speed control (for example, control for maintaining a certain speed), vehicle collision warning, lane departure warning. As a non-limiting example of the driving automation, there is traveling that does not depend on the operation of the driver or the remote driver.

[0061] In some embodiments, the DMS 30 may perform authentication processing of the driver, recognition processing of the state of the driver, and / or the like on the basis of sensor data from the in-vehicle sensor 26, input data input to the HMI 31 to be described later, and / or the like. Non-limiting examples of the state of the driver that may be recognized include a physical condition, an alertness level, a concentration level, a fatigue level, a line-of-sight direction, a drunkenness level, a driving operation, a posture, and the like.

[0062] In some embodiments, the DMS 30 may perform authentication processing of a user (for example, a passenger) other than the driver and / or recognition processing of a state of the user. In some embodiments, the DMS 30 may perform recognition processing of the internal situation of the vehicle 1 on the basis of the sensor data from the in-vehicle sensor 26. Non-limiting examples of features of the internal situation of the vehicle 1 that can be a recognition target include temperature, humidity, brightness, smell, and the like.

[0063] The HMI 31 receives various data, instructions, and the like as inputs, and presents various data to the user.

[0064] Data input by the HMI 31 will be schematically described. The HMI 31 includes an input device for a person to input data, instructions, or the like. The HMI 31 generates an input signal on the basis of data, instructions, or the like input by an input device, and supplies the input signal to one or more (or all in some cases) units of the vehicle control system 11. In some embodiments, the HMI 31 may include a touch panel, a button, a switch, and / or a lever as an input device. The present invention is not limited thereto, and the HMI 31 may include an input device capable of inputting information by a method other than manual operation by voice, gesture, or the like. In some embodiments, the HMI 31 may include, as an input device, a remote control device using infrared rays and / or radio waves, or an external connection device corresponding to an operation of the vehicle control system 11. Non-limiting examples of the external connection device include a mobile device (for example, a smartphone) and a wearable device (for example, a smart watch).

[0065] The presentation of data by the HMI 31 will be schematically described. The HMI 31 generates visual information, auditory information, and / or tactile information for the user and / or a person outside the vehicle 1. Furthermore, the HMI 31 may perform output control for controlling output, output content, output timing, output method, and / or the like of each piece of generated information. Non-limiting examples of the visual information that can be generated and output by the HMI 31 include an operation screen, a state display of the vehicle 1, a warning display, an image such as a monitor image indicating a situation around the vehicle 1, and information indicated by light. Non-limiting examples of the auditory information that can be generated and output by the HMI 31 include voice guidance, warning sound, warning message, and the like. Non-limiting examples of the tactile information that can be generated and output by the HMI 31 include information given to the tactile sense of the user by force, vibration, motion, or the like.

[0066] In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device that presents visual information by displaying an image by itself, or a projector device that presents visual information by projecting an image. In some embodiments, the display device may be a device that displays the visual information in the field of view of the user, such as a head-up display, a transmissive display, or a wearable device having an augmented reality (AR) function, for example, in addition or as an alternative to a normal display device. In some embodiments, in the HMI 31, a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 1 can also be included as an output device that can output visual information.

[0067] In some embodiments, the HMI 31 may include an audio speaker, a headphone, and an earphone as an output device capable of outputting auditory information.

[0068] In some embodiments, the HMI 31 may include a haptic element using haptics technology as an output device capable of outputting tactile information. The haptic element may be provided, for example, at a portion of the vehicle 1 with which the user comes into contact, such as the steering wheel or the seat.

[0069] The vehicle control unit 32 controls one or more (or all in some cases) units of the vehicle 1. The vehicle control unit 32 includes the steering control unit 81, the brake control unit 82, the drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0070] The steering control unit 81 performs detection, control, and / or the like of the state of a steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel and the like, an electric power steering, and the like. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, and the like.

[0071] The brake control unit 82 performs detection, control, and / or the like of the state of a brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal and the like, an antilock brake system (ABS), a regenerative brake mechanism, and the like. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, and the like.

[0072] The drive control unit 83 performs detection, control, and / or the like of the state of a drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a driving force generation device for generating a driving force such as an internal combustion engine or a driving motor, a driving force transmission mechanism for transmitting the driving force to wheels, and the like. The drive control unit 83 includes, for example, a drive ECU that controls the drive system, an actuator that drives the drive system, and the like.

[0073] The body system control unit 84 performs detection, control, and / or the like of the state of a body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, a power window device, a power seat, an HVAC system, an airbag, a seat belt, a shift lever, and the like. The body system control unit 84 includes, for example, a body system ECU that controls the body system, an actuator that drives the body system, and the like.

[0074] The light control unit 85 performs detection, control, and / or the like of the states of various lights of the vehicle 1. Non-limiting examples of lights that can be controlled by the light control unit 85 include headlights, backlights, fog lights, turn signals, brake lights, projector lights, and bumper displays. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0075] The horn control unit 86 performs detection, control, and / or the like of the state of a car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0076] Fig. 2 is a diagram illustrating examples of sensing areas of the camera 51, the radar 52, the LiDAR 53, the ultrasonic sensor 54, and the like of the external recognition sensor 25 in Fig. 1. Fig. 2 schematically illustrates a state in which the vehicle 1 is viewed from above.

[0077] A sensing area 101F and a sensing area 101B indicate examples of the sensing area of the ultrasonic sensor 54. The sensing area 101F (for example, the sensing areas of the plurality of ultrasonic sensors 54) covers the periphery of the front end of the vehicle 1. The sensing area 101B (for example, the sensing areas of the plurality of ultrasonic sensors 54) covers the periphery of the rear end of the vehicle 1.

[0078] Sensing results in the sensing area 101F and / or the sensing area 101B may be used for parking assistance for the vehicle 1, and the like, for example.

[0079] A sensing area 102F, a sensing area 102B, a sensing area 102L, and a sensing area 102R illustrate examples of the sensing area of the radar 52 for a short distance or a middle distance. The sensing area 102F covers an area extending farther than the sensing area 101F in front of the vehicle 1. The sensing area 102B covers an area extending farther than the sensing area 101B behind the vehicle 1. The sensing area 102L covers an area around the rear left side of the vehicle 1. The sensing area 102R covers an area around the rear right side of the vehicle 1.

[0080] A sensing result in the sensing area 102F may be used for detection of a vehicle, a pedestrian, or the like present in front of the vehicle 1, and the like, for example. A sensing result in the sensing area 102B may be used for a rear collision prevention function of the vehicle 1, and the like, for example. The sensing results in the sensing area 102L and / or the sensing area 102R may be used, for example, for detection of one or more objects in a blind spot on the left side and / or the right side of the vehicle 1.

[0081] A sensing area 103F, a sensing area 103B, a sensing area 103L, and a sensing area 103R indicate examples of sensing areas by the camera 51. The sensing area 103F covers an area extending farther than the sensing area 102F in front of the vehicle 1. The sensing area 103B covers an area extending farther than the sensing area 102B behind the vehicle 1. The sensing area 103L covers an area around the left side of the vehicle 1. The sensing area 103R covers an area around the right side of the vehicle 1.

[0082] A sensing result in the sensing area 103F may be used for, for example, recognition of a traffic light or a traffic sign, a lane departure prevention support system, and an automated headlight control system. A sensing result in the sensing area 103B may be used for, for example, parking support and / or a surround view system. Sensing results in the sensing area 103L and / or the sensing area 103R may be used for, for example, the surround view system.

[0083] A sensing area 104 indicates an example of the sensing area of the LiDAR 53. The sensing area 104 covers an area extending farther than the sensing area 103F in front of the vehicle 1. On the other hand, the sensing area 104 has a narrower range in a left-right direction of the vehicle 1 than that of the sensing area 103F.

[0084] A sensing result in the sensing area 104 may be used for, for example, detection of an object such as a vehicle around.

[0085] A sensing area 105 indicates an example of the sensing area of the long-range radar 52. The sensing area 105 covers an area extending farther than the sensing area 104 in front of the vehicle 1. On the other hand, the sensing area 105 has a narrower range in the left-right direction of the vehicle 1 than that of the sensing area 104.

[0086] A sensing result in the sensing area 105 may be used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, or the like.

[0087] In some embodiments, the sensing area of each sensor of the external recognition sensor 25 (for example, the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54) may have various configurations other than the configuration illustrated in Fig. 2. Specifically, in some embodiments, the ultrasonic sensor 54 may also perform sensing on the sides of the vehicle 1, or the LiDAR 53 may perform sensing on the rear side of the vehicle 1. Furthermore, the installation position of each sensor is not limited to the above-described example. Furthermore, each sensor may be provided in a single unit or in plural.

[0088] <2. Overview of present technology> (Conceivable Problems) In recent years, evaluation criteria for recognition performance in the in-vehicle systems have been formulated by EURO NCAP, AI-SEE, and the like. Since the recognition performance under bad weather is different from that under good weather, it is necessary to change the evaluation criteria. On the other hand, the recognition sensor and the system need to be able to cope with all scenes. Therefore, it is necessary to evaluate the recognition sensor and the system by determining a situation such as bad weather.

[0089] (Solution) In the present technology, the following solutions to the above-described problems are realized.

[0090] (1) Determination of bad weather using in-cabin sensor An image (rear seat, rear window, side mirror, etc.) captured by an in-cabin camera and data sensed by other sensors are used as a weather data set and learned by deep learning. Then, using the learning result, bad weather is determined based on images of the back seat, the rear window, the side mirror, and the like and sensing data of other sensors.

[0091] For example, in a case where the setting sun or high beam is included in the image, saturation (overexposure) is determined from the image, and it can be determined that the saturation is caused by the setting sun if the photographing time is the evening, and the saturation is caused by the high beam if the photographing time is the night. As described above, in the case of determining the weather or the like from the image, information other than the camera, such as time, season, or position information, can be used.

[0092] Furthermore, for example, a pattern of the face image may be analyzed from the image of the passenger, and it may be estimated what kind of external environment of the vehicle the pattern of the face image corresponds to. By using artificial intelligence in which a correlation between a pattern of a face image and an external environment of the vehicle is learned in advance, it is possible to estimate an external environment of the vehicle such as a setting sun or a high beam from a glaring face image, darkness or difficulty in seeing from a narrowed face image, and an obstacle from a surprised face image. Note that the pattern of the face image can be configured by a combination of face parts such as eyebrows, eyes, nose, mouth, and cheeks, but can also be configured by an image of the entire face image without being divided into individual face parts. The learning model in the present disclosure may be configured using a learned machine learning model such as a convolutional neural network (CNN), and may be, for example, a convolutional neural network such as a U-net neural network, a Senet neural network, or an SKnet neural network.

[0093] It is conceivable to adopt a 360° camera or a plurality of cameras as an in-cabin camera that captures images of a rear seat, a rear window, a side mirror, and the like. Normally, the in-cabin camera is focused on the inside of the vehicle, but in order to determine the weather, the in-cabin camera may be focused on a window glass. The in-cabin camera cannot image the window glass in a determinable manner at night. On the other hand, the weather can be determined by sensing the window glass from the inside of the vehicle by an infrared (IR) sensor, an event-base vision sensor (EVS), or the like, and inputting the rainfall amount (snowfall amount) from the size and density of water droplets (snow particles) to an estimation model.

[0094] (2) Bad weather determination by road surface detection The bad weather can be determined on the basis of the reflection waveform from the road surface. The distance from the LiDAR or radar of the in-vehicle sensor to the road surface is always constant, and the road surface condition can be estimated by inputting the reflection waveform to an estimation model in which the road surface condition is learned in advance. Furthermore, it is also possible to determine whether or not it is an ice road from a sound emitted from a tire or a difference between a road and a road side. A large amount of weather data sets and road surface data can be learned by deep learning, and the weather can be comprehensively estimated including not only camera information but also time, season, position information, and the like.

[0095] (Obtained Effects) According to the present technology, it is possible to accurately determine specific bad weather. Furthermore, by determining the bad weather for each scene and changing the recognition performance, it is possible to accurately recognize an object or the like and contribute to safe driving.

[0096] <3. Sensing information that can be acquired from inside the vehicle> Sensing information that can be acquired from the inside of the vehicle 1 will be described with reference to Fig. 3.

[0097] In the vehicle 1 illustrated in Fig. 3, an in-cabin sensor 101 is provided in the front of the vehicle interior, and an in-cabin sensor 102 is provided in the rear of the vehicle interior. The in-cabin sensors 101 and 102 include any one of a visible light camera, a ToF sensor, an IR sensor, a SWIR sensor, an EVS, a polarization image sensor, LiDAR, and a radar, and may have a light source as appropriate. In a case where the in-cabin sensors 101 and 102 are configured as IR sensors or SWIR sensors, it is possible to sense the inside of the vehicle even in a dark place.

[0098] The in-cabin sensor 101 performs vehicle interior sensing (in-cabin sensing) with a wide range from the front center toward the rear in the vehicle as a sensing range 101v. The in-cabin sensor 102 performs vehicle interior sensing (in-cabin sensing) with a wide range from the rear center toward the front in the vehicle as a sensing range 102v.

[0099] In the vehicle 1, a driver 111 seated in a driver's seat and a driving assistant 112 seated in a passenger seat are on board as passengers. The driver 111 drives the vehicle 1 with the front of the vehicle 1 as a visual field range 111v. The driving assistant 112 assists the driving of the driver 111 in a direction different from the direction of the driver 111, such as the right front of the vehicle 1, as the visual field range 112v.

[0100] A mirror BS is provided in a pillar (B pillar) in the vehicle 1. According to the mirror BS, it is also possible to sense a range that is a blind spot from the in-cabin sensors 101 and 102 by the passenger (driver 111, driving assistant 112) or the seat ST where the passenger is seated.

[0101] Furthermore, although not illustrated, the vehicle 1 may be provided with an in-cabin sensor including a wide-angle lens on the ceiling in the vehicle, or may be provided with mirrors BS at a plurality of locations other than the pillars in the vehicle.

[0102] In the present technology, passenger state information indicating a state of a passenger, window state information indicating a state of a window of the vehicle 1, seat state information indicating a state of a seat in the vehicle, and vehicle exterior state information indicating a state outside the vehicle can be acquired as sensing information on the basis of data acquired by the in-cabin sensors 101 and 102 installed in the vehicle 1. Specifically, sensing information indicated by circled numbers 1 to 9 in Fig. 3 can be acquired.

[0103] The sensing information indicated by the circled number 1 is one of the passenger state information, and is the expression information of the face of the driver 111 based on the data acquired by the in-cabin sensor 101. The facial expression information may include information indicating the state of the iris of the eye, emotions such as delight, anger, sorrow, and pleasure, and the like.

[0104] The sensing information indicated by the circled number 2 is one of the passenger state information, and is posture information of the driver 111 based on data acquired by the in-cabin sensor 101 or the in-cabin sensor 102. The posture information may include information indicating a distance between the driver 111 and the steering wheel HD.

[0105] The sensing information indicated by the circled number 3 is one of the window state information, and is the state information of the rear window RW based on the data acquired by the in-cabin sensor 101. The state information of the rear window RW may include information indicating the size and type of the attachable matter attached to the rear window RW, specifically, the size and presence or absence of raindrops, snow particles, and mud splashes, and the irradiation condition of intense light such as the setting sun and high beam.

[0106] The sensing information indicated by the circled number 4 is one of the seat state information, and is illuminance information of the surface of the seat ST based on data acquired by the in-cabin sensor 101 or the in-cabin sensor 102. The illuminance information on the surface of the seat ST may include information indicating how the surface of the seat ST is illuminated by intense light such as the setting sun or high beam.

[0107] The sensing information indicated by the circled number 5 is one of the passenger state information, and is the expression information of the face and the posture information of the driving assistant 112 based on the data acquired by the in-cabin sensor 101.

[0108] The sensing information indicated by the circled number 6 is one of the vehicle exterior state information, and is position and posture information of a vehicle exterior obstacle 121 existing outside the vehicle based on the data acquired by the in-cabin sensor 101 or the in-cabin sensor 102. Here, the vehicle exterior obstacle 121 is in a blind spot of the visual field range 111v of the driver 111 and the visual field range 112v of the driving assistant 112, and the position and posture information of the vehicle exterior obstacle 121 may include information indicating the position, posture, presence or absence, and the like of the vehicle exterior obstacle 121.

[0109] The sensing information indicated by the circled number 7 is one of the window state information, and is the state information of the windshield FW based on the data acquired by the in-cabin sensor 102. The state information of the windshield FW may include information indicating the size and type of the attachable matter attached to the windshield FW, specifically, the size and presence or absence of raindrops, snow particles, and mud splashes, and the irradiation condition of intense light such as the setting sun and high beam.

[0110] The sensing information indicated by the circled number 8 is one of the vehicle exterior state information, and is mirror image information reflected on the side mirror SM of the vehicle 1 based on the data acquired by the in-cabin sensor 102. The mirror image information reflected on the side mirror SM may include information indicating the presence or absence of the vehicle exterior obstacle 121 reflected on the side mirror SM, the presence or absence of raindrops or snow particles attached to the side mirror SM, the irradiation condition of intense light such as the setting sun or high beam, the road surface condition of the road reflected on the side mirror SM, and the like.

[0111] The sensing information indicated by the circled number 9 is one of vehicle exterior state information, and is road surface information of a road on which the vehicle 1 travels based on data acquired by the in-cabin sensor 102. The road surface information may include information indicating an irradiation condition of intense light such as the setting sun or a high beam reflected on the road surface, a road surface condition of the road, and the like.

[0112] In the present technology, in a case where the sensing by the vehicle exterior sensor of the vehicle 1 cannot be sufficiently used, the recognition performance of the driver 111 and the vehicle exterior sensor can be determined on the basis of the sensing information acquired on the basis of the in-cabin sensing by the in-cabin sensors 101 and 102.

[0113] <4. Configuration and operation outline of vehicle control system according to present technology> (Configuration of Vehicle Control System) Fig. 4 is a block diagram illustrating a functional configuration example of a vehicle control system to which the present technology is applied.

[0114] A vehicle control system 300 illustrated in Fig. 4 is realized by the vehicle control system 11 provided in the vehicle 1 described with reference to Fig. 1. The vehicle control system 300 includes in-cabin sensors 310-1,..., and 310-N, an information acquisition unit 320, a recognition performance determination unit 330, and a feedback output unit 340. The information acquisition unit 320, the recognition performance determination unit 330, and the feedback output unit 340 may be implemented in one vehicle control device or may be implemented across a plurality of in-vehicle devices.

[0115] The in-cabin sensors 310-1,..., and 310-N are a plurality of in-cabin sensors installed in the vehicle 1, and correspond to the in-vehicle sensor 26 in Fig. 1 and the in-cabin sensors 101 and 102 in Fig. 3. As described above, the in-cabin sensors 310-1,..., and 310-N may be any one of a visible light camera, a ToF sensor, an IR sensor, a SWIR sensor, an EVS, a polarization image sensor, LiDAR, and a radar. In a case where the in-cabin sensors 310-1,..., and 310-N are not individually distinguished, they are simply referred to as in-cabin sensors 310. The in-cabin sensor 310 supplies sensor data acquired by sensing the inside of the vehicle 1 to the information acquisition unit 320.

[0116] The information acquisition unit 320 acquires sensing information on the basis of sensor data acquired by the plurality of in-cabin sensors 310. The sensing information is, for example, the sensing information indicated by the circled numbers 1 to 9 described with reference to Fig. 3.

[0117] Furthermore, the information acquisition unit 320 acquires auxiliary information on the basis of vehicle exterior data supplied from the outside of the vehicle 1 in addition to the sensing information. The vehicle exterior data may include sensor data acquired by sensing by a vehicle exterior sensor of the vehicle 1, and communication data supplied by wireless communication with another vehicle or wireless communication with an information processing device such as a cloud server via a network. The auxiliary information can include weather information, temperature information, and the like in addition to posture information, driving route information, and position information of the vehicle 1.

[0118] The information acquisition unit 320 supplies the acquired sensing information and auxiliary information to the recognition performance determination unit 330.

[0119] On the basis of the sensing information from the information acquisition unit 320, the recognition performance determination unit 330 determines deterioration in recognition performance due to an environmental state outside the vehicle 1. The recognition performance here may be the recognition capability of the driver 111 or the driving assistant 112 as a passenger, or may be the detection capability of a vehicle exterior sensor installed outside the vehicle. For example, the recognition performance determination unit 330 determines whether or not the recognition capability of the passenger and the detection capability of the vehicle exterior sensor have deteriorated due to bad weather outside the vehicle. The recognition performance determination unit 330 can also determine deterioration in recognition performance due to an environmental state outside the vehicle 1 by further using auxiliary information from the information acquisition unit 320 as necessary.

[0120] The feedback output unit 340 functions as a feedback control unit that performs feedback control related to traveling of the vehicle 1 according to a determination result of deterioration in recognition performance by the recognition performance determination unit 330.

[0121] (Operation outline of vehicle control system) An operation outline of the vehicle control system 300 will be described with reference to flowcharts of Fig. 5. The processing of Fig. 5 may be executed while the vehicle 1 is traveling.

[0122] In step S1, the information acquisition unit 320 acquires sensing information on the basis of sensor data acquired by the in-cabin sensor 310.

[0123] In step S2, on the basis of the sensing information acquired by the information acquisition unit 320, the recognition performance determination unit 330 estimates, as the environmental state outside the vehicle 1, the presence or absence of intense light, hydrometeors such as rain, snow, and fog, a road surface condition, whether or not the environmental state is in a dark place, and the like. Details of the environmental state estimated on the basis of the sensing information will be described later.

[0124] In step S3, the recognition performance determination unit 330 optimizes the sensing information used for determination of deterioration in recognition performance according to the estimated environmental state. For example, the recognition performance determination unit 330 may select sensing information to be used for determination of deterioration in recognition performance or may weight the reliability of the sensing information to be used for determination of deterioration in recognition performance according to the estimated environmental state.

[0125] In step S4, the recognition performance determination unit 330 determines whether the recognition performance of the driver 111 / vehicle exterior sensor is deteriorated (whether or not the recognition performance is deteriorated) due to the estimated environmental state on the basis of the optimized sensing information.

[0126] That is, in a case where it is determined in step S5 that the recognition performance of the driver 111 / vehicle exterior sensor has deteriorated, the process proceeds to step S6, and the feedback output unit 340 outputs feedback related to traveling of the vehicle 1 (performs feedback control).

[0127] For example, the feedback output unit 340 controls a driving support function of the vehicle 1 as feedback control regarding traveling of the vehicle 1. The driving support function of the vehicle 1 includes, for example, irradiation control of a headlight, speed restriction, automatic stop, driving mode switching, and the like. Furthermore, the feedback output unit 340 controls information presentation to the driver 111 and the driving assistant 112 as passengers and controls information presentation to other vehicles as feedback control regarding traveling of the vehicle 1. Thereafter, the process returns to step S1, and the subsequent processes are repeated. Details of the feedback control against the deterioration in the recognition performance will be described later.

[0128] On the other hand, in a case where it is determined in step S5 that the recognition performance of the driver 111 / vehicle exterior sensor has not deteriorated, the process returns to step S1, and the subsequent processes are repeated.

[0129] According to the above processing, in a case where it is determined that the recognition performance is deteriorated due to the environmental state on the basis of the sensing information acquired on the basis of the data acquired by the in-cabin sensor installed in the vehicle, the feedback regarding the traveling of the vehicle is output. That is, even in a situation where sufficient information cannot be acquired only by the vehicle exterior sensor and a sufficient driving support function cannot be realized, a more suitable driving support function can be realized by using the information acquired on the basis of the in-cabin sensor.

[0130] <5. Example of environmental state> An example of the environmental state estimated on the basis of the sensing information will be described with reference to Fig. 6.

[0131] Fig. 6 illustrates eight patterns of “setting sun / high beam”, “setting sun / high beam (predicted)”, “vehicle body side obstacle”, “rain / snow / fog”, “road surface condition (frozen)”, “road surface condition (accumulated snow)”, “road surface condition (irregular ground)”, and “dark place” as the environmental state estimated on the basis of the sensing information indicated by the circled numbers 1 to 9 described with reference to Fig. 3.

[0132] (1) Setting sun / high beam In a case where sensing information such as the driver's expression of “squinting”, the driver's posture of “leaning forward”, the illuminance of the seat of “high illuminance”, the expression and posture of the driving assistant of “squinting”, and the state of the windshield of “flare has occurred” is acquired, it is estimated that intense light such as the setting sun which is sunlight or a high beam which is a headlight beam of another vehicle strikes.

[0133] In a case where the driver or the driving assistant wears sunglasses, whether or not the eyes squint can be determined by sensor data from the IR sensor or the SWIR sensor. In a case where the illuminance of the seat is high, the irradiation angle can also be estimated from the luminance distribution in the image as the sensor data. In particular, in a case where a difference in brightness between the driver's seat or the passenger seat and the rear seat is large, it is estimated that the setting sun or the high beam is hitting.

[0134] As illustrated in Fig. 6, whether or not flare has occurred can be determined by whether or not there is an area SL in which pixels are saturated (overexposure) due to intense light in an image IMG1 imaging the windshield. Whether the intense light is the setting sun or the high beam can be determined on the basis of the vehicle body posture and the driving route acquired as the auxiliary information, the sunset time and the current time, other sensor data obtained by vehicle exterior sensing, and the like. For example, in a case where the current time is close to the sunset time, it may be determined that the beam is the setting sun, and in a case where the current time is the nighttime, it may be determined that the beam is the high beam. Furthermore, in a case where the position of the region SL saturated in the image IMG1 is a position where an oncoming vehicle can exist, it may be determined as a high beam, and otherwise, it may be determined as the setting sun. Positions where oncoming vehicles may be present may be determined on the basis of other sensor data.

[0135] (2) Setting sun / high beam (estimation) In a case where sensing information is acquired such that the state of the rear window is “flare has occurred”, the illuminance of the seat is “high”, the expression and posture of the driving assistant is “squinting”, the state of the front window is “flare has occurred”, the mirror image information of the side mirror is “flare has occurred”, and the road surface information via the front window is “high brightness light source reflected on the ground”, it is estimated that the setting sun will hit in a few seconds.

[0136] In a case where the illuminance of the seat is high, the irradiation angle can also be estimated from the luminance distribution in the image as the sensor data. Here, in a case where there is a difference in brightness different from normal, such as a difference in brightness between the driver's seat and the passenger's seat, or a difference in brightness between the upper part and the lower part of the driver's body, it is estimated that the setting sun or the high beam will hit the vehicle in a few seconds.

[0137] Whether the intense light is the setting sun or the high beam can be determined on the basis of the vehicle body posture and the driving route, the sunset time and the current time, and the like acquired as the auxiliary information.

[0138] (3) Vehicle body side obstacle In a case where the sensing information such as the expression and posture of the driving assistant is “interest in the obstacle”, the position and posture of the vehicle exterior obstacle are “presence or absence of the obstacle”, and the mirror image information of the side mirror is “presence or absence of the obstacle” is acquired, it is estimated that there is an obstacle on the side of the vehicle body.

[0139] (4) Rain / snow / fog In a case where sensing information is acquired, such as at least one of the driver's expression of “squinting”, the driver's posture of “leaning forward”, the state of the rear window of “raindrops present”, the expression and posture of the driving assistant of “squinting”, the state of the windshield of “raindrops present”, “snow touches the windshield”, and “the wiper is moving”, or at least one of the mirror image information of the side mirror of “raindrops present”, the road surface information via the windshield of “road surface is wet”, and “there is a puddle”, it is estimated that the weather is hydrometeors such as rain, snow, or fog.

[0140] Since whether or not to operate the wiper is determined by the driver, it is not possible to estimate that the weather is rain or snow, the rainfall amount, or the snowfall amount only by “the wiper is moving”. Therefore, by using the presence, absence, and size of raindrops and snow particles on the windshield, the rear window, the side window, and the side mirror that can be captured by the in-vehicle camera as the in-cabin sensor, it is possible to estimate whether the weather is rain or snow, and the rainfall amount or the snowfall amount. At this time, in particular, the presence or absence and size of raindrops or snow particles in the rear window or the side window without the wiper may be used. Furthermore, the rainfall amount or the snowfall amount may be estimated according to deterioration in the resolution or the recognition rate of an object obtained as mirror image information of a side mirror. Whether or not the road surface is wet and whether or not there is a puddle can be determined by sensor data (degree of polarization of the road surface) from the polarization sensor.

[0141] The rainfall amount, the snowfall amount, the presence or absence of puddles, and road flooding can be estimated using a learned estimation model. As illustrated in Fig. 8, for example, an image IMG2 captured by an in-vehicle camera focused on the windshield is input to the learned estimation model EM with respect to the size of raindrops, the density of raindrops, and the rainfall amount with respect to the traveling speed of the vehicle. Therefore, the rainfall amount can be estimated from the image IMG2. At night, a normal visible light camera cannot capture raindrops of the window glass, but can sense the raindrops of the window glass from inside the vehicle by using an IR sensor, an EVS, or the like.

[0142] In this manner, the weather can be comprehensively estimated by combining the learning by the deep learning with the image captured by the in-vehicle camera as the in-cabin sensor, and the sensor data obtained by integrating the LiDAR, the radar, the rainfall sensor, and the like.

[0143] Whether or not the weather is rain, snow, or fog can also be determined on the basis of global positioning system (GPS) information, weather information, or the like acquired as the auxiliary information.

[0144] (5) Road surface condition (frozen) In a case where sensing information such as the driver’s expression of “stiff”, the driver’s posture of “force is applied to the steering wheel”, the expression and posture of the driving assistant of “stiff”, the mirror image information of the side mirror of “(frozen) road surface condition”, and the road surface information via the windshield of “(frozen) road surface condition” is acquired, it is estimated that the road surface on which the vehicle 1 travels is in the frozen state.

[0145] Whether or not the expression of the driver or the driving assistant is stiff and whether or not the posture of the driver applies force to the steering wheel can be determined by machine learning. Whether or not the road surface is frozen can also be determined by machine learning. Whether or not the road surface is frozen may be determined by sensor data (degree of polarization of the road surface) from the polarization sensor.

[0146] JP 2022-14729 A discloses a bad environment determination method for determining a type of a bad environment by combining environmental supplementary information acquired from a sensor / device other than the front camera and a recognition situation in the front camera. Although detailed description is omitted, as illustrated in Fig. 9, the configuration of the technology specifies the environment type from the recognition situation by the front camera of various features on the basis of the fact that the appearance of various features differs according to the environment type such as the setting sun, fog, and heavy rain.

[0147] For example, if a landmark such as a sign and a dividing line such as a boundary line of a lane can be recognized by the front camera at a short distance, the fog is specified in consideration of environmental supplementary information such as temperature and humidity. Furthermore, if only the dividing line can be recognized by the front camera at a short distance (for example, 8.5 m ahead), it is specified that it is the setting sun in consideration of environmental supplementary information such as time and traveling direction. Moreover, if neither the landmark nor the dividing line can be recognized by the front camera in either the short distance or the far distance, it is specified that it is a heavy rain in consideration of environmental supplementary information such as wiper operation information and weather forecast. However, in this technique, even if the front camera can recognize the landmark and the dividing line at either a short distance or a long distance, it is not possible to specify whether or not the road surface is frozen.

[0148] On the other hand, in the present technology, it is possible to estimate whether or not the road surface is frozen from the sound of the tire, the traveling position (whether or not it is a position where an ice road is likely to occur), and the state of the road side (difference between the road and the road side) acquired from the sensor data sensed by the in-cabin sensor in addition to the temperature and the time as the environmental supplementary information.

[0149] Whether or not the road surface is frozen can also be determined on the basis of a vehicle exterior temperature or the like acquired as the auxiliary information.

[0150] (6) Road surface condition (accumulated snow) In a case where sensing information such as the driver’s expression of “stiff”, the driver’s posture of “force is applied to the steering wheel”, the expression and posture of the driving assistant of “stiff”, the state of the windshield of “presence of falling snow”, the mirror image information of the side mirror of “road surface condition (accumulated snow)”, and the road surface information via the windshield of “road surface condition (accumulated snow)” is acquired, it is estimated that the road surface on which the vehicle 1 travels is in the snowy state.

[0151] Whether or not the expression of the driver or the driving assistant is stiff and whether or not the posture of the driver applies force to the steering wheel can be determined by machine learning. Whether or not the road surface has accumulated snow can also be determined by machine learning. Whether or not the road surface has accumulated snow may be determined according to whether or not snow is piled on the road shoulder in the image captured by the in-vehicle camera.

[0152] Whether or not the road surface has accumulated snow can also be determined on the basis of the vehicle exterior temperature or the like acquired as the auxiliary information.

[0153] (7) Road surface condition (irregular ground) In a case where sensing information such as at least one of the driver’s expression of “stiff”, the driver’s posture of “force is applied to the steering wheel”, the state of the rear window of “mud (attached)”, the expression and posture of the driving assistant of “stiff”, the state of the front window of “mud (attached)”, the mirror image information of the side mirror of at least any one of “road surface condition (of irregular ground)” and “weed, crack”, and the road surface information via the front window of “road surface condition (of irregular ground)” are acquired, it is estimated that the road surface on which the vehicle 1 travels is an irregular ground.

[0154] Whether or not the expression of the driver or the driving assistant is stiff and whether or not the posture of the driver applies force to the steering wheel can be determined by machine learning.

[0155] Whether or not the road surface is an irregular ground can also be determined on the basis of position information, map information, and the like acquired as the auxiliary information.

[0156] (8) Dark place In a case where sensing information is acquired such as the driver's expression of “squinting”, the driver's posture of “leaning forward”, the expression and posture of the driving assistant of “squinting”, the state of the windshield of “dark”, the mirror image information of the side mirror of “dark”, and the road surface information through the windshield of “the signal level (of the sensor data) is low”, it is estimated that the travel environment is a dark place.

[0157] As described above, in the vehicle control system to which the present technology is applied, it is possible to estimate various environmental states that can lower the recognition performance of the driver and the vehicle exterior sensor on the basis of the sensing information acquired on the basis of the in-cabin sensing. To determine whether the illumination is too high (high illuminance) or too low (darkness), conventional image processing techniques may be applied to the pixels within the Region of Interest (ROI). These include thresholding based on the mean luminance, as well as thresholding based on the ratio of high-intensity or low-intensity pixels derived from the histogram.

[0158] <6. Example of feedback control> With reference to Fig. 10, an example of feedback control for various environmental states that can lower the recognition performance of the driver and the vehicle exterior sensor, estimated on the basis of the sensing information, will be described.

[0159] In the feedback control for the environmental state, information obtained from the in-cabin sensor may be fed back to the reliability of information obtained from the vehicle exterior sensor such as LiDAR or radar. In an environment where each of the in-cabin sensors is not good at, operation control can be performed by increasing reliability of information obtained from other in-cabin sensors.

[0160] Fig. 10 illustrates specific examples of feedback control to the “driving support function” of the vehicle 1, the “driver”, and the “other vehicle” to be fed back for the respective environmental states described with reference to Fig. 6.

[0161] (1) Setting sun / high beam At least one of “speed limit” and “high / low beam switching” is performed as feedback control to the driving support function. In particular, in the image captured by the in-vehicle camera, the high beam of the setting sun or another vehicle becomes backlight and the subject disappears. Therefore, it is desirable to perform driving control such as speed limit using a recognition result based on information acquired from another in-cabin sensor.

[0162] As the feedback control to the driver, “warning” to the driver is issued. The “warning” to the driver may be, for example, a notification of the direction of the high beam of the setting sun or another vehicle, or a notification of the presence or absence of a subject around the vehicle 1 on the basis of information acquired from another in-cabin sensor.

[0163] As the feedback control to the other vehicle, a “warning” to the driver of the other vehicle is issued. The “warning” to the driver of another vehicle may be, for example, an instruction to switch a high beam to a low beam.

[0164] (2) Setting sun / high beam (estimation) At least one of “speed limit” and “high / low beam switching” is performed as feedback control to the driving support function.

[0165] As the feedback control to the driver, “warning” to the driver is issued.

[0166] As the feedback control to the other vehicle, a “warning” to the driver of the other vehicle is issued. The “warning” to the driver of another vehicle may be, for example, a warning notifying that the setting sun will hit in a few seconds.

[0167] (3) Vehicle body side obstacle As the feedback control to the driving support function, “automatic stop for preventing involvement in an accident” is performed.

[0168] As the feedback control to the driver, “warning” to the driver is issued. The “warning” to the driver may be a notification of presence or absence of an obstacle on the side of the vehicle 1 on the basis of information acquired from another in-cabin sensor.

[0169] As the feedback control to the other vehicle, “detected object information” is notified to a driver of the other vehicle. The “detected object information” may be notified according to information acquired from another in-cabin sensor.

[0170] (4) Rain / snow / fog As feedback control to the driving support function, “speed limit” is performed. In particular, since there is a possibility that the recognition rate of the surrounding environment of the vehicle exterior sensor decreases due to a puddle, accumulated snow, or the like, it is desirable to perform driving control such as speed limit using information acquired from the in-cabin sensor as feedback information.

[0171] As the feedback control to the driver, “brake assist” is performed. Specifically, a sudden accelerator suppression function is started as “brake assist”.

[0172] As the feedback control to the other vehicle, a “warning” to the driver of the other vehicle is issued. The “warning” to the driver of the other vehicle may be, for example, a notification that a puddle or accumulated snow exists in the surroundings.

[0173] (5) Road surface condition (frozen) As the feedback control to the driving support function, “switching to the frozen driving mode” is performed. In particular, since there is a possibility that the recognition rate of the surrounding environment of the vehicle exterior sensor decreases due to a puddle or the like, it is desirable to perform driving control such as speed limit using information acquired from the in-cabin sensor as feedback information.

[0174] As the feedback control to the driver, “warning” is issued. The “warning” to the driver may be, for example, a notification for suppressing a sudden operation of the driver on the accelerator pedal.

[0175] As the feedback control to the other vehicle, “warning” is issued. The “warning” to the other vehicle may be, for example, a notification for suppressing a sudden operation of the driver of the other vehicle on the accelerator pedal.

[0176] (6) Road surface condition (accumulated snow) As the feedback control to the driving support function, “switching to the frozen driving mode” is performed. In particular, since there is a possibility that the recognition rate of the surrounding environment of the vehicle exterior sensor decreases due to accumulated snow or the like, it is desirable to perform driving control such as speed limit using information acquired from the in-cabin sensor as auxiliary information.

[0177] As the feedback control to the driver, “warning” is issued. The “warning” to the driver may be, for example, a notification for suppressing a sudden operation of the driver on the accelerator pedal.

[0178] As the feedback control to the other vehicle, “warning” is issued. The “warning” to the other vehicle may be, for example, a notification for suppressing a sudden operation of the driver of the other vehicle on the accelerator pedal.

[0179] (7) Road surface condition (irregular ground) As feedback control to the driving support function, “switching from 2WD to 4WD” is performed.

[0180] As the feedback control to the driver, “warning” is issued. The “warning” to the driver may be, for example, one that notifies the driver that the road surface is an irregular ground.

[0181] As the feedback control to the other vehicle, “warning” is issued. The “warning” to the other vehicle may be, for example, a notification notifying a driver of the other vehicle that the road surface is an irregular ground.

[0182] (8) Dark place As the feedback control to the driving support function, at least one of “control of headlight irradiation range”, “speed limit”, and “high / low beam switching” is performed. The illumination range of the headlight and the high / low beam switching may be mutually controlled.

[0183] As the feedback control to the driver, “eye tracking and control of the emission direction of the headlight” are performed. That is, the emission direction of the headlight is controlled in accordance with the line-of-sight direction of the driver.

[0184] As the feedback control to the other vehicle, “detected object information” is notified to a driver of the other vehicle. The “detected object information” may be notified according to information acquired from another in-cabin sensor.

[0185] As described above, in the vehicle control system to which the present technology is applied, it is possible to perform appropriate feedback control for various environmental states estimated on the basis of in-cabin sensing and capable of deteriorating the recognition performance of the driver and the vehicle exterior sensor. As a result, even in a situation where sufficient information cannot be acquired only by the vehicle exterior sensor and a sufficient driving support function cannot be realized, a more suitable driving support function can be realized by using the information acquired on the basis of the in-cabin sensor.

[0186] Note that the feedback control to the driver may be performed in a case where it is determined whether or not the driver is aware of an environmental state that may deteriorate recognition performance on the basis of the in-cabin sensing and it is determined that the driver is not aware of the environmental state.

[0187] <7. Configuration example of computer> The above-described series of processing can be executed by hardware or software. In a case where the series of processing is executed by software, a program constituting the software is installed from a program recording medium to a computer incorporated in dedicated hardware, a general-purpose personal computer, or the like.

[0188] Fig. 11 is a block diagram illustrating an example of configuration of hardware of a computer that executes the above-described series of processing steps using a program.

[0189] The above-described vehicle control system 300 is realized by a computer 500 having the configuration illustrated in Fig. 11.

[0190] A CPU 501, a ROM 502, and a RAM 503 are connected to one another by a bus 504.

[0191] An input / output interface 505 is further connected to the bus 504. An input unit 506 including a keyboard, a mouse, and the like and an output unit 507 including a display, a speaker, and the like are connected to the input / output interface 505. Furthermore, a storage unit 508 including a hard disk, a nonvolatile memory, or the like, a communication unit 509 including a network interface or the like, and a drive 510 that drives a removable medium 511 are connected to the input / output interface 505.

[0192] In the computer 500 configured as described above, for example, the CPU 501 loads a program stored in the storage unit 508 into the RAM 503 via the input / output interface 505 and the bus 504 and executes the program to perform the above-described series of processing.

[0193] The program executed by the CPU 501 is provided, for example, by being recorded in the removable medium 511 or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 508.

[0194] Note that the program to be executed by the computer 500 may be a program in which processing is executed in a time series in the order described herein, or may be a program in which processing is executed in parallel or at a necessary timing such as a timing when a call is made, or the like.

[0195] In the present specification, a system refers to a set of a plurality of components (devices, modules (parts), or the like), and it does not matter whether or not all the components are located in the same housing. Therefore, a plurality of devices stored in different housings and connected via a network and one device in which a plurality of modules is stored in one housing are both systems.

[0196] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

[0197] Furthermore, the effects described in the present specification are merely examples and are not limited, and other effects may be provided.

[0198] Moreover, the present technology may have the following configurations. (1) A vehicle control system including: an information acquisition unit that acquires sensing information on the basis of data acquired by a sensor installed in a vehicle interior; a recognition performance determination unit that determines deterioration in recognition performance due to an environmental state outside the vehicle on the basis of the sensing information; and a feedback control unit that performs feedback control related to traveling of the vehicle according to a determination result of deterioration in the recognition performance. (2) The vehicle control system according to (1), in which the sensor includes at least one of a visible light camera, a ToF sensor, an IR sensor, a SWIR sensor, an EVS, a polarization image sensor, a LiDAR, and a radar. (3) The vehicle control system according to (1) or (2), in which the information acquisition unit acquires at least one of: passenger state information indicating a state of a passenger as the sensing information; window state information indicating a state of a window of the vehicle; seat state information indicating a state of a seat in the vehicle; and vehicle exterior state information indicating a state outside the vehicle. (4) The vehicle control system according to (3), in which the passenger state information includes expression information of a face of the passenger and posture information of the passenger. (5) The vehicle control system according to (3), in which the seat state information includes illuminance information of a surface of the seat. (6) The vehicle control system according to (3), in which the vehicle exterior state information includes position and posture information of a vehicle exterior obstacle existing outside the vehicle, mirror image information reflected on a side mirror of the vehicle, and road surface information on which the vehicle travels. (7) The vehicle control system according to any one of (1) to (6), in which the recognition performance determination unit determines at least one of deterioration in recognition capability of a passenger and deterioration in detection capability of a vehicle exterior sensor installed outside the vehicle. (8) The vehicle control system according to any one of (1) to (7), in which the recognition performance determination unit optimizes the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state. (9) The vehicle control system according to (8), in which the recognition performance determination unit selects the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state. (10) The vehicle control system according to (8), in which the recognition performance determination unit weights a reliability of the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state. (11) The vehicle control system according to any one of (1) to (10), in which the recognition performance includes at least one of a recognition capability of a passenger and a detection capability of a vehicle exterior sensor installed outside the vehicle. (12) The vehicle control system according to any one of (1) to (11), in which the environmental state includes at least one of presence or absence of intense light, hydrometeors, a road surface condition, and presence or absence in a dark place. (13) The vehicle control system according to (12), in which the intense light includes at least one of sunlight and a headlight beam of another vehicle. (14) The vehicle control system according to (12), in which the road surface condition includes that a road surface on which the vehicle travels is at least one of a frozen state, a snowy state, and an irregular ground. (15) The vehicle control system according to any one of (1) to (14), in which the feedback control unit controls a driving support function of the vehicle according to the determination result. (16) The vehicle control system according to (15), in which the driving support function includes at least one of irradiation control of a headlight, speed restriction, automatic stop, and driving mode switching. (17) The vehicle control system according to any one of (1) to (14), in which the feedback control unit controls information presentation to a passenger according to the determination result. (18) The vehicle control system according to any one of (1) to (14), in which the feedback control unit controls information presentation to another vehicle according to the determination result. (19) A vehicle control device including: an information acquisition unit that acquires sensing information on the basis of data acquired by a sensor installed in a vehicle interior; a recognition performance determination unit that determines deterioration in recognition performance due to an environmental state outside the vehicle on the basis of the sensing information; and a feedback control unit that performs feedback control related to traveling of the vehicle according to a determination result of deterioration in the recognition performance. (20) A vehicle control method including: acquiring sensing information on the basis of data acquired by a sensor installed in a vehicle interior; determining deterioration in recognition performance due to an environmental state outside the vehicle on the basis of the sensing information; and performing feedback control related to traveling of the vehicle according to a determination result of deterioration in the recognition performance.

[0199] (1) A vehicle control system comprising: circuitry configured to acquire sensing information based on data acquired by a sensor installed in a vehicle interior, determine an environmental state outside the vehicle based on the sensing information, and perform feedback control related to traveling of the vehicle according to the determined environmental state. (2) The vehicle control system according to (1), wherein the circuitry is further configured to acquire at least one of: passenger state information indicating a state of a passenger as the sensing information; window state information indicating a state of a window of the vehicle; seat state information indicating a state of a seat in the vehicle; and vehicle exterior state information indicating a state outside the vehicle. (3) The vehicle control system according to (2), wherein the passenger state information includes expression information of a face of the passenger and posture information of the passenger. (4) The vehicle control system according to (2), wherein the seat state information includes illuminance information of a surface of the seat. (5) The vehicle control system according to (2), wherein the vehicle exterior state information includes position and posture information of a vehicle exterior obstacle existing outside the vehicle, mirror image information reflected on a side mirror of the vehicle, and road surface information for a road surface on which the vehicle travels. (6) The vehicle control system according to (1), wherein the circuitry is further configured to determine whether recognition performance has deteriorated due to the environmental state by determining at least one of deterioration in recognition capability of a passenger and deterioration in detection capability of a vehicle exterior sensor installed outside the vehicle, and perform the feedback control according to a determination that the recognition performance has deteriorated. (7) The vehicle control system according to (6), wherein the circuitry is further configured to optimize the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state. (8) The vehicle control system according to (7), wherein the circuitry is further configured to select the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state. (9) The vehicle control system according to (7), wherein the circuitry is further configured to weight a reliability of the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state. (10) The vehicle control system according to (1), wherein the recognition performance includes at least one of a recognition capability of a passenger and a detection capability of a vehicle exterior sensor installed outside the vehicle. (11) The vehicle control system according to (1), wherein the environmental state includes at least one of presence or absence of intense light, hydrometeors, a road surface condition, and presence or absence of darkness. (12) The vehicle control system according to (1), wherein the circuitry is further configured to control a driving support function of the vehicle according to the determination result. (13) The vehicle control system according to (12), wherein the driving support function includes at least one of irradiation control of a headlight, speed restriction, automatic stop, and driving mode switching. (14) The vehicle control system according to (1), wherein the circuitry is further configured to control information presentation to a passenger according to the determination result. (15) The vehicle control system according to (1), herein the circuitry is further configured to control information presentation to another vehicle according to the determination result. (16) The vehicle control system according to (1), wherein the sensor includes a polarization image sensor, and the circuitry is further configured to determine a road surface condition based on a degree of polarization of a road surface acquired by the polarization image sensor. (17) The vehicle control system according to (16), wherein the circuitry is further configured to determine at least one of whether the road surface is wet and whether a puddle exists on the road surface, based on the degree of polarization of the road surface. (18) The vehicle control system according to (16), wherein the circuitry is further configured to determine whether the road surface is frozen based on the degree of polarization of the road surface. (19) A vehicle control device comprising: circuitry configured to acquire sensing information based on data acquired by a sensor installed in a vehicle interior, determine an environmental state outside the vehicle based on the sensing information, and perform feedback control related to traveling of the vehicle according to the determined environmental state. (20) A vehicle control method comprising: acquiring sensing information based on data acquired by a sensor installed in a vehicle interior; determining an environmental state outside the vehicle based on the sensing information; and performing feedback control related to traveling of the vehicle according to the determined environmental state.

[0200] 1 Vehicle 11 Vehicle control system 26 In-vehicle sensor 300 Vehicle control system 310, 310-1 to 310-N In-cabin sensor 320 Information acquisition unit 330 Recognition performance determination unit 340 Feedback output unit

Claims

A vehicle control system comprising:circuitry configured toacquire sensing information based on data acquired by a sensor installed in a vehicle interior,determine an environmental state outside the vehicle based on the sensing information, andperform feedback control related to traveling of the vehicle according to the determined environmental state.The vehicle control system according to claim 1, wherein the circuitry is further configured toacquire at least one of:passenger state information indicating a state of a passenger as the sensing information;window state information indicating a state of a window of the vehicle;seat state information indicating a state of a seat in the vehicle; andvehicle exterior state information indicating a state outside the vehicle.The vehicle control system according to claim 2, whereinthe passenger state information includes expression information of a face of the passenger and posture information of the passenger.The vehicle control system according to claim 2, whereinthe seat state information includes illuminance information of a surface of the seat.The vehicle control system according to claim 2, whereinthe vehicle exterior state information includes position and posture information of a vehicle exterior obstacle existing outside the vehicle, mirror image information reflected on a side mirror of the vehicle, and road surface information for a road surface on which the vehicle travels.The vehicle control system according to claim 1, wherein the circuitry is further configured todetermine whether recognition performance has deteriorated due to the environmental state by determining at least one of deterioration in recognition capability of a passenger and deterioration in detection capability of a vehicle exterior sensor installed outside the vehicle, andperform the feedback control according to a determination that the recognition performance has deteriorated.The vehicle control system according to claim 6, wherein the circuitry is further configured tooptimize the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state.The vehicle control system according to claim 7, wherein the circuitry is further configured toselect the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state.The vehicle control system according to claim 7, wherein the circuitry is further configured toweight a reliability of the sensing information to be used for determination of deterioration in the recognition performance according to the environmental state.The vehicle control system according to claim 1, whereinthe recognition performance includes at least one of a recognition capability of a passenger and a detection capability of a vehicle exterior sensor installed outside the vehicle.The vehicle control system according to claim 1, whereinthe environmental state includes at least one of presence or absence of intense light, hydrometeors, a road surface condition, and presence or absence of darkness.The vehicle control system according to claim 1, wherein the circuitry is further configured tocontrol a driving support function of the vehicle according to the determination result.The vehicle control system according to claim 12, whereinthe driving support function includes at least one of irradiation control of a headlight, speed restriction, automatic stop, and driving mode switching.The vehicle control system according to claim 1, wherein the circuitry is further configured tocontrol information presentation to a passenger according to the determination result.The vehicle control system according to claim 1, wherein the circuitry is further configured tocontrol information presentation to another vehicle according to the determination result.The vehicle control system according to claim 1, wherein the sensor includes a polarization image sensor, and the circuitry is further configured todetermine a road surface condition based on a degree of polarization of a road surface acquired by the polarization image sensor.The vehicle control system according to claim 16, wherein the circuitry is further configured todetermine at least one of whether the road surface is wet and whether a puddle exists on the road surface, based on the degree of polarization of the road surface.The vehicle control system according to claim 16, wherein the circuitry is further configured todetermine whether the road surface is frozen based on the degree of polarization of the road surface.A vehicle control device comprising:circuitry configured toacquire sensing information based on data acquired by a sensor installed in a vehicle interior,determine an environmental state outside the vehicle based on the sensing information, andperform feedback control related to traveling of the vehicle according to the determined environmental state.A vehicle control method comprising:acquiring sensing information based on data acquired by a sensor installed in a vehicle interior;determining an environmental state outside the vehicle based on the sensing information; andperforming feedback control related to traveling of the vehicle according to the determined environmental state.