Information processing method and information processing system

Abstract

[Problem] To provide a technology capable of controlling a vehicle so as to perform more appropriate lane changes. [Solution] Provided is an information processing method that is executed by a processor and includes: acquiring a trained model generated by machine learning based on first sensor data obtained when a first vehicle changes lanes; and determining, on the basis of second sensor data and the trained model, whether a lane change is to be carried out by a second vehicle that is the same as or different from the first vehicle.

Description

Information processing method and information processing system

[0001] This disclosure relates to an information processing method and an information processing system.

[0002] In recent years, advancements have been made in the development of technologies related to autonomous driving of vehicles. For example, technologies for proactively avoiding vehicle risks have been developed. For instance, Patent Document 1 discloses a technology for proactively avoiding vehicle risks by calculating risk values ​​for each location on the road and calculating a driving route corresponding to the risk value for each location.

[0003] Japanese Patent Publication No. 2021-123254

[0004] When a vehicle is in autonomous driving mode, it will automatically change lanes. However, controlling the vehicle to ensure that lane changes are performed appropriately can be challenging.

[0005] Therefore, it is desirable to have technology that can control vehicles to enable more appropriate lane changes.

[0006] According to this disclosure, there is an information processing method performed by a processor that includes: obtaining a trained model generated by machine learning based on first sensor data obtained when a first vehicle changes lanes; and determining whether a second vehicle, which is the same as or different from the first vehicle, will change lanes based on second sensor data and the trained model.

[0007] Furthermore, according to this disclosure, an information processing system is provided which includes a processor that acquires a trained model generated by machine learning based on first sensor data obtained when the first vehicle changes lanes, and determines whether or not to perform a lane change with a second vehicle that is the same as or different from the first vehicle, based on second sensor data and the trained model.

[0008] This is a block diagram showing an example of the configuration of a vehicle control system. This is a diagram showing an example of the sensing area of ​​the external recognition sensor of the vehicle control system in Figure 1. This is a diagram showing an example of the configuration of the information processing system 10 according to the embodiment of this disclosure. This is a diagram showing an example of the configuration of the vehicle control ECU 21 according to the embodiment of this disclosure. This is a diagram showing an example of the configuration of the server 2 according to the embodiment of this disclosure. This is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the server 2. This is a flowchart showing an example of processing by the information processing system 10 when the mode of vehicle 1 is manual driving mode. This is a flowchart showing the details of data recording processing (S15). This is a diagram for explaining an example of collision risk determination. This is a diagram for explaining an example of attributes. This is a diagram for explaining an example of learning the vehicle risk calculation model M31. This is a flowchart showing an example of processing by the information processing system 10 when the mode of vehicle 1 is automatic driving mode. This is a flowchart showing the details of lane risk calculation processing (S41). This is a flowchart showing the details of lane detection processing (S412). This is a diagram for explaining an example of vehicle risk calculation. This is a flowchart showing the details of lane change cost calculation processing (S42). This is a diagram for explaining an example of operation by a user. This is a flowchart showing an example of annotation processing. This is a flowchart showing the details of object of interest identification processing (S23). This is a diagram illustrating an example of the driver's state. This is a diagram illustrating an example of the trajectory Tr1 of the gaze direction. This is the first diagram illustrating the relationship between the gaze direction and an object. This is the second diagram illustrating the relationship between the gaze direction and an object. This is a diagram illustrating the process of thinning out collected data. This is a flowchart illustrating an example of the learning process according to the embodiment of this disclosure. This is a diagram illustrating the generation of the category estimator M12 and the lane change presence / absence estimator M22. This is a diagram illustrating the inference process according to the embodiment of this disclosure.

[0009] Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0010] The explanation will proceed in the following order: 1. Example of a vehicle control system configuration 2. Example of an information processing system configuration 3. Generation of a vehicle risk estimator 4. Example of lane change control 5. Annotation processing 6. Learning processing 7. Inference processing 8. Effects 9. Modifications

[0011] <<1. Example of Vehicle Control System Configuration>> Figure 1 is a block diagram showing an example of the configuration of a vehicle control system 11, which is a non-limiting example of a mobile device control system to which this technology is applied.

[0012] The vehicle control system 11 is installed in the vehicle 1 and performs processing related to the automation of the vehicle's operation. This automation includes Level 1 to Level 5 driving automation, as well as remote driving and / or remote assistance of the vehicle 1 by a remote driver. The levels of driving automation may refer to the Society of Automotive Engineers (SAE) J3016™ APL2021 Levels of Driving Automation, where SAE Level 0 represents the lowest level of driving automation and SAE Level 5 represents the highest level of driving automation. For example, SAE Level 1 driving automation may consist of driver assistance functions that provide the driver with steering or brake / acceleration support, and SAE Level 5 driving automation may consist of an automated driving function that can drive the vehicle under all conditions.

[0013] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, an automated driving control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0014] Two or more (or, in some cases, all) of the following components are connected to communicate with each other via a communication network 41: the vehicle control ECU 21, the communication unit 22, the map information storage unit 23, the location information acquisition unit 24, the external recognition sensor 25, the in-vehicle sensor 26, the vehicle sensor 27, the memory unit 28, the driving automation control unit 29, the DMS 30, the HMI 31, and the vehicle control unit 32. The communication network 41 is composed of an in-vehicle communication network or bus that conforms to digital bidirectional communication standards such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), FlexRay®, and Ethernet®. In some embodiments, the communication network 41 may have two or more types of communication networks, and different types of communication networks may be used depending on the type of data being transmitted. For example, CAN may be applied to data related to vehicle control, and Ethernet may be applied to large-capacity data. In some embodiments, two or more (or possibly all) units of the vehicle control system 11 may be directly connected using wireless communication (e.g., relatively short-range communication) without going through the communication network 41. In some embodiments, the wireless communication may use near-field wireless communication technology. Non-limiting examples of near-field wireless communication technology include near-field communication (NFC) and Bluetooth®. In some embodiments, two or more (or possibly all) units of the vehicle control system 11 may be connected using the communication network 41 and wireless communication technology (e.g., near-field wireless communication technology).

[0015] In the following embodiment, where two or more units of the vehicle control system 11 communicate via the communication network 41, the description of the communication network 41 will be omitted. For example, in an embodiment where the vehicle control ECU 21 and the communication unit 22 communicate via the communication network 41, it will simply be described as the vehicle control ECU 21 and the communication unit 22 communicating.

[0016] The vehicle control ECU 21 is composed of various processors, such as a CPU (Central Processing Unit) and an MPU (Micro Processing Unit). The vehicle control ECU 21 controls the functions of the entire vehicle control system 11 or a part of it.

[0017] The communication unit 22 communicates with various devices inside the vehicle 1 (hereinafter referred to as in-vehicle devices), various devices outside the vehicle 1 (hereinafter referred to as external devices), other vehicles, base stations, etc., and transmits and receives various types of data. In some embodiments, the communication unit 22 may use multiple communication technologies to perform communication.

[0018] A non-limiting example of communication between the communication unit 22 and external equipment will be briefly described. In some embodiments, the communication unit 22 may communicate with servers (hereinafter referred to as "external servers") located on an external network via a base station or access point using wireless communication technology. Examples of non-limiting wireless communication technologies include 5G (fifth-generation mobile communication system), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc. External networks that the communication unit 22 can communicate with may include, for example, the internet, a cloud network, or a network specific to a carrier. The communication technology used by the communication unit 22 to communicate with an external network is not particularly limited, as long as it is a wireless communication technology that enables digital two-way communication at a predetermined communication speed and over a predetermined distance.

[0019] In some embodiments, the communication unit 22 may communicate with terminals located near the vehicle using P2P (Peer To Peer) technology. Terminals located near the vehicle include, for example, terminals worn by relatively slow-moving objects such as pedestrians and cyclists, terminals installed in fixed locations such as stores, and / or MTC (Machine Type Communication) terminals. In some embodiments, the communication unit 22 may perform V2X (Vehicle to Everything) communication. V2X communication generally refers to communication between the vehicle and other entities. Non-exclusive examples of V2X communication include vehicle-to-vehicle communication with other vehicles, vehicle-to-infrastructure communication with roadside devices, etc., vehicle-to-home communication with homes, and vehicle-to-pedestrian communication with terminals carried or worn by pedestrians.

[0020] In some embodiments, the communication unit 22 may receive a program from outside the vehicle 1 to update the software that controls the operation of the vehicle control system 11 (for example, over the air). In some embodiments, the communication unit 22 may receive map information, traffic information, information about the vehicle 1's surroundings, etc., from outside the vehicle 1. In some embodiments, the communication unit 22 may transmit information about the vehicle 1, information about the vehicle 1's surroundings, etc., to an external device or external network. Non-limiting examples of information about the vehicle 1 that the communication unit 22 transmits to an external device or external network include data indicating the status of the vehicle 1, recognition results from the recognition unit 73, etc. In some embodiments, the communication unit 22 may communicate with a vehicle emergency call system. Non-limiting examples of a vehicle emergency call system include e-Call, etc.

[0021] In some embodiments, the communication unit 22 may receive electromagnetic waves transmitted by a road traffic information communication system. In some embodiments, such electromagnetic waves may be transmitted using radio beacons, optical beacons, FM multiplex broadcasting, etc.

[0022] A non-limiting example of communication with in-vehicle equipment that the communication unit 22 can perform will be outlined below. In some embodiments, the communication unit 22 may communicate with in-vehicle equipment using wireless communication. For example, in some embodiments, the communication unit 22 may communicate with in-vehicle equipment wirelessly using wireless communication technology that enables digital bidirectional communication at a predetermined or higher communication speed. Non-limiting examples of wireless communication technologies include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). Not limited to these, the communication unit 22 may also communicate with in-vehicle equipment using wired communication (in addition to or as an alternative to wireless communication). For example, in some embodiments, the communication unit 22 may communicate with in-vehicle equipment via wired communication through a cable connected to a connection terminal (not shown). In some embodiments, the communication unit 22 may communicate with in-vehicle equipment using wired communication technology that enables digital bidirectional communication at a predetermined or higher communication speed. Non-exclusive examples of wired communication technologies include USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), and MHL (Mobile High-definition Link).

[0023] Here, in-vehicle equipment refers to, for example, equipment located inside vehicle 1 that is not connected to the communication network 41. In-vehicle equipment is divided into equipment that constitutes the vehicle control system 11 and equipment that does not. Non-exclusive examples of in-vehicle equipment that does not constitute the vehicle control system 11 include mobile devices and wearable devices owned by users of vehicle 1 (e.g., the driver, passengers), and information equipment temporarily installed inside vehicle 1. These devices can, for example, be moved outside vehicle 1 and become external equipment.

[0024] The map information storage unit 23 stores maps acquired from external devices or external networks and / or maps created by the vehicle 1. For example, the map information storage unit 23 may store three-dimensional high-precision maps, global maps with lower precision than high-precision maps but covering a wide area, etc.

[0025] High-precision maps include, for example, dynamic maps, point cloud maps, and vector maps. A dynamic map may be a map consisting of four layers: dynamic information, semi-dynamic information, semi-static information, and static information, and may be provided to vehicle 1 from an external server or the like. A point cloud map may be a map composed of point clouds (point cloud data). A vector map may be a map adapted for automated driving by associating traffic information, such as the locations of lanes and traffic lights, with a point cloud map.

[0026] The point cloud map and vector map may be provided from, for example, an external server, or they may be created in the vehicle 1 as maps for matching with the local map described later, based on sensing results from the camera 51, radar 52, LiDAR 53, etc., and stored in the map information storage unit 23. In addition, if high-precision maps are provided from an external server, in order to reduce communication capacity, map data of, for example, several hundred meters square, relating to the planned route that the vehicle 1 will travel may be obtained from the external server.

[0027] The location information acquisition unit 24 acquires location information of the vehicle 1. The acquired location information may be supplied to the driving automation control unit 29. In some embodiments, the location information acquisition unit 24 may receive GNSS (Global Navigation Satellite System) signals from GNSS satellites. In some embodiments, the location information acquisition unit 24 may receive signals from beacons or the like.

[0028] The external recognition sensor 25 is equipped with various sensors used to recognize the external conditions of the vehicle 1, and supplies sensor data from one or more (or, in some cases, all) sensors to one or more (or, in some cases, all) units of the vehicle control system 11. The types and number of sensors equipped in the external recognition sensor 25 are arbitrary.

[0029] In some embodiments, the external recognition sensor 25 may include a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. However, the external recognition sensor 25 may also be configured to include one or more of the cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54. The number of cameras 51, radar 52, LiDAR 53, and ultrasonic sensor 54 is not particularly limited as long as it is a number that can be realistically installed in the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of the sensing areas of each sensor included in the external recognition sensor 25 will be described later.

[0030] Camera 51 can use any suitable shooting method. In some embodiments, camera 51 may use a shooting method capable of distance measurement. Non-limiting examples of cameras using a shooting method capable of distance measurement include ToF (Time of Flight) cameras, stereo cameras, monocular cameras, and infrared cameras. However, camera 51 may not be limited to these and may simply be for acquiring images, regardless of distance measurement.

[0031] In some embodiments, the external recognition sensor 25 may include environmental sensors for detecting characteristics of the environment around the vehicle 1. Non-limiting examples of detectable environmental characteristics include weather, climate, brightness, etc. In some embodiments, the environmental sensors may include various sensors such as raindrop sensors, fog sensors, sunshine sensors, snow sensors, and illuminance sensors.

[0032] In some embodiments, the external recognition sensor 25 may include a microphone used for detecting sounds around the vehicle 1 and the location of sound sources.

[0033] The in-vehicle sensor 26 is equipped with various sensors for detecting information inside the vehicle 1, and supplies sensor data from one or more (or, in some cases, all) sensors to one or more (or, in some cases, all) units of the vehicle control system 11. The types and number of sensors equipped in the in-vehicle sensor 26 are not particularly limited, as long as they are of a type and number that can be realistically installed in the vehicle 1.

[0034] In some embodiments, the in-vehicle sensor 26 may include one or more sensors from among a camera, radar, seat sensor, microphone, and biosensor. In some embodiments, the camera included in the in-vehicle sensor 26 may use a distance-measuring shooting method. Non-limiting examples of cameras using a distance-measuring shooting method include ToF cameras, stereo cameras, monocular cameras, and infrared cameras. However, the camera included in the in-vehicle sensor 26 may not be for distance measurement and may simply be for acquiring captured images. The biosensor included in the in-vehicle sensor 26 may be installed, for example, on the seat or steering wheel, and may detect various biometric information of the user.

[0035] The vehicle sensor 27 is equipped with various sensors for detecting the state of the vehicle 1 and supplies sensor data from one or more (or, in some cases, all) sensors to one or more (or, in some cases, all) units of the vehicle control system 11. The types and number of sensors equipped in the vehicle sensor 27 are not particularly limited, as long as they are of a type and number that can be realistically installed on the vehicle 1.

[0036] In some embodiments, the vehicle sensor 27 may comprise a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and / or an inertial measurement unit (IMU (Inertial Measurement Unit)) integrating these sensors. In some embodiments, the vehicle sensor 27 includes a steering angle sensor that detects the steering angle of a steering wheel, a yaw rate sensor, an accelerator sensor that detects an operation amount of an accelerator pedal (e.g., pedal force, pedal stroke), and / or a brake sensor that detects an operation amount of a brake pedal (e.g., pedal force, pedal stroke) may be provided. In some embodiments, the vehicle sensor 27 may include a rotation sensor that detects the rotation speed of an engine or a motor, an air pressure sensor that detects tire air pressure, a slip rate sensor that detects a tire slip rate, and / or a wheel speed sensor that detects the rotation speed of a wheel. In some embodiments, the vehicle sensor 27 may comprise a battery sensor that detects the remaining amount and temperature of a battery, and / or an impact sensor capable of detecting an impact from the outside.

[0037] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Non-limiting examples of the storage medium include magnetic storage devices such as EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access Memory) and / or HDD (Hard Disc Drive), semiconductor storage devices, optical storage devices, and magneto-optical storage devices. The storage unit 28 stores various programs and data used by one or more (or in some cases all) units of the vehicle control system 11. In some embodiments, the storage unit 28 may include an EDR (Event Data Recorder) or a DSSAD (Data Storage System for Automated Driving), and store information of the vehicle 1 before and after an event such as an accident or information acquired by the in-vehicle sensor 26.

[0038] The driving automation control unit 29 controls the driving automation function of the vehicle 1. In some embodiments, the driving automation control unit 29 may include an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0039] The analysis unit 61 performs analysis processing on the vehicle 1 and / or surrounding conditions. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0040] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on sensor data from the external recognition sensor 25 and the high-definition map stored in the map information storage unit 23. For example, the self-position estimation unit 71 may generate a local map based on sensor data from the external recognition sensor 25, and estimate the self-position of the vehicle 1 by performing matching between the local map and the high-definition map. For example, the center of the rear wheel axle may be used as the reference for the position of the vehicle 1.

[0041] In some embodiments, the local map may be a three-dimensional high-definition map created using a technology such as SLAM (Simultaneous Localization and Mapping), an Occupancy Grid Map, or the like. The three-dimensional high-definition map may be, for example, the above-mentioned point cloud map or the like. The occupancy grid map may be a map that divides the three-dimensional or two-dimensional space around the vehicle 1 into grids of a predetermined size, and indicates the occupation state of an object in grid units. The occupation state of an object may be indicated by, for example, the presence or absence of the object or the existence probability of the object. In some embodiments, the local map may also be used for, for example, detection processing and / or recognition processing of the external situation of the vehicle 1 by the recognition unit 73.

[0042] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and / or sensor data from the vehicle sensor 27.

[0043] The sensor fusion unit 72 performs sensor fusion processing to obtain information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data are not limited to these, but include composite, integrated, fused, and combined methods.

[0044] The recognition unit 73 performs a detection process to detect the external conditions of the vehicle 1, and / or a recognition process to recognize the external conditions of the vehicle 1.

[0045] For example, the recognition unit 73 may perform detection and / or recognition processing of the external conditions of the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0046] Specifically, for example, the recognition unit 73 may perform detection and / or recognition processing of objects around the vehicle 1. Object detection processing may include, for example, detecting the presence, size, shape, position, and movement of an object. Object recognition processing may include, for example, recognizing the type or attributes of an object, or identifying a specific object. Detection processing and recognition processing are not necessarily clearly separated, and at least some overlap may occur.

[0047] In some embodiments, the recognition unit 73 may detect objects around the vehicle 1 by performing clustering, which classifies the point cloud based on sensor data from the radar 52 and / or LiDAR 53 into clusters of points. This allows for the detection of the presence, size, shape, and position of objects around the vehicle 1.

[0048] In some embodiments, the recognition unit 73 may detect the movement of objects around the vehicle 1 by tracking the movement of clusters of points classified by clustering. This allows for the detection of the velocity and / or direction of travel (movement vector) of objects around the vehicle 1.

[0049] In some embodiments, the recognition unit 73 may detect and / or recognize vehicles (including bicycles), people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc., based on image data supplied from the camera 51. In some embodiments, the recognition unit 73 may recognize the types of objects around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0050] In some embodiments, the recognition unit 73 may perform a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the self-position estimation result by the self-position estimation unit 71, and / or the recognition result of objects around the vehicle 1 by the recognition unit 73. Through this process, the recognition unit 73 may recognize the location and / or status of traffic signals, the content of traffic signs and / or road markings, the content of traffic regulations, and / or drivable lanes.

[0051] In some embodiments, the recognition unit 73 may perform recognition processing of the environment surrounding the vehicle 1. In some embodiments, the recognition unit 73 may recognize weather characteristics (temperature, humidity, brightness), and / or road surface conditions, etc.

[0052] The action planning unit 62 creates an action plan for vehicle 1. For example, the action planning unit 62 may create an action plan by performing route planning and route following.

[0053] In some embodiments, the path planning may include global path planning and local path planning. Global path planning may include the process of planning a rough route from the start to the goal. Local path planning, also called trajectory planning, may include the generation of a trajectory that allows the vehicle 1 to travel safely and smoothly along the planned route in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1 and the presence of any obstacles.

[0054] In some embodiments, route following may involve planning actions to safely and accurately travel along the route planned by the route planner within a planned time. The action planning unit 62 may, for example, calculate the target speed and / or target angular velocity of the vehicle 1 based on the results of this route following process.

[0055] The motion control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62.

[0056] For example, in some embodiments, the motion control unit 63 may control the steering control unit 81, brake control unit 82, and / or drive control unit 83, which are included in the vehicle control unit 32 described later, to perform lateral vehicle motion control and / or longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the motion control unit 63 may perform one or more driver assistance functions and / or control for the purpose of driving automation (e.g., lateral vehicle motion control, longitudinal vehicle motion control). Non-limited examples of driver assistance functions include collision avoidance or impact mitigation, inter-vehicle distance control (e.g., control to maintain a specific distance from a vehicle traveling in front of the vehicle 1), vehicle speed control (e.g., control to maintain a specific speed), vehicle collision warning, and lane departure warning. Non-limited examples of driving automation include driving without operation by the driver or remote driver.

[0057] In some embodiments, the DMS 30 may perform driver authentication processing and / or driver status recognition processing based on sensor data from the in-vehicle sensor 26 and / or input data input to the HMI 31, which will be described later. Non-limited examples of driver status that may be recognized include physical condition, alertness level, concentration level, fatigue level, gaze direction, intoxication level, driving operation, posture, etc.

[0058] In some embodiments, the DMS 30 may perform authentication processing for users other than the driver (e.g., passengers) and / or recognition processing for the status of such users. In some embodiments, the DMS 30 may perform recognition processing for the internal conditions of the vehicle 1 based on sensor data from the in-vehicle sensors 26. Non-limiting examples of characteristics of the internal conditions of the vehicle 1 that may be recognized include temperature, humidity, brightness, odor, etc.

[0059] HMI31 receives various data and instructions as input and presents various data to the user.

[0060] A brief overview of data input to the HMI 31 is provided. The HMI 31 is equipped with an input device for a person to input data, instructions, etc. Based on the data, instructions, etc. input by the input device, the HMI 31 generates an input signal and supplies it to one or more (or, in some cases, all) units of the vehicle control system 11. In some embodiments, the HMI 31 may be equipped with a touch panel, buttons, switches, and / or levers as input devices. Not limited to these, the HMI 31 may be equipped with an input device that allows information to be input by methods other than manual operation, such as voice or gestures. In some embodiments, the HMI 31 may be equipped with a remote control device using infrared and / or radio waves, or an external connection device that corresponds to the operation of the vehicle control system 11, as an input device. Non-limited examples of external connection devices include mobile devices (e.g., smartphones) and wearable devices (e.g., smartwatches).

[0061] A brief explanation of data presentation by HMI31 is provided below. HMI31 generates visual, auditory, and / or tactile information for the user and / or people outside of vehicle 1. HMI31 may also perform output control to control the output, output content, output timing, and / or output method of each generated piece of information. Non-limited examples of visual information that can be generated and output by HMI31 include information shown by images and light, such as operation screens, vehicle 1 status displays, warning displays, and monitor images showing the surroundings of vehicle 1. Non-limited examples of auditory information that can be generated and output by HMI31 include voice guidance, warning sounds, and warning messages. Non-limited examples of tactile information that can be generated and output by HMI31 include information given to the user's sense of touch through force, vibration, movement, etc.

[0062] In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device that presents visual information by displaying an image itself, or a projector device that presents visual information by projecting an image. In some embodiments, the display device may be a device that displays visual information within the user's field of view, such as a head-up display, a transparent display, or a wearable device with AR (Augmented Reality) functionality, in addition to or as an alternative to a normal display device. In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device provided in the vehicle 1, such as a navigation device, instrument panel, CMS (Camera Monitoring System), electronic mirror, lamp, etc.

[0063] In some embodiments, the HMI31 may include an audio speaker, headphones, or earphones as an output device capable of outputting auditory information.

[0064] In some embodiments, the HMI 31 may include a haptic element using haptic technology as an output device capable of outputting tactile information. The haptic element may be provided, for example, on a part of the vehicle 1 that the user comes into contact with, such as the steering wheel or the seat.

[0065] The vehicle control unit 32 controls one or more (or, in some cases, all) units of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0066] The steering control unit 81 detects and / or controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism with a steering wheel, an electric power steering system, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0067] The brake control unit 82 detects and / or controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an ABS (Antilock Brake System), a regenerative braking mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0068] The drive control unit 83 detects and / or controls the state of the vehicle 1's drive system. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating driving force such as an internal combustion engine or drive motor, and a drive force transmission mechanism for transmitting driving force to the wheels. The drive control unit 83 also includes, for example, a drive ECU for controlling the drive system and an actuator for driving the drive system.

[0069] The body system control unit 84 detects and / or controls the state of the body system of the vehicle 1. The body system includes, for example, a keyless entry system, a smart key system, power window devices, power seats, an air conditioning system, airbags, seat belts, a shift lever, etc. The body system control unit 84 also includes, for example, a body system ECU that controls the body system, actuators that drive the body system, etc.

[0070] The light control unit 85 detects and / or controls the state of various lights on the vehicle 1. Non-exclusive examples of lights that can be controlled by the light control unit 85 include headlights, taillights, fog lights, turn signals, brake lights, projector lights, bumper indicators, etc. The light control unit 85 includes a light ECU for controlling the lights, actuators for driving the lights, etc.

[0071] The horn control unit 86 detects and / or controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU for controlling the car horn, an actuator for driving the car horn, and the like.

[0072] Figure 2 shows an example of the sensing area of ​​the external recognition sensor 25 in Figure 1, including the camera 51, radar 52, LiDAR 53, and ultrasonic sensor 54. In Figure 2, a schematic view of the vehicle 1 from above is shown.

[0073] Sensing regions 101F and 101B show examples of sensing regions for ultrasonic sensors 54. Sensing region 101F (for example, the sensing region of multiple ultrasonic sensors 54) covers the area around the front end of vehicle 1. Sensing region 101B (for example, the sensing region of multiple ultrasonic sensors 54) covers the area around the rear end of vehicle 1.

[0074] The sensing results in sensing region 101F and / or sensing region 101B may be used, for example, for parking assistance of vehicle 1.

[0075] Sensing areas 102F, 102B, 102L, and 102R are examples of sensing areas for short-range or medium-range radar 52. Sensing area 102F covers a position further in front of vehicle 1 than sensing area 101F. Sensing area 102B covers a position further in rear of vehicle 1 than sensing area 101B. Sensing area 102L covers the area around the left rear of vehicle 1. Sensing area 102R covers the area around the right rear of vehicle 1.

[0076] The sensing results in sensing region 102F may be used, for example, to detect vehicles or pedestrians in front of vehicle 1. The sensing results in sensing region 102B may be used, for example, to prevent collisions behind vehicle 1. The sensing results in sensing region 102L and / or sensing region 102R may be used, for example, to detect one or more objects in the blind spots on the left and / or right sides of vehicle 1.

[0077] Sensing areas 103F, 103B, 103L, and 103R show examples of sensing areas by camera 51. Sensing area 103F covers a position further in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position further in rear of vehicle 1 than sensing area 102B. Sensing area 103L covers the left side of vehicle 1. Sensing area 103R covers the right side of vehicle 1.

[0078] The sensing results in sensing region 103F may be used, for example, for recognition of traffic lights and traffic signs, lane departure prevention support systems, and automatic headlight control systems. The sensing results in sensing region 103B may be used, for example, for parking assistance and / or surround view systems. The sensing results in sensing region 103L and / or sensing region 103R may be used, for example, for surround view systems.

[0079] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position further in front of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction of vehicle 1 than sensing area 103F.

[0080] The sensing results in the sensing region 104 may be used, for example, to detect objects such as surrounding vehicles.

[0081] Sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. Sensing area 105 covers a position further in front of vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction of vehicle 1 than sensing area 104.

[0082] The sensing results in the sensing area 105 may be used, for example, for ACC (Adaptive Cruise Control), emergency braking, collision avoidance, etc.

[0083] In some embodiments, the sensing areas of each sensor of the external recognition sensor 25 (e.g., camera 51, radar 52, LiDAR 53, ultrasonic sensor 54) may take various configurations other than those shown in Figure 2. Specifically, in some embodiments, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, or the LiDAR 53 may be configured to sense the rear of the vehicle 1. Furthermore, the installation positions of each sensor are not limited to the examples described above. Also, the number of sensors may be one or multiple.

[0084] The above describes an example of the configuration of the vehicle control system 11 according to the embodiment of this disclosure.

[0085] <<2. Example of Information Processing System Configuration>> Next, an example of the configuration of the information processing system 10 according to the embodiment of this disclosure will be described with reference to Figure 3. Figure 3 is a diagram showing an example of the configuration of the information processing system 10 according to the embodiment of this disclosure.

[0086] As shown in Figure 3, the information processing system 10 according to the embodiment of this disclosure comprises a vehicle control system 11 and a server 2. The vehicle control system 11 is installed in the vehicle 1. The vehicle control system 11 and the server 2 can communicate via a network 3. For example, the network 3 may correspond to the external network described above. The server 2 may be implemented by a computer. The server 2 may correspond to the external server described above. The information processing system 10 may be composed of multiple devices as shown in Figure 3, or it may be composed of one device.

[0087] Vehicle 1 travels on a road. The vehicle control system 11 then performs operations according to either an automatic driving mode or a manual driving mode. For example, the automatic driving mode may be a driving mode corresponding to SAE levels 1 to 5 as described above. On the other hand, the manual driving mode may be a driving mode corresponding to SAE level 0 as described above. Switching between the automatic driving mode and the manual driving mode may be done manually by the user, or it may be done automatically based on the fulfillment of predetermined switching conditions.

[0088] As part of the overall processing flow executed by the information processing system 10, the vehicle control system 11 collects sensor data for training. The vehicle control system 11 sends the sensor data for training to the server 2. The server 2 generates a trained model using machine learning based on the sensor data for training. The server 2 then sends the generated trained model to the vehicle control system 11. The vehicle control system 11 receives the trained model, performs inference based on the received trained model and the sensor data for inference, and implements an autonomous driving function based on the inference results.

[0089] In the following explanation, we primarily assume that the process of obtaining training sensor data and the process of performing inference are executed by the vehicle control system 11 of the same vehicle 1. However, there may be multiple vehicles 1 and multiple vehicle control systems 11. In such cases, the process of obtaining training sensor data and the process of performing inference do not have to be executed by the vehicle control system 11 of the same vehicle, but may be executed by the vehicle control systems 11 of different vehicles.

[0090] Alternatively, each of the multiple vehicle control systems 11 may transmit sensor data for training to the server 2. The server 2 may then generate a trained model based on the sensor data for training transmitted from each of the multiple vehicle control systems 11. The server 2 may then transmit the generated trained model to each of the multiple vehicle control systems 11. Each of the multiple vehicle control systems 11 may receive the trained model, perform inference based on the received trained model and the sensor data for inference, and implement an autonomous driving function based on the inference result.

[0091] A vehicle equipped with a vehicle control system 11 that processes sensor data for training is referred to as the first vehicle. On the other hand, a vehicle equipped with a vehicle control system 11 that performs inference based on a trained model and sensor data for inference is referred to as the second vehicle. In the following description, the sensor data for training is the sensor data used by the model generation unit 253 and may correspond to the first sensor data. On the other hand, the sensor data for inference is the sensor data used by the inference unit 216 and may correspond to the second sensor data.

[0092] (Example of Functional Configuration of Vehicle Control ECU 21) Next, an example of the functional configuration of the vehicle control ECU 21 according to the embodiment of the present disclosure will be described with reference to Figure 4. Figure 4 is a diagram showing an example of the configuration of the vehicle control ECU 21 according to the embodiment of the present disclosure. As shown in Figure 4, the vehicle control ECU 21 includes a data provision unit 212, a model acquisition unit 214, and an inference unit 216.

[0093] The data provision unit 212 acquires sensor data obtained by the external recognition sensor 25 when the vehicle 1 changes lanes. The data provision unit 212 transmits the sensor data to the server 2 via the communication unit 22. The sensor data transmitted to the server 2 is used by the server 2 to generate a trained model. The trained model generated by the server 2 is transmitted from the server 2 to the vehicle control system 11.

[0094] The model acquisition unit 214 acquires the trained model transmitted from the server 2 via the communication unit 22. The model acquisition unit 214 then stores the acquired trained model in the storage unit 28.

[0095] After the model acquisition unit 214 acquires a trained model, the inference unit 216 acquires sensor data obtained from the external recognition sensor 25. Based on the acquired sensor data and the trained model stored in the storage unit 28, the inference unit 216 determines whether or not to change lanes with vehicle 1. Once it is determined whether or not to change lanes, the vehicle 1 can be controlled according to the determination result to make a more appropriate lane change.

[0096] A lane is also referred to as a "lane" and can be a demarcated area for vehicles to travel in. A lane change can mean changing the lane a vehicle is currently traveling in from the current lane to another lane. In the following explanation, changing lanes will be referred to as "with a lane change," and not changing lanes will be referred to as "without a lane change."

[0097] (Example of Server 2 Functional Configuration) Next, an example of the functional configuration of Server 2 according to the embodiment of this disclosure will be described with reference to Figure 5. Figure 5 is a diagram showing an example of the configuration of Server 2 according to the embodiment of this disclosure. As shown in Figure 5, Server 2 comprises a control unit 251, a storage unit 258, and a communication unit 259. The control unit 251 also comprises a data acquisition unit 252, a model generation unit 253, and a model provision unit 254.

[0098] The data acquisition unit 252 acquires sensor data transmitted from the vehicle control ECU 21 of the vehicle control system 11 of vehicle 1 via the communication unit 259. The data acquisition unit 252 then stores the acquired sensor data in the storage unit 258. The storage unit 258 may store not only sensor data transmitted from vehicle 1, but also sensor data transmitted similarly from vehicles other than vehicle 1.

[0099] The model generation unit 253 generates a trained model through machine learning based on sensor data stored in the memory unit 258. The machine learning algorithm used by the model generation unit 253 is not limited to a specific algorithm. For example, the machine learning algorithm used by the model generation unit 253 may be a neural network. Furthermore, the timing of the generation of the trained model by the model generation unit 253 is not limited. For example, the generation of the trained model may be performed periodically, or when the amount of data stored in the memory unit 258 reaches a predetermined amount.

[0100] The model provisioning unit 254 transmits the trained model generated by the model generation unit 253 to the vehicle control ECU 21 of the vehicle control system 11 of vehicle 1 via the communication unit 259. The model provisioning unit 254 may transmit the trained model not only to vehicle 1, but also to other vehicles.

[0101] In the following explanation, we primarily assume that both the category estimator M12 (Figure 27) and the lane change presence / absence estimator M22 (Figure 27) are generated by Server 2 as examples of trained models. However, it is sufficient that at least the lane change presence / absence estimator M22 is generated by Server 2; the category estimator M12 does not necessarily have to be generated. Note that the category estimator M12 may correspond to the first trained model. Also, the lane change presence / absence estimator M22 may correspond to the second trained model.

[0102] (Example of Server 2 Hardware Configuration) The information processing performed by Server 2 is realized through the cooperation of software and hardware. Below, an example of a computer 1000 hardware configuration that can be applied to Server 2 according to the embodiment of this disclosure will be described.

[0103] Figure 6 is a hardware configuration diagram showing an example of a computer 1000 that implements the functions of server 2. Computer 1000 includes a processing circuitry 1100, RAM 1200, ROM 1300, secondary storage device 1400, communication interface 1500, input / output interface 1600, display unit 1700, camera unit 1800, microphone 1900, and speaker 2000. The various parts of computer 1000 are connected by a bus 1050.

[0104] The processing circuit 1100 operates based on a program stored in the ROM 1300 or secondary storage device 1400, and controls each part. For example, the processing circuit 1100 loads the program stored in the ROM 1300 or secondary storage device 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0105] ROM 1300 stores boot programs such as the BIOS (Basic Input Output System) that are executed by the processing circuit 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0106] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the server 2 according to the embodiment of this disclosure, which is an example of program data 1450.

[0107] The communication interface 1500 is an interface for the computer 1000 to connect to the external network 1550. For example, the processing circuit 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0108] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from input devices such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to output devices such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0109] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescent display (Organic Electro Luminescence Display). Alternatively, the display unit 1700 may be a touch panel display device or an image projection device.

[0110] The camera unit 1800 is an interface for the computer 1000 to capture images. The microphone 1900 is an interface for the computer 1000 to capture sound. The speaker 2000 is an interface for the computer 1000 to output processed sound. The various parts of the computer 1000 are connected by the bus 1050. Each interface does not necessarily have to be located inside the computer 1000, but may be located outside the computer 1000 via a network or the like. Furthermore, each part of the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100, but by a circuit dedicated to display processing provided within the display unit 1700.

[0111] For example, when computer 1000 functions as server 2 according to an embodiment of this disclosure, the processing circuit 1100 of computer 1000 functions as a control unit (not shown) by executing a program loaded onto RAM 1200. The secondary storage device 1400 stores the information processing program according to this disclosure and various data stored in the map storage unit 120 and the prior knowledge storage unit 140. The processing circuit 1100 reads and executes the program data 1450 from the secondary storage device 1400, but as another example, these programs may be obtained from other devices via an external network 1550. In other words, the secondary storage device 1400 is not limited to being inside computer 1000, but may be located outside computer 1000. The processing circuit 1100 is an example of an integrated circuit, and CPU, MPU, GPU, APU, ASIC, and FPGA can all be considered integrated circuits.

[0112] The above describes an example of the configuration of the information processing system 10 according to the embodiment of this disclosure.

[0113] <<3. Generation of Vehicle Risk Estimator>> In order to determine whether or not a lane change has occurred, a risk value (hereinafter also referred to as "vehicle risk"), which is the magnitude of the risk that the vehicle poses to its surroundings, may be used. Vehicle risk can be calculated by the vehicle risk estimator M32 (Figure 15). The vehicle risk estimator M32 can be generated by updating the parameters of the vehicle risk calculation model M31 (Figure 11). An example of generating the vehicle risk estimator M32 will be explained with reference to Figures 7 to 11 (and other figures as appropriate).

[0114] Figure 7 is a flowchart showing an example of processing by the information processing system 10 when the mode of vehicle 1 is manual driving mode. However, the processing example shown in Figure 7 may also be executed when the mode of vehicle 1 is automatic driving mode, after S16 is replaced with "Automatic driving ended?". Alternatively, the processing example shown in Figure 12 may be executed when the mode of vehicle 1 is automatic driving mode. As shown in Figure 7, when the mode of vehicle 1 is manual driving mode, the data provision unit 212 acquires sensor data obtained by the external recognition sensor 25 (S11). Then, the recognition of the driver's state by the DMS 30 is also started.

[0115] Here, we primarily assume that the driver's state is represented by the direction of the driver's gaze, as recognized from image data obtained by the camera equipped with the in-vehicle sensor 26. However, the driver's state does not have to be represented by the direction of the gaze. For example, the orientation of the driver's face, as recognized from image data obtained by the camera equipped with the in-vehicle sensor 26, may be used as the driver's state. Alternatively, the content of the driver's speech, as recognized from sound detected by the microphone equipped with the in-vehicle sensor 26, may be used as the driver's state.

[0116] The data supply unit 212 then performs lane detection processing (S12) based on the sensor data. In the lane detection processing, the data supply unit 212 detects one or more lanes. Details of the lane detection processing (S12) will be explained later with reference to Figure 14.

[0117] The detected lanes may include the lane in which vehicle 1 is traveling, and either or both of the left-hand lane and the right-hand lane of the lane in which vehicle 1 is traveling. The lane in which vehicle 1 is traveling may be considered the first lane. In addition, either or both of the left-hand lane and the right-hand lane of the lane in which vehicle 1 is traveling (hereinafter also referred to as the "candidate lane for change") may be considered the second lane.

[0118] The data provision unit 212 estimates the position of the detected lane (S131). For example, the position of the lane estimated by the data provision unit 212 may be the three-dimensional position of the lane. Note that the process of detecting the lane and the process of estimating the position of the lane may be performed in parallel.

[0119] Next, the recognition unit 73 detects objects around vehicle 1 (S132). Furthermore, the recognition unit 73 recognizes the type of object. The types of objects recognized by the recognition unit 73 may include vehicles. That is, the recognition unit 73 may detect vehicles other than vehicle 1. The recognition unit 73 detects the position of the detected vehicle. In addition, the types of objects recognized by the recognition unit 73 may include intersections and obstacles. The recognition unit 73 detects the position of the detected intersection and the position of the obstacle.

[0120] Furthermore, the recognition unit 73 estimates the type of vehicle detected as a vehicle type. Examples of vehicle types include passenger cars, trucks, emergency vehicles, motorcycles, or bicycles. The recognition unit 73 also estimates the attributes of the detected vehicle (S133). Examples of vehicle attributes will be explained later with reference to Figure 10.

[0121] In the vehicle control ECU 21 of the vehicle control system 11 of vehicle 1, the data provision unit 212 determines whether or not a lane change by vehicle 1 has been detected (S14). Note that the lane change may be detected in any way.

[0122] For example, a change in the position of vehicle 1 to straddle the boundary between two different detected lanes may be detected as a lane change.

[0123] Alternatively, the activation of the turn signal may be detected as a lane change. The activation of the turn signal may be detected by receiving an instruction signal indicating that the turn signal has been activated, or it may be recognized based on the sound detected by the microphone provided by the in-vehicle sensor 26.

[0124] Alternatively, a lane change may be detected when vehicle 1 moves to the left or to the right. The movement of vehicle 1 to the left or to the right may be detected by the IMU (Infrared Monitoring Unit) of the vehicle sensor 27. Alternatively, a lane change may be detected based on image data obtained by the camera of the external recognition sensor 25.

[0125] If the data provision unit 212 does not detect a lane change by vehicle 1 (NO in S14), it returns to S14. On the other hand, if the data provision unit 212 detects a lane change by vehicle 1 (YES in S14), it performs data recording processing (S15).

[0126] Figure 8 is a flowchart detailing the data recording process (S15). As shown in Figure 8, in the data recording process, the data provision unit 212 records collected data in the server 2, including the position of the lane, the type of vehicle detected, the vehicle's position and attributes, the position of the intersection, the position of the obstacle, sensor data obtained by the external recognition sensor 25, and the driver's status (S150).

[0127] Furthermore, the sensor data, driver status, and environmental recognition results recorded on server 2 may each be data for a predetermined time period before and after lane change detection. The predetermined time period before and after lane change detection may be the time from N (where N > 0) seconds before lane change detection to N seconds after lane change detection.

[0128] Next, the data provision unit 212 determines the risk of collision between vehicle 1 and another vehicle (hereinafter also referred to as "collision risk") (S151). An example of collision risk determination will be explained with reference to Figure 9.

[0129] Figure 9 is a diagram illustrating an example of collision risk determination. Figure 9 shows state St1, in which vehicle 1 is traveling. Referring to state St1, other vehicles 4 to 6 are also traveling. For example, the risk of collision between vehicle 1 and other vehicle 6, which is traveling in the lane to the left of the lane in which vehicle 1 is traveling, may be below the threshold. On the other hand, the risk of collision between vehicle 1 and other vehicle 4, which is traveling in front of vehicle 1, and the risk of collision between vehicle 1 and other vehicle 5, which is traveling in the lane to the right, which is the lane that vehicle 1 is changing lanes to, may be above the threshold.

[0130] Specifically, the data provision unit 212 may determine whether the collision risk between the vehicle 1 and the other vehicle is above a threshold, based on whether it has detected that the distance between the position of the other vehicle and the position of the vehicle 1 is closer than a predetermined distance, based on the position of the other vehicle recognized by the recognition unit 73.

[0131] Alternatively, the data provision unit 212 may determine whether the risk of collision between its own vehicle 1 and the other vehicle is above a threshold, depending on whether or not it has detected a horn sound from another vehicle from the sound detected by the microphone provided by the external recognition sensor 25.

[0132] Alternatively, the data provision unit 212 may determine whether the collision risk between vehicle 1 and the other vehicle is above a threshold value, depending on whether the collision sensor provided by the vehicle sensor 27 detects that another vehicle has collided with vehicle 1.

[0133] Alternatively, the data provision unit 212 may determine whether the collision risk between its own vehicle 1 and another vehicle is above a threshold value, depending on whether the value of the detection data detected by the various sensors provided by the vehicle sensor 27 exceeds a predetermined value. The detection data may include the amount of brake pedal operation detected by the brake sensor provided by the vehicle sensor 27, the steering angle of the steering wheel detected by the steering angle sensor provided by the vehicle sensor 27, the acceleration detected by the acceleration sensor provided by the vehicle sensor 27, or the angular velocity detected by the angular velocity sensor (gyro sensor) provided by the vehicle sensor 27.

[0134] The data provision unit 212 terminates the data recording process if the collision risk between its own vehicle 1 and the other vehicle is below a threshold (NO in S153). On the other hand, if the collision risk between its own vehicle 1 and the other vehicle is above a threshold (YES in S153), the data provision unit 212 recognizes the other vehicle as a high-risk vehicle, records the vehicle type, location, and attributes of the high-risk vehicle in the server 2 (S154), and terminates the data recording process.

[0135] The vehicle type, location, and attributes of high-risk vehicles may be recognized by the recognition unit 73 based on sensor data stored in the storage unit 28 (for example, sensor data for a predetermined time before and after a lane change). Alternatively, the sensor data stored in the storage unit 28 may be transmitted to the server 2, and the server 2 may recognize the vehicle type, location, and attributes of high-risk vehicles based on the sensor data.

[0136] For example, if image data obtained by the camera 51 of the external recognition sensor 25 is stored in the storage unit 28, the vehicle recognized from the image data stored in the storage unit 28 may be recognized as a vehicle with a high risk. Information recorded in an EDR or DSSAD may be used to recognize the vehicle type, location, and attributes of a vehicle with a high risk.

[0137] Examples of attributes will be explained with reference to Figure 10. Figure 10 is a diagram illustrating examples of attributes. As shown in Figure 10, examples of attributes include vehicle "marks," "speed," "acceleration," "turn signals," "hazard lights," and "brake lights." Attributes may include some or all of these.

[0138] Possible values ​​for the "Mark" attribute include "New Driver" to indicate that the driver is a novice driver, "Elderly Driver" to indicate that the driver is an elderly driver, "Hearing Impaired Driver" to indicate that the driver is a hearing impaired person, "Physically Disabled Driver" to indicate that the driver is a physically disabled person, and "Child" to indicate that a child is on board.

[0139] Possible values ​​for the "velocity" attribute include the velocity vector v(vx, vy). Possible values ​​for the "acceleration" attribute include the acceleration vector a(ax, ay). Note that the velocity vector v and acceleration vector a may also be expressed as a combination of the x and y components in the xy-plane parallel to the road plane.

[0140] Possible values ​​for the "Turn Signal" attribute include a value indicating whether the vehicle's turn signal indicates "left" or "right". Possible values ​​for the "Hazard Lights" attribute include a value indicating whether the hazard lights are "ON" or "OFF". Possible values ​​for the "Brake Lights" attribute include a value indicating whether the brake lights are "ON" or "OFF".

[0141] When the data recording process is complete, the data supply unit 212 returns to Figure 7 to determine whether or not to terminate the manual operation mode (S16). If the data supply unit 212 determines not to terminate the manual operation mode (NO in S16), it returns to S14. On the other hand, if the data supply unit 212 determines to terminate the manual operation mode (YES in S16), it terminates the manual operation mode.

[0142] In the example shown in Figure 7, the server 2 primarily records the vehicle type, location, and attributes of vehicles with a high risk of collision during manual driving. However, if an automated driving simulation is performed using a driving simulator or the like, and the collision risk is determined to be above a threshold during the automated driving simulation, the server 2 may also record the vehicle type, location, and attributes of vehicles with a high risk of collision.

[0143] Figure 11 is a diagram illustrating an example of training the vehicle risk calculation model M31. As shown in Figure 11, the vehicle risk calculation model M31 is pre-stored in the storage unit 258. The data acquisition unit 252 acquires the vehicle type, vehicle location, and attributes of high-risk vehicles transmitted from vehicle 1. The model generation unit 253 trains the vehicle risk calculation model M31 by updating its parameters based on the vehicle type, vehicle location, and attributes.

[0144] Specifically, the model generation unit 253 may assign a value indicating a high vehicle risk to the vehicle type, vehicle location, and attributes as the correct answer value, and store data in the storage unit 258 that associates the vehicle type, vehicle location, attributes, and the correct answer value indicating a high vehicle risk. In addition, the storage unit 258 may store data in which the correct answer value indicating a high vehicle risk is associated to the vehicle type, vehicle location, and attributes of vehicles transmitted from other vehicles.

[0145] Furthermore, the memory unit 258 may store data that associates the vehicle type, vehicle location, attributes, and correct values ​​indicating low vehicle risk for vehicles with low vehicle risk. The model generation unit 253 may perform training on the vehicle risk calculation model M31 by calculating the error between the output from the vehicle risk calculation model M31 and the correct values ​​when the vehicle type, vehicle location, and attributes are input to the vehicle risk calculation model M31, and updating the parameters of the vehicle risk calculation model M31 to reduce the error.

[0146] Furthermore, it is not necessary for all of the vehicle type, vehicle location, and attributes to be input into the vehicle risk calculation model M31. In other words, one or two of the vehicle type, vehicle location, and attributes may be input into the vehicle risk calculation model M31.

[0147] For example, the correct value indicating high vehicle risk may be "1," and the correct value indicating low vehicle risk may be "0." The vehicle risk calculation model M31 may be constructed using a neural network, and the error calculated by the model generation unit 253 may be the mean squared error. However, the learning method for the vehicle risk calculation model M31 is not limited to this method. A vehicle risk estimator M32 (Figure 15) can be generated by learning such a vehicle risk calculation model M31.

[0148] The model provisioning unit 254 transmits the vehicle risk estimator M32 to the vehicle control ECU 21 of the vehicle control system 11 of vehicle 1 via the communication unit 259. The model provisioning unit 254 may transmit the vehicle risk estimator M32 not only to vehicle 1, but also to other vehicles. The vehicle risk estimator M32 transmitted to the vehicle control ECU 21 is stored in the storage unit 28 and can be used to calculate vehicle risk.

[0149] The above describes an example of generating a vehicle risk estimator M32 (Figure 15) according to the embodiment of this disclosure.

[0150] <<4. Examples of Lane Change Control>> Next, we will explain examples of lane change control with reference to Figures 12 to 17 (and other figures as appropriate). The lane change control examples explained with reference to Figures 12 to 17 may be implemented when the lane change presence / absence estimator M22 (Figure 27), which will be explained later, is used to determine whether or not a lane change is occurring, or they may be implemented during automated driving when the lane change presence / absence estimator M22 (Figure 27) is not used to determine whether or not a lane change is occurring.

[0151] Figure 12 is a flowchart showing an example of processing by the information processing system 10 when the mode of vehicle 1 is automatic driving mode. As shown in Figure 12, when the mode of vehicle 1 is automatic driving mode, the data provision unit 212 in the vehicle control ECU 21 of the vehicle control system 11 of vehicle 1 performs lane risk calculation processing (S41). The details of the lane risk calculation processing (S41) will be explained with reference to Figure 13.

[0152] Figure 13 is a flowchart showing the details of the lane risk calculation process (S41). As shown in Figure 13, the data provision unit 212 acquires sensor data obtained by the external recognition sensor 25 (S411). Then, the data provision unit 212 performs lane detection processing (S412) based on the sensor data. The details of the lane detection processing (S412) will be explained with reference to Figure 14.

[0153] Figure 14 is a flowchart detailing the lane detection process (S412). As shown in Figure 14, the data supply unit 212 determines whether a predetermined event has occurred (S4121). If the data supply unit 212 determines that the predetermined event has not occurred (NO in S4121), it returns to S4121. On the other hand, if the data supply unit 212 determines that the predetermined event has occurred (YES in S4121), it detects multiple lanes based on the sensor data (S4122).

[0154] When lane detection is performed, the lane detection process (S412) ends. In this way, lanes are detected when a predetermined event occurs, and not detected when the predetermined event does not occur, thus reducing the processing cost of the vehicle control ECU 21. Note that the type of event is not limited. For example, an event may include the elapsed time with a predetermined time interval as its period.

[0155] Alternatively, the event may include the fact that the road Vehicle 1 is traveling on has changed from one road to another. Such a change in the road Vehicle 1 is traveling on may be determined based on the position of Vehicle 1 on the SD (Standard-Definition) map. Alternatively, since there is no need to change lanes if the road Vehicle 1 is traveling on has only one lane, the event may also include the fact that the road Vehicle 1 is traveling on has multiple lanes.

[0156] Alternatively, the event may include the activation of a turn signal. The activation of a turn signal may be detected by receiving an instruction signal indicating that the turn signal has been activated, or it may be recognized based on sound detected by a microphone provided by the in-vehicle sensor 26. Alternatively, the event may include the vehicle 1 moving to the left or to the right. The vehicle 1 moving to the left or to the right may be detected by the IMU provided by the vehicle sensor 27.

[0157] Returning to Figure 13, the explanation continues. Similar to S131 (Figure 7), the data provision unit 212 estimates the position of the detected lane (S413). Subsequently, similar to S132 (Figure 7), the recognition unit 73 detects objects around vehicle 1 (S414). Furthermore, the recognition unit 73 recognizes the type of object. The types of objects recognized by the recognition unit 73 may include vehicles. That is, the recognition unit 73 may detect vehicles other than vehicle 1. The recognition unit 73 then detects the position of the detected vehicle.

[0158] Furthermore, the recognition unit 73 estimates the type of vehicle detected as a vehicle type. Also, similar to S133 (Figure 7), the recognition unit 73 estimates the attributes of the detected vehicle (S415). Examples of vehicle attributes are as explained with reference to Figure 10.

[0159] Next, the data provision unit 212 calculates the vehicle risk of the detected vehicle based on the vehicle type, vehicle location, attributes, and vehicle risk estimator M32 (Figure 15) (S416). An example of vehicle risk calculation will be explained with reference to Figure 15.

[0160] Figure 15 is a diagram illustrating an example of vehicle risk calculation. Referring to Figure 15, the vehicle risk estimator M32 is shown. The data supply unit 212 may calculate the vehicle risk as the output from the vehicle risk estimator M32 when the detected vehicle type, vehicle location, and attributes are input to the vehicle risk estimator M32.

[0161] Furthermore, similar to the case where input is made to the vehicle risk calculation model M31, it is not necessary for all of the vehicle type, vehicle location, and attributes to be input to the vehicle risk estimator M32. In other words, one or two of the vehicle type, vehicle location, and attributes may be input to the vehicle risk estimator M32.

[0162] Returning to Figure 13, the explanation continues. The data provision unit 212 associates the candidate lane to be changed with the vehicle based on the estimated lane position and the detected vehicle position (S417). That is, the data provision unit 212 determines which vehicle is in which candidate lane to be changed based on the estimated lane position and the detected vehicle position. Based on the association between the candidate lane to be changed and the vehicle, the data provision unit 212 calculates the risk for each lane as the lane risk (S418). The lane risk can be calculated based on the vehicle risk for each lane.

[0163] Here, let i be the detected candidate lane to change to (1 ≤ i ≤ L), and let R_lane be the lane risk. ｉ The distance between the vehicle 1 (the vehicle itself) and the detected vehicle k is defined as D_car ｋ The vehicle risk of vehicle k is R_car ｋ Therefore, lane risk R_lane ｉ This can be calculated as shown in equation (1) below.

[0164] ... (1)

[0165] In the above equation (1), the distance D_car ｋ The smaller the value, the lower the vehicle risk R_car ｋ The weight of R_car is increasing. This means that the vehicle risk of vehicles closer to the vehicle itself (vehicle 1) can be more significantly reflected in the lane risk of that lane. However, vehicle risk R_car ｋThe weight is not limited to such examples. For example, the vehicle risk R_car ｋ may be a constant value independent of the vehicle.

[0166] Returning to FIG. 12, the description will be continued. When the lane risk calculation process (S41) is executed, the data providing unit 212 executes a lane change cost calculation process (S42). The lane change cost is a cost required for the lane change by the vehicle 1. Details of the lane change cost calculation process (S42) will be described with reference to FIG. 16.

[0167] FIG. 16 is a flowchart showing details of the lane change cost calculation process (S42). As shown in FIG. 16, the data providing unit 212 sets the current time as t, sets "1" to the current time t, and reads the presence / absence of lane changes at past times from the storage unit 28 (S421).

[0168] For example, if the reading time (predetermined time) is M, and the presence / absence of a lane change at time j is change(j), the presence / absence of vehicle lane changes at past times can be expressed as change(t−1), ..., change(t−M). However, when the number of recorded vehicle lane changes stored in the storage unit 28 is less than M, all the presence / absence of lane changes stored in the storage unit 28 only need to be read. As an example, a value indicating that a lane change has been performed may be "1". Further, a value indicating that no lane change has been performed may be "0".

[0169] Subsequently, the data providing unit 212 calculates an intra-time lane change amount based on the presence / absence of vehicle lane changes at past times (S422). For example, the greater the number of lane changes at past times, the greater the lane change cost may be. This can reduce deterioration of riding comfort for the vehicle 1. For example, let the intra-time lane change amount at the current time t be ChangeCost(t, M), and the weight for the presence / absence of a lane change at time j be w ｊ Then, the intra-time lane change amount at the current time t can be calculated as shown in the following formula (2).

[0170] ... (2)

[0171] The weight w for the presence / absence of a lane change ｊThis can be set in any way. For example, the closer time i is to the current time t, the greater the weight given to whether or not a lane change occurred. ｊ This can be large. This allows the presence or absence of a lane change at time j close to the current time t to be significantly reflected in the amount of lane changes within the time, ChangeCost(t,M). However, the weight w of the presence or absence of a lane change ｊ This does not have to be limited to such examples. For example, the weight given to whether or not a lane change occurred. ｊ j may be a constant value that does not depend on time j.

[0172] Next, the data provision unit 212 calculates the necessity of changing lanes (S423). For example, the data provision unit 212 may calculate "high necessity" as the necessity of changing lanes if an emergency vehicle is near vehicle 1 or if the necessity of changing lanes is recognized based on the route plan. On the other hand, the data provision unit 212 may calculate "low necessity" as the necessity of changing lanes if an emergency vehicle is not near vehicle 1 and the necessity of changing lanes is not recognized based on the route plan.

[0173] "High necessity" may be a larger value than "Low necessity." For example, "High necessity" may be "1" and "Low necessity" may be "0."

[0174] Furthermore, the presence of an emergency vehicle near vehicle 1 may be detected by the recognition unit 73 recognizing the siren sound emitted by the emergency vehicle from the sound detected by the microphone equipped with the external recognition sensor 25. In addition, the route plan is created by the action planning unit 62, and the need for a lane change based on the route plan may be recognized, for example, when the route plan detects that vehicle 1 is scheduled to turn right or left.

[0175] Next, the data provision unit 212 calculates the lane change cost (S424). As an example, the lane risk of lane i at the current time t is calculated as R_lane ｉ Let (t) be the necessity of changing lanes at the current time t, and let Necessity(t) be the necessity of changing lanes at the current time t. Then the cost of changing lanes i at the current time t is Cost. ｉ(t) can be calculated as shown in equation (3) below.

[0176] Cost ｉ (t)=R_lane ｉ (t) + ChangeCost (t, M) - Necessity (t) ... (3)

[0177] Returning to Figure 12, the explanation continues. The data provision unit 212 determines whether or not there is a candidate lane to change to where the lane change cost is below a threshold (S43). If there is a candidate lane to change to where the lane change cost is below a threshold (YES in S43), the data provision unit 212 controls the vehicle control unit 32 so that vehicle 1 changes lanes to that candidate lane (S46). The data provision unit 212 then sets the value indicating the presence or absence of a lane change to Change(t) (S47).

[0178] On the other hand, if there are no candidate lanes for lane change where the lane change cost is below a threshold (NO in S43), the data provision unit 212 controls the vehicle control unit 32 so that the lane in which vehicle 1 is traveling is maintained (S44). The data provision unit 212 then sets the value indicating no lane change to Change(t) (S45).

[0179] Although not shown in Figure 12, the data provision unit 212 may also control the vehicle control unit 32 so that the vehicle 1 makes a lane change based on the operation of the vehicle 1 user (e.g., driver, passenger), even if there are no candidate lanes for lane change where the lane change cost is below a threshold (NO in S43). An example of operation by the user will be explained with reference to Figure 17.

[0180] Figure 17 is a diagram illustrating an example of operation by a user. Referring to Figure 17, a navigation device 312 is shown. The navigation device 312 also includes a display device 314 and a touch panel 316. The data provision unit 212 may be controlled to output a message asking whether to attempt a lane change if there are no candidate lanes for lane change where the lane change cost is below a threshold (NO in S43).

[0181] For example, as shown in Figure 17, the message asking whether to attempt a lane change may be a message such as "Do you want to change lanes or keep your lane?". The data provision unit 212 may control the display device 314 so that such a message is displayed by the display device 314.

[0182] The data provision unit 212 may control the vehicle control unit 32 so that the vehicle 1 makes a lane change when the user of the vehicle 1 inputs an operation to select to attempt a lane change. This allows the vehicle 1 to make a lane change according to the user's intention, for example, when a lane change is permitted by the user of another vehicle.

[0183] Alternatively, the data provision unit 212 may reduce the lane change cost by a predetermined value instead of controlling the lane change to occur. The data provision unit 212 may then determine whether or not there are any candidate lanes for lane change where the lane change cost after the reduction by the predetermined value is below a threshold.

[0184] The data provision unit 212 may control the vehicle control unit 32 so that if there is a candidate lane to change to where the lane change cost after being reduced by a predetermined value is below a threshold, the vehicle 1 changes lanes to that candidate lane. On the other hand, if there is no candidate lane to change to where the lane change cost after being reduced by a predetermined value is below a threshold, the data provision unit 212 may control the vehicle control unit 32 so that the lane in which the vehicle 1 is traveling is maintained.

[0185] The operation to select to attempt a lane change may be entered via the touch panel 316 or by voice.

[0186] For example, consider a case where the positions of three lanes, including the lane in which vehicle 1 is currently traveling, are detected. In this case, the data provision unit 212 may assign different colors to the display areas L1 to L3 of the display device 314 corresponding to the positions of each of the three lanes. The operation of selecting to attempt a lane change may be a selection operation in which one of the display areas L1 to L3 is selected.

[0187] The selection operation may be an operation to touch the area of ​​the touch panel 316 corresponding to the display area. Alternatively, the selection operation may be an operation to input the voice of the color of the display area into the microphone. If the lane corresponding to the display area selected by the user is different from the lane in which the vehicle 1 is currently traveling, the data provision unit 212 may control the vehicle control unit 32 so that the vehicle changes lanes to the lane corresponding to the selected display area.

[0188] Returning to Figure 12, the explanation continues. If the data supply unit 212 does not terminate its operation (NO in S48), it advances to the next time by adding 1 to the current time t (S49) and returns to S41. On the other hand, if the operation is terminated (YES in S48), it terminates its operation in automatic driving mode.

[0189] The above describes an example of lane change control according to the embodiment of this disclosure.

[0190] <<5. Annotation Processing>> Next, the annotation processing will be explained with reference to Figures 18 to 24 (and other figures as appropriate). Note that annotation mainly refers to the process of attaching correct data to sensor data. Furthermore, as will be explained later, the correct data mainly includes correct categories corresponding to the correct data for categories and correct labels corresponding to the correct data for whether or not a lane change occurred.

[0191] The storage unit 258 of server 2 stores the collected data transmitted from vehicle 1. The storage unit 258 of server 2 may also store collected data transmitted from vehicles other than vehicle 1. As described above, the collected data includes sensor data. The collected data also includes the driver's status and environmental recognition results corresponding to the sensor data. The environmental recognition results include the lane position, the type of vehicle detected, the vehicle's position and attributes, the intersection position, and the location of obstacles.

[0192] Furthermore, the storage unit 258 of server 2 may store sensor data obtained when a vehicle that is not changing lanes is present in the sensing area. Such sensor data may correspond to the third type of sensor data. In addition, such sensor data may have a value indicating no lane change as the correct label. Furthermore, in addition to the sensor data with a value indicating no lane change as the correct label, the storage unit 258 may also store environmental recognition results and the driver's status corresponding to the sensor data.

[0193] Figure 18 is a flowchart illustrating an example of annotation processing. As shown in Figure 18, the data acquisition unit 252 acquires collected data from the storage unit 258 (S21). As described above, the sensor data, driver status, and environmental recognition results included in the collected data may each be data for a predetermined time period before and after lane change detection. The predetermined time period before and after lane change detection may be the time from N seconds before the lane change detection to N seconds after the lane change detection.

[0194] The model generation unit 253 determines whether the vehicle turned right or left after changing lanes (S22). Note that turning right or left can mean either a right turn or a left turn.

[0195] For example, the model generation unit 253 may determine whether the vehicle turned right or left after changing lanes based on whether the location of an intersection is included in the environmental recognition results for a predetermined time period. Alternatively, the model generation unit 253 may determine whether the vehicle turned right or left after changing lanes based on sensor data for a predetermined time period based on whether the IMU detects that the change in the direction of the vehicle to the right or left is greater than a predetermined angle. Alternatively, the model generation unit 253 may determine whether the vehicle turned right or left after changing lanes based on the map stored in the map information storage unit 23 and the self-position estimation result by the self-position estimation unit 71.

[0196] If the model generation unit 253 determines that the vehicle turned right or left after changing lanes (YES in S22), it proceeds to S25. On the other hand, if the model generation unit 253 determines that the vehicle did not turn right or left after changing lanes (NO in S22), it proceeds to S23. The model generation unit 253 then performs the object identification process (S23). The details of the object identification process will be explained with reference to Figures 19 to 23.

[0197] Figure 19 is a flowchart detailing the process of identifying the object of interest (S23). As shown in Figure 19, the model generation unit 253 counts the number of times the driver pays attention to each object (S232). Note that driver attention to an object may mean that the driver focuses their awareness on the object. Alternatively, attention time may be used instead of the number of attentions.

[0198] The model generation unit 253 determines whether or not there are any objects that the driver has paid attention to more than a threshold number of times. If the model generation unit 253 determines that there are objects that the driver has paid attention to more than a threshold number of times (YES in S233), it identifies those objects as objects of interest (S234). On the other hand, if the model generation unit 253 determines that there are no objects of interest that the driver has paid attention to more than a threshold number of times (NO in S233), it determines that there are no objects of interest (S235).

[0199] The model generation unit 253 may also detect when the driver's line of sight intersects with an object, based on the driver's line of sight and the object's position, as an indication of the driver's attention to the object. Alternatively, the model generation unit 253 may detect when the driver's face direction intersects with an object, based on the face direction and the object's position, as an indication of the driver's attention to the object. Alternatively, the model generation unit 253 may detect the driver's attention to an object based on the driver's spoken voice.

[0200] Figure 20 is a diagram illustrating an example of the driver's state. Referring to Figure 20, driver Dr1 is shown. Also shown are the camera 261 of the in-vehicle sensor 26 and the camera 51 of the external recognition sensor 25. Camera 261 provides image data of driver Dr1 to the DMS 30, and the DMS 30 detects the driver Dr1's line of sight direction E0. Camera 51 obtains image data of the surroundings of vehicle 1.

[0201] Note that the image data is an example of sensor data. Therefore, sensor data obtained by other sensors provided by the external recognition sensor 25 may be used instead of the image data.

[0202] Figure 21 shows an example of a line of sight trajectory Tr1. Referring to Figure 21, image data Im1 obtained by the camera 51 of the external recognition sensor 25 is shown. Image data Im1 shows vehicle 4. Image data Im1 also shows the line of sight trajectory Tr1 of driver Dr1. In the example shown in Figure 21, the line of sight trajectory Tr1 of driver Dr1 is focused on vehicle 4, so it can be seen that driver Dr1 is paying attention to vehicle 4. If the number of times the driver pays attention to vehicle 4 exceeds a threshold, vehicle 4 is identified as an object of attention.

[0203] Figure 22 is the first diagram illustrating the relationship between line of sight direction and objects. Referring to Figure 22, image data Im2 obtained by the camera 51 of the external recognition sensor 25 is shown. In addition to the vehicle 4, object B1 and object B2 are visible in the image data Im2. Furthermore, the image data Im2 shows the driver's line of sight directions E1 to E4.

[0204] Figure 23 is a second diagram illustrating the relationship between line of sight and objects. Referring to Figure 23, the state St shown is a view of vehicle 1 from outside vehicle 1, as seen in the image data Im2 shown in Figure 22. In addition to vehicle 4, object B1 and object B2 are also shown in the image data Im2. Furthermore, the driver's line of sight directions E1 to E4 are shown in the image data Im2.

[0205] In the examples shown in Figures 22 and 23, line of sight directions E1 and E2 intersect with object B1, line of sight direction E3 intersects with object B2, and line of sight direction E4 intersects with vehicle 4. Therefore, the driver pays attention to object B1 the most often. At this time, the model generation unit 253 may identify object B1 as the object of interest.

[0206] Returning to Figure 18, the explanation continues. If the model generation unit 253 fails to identify the object of interest (NO in S24), it returns to S21 to acquire the next collected data (S21) and proceeds with the processing from S22 onward. On the other hand, if the model generation unit 253 successfully identifies the object of interest (YES in S24), it classifies the sensor data into one of several categories (S25) and assigns the category to which the sensor data belongs as the correct category to the sensor data.

[0207] For example, if the model generation unit 253 detects that the vehicle has turned right or left after changing lanes, it may determine that the sensor data belongs to a category indicating a right or left turn scene, and assign the category indicating a right or left turn scene to the sensor data as the correct category.

[0208] On the other hand, if the object of interest is identified, the model generation unit 253 may determine that the sensor data belongs to a category related to the object of interest and assign the category related to the object of interest to the sensor data as the correct category. The category related to the object of interest may include all of the type, attributes, and location of the object of interest, or it may include some of them (for example, any two or any one of them).

[0209] For example, if the type of object of interest is "vehicle" and the attribute of the object of interest, "mark," is "beginner driver mark," the correct category could be "avoiding a vehicle with a beginner driver mark." Also, if the type of object of interest is "traffic cone" and the location of the object of interest is "on the roadway," the correct category could be "obstacle."

[0210] If there is unprocessed collected data (YES in S26), the model generation unit 253 returns to S21 to acquire the next collected data (S21) and performs the processing from S22 onwards. On the other hand, if there is no unprocessed collected data (NO in S26), the model generation unit 253 performs a process to thin out the collected data (S27). The process of thinning out the collected data will be explained with reference to Figure 24.

[0211] Figure 24 is a diagram illustrating the process of thinning collected data. Referring to Figure 24, graph Gr1 shows the number of data points for each correct answer category before the collected data is thinned, and graph Gr2 shows the number of data points for each correct answer category after the collected data is thinned. As shown in graphs Gr1 and Gr2, the model generation unit 253 may perform a process to thin the collected data so that the number of data points for each correct answer category is equal.

[0212] This can help prevent learning from becoming biased towards specific correct answer categories. Note that the correct answer category "Right / Left Turn Scene" refers to a scene where a vehicle turns right or left. The correct answer categories "Vehicle Type A, Relative Speed ​​Less Than X", "Vehicle Type A, Relative Speed ​​X to Y", and "Vehicle Type B, Relative Speed ​​Less Than X" indicate combinations of the type of object of interest and its attributes. The correct answer category "Bicycle / Pedestrian" indicates the type of each of the two objects of interest. The correct answer category "Falling Object" indicates the type of object of interest.

[0213] The annotation processing according to the embodiments of this disclosure has been described above.

[0214] <<6. Learning Process>> Next, the learning process according to the embodiment of this disclosure will be described, mainly with reference to Figures 25 and 26 (and other figures as appropriate). Figure 25 is a flowchart of an example of the learning process according to the embodiment of this disclosure.

[0215] The model generation unit 253 acquires sensor data from the collected data stored in the storage unit 258 (S31). Furthermore, the model generation unit 253 acquires the correct category related to the category to which the sensor data belongs and the correct label related to whether or not a lane change occurred from the collected data as correct data corresponding to the sensor data (S32). In addition to the sensor data, correct category and correct label, the model generation unit 253 may also acquire the driver's state and environmental recognition results from the collected data.

[0216] Next, the model generation unit 253 generates a category estimator M12 (Figure 27) using machine learning based on sensor data and correct categories.

[0217] Furthermore, the model generation unit 253 generates a lane change presence / absence estimator M22 (Figure 27) through machine learning based on sensor data. Specifically, the model generation unit 253 generates the lane change presence / absence estimator M22 through machine learning based on combinations of correct labels indicating lane change presence and sensor data with said correct labels, and combinations of correct labels indicating no lane change presence and sensor data with said correct labels.

[0218] The generation of the category estimator M12 and the lane change presence / absence estimator M22 will be explained with reference to Figure 26.

[0219] Figure 26 is a diagram illustrating the generation of the category estimator M12 and the lane change presence / absence estimator M22. Referring to Figure 26, the category estimation model M11 and the lane change presence / absence estimation model M21 are shown.

[0220] The model generation unit 253 calculates the error between the output from the category estimation model M11 and the correct category when the sensor data is input to the category estimation model M11. Furthermore, the model generation unit 253 calculates the error between the output from the lane change presence / absence estimation model M21 and the correct label when the sensor data and the output from the category estimation model M11 are input to the lane change presence / absence estimation model M21 (S33).

[0221] The model generation unit 253 generates a category estimator M12 by updating the parameters of the category estimation model M11 so that the error between the output of the category estimation model M11 and the correct category is reduced. Furthermore, the model generation unit 253 generates a category estimator M12 by updating the parameters of the category estimation model M11 so that the error between the output of the lane change presence / absence estimation model M21 and the correct label is reduced (S34).

[0222] The input to the lane change presence / absence estimation model M21 may include the driver's state and environmental recognition results. Also, as described above, it is sufficient for the server 2 to generate at least the lane change presence / absence estimator M22, and the category estimator M12 does not necessarily have to be generated. If the model generation unit 253 wants to continue learning (NO in S35), it returns to S31. On the other hand, if the model generation unit 253 wants to end learning (YES in S35), it terminates learning.

[0223] The learning process according to the embodiments of this disclosure has been described above.

[0224] <<7. Inference Processing>> Next, the inference processing according to the embodiment of this disclosure will be described, mainly with reference to Figure 27 (and other figures as appropriate). Figure 27 is a diagram for explaining the inference processing according to the embodiment of this disclosure.

[0225] The model provisioning unit 254 transmits the category estimator M12 and the lane change presence / absence estimator M22, generated by the model generation unit 253, to the vehicle control ECU 21 of the vehicle control system 11 of vehicle 1 via the communication unit 259. The model provisioning unit 254 may transmit the trained model not only to vehicle 1, but also to other vehicles.

[0226] The model acquisition unit 214 acquires the category estimator M12 and the lane change presence / absence estimator M22 transmitted from the server 2 via the communication unit 22. The model acquisition unit 214 then stores the acquired category estimator M12 and lane change presence / absence estimator M22 in the storage unit 28.

[0227] After the category estimator M12 and the lane change presence / absence estimator M22 are acquired by the model acquisition unit 214, the inference unit 216 acquires sensor data obtained by the external recognition sensor 25 from the external recognition sensor 25.

[0228] The inference unit 216 determines whether or not vehicle 1 will change lanes based on the acquired sensor data and the category estimator M12 and lane change presence / absence estimator M22 stored in the storage unit 28. Once it is determined whether or not to change lanes, the inference unit 216 can instruct the vehicle control unit 32 to perform a more appropriate lane change according to the determination result.

[0229] Specifically, the inference unit 216 can obtain the output from the category estimator M12 when the sensor data is input to the category estimator M12, and the output from the lane change presence / absence estimator M22 when the sensor data is input to the lane change presence / absence estimator M22, as a determination result indicating whether or not to change lanes.

[0230] Furthermore, the input to the lane change presence / absence estimator M22 may include the driver's state and environmental recognition results. As mentioned above, it is conceivable that the category estimator M12 may not be generated. In such cases, the inference unit 216 may obtain the output from the lane change presence / absence estimator M22, when sensor data is input to the lane change presence / absence estimator M22, as a determination result indicating whether or not a lane change will be performed.

[0231] The inference processing according to the embodiments of this disclosure has been described above.

[0232] <<8. Effects>> According to the information processing system 10 of the embodiment of this disclosure, it is determined whether or not the vehicle 1 will change lanes based on sensor data and the lane change presence / absence estimator M22. Once it is determined whether or not to change lanes, the vehicle 1 can be controlled to make a more appropriate lane change according to the determination result.

[0233] The effects of the information processing system 10 according to the embodiment of this disclosure have been described above.

[0234] <<9. Modifications>> Although preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to these examples. It is clear that a person with ordinary skill in the art of the present disclosure may conceive of various modifications or alterations within the scope of the technical ideas described in the claims, and these will naturally also be understood to fall within the technical scope of the present disclosure.

[0235] For example, the above mainly described an example in which the embodiments of the present disclosure are applied to the control of lane changes by vehicles. However, the embodiments of the present disclosure can also be applied to objects moving in lanes. That is, the embodiments of the present disclosure can also be applied to the control of lane changes by moving objects. The objects moving in lanes may be parts or products moving in lanes on a factory production line, for example.

[0236] Furthermore, the above mainly described the case in which sensor data for learning and estimating whether or not a lane change has occurred is obtained by an external recognition sensor 25 equipped on vehicle 1. However, sensor data for learning and estimating whether or not a lane change has occurred may also be obtained by sensors fixed to the outside of vehicle 1. For example, sensor data for learning and estimating whether or not a lane change has occurred may be obtained by sensors installed on the side of the road or above the road.

[0237] Furthermore, among the processes described in the embodiments of this disclosure described above, all or part of the processes described as being performed automatically may be performed manually, or all or part of the processes described as being performed manually may be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above documents and drawings may be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0238] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0239] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas where the processing content is not contradictory. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0240] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0241] Furthermore, the following configurations also fall within the technical scope of this disclosure: (1) An information processing method executed by a processor, comprising: acquiring a trained model generated by machine learning based on first sensor data obtained when a first vehicle changes lanes; and determining whether a second vehicle, identical or different from the first vehicle, makes a lane change based on second sensor data and the trained model. (2) The information processing method according to (1), wherein the trained model comprises a first trained model for estimating categories, the first sensor data is assigned a category to which the first sensor data belongs as a correct category, and the first trained model is generated by machine learning based on the first sensor data and the correct category. (3) The information processing method according to (2), wherein the correct category includes a category indicating a right or left turn scene if it is detected that the first vehicle made a right or left turn after changing lanes by the first vehicle. (4) The information processing method according to (2), wherein the correct answer category includes a category relating to the object of interest identified by the driver of the first vehicle. (5) The information processing method according to any one of (2) to (4), wherein the first trained model is generated by updating the parameters of the first model so that the error between the output from the first model and the correct answer category when the first sensor data is input to the first model is small. (6) The information processing method according to any one of (1) to (5), wherein the lane change by the first vehicle is performed when the lane change cost, which is the cost incurred by the lane change by the first vehicle calculated based on the first sensor data, is less than or equal to a threshold. (7) The information processing method according to (6), wherein the lane change cost increases as the number of lane changes by the first vehicle in a predetermined past time period increases. (8) The information processing method according to (6) or (7), wherein if the cost of changing lanes exceeds the threshold, a message is output asking whether to attempt to change lanes.(9) The information processing method according to (8), wherein if an operation to select to attempt the lane change is input, the first vehicle performs a lane change. (10) The information processing method according to any one of (6) to (9), wherein the lane change cost is calculated based on the lane risk, which is the risk value of a second lane different from the first lane in which the first vehicle is traveling, and the first lane and the second lane are detected based on the occurrence of a predetermined event. (11) The information processing method according to (10), wherein the lane risk is calculated based on the vehicle risk, which is the risk value of a vehicle traveling in the second lane. (12) The information processing method according to any one of (1) to (11), wherein the trained model includes a second trained model for estimating whether or not to change lanes, and the second trained model is generated by machine learning based on a combination of the first sensor data and a ground truth label indicating that a lane change has occurred, and a combination of third sensor data obtained when no lane change has occurred and a ground truth label indicating that no lane change has occurred. (13) An information processing system comprising a processor that acquires a trained model generated by machine learning based on first sensor data obtained when a lane change is made by a first vehicle, and determines whether or not a lane change is made by a second vehicle that is the same as or different from the first vehicle, based on the second sensor data and the trained model. (14) The information processing system according to (13), wherein the trained model includes a first trained model for estimating categories, the first sensor data is assigned the category to which the first sensor data belongs as a ground truth category, and the first trained model is generated by machine learning based on the first sensor data and the ground truth category. (15) The information processing system according to (13) or (14), wherein the lane change by the first vehicle is performed when the lane change cost, which is the cost incurred for the lane change by the first vehicle calculated based on the first sensor data, is less than or equal to a threshold.(16) The information processing system according to (15), wherein the lane change cost increases with the number of times the first vehicle has changed lanes in a predetermined past period of time. (17) The information processing system according to (15) or (16), wherein if the lane change cost exceeds the threshold, a message is output asking whether to attempt a lane change. (18) The information processing system according to (17), wherein if an operation to select to attempt a lane change is input, the first vehicle makes a lane change. (19) The information processing system according to any one of (15) to (18), wherein the lane change cost is calculated based on the lane risk, which is the risk value of a second lane different from the first lane in which the first vehicle is traveling, and the first lane and the second lane are detected based on the occurrence of a predetermined event. (20) The information processing system according to any one of (13) to (19), wherein the trained model includes a second trained model for estimating whether or not to change lanes, the second trained model being generated by machine learning based on a combination of the first sensor data and a ground truth label indicating that a lane change has occurred, and a combination of third sensor data obtained when no lane change has occurred and a ground truth label indicating that no lane change has occurred.

[0242] 1 Vehicle 10 Information Processing System 11 Vehicle Control System 2 Server 21 Vehicle Control ECU 22 Communication Unit 23 Map Information Storage Unit 24 Location Information Acquisition Unit 25 External Recognition Sensor 26 In-Vehicle Sensor 27 Vehicle Sensor 28 Memory Unit 29 Driving Automation Control Unit 30 DMS 31 HMI 32 Vehicle Control Unit 212 Data Provision Unit 214 Model Acquisition Unit 216 Inference Unit 251 Control Unit 252 Data Acquisition Unit 253 Model Generation Unit 254 Model Provision Unit 258 Memory Unit 259 Communication Unit

Claims

1. An information processing method executed by a processor, comprising: acquiring a trained model generated by machine learning based on first sensor data obtained when a first vehicle changes lanes; and determining whether or not a second vehicle, identical or different from the first vehicle, will change lanes based on second sensor data and the trained model.

2. The information processing method according to claim 1, wherein the trained model comprises a first trained model for estimating categories, the first sensor data is assigned a category to which the first sensor data belongs as a ground truth category, and the first trained model is generated by machine learning based on the first sensor data and the ground truth category.

3. The information processing method according to claim 2, wherein the correct answer category includes a category indicating a right or left turn scene when it is detected that the first vehicle has turned right or left after changing lanes by the first vehicle.

4. The information processing method according to claim 2, wherein the correct answer category includes a category relating to the object of interest identified by the driver of the first vehicle.

5. The information processing method according to claim 2, wherein the first trained model is generated by updating the parameters of the first model so that the error between the output from the first model and the correct category when the first sensor data is input to the first model is reduced.

6. The information processing method according to claim 1, wherein the lane change by the first vehicle is performed when the lane change cost, which is the cost incurred by the lane change by the first vehicle calculated based on the first sensor data, is less than or equal to a threshold.

7. The information processing method according to claim 6, wherein the lane change cost increases as the number of lane changes by the first vehicle in a predetermined past period of time increases.

8. The information processing method according to claim 6, wherein if the lane change cost exceeds the threshold, a message is output asking whether to attempt a lane change.

9. The information processing method according to claim 8, wherein when an operation to select to attempt the lane change is input, the first vehicle performs the lane change.

10. The information processing method according to claim 6, wherein the lane change cost is calculated based on the lane risk, which is the risk value of a second lane different from the first lane in which the first vehicle is traveling, and the first lane and the second lane are detected based on the occurrence of a predetermined event.

11. The information processing method according to claim 10, wherein the lane risk is calculated based on the vehicle risk, which is the risk value of a vehicle traveling in the second lane.

12. The information processing method according to claim 1, wherein the trained model includes a second trained model for estimating whether or not to change lanes, the second trained model being generated by machine learning based on a combination of the first sensor data and a ground truth label indicating that a lane change has occurred, and a combination of third sensor data obtained when no lane change has occurred and a ground truth label indicating that no lane change has occurred.

13. An information processing system comprising a processor that acquires a trained model generated by machine learning based on first sensor data obtained when a first vehicle changes lanes, and determines whether or not to perform a lane change with a second vehicle that is the same as or different from the first vehicle, based on second sensor data and the trained model.

14. The information processing system according to claim 13, wherein the trained model comprises a first trained model for estimating categories, the first sensor data is assigned a category to which the first sensor data belongs as a ground truth category, and the first trained model is generated by machine learning based on the first sensor data and the ground truth category.

15. The information processing system according to claim 13, wherein the lane change by the first vehicle is performed when the lane change cost, which is the cost incurred by the first vehicle to change lanes, calculated based on the first sensor data, is less than or equal to a threshold.

16. The information processing system according to claim 15, wherein the lane change cost increases as the number of lane changes by the first vehicle in a predetermined past period of time increases.

17. The information processing system according to claim 15, wherein if the lane change cost exceeds the threshold, a message is output asking whether to attempt a lane change.

18. The information processing system according to claim 17, wherein when an operation to select to attempt the lane change is input, the first vehicle performs the lane change.

19. The information processing system according to claim 15, wherein the lane change cost is calculated based on the lane risk, which is the risk value of a second lane different from the first lane in which the first vehicle is traveling, and the first lane and the second lane are detected based on the occurrence of a predetermined event.

20. The information processing system according to claim 13, wherein the trained model includes a second trained model for estimating whether or not to change lanes, the second trained model being generated by machine learning based on a combination of the first sensor data and a ground truth label indicating that a lane change has occurred, and a combination of third sensor data obtained when no lane change has occurred and a ground truth label indicating that no lane change has occurred.

Images