Information processing system and information processing method

Abstract

[Problem] To obtain, by a recognition model, a recognition result related to a lane change. [Solution] An information processing system comprises: an acquisition part that acquires a sensing result of the surrounding environment of a vehicle; and a recognition processing part that obtains lane information, which is information on each of two or more lanes, and a recognition result regarding a lane change related to a lane in which the vehicle travels, from a recognition model that receives the sensing result as an input and outputs the lane information and the recognition result.

Description

Information processing system and information processing method

[0001] This disclosure relates to an information processing system and an information processing method.

[0002] Various systems have been proposed to assist drivers of automobiles. For example, technologies related to autonomous driving have also been developed.

[0003] Autonomous driving utilizes data obtained from cameras installed on the vehicle, data obtained from various sensors that detect obstacles around the vehicle, and high-precision 3D map data (HD (Hi-Definition) Map). The HD Map stores a large amount of data necessary for controlling the vehicle, such as road information for each lane, traffic lights, road signs, and pedestrian crossing information.

[0004] Here, developing HD Maps takes time and money, and currently, they do not cover all roads. Patent Document 1 below describes a situation where areas with HD Maps (i.e., areas where autonomous driving is possible) and areas without HD Maps (i.e., areas where autonomous driving is not possible) coexist, and the driver is notified that there are parts of the guidance route to the destination where autonomous driving is not possible, thereby reducing the driver's discomfort when autonomous driving suddenly becomes impossible.

[0005] Japanese Patent Publication No. 6358156

[0006] In cases where using HD Map is difficult, recent research has focused on using AI models (recognition models) to recognize external environments such as lanes, using sensing data obtained from various sensors such as cameras and distance sensors mounted on the vehicle. Furthermore, not limited to autonomous driving, it is possible to assist drivers by presenting lane recognition results to them in real time.

[0007] However, while AI-based lane recognition can obtain geometric lane information, it cannot obtain semantic information such as whether or not a lane change is permitted.

[0008] Therefore, this disclosure proposes an information processing system and information processing method that can obtain recognition results regarding lane changes using a recognition model.

[0009] According to this disclosure, an information processing system is provided, comprising: an acquisition unit that acquires sensing results of the surrounding environment of a vehicle; and a recognition processing unit that takes the sensing results as input and obtains lane information, which is information of each of two or more lanes, and recognition results regarding lane changes related to the lane in which the vehicle is traveling, from a recognition model.

[0010] Furthermore, the present disclosure provides an information processing method that includes a processor acquiring sensing results of the surrounding environment of a vehicle, and obtaining lane information and recognition results from a recognition model that takes the sensing results as input and outputs lane information, which is information for each of two or more lanes, and recognition results regarding lane changes related to the lane in which the vehicle is traveling.

[0011] This figure shows an example of the configuration of the vehicle 10 according to this embodiment. This block diagram shows an example of the detailed configuration of the control unit 20 according to this embodiment. This flowchart shows an example of the flow of the AI-based lane change feasibility inference process according to this embodiment. This figure shows an example of the display of lane change not being possible according to this embodiment. This figure shows another example of the display of lane change not being possible according to this embodiment. This figure shows lane-related information inferred by the AI ​​model 212 according to this embodiment. This figure shows the details of the lane segment information 421 according to this embodiment. This figure shows a detailed classification of each line segment included in the lane segment information 421 according to this embodiment. This figure shows a table of lane topology information 430 according to this embodiment. This figure shows a table of lane change topology according to this embodiment. This figure shows an example of the driving environment and lane information of the vehicle 10. This figure shows an example of lane topology information and lane change topology information output by the AI ​​model 212 from the driving environment shown in Figure 11. This figure shows another example of the driving environment and lane information of the vehicle 10. This figure shows a table of lane topology and lane change topology output by the AI ​​model 212 from the driving environment shown in Figure 13. This figure is for explaining the generation of rule-based lane change topology information 442 according to this embodiment. This is a flowchart illustrating an example of the process flow for generating rule-based lane change topology according to this embodiment. This is a diagram illustrating the rule-based inference of whether a lane change is permissible according to this embodiment. This is a diagram illustrating an example of the lane change permissibility table 232 according to this embodiment. This is a diagram illustrating the generation of lane change topology information 450 based on integration according to this embodiment. This is a diagram illustrating an example of the integration of AI-based lane change topology information 440a and rule-based lane change topology information 442a according to this embodiment. This is a flowchart illustrating an example of the process flow for generating lane change topology information based on integration according to this embodiment. This is a block diagram illustrating an example of the configuration of the information processing device 50 according to this embodiment. This is a diagram illustrating the generation of lane change topology information used as training data according to this embodiment.This is a flowchart showing an example of the process flow for generating lane change topology information used as training data according to this embodiment. This is a diagram showing an example of a lane structure according to this embodiment. This is a diagram for explaining the training of the AI ​​model 212 according to this embodiment. This is a flowchart showing an example of the training process flow according to this embodiment. This is a diagram showing an example of the loss function calculation formula according to this embodiment. This is a block diagram showing an example of the hardware configuration of an information processing device 900 according to one embodiment of the present disclosure.

[0012] Preferred embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant descriptions will be omitted.

[0013] Furthermore, the explanation will be given in the following order: 1. Configuration 1-1. Configuration of Vehicle 10 1-2. Detailed Configuration of Control Unit 20 2. Inference of Lane Change Feasibility 2-1. AI-based Inference of Lane Change Feasibility 2-2. Rule-based Inference of Lane Change Feasibility 2-3. Inference of Lane Change Feasibility by Integrated Processing 3. Training of AI Model 212 3-1. Configuration 3-2. Preparation of Training Data 3-3. Training Process 4. Example Hardware Configuration 5. Supplementary Information

[0014] <1. Structure> As one embodiment of this disclosure, a mechanism for obtaining recognition results regarding lane changes using a recognition model will be described.

[0015] <<1-1. Configuration of Vehicle 10>> Figure 1 is a diagram showing an example of the configuration of vehicle 10 according to this embodiment. As shown in Figure 1, vehicle 10 is equipped with a vehicle control system 11. The vehicle control system 11 acquires information from various sensors and communication units provided in vehicle 10 and controls vehicle 10.

[0016] The vehicle control system 11 includes a communication unit 21, a control unit 20, an external sensor 22, a location information acquisition unit 23, an input unit 24, an in-vehicle sensor 25, a vehicle sensor 26, a storage unit 27, a display unit 28, and a speaker 29.

[0017] The control unit 20 controls the operation of the entire vehicle control system 11. The control unit 20 includes one or more ECUs (Electronic Control Units).

[0018] The communication unit 21 is equipped with various communication devices and communicates with various devices inside and outside the vehicle, other vehicles, servers, base stations, etc., and transmits and receives various types of data. The communication unit 21 is an example of an information acquisition unit.

[0019] The communication unit 21 can communicate using multiple communication methods as needed. For example, the communication unit 21 can communicate with external devices or the Internet using wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network, etc.

[0020] For example, the communication unit 21 transmits and receives information to and from a server on the network via wireless communication.

[0021] The external environment sensor 22 includes various sensors used to detect various types of information from the outside world (external environment) of the vehicle 10. The external environment sensor 22 is an example of an acquisition unit that acquires sensing results of the vehicle's surrounding environment. Sensor data (sensing results) from each sensor are output to the control unit 20. The types or number of sensors included in the external environment sensor 22 are not particularly limited.

[0022] For example, the external sensor 22 includes a camera 31, a microphone 32, a radar 33, and a LiDAR (Light Detection and Ranging) 34. The number of cameras 31, microphones 32, radars 33, and LiDARs 34 is not particularly limited.

[0023] The shooting method of camera 31 is not particularly limited. For example, various types of cameras with shooting methods capable of distance measurement, such as ToF (Time of Flight) cameras, stereo cameras, monocular cameras, and infrared cameras, can be applied to camera 31 as needed. However, camera 31 may simply be for acquiring captured images. Microphone 32 acquires ambient sounds.

[0024] The radar 33 and LiDAR 34 detect the positions of surrounding lanes, vehicles, obstacles, etc.

[0025] The location information acquisition unit 23 calculates the current position of the vehicle 10. For example, the location information acquisition unit 23 may calculate its own position (the current position of the vehicle 10) based on GNSS (Global Navigation Satellite System) signals received from GNSS satellites.

[0026] The input unit 24 has an input device for passengers to input data, instructions, etc. The input unit 24 generates an input signal based on the data, instructions, etc. input by the input device and outputs it to the control unit 20. The input unit 24 is not limited to an input device such as a touch panel or buttons that the passenger directly operates, but may also have an input device that allows passengers to input instructions, etc. by voice or gesture.

[0027] The in-vehicle sensor 25 detects the conditions inside the vehicle. For example, the in-vehicle sensor 25 has a camera and a microphone. The number and type of cameras and microphones are not particularly limited. Also, the method of shooting with the camera is not particularly limited, similar to the camera 31 of the external sensor 22. The in-vehicle sensor 25 can, for example, detect the condition of the passengers.

[0028] The vehicle sensor 26 detects the overall state of the vehicle. For example, the vehicle sensor 26 is equipped with various sensors and outputs sensing data from each sensor to the control unit 20. The types and number of sensors in the vehicle sensor 26 are not particularly limited.

[0029] For example, the vehicle sensor 26 includes a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and an inertial measurement unit (IMU) that integrates them. For example, the vehicle sensor 26 includes a steering angle sensor for detecting the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor for detecting the amount of operation of the accelerator pedal, and a brake sensor for detecting the amount of operation of the brake pedal. For example, the vehicle sensor 26 includes a rotation sensor for detecting the rotation speed of the engine or motor, an air pressure sensor for detecting the air pressure of the tires, a slip ratio sensor for detecting the slip ratio of the tires, and a wheel speed sensor for detecting the rotation speed of the wheels. For example, the vehicle sensor 26 includes a battery sensor for detecting the remaining charge and temperature of the battery, and an impact sensor for detecting external impacts.

[0030] The memory unit 27 stores information necessary for the operation of the vehicle control system 11.

[0031] The display unit 28 consists of various display devices and presents visual information to the passenger. The number and types of display devices in the display unit 28 are not particularly limited. For example, the display unit 28 may be equipped with a display device that displays images. In addition to display devices with ordinary displays, the display device may also be a device that presents visual information within the passenger's field of view, such as a head-up display, a transparent display, or a wearable device equipped with AR (Augmented Reality) functionality.

[0032] Speaker 29 provides auditory information to the passengers.

[0033] The configuration of the vehicle 10 according to this embodiment has been described above. The configuration shown in Figure 1 is just one example, and this embodiment is not limited thereto.

[0034] <<1-2. Detailed Configuration of Control Unit 20>> Figure 2 is a block diagram showing an example of the detailed configuration of the control unit 20 according to this embodiment. As shown in Figure 2, it has a recognition processing unit 210, a rule-based decision unit 230, an integrated processing unit 240, and a display control unit 220.

[0035] (Recognition Processing Unit 210) The recognition processing unit 210 receives information from an external sensor 22 mounted on a vehicle 10 and analyzes the conditions of the external environment. For example, the recognition processing unit 210 inputs information (sensor data) from the external sensor 22 to an AI model (an example of a recognition model), and outputs lane-related information such as Lane segment information (an example of lane information), Lane topology information (an example of lane connection information), and AI-based lane change Topology information (an example of a recognition result regarding whether lane change is allowable). In other words, the recognition processing unit 210 can infer Lane segment information, Lane topology information, and AI-based lane change Topology information using the AI model.

[0036] A Lane segment is one of a plurality of divisions of a lane (specifically, a roadway on which a vehicle travels, which is an example of a lane) divided into a certain range, and is also referred to as a "lane segment" in the present specification. A lane segment may correspond to a "lane". In the present specification, the term "lane" is a concept encompassing lane segments and roadways. Lane segment information includes information related to each lane, more specifically, information of each lane segment. For example, the Lane segment information includes the position (p) of the center line of each lane segment, and the position of the boundary line with other lanes (the position of the left boundary line: p l , the position of the right boundary line: p r ), and indicates the color, shape, and the like of the boundary line. More specifically, the Lane segment information includes three-dimensional coordinate points (p1 to p n ) of a Center line which is a line passing through the center of a lane (roadway), and three-dimensional coordinate points (p l 1 to p l n , p r 1 to p r n ) of a Left lane (left boundary line) and a Right lane (right boundary line) which are boundary lines respectively located on the left and right of the lane (roadway), and includes information on the colors (white, yellow, etc.) and shapes (solid line, broken line, etc.) of the Center line, Left lane, and Right lane.

[0037] The Lane topology information is information indicating the connection relationship between lanes (here, between Lane segments). For example, the Lane topology information indicates whether each combination of two lanes among a plurality of lanes (here, Lane segments) is in a series connection relationship or a parallel (adjacent) connection relationship. The series connection relationship further includes a connection in the same direction as the traveling direction and a connection in the opposite direction to the traveling direction. The parallel connection relationship further includes whether the parallel lanes are in the same direction as the traveling direction or the opposite direction to the traveling direction.

[0038] The AI-based lane change Topology information indicates whether lane change is allowed between lanes (here, between Lane segments). Lane change means changing the lane segment (or roadway) on which a vehicle travels. For example, the lane change Topology information indicates whether lane change is allowed for each combination of two lanes among a plurality of lanes (here, between Lane segments). The lane change permission indicated by the lane change Topology information includes, for example, lane change allowed, lane change with caution (allowed but requires attention), and lane change not allowed.

[0039] (Rule-based judging unit 230) The rule-based judging unit 230 receives Lane segment information and Lane topology information from the recognition processing unit 210, and judges whether lane change is allowed based on predetermined rules. It is assumed that the rules to be used are based on traffic regulations, such as "lane change is not allowed when the boundary line is yellow". The rule-based judging unit 230 outputs rule-based lane change Topology information as a judgment result.

[0040] (Integrated Processing Unit 240) The integrated processing unit 240 integrates the AI-based lane change topology information obtained by the recognition processing unit 210 and the rule-based lane change topology information obtained by the rule-based decision unit 230 to obtain lane change topology information. The integration process corrects the uncertainty of the inference results by the AI ​​model, enabling a more robust determination of whether or not a lane change is possible. The integrated processing unit 240 outputs the lane change topology information as a processing result.

[0041] (Display Control Unit 220) The display control unit 220 generates a display screen to be shown on the display unit 28 by superimposing a display indicating guidance regarding lane changes onto the video (captured image) of the area around the vehicle 10. For example, if lane changes are prohibited, the display control unit 220 displays a warning mark on the display unit 28 to improve the driver's visibility. The display control unit 220 also provides information to encourage appropriate lane changes when lane changes are permitted.

[0042] The display control unit 220 may generate a display screen that displays guidance regarding lane changes based on the AI-based lane change topology information obtained by the recognition processing unit 210.

[0043] Furthermore, the display control unit 220 may generate a display screen that displays guidance regarding lane changes based on the rule-based lane change topology information obtained by the rule-based determination unit 230.

[0044] Furthermore, the display control unit 220 may generate a display screen that displays guidance regarding lane changes based on the lane change topology information obtained by the integrated processing unit 240.

[0045] The configuration of the control unit 20 according to this embodiment has been described above. The configuration shown in Figure 2 is just one example, and this embodiment is not limited thereto. The configuration of the control unit 20 does not have to include all the configurations shown in Figure 2. For example, the configuration of the control unit 20 may include a recognition processing unit 210 and a display control unit 220, or it may include a recognition processing unit 210, a rule-based decision unit 230, and a display control unit 220.

[0046] <2. Inference of whether a lane change is permissible> <<2-1. AI-based inference of whether a lane change is permissible>> The AI-based inference of whether a lane change is permissible according to this embodiment will be explained in detail. The AI-based inference of whether a lane change is permissible is realized by the recognition processing unit 210 shown in Figure 2.

[0047] (Overall flow) Figure 3 is a flowchart showing an example of the flow of the AI-based lane change inference process according to this embodiment.

[0048] As shown in Figure 3, first, the external sensor 22 of the vehicle 10 senses the surrounding environment (S103).

[0049] Next, the recognition processing unit 210 inputs the output from the external sensor 22 (sensor data) and the information from the pre-stored standard-definition map data, the SD (Standard-Definition) Map, into the AI ​​model (S106). The sensor data includes captured images (RGB images, infrared images, depth images, etc.), depth data, lane recognition results from radar, vehicle position information, etc. The SD Map includes lane structure and road restriction information, and its integration with the sensor data enables more accurate environmental recognition.

[0050] Next, the recognition processing unit 210 uses the AI ​​model 212 (see Figure 6) to infer lane segment information, lane topology information, and AI-based lane change topology information within a certain spatial range (S109). Details of the inference process will be described later with reference to Figures 6 to 14.

[0051] The display control unit 220 then controls the display unit 28 to display guidance regarding lane changes based on the obtained data, specifically the lane segment information and the AI-based lane change topology information (S112). This allows the driver to intuitively understand whether or not a lane change is permissible, and the system can support safe driving.

[0052] (Example Display) Next, we will explain examples of signs indicating lane changes with reference to Figures 4 and 5.

[0053] Figure 4 shows an example of a display indicating that lane changes are not permitted according to this embodiment. In the display screen 310 shown in Figure 4, an "X" mark is superimposed on the image of the area around the vehicle 10 from the front camera's perspective. More specifically, a yellow boundary line, represented by dot hatching, exists between the roadway in which the vehicle 10 is traveling and the roadway to its left, and an "X" mark is superimposed on this boundary line. This "X" mark indicates that the vehicle 10 cannot change lanes from its current lane (roadway) to the lane (roadway) to its left, that is, lane changes are not permitted. This allows the driver to intuitively understand whether or not lane changes are permitted.

[0054] Figure 5 shows another example of the display indicating that lane changes are not permitted according to this embodiment. In the display screen 320 shown in Figure 5, an "X" mark is superimposed on the image from an overhead view of the vehicle 10. More specifically, a yellow boundary line, represented by dot hatching, exists between the lane (roadway) in which the vehicle 10 is traveling and the lane (roadway) to its left, and an "X" mark is superimposed on this boundary line. This allows the driver to intuitively understand whether or not lane changes are permitted.

[0055] (AI-based inference processing) Next, we will explain the details of the inference processing shown in S109 of Figure 3.

[0056] Figure 6 shows the lane-related information inferred by the AI ​​model 212 according to this embodiment. As shown in Figure 6, the recognition processing unit 210 inputs the input data 410 to the AI ​​model 212 and obtains Lane segment information 420, Lane topology information 430, and AI-based lane change topology information 440 from the AI ​​model 212.

[0057] Figure 7 shows the details of the Lane segment information 421 according to this embodiment. As shown in Figure 7, a lane (roadway) is composed of multiple lane segments V (V1, V2, V3, ..., V8), and each lane segment V is indicated by an arrow indicating the direction of travel. The range of the multiple lane segments V that make up the lane is not particularly limited and may be divided at points where the meaning changes, such as straight lines, curves, and merges.

[0058] In the enlarged portion of Figure 7, the lane segment V is defined as a set of line segments {V_center, V_right, V_left}, and each line segment is a point cloud (sequence of points) {p1, p2, p3, p4}, {p r 1, p r 2,p r 3,p r 4}, {p l 1, p l 2,p l 3,p l This is represented by {4}. Point p is a three-dimensional coordinate point. V_right represents the right boundary line located to the right in the direction of travel on the roadway, V_center represents the line passing through the center of the roadway, and V_left represents the left boundary line located to the left in the direction of travel on the roadway.

[0059] Figure 8 shows a detailed classification of each line segment included in the Lane segment information 421 according to this embodiment. As shown in Figure 8, a lane (roadway) is composed of multiple lane segments V (V1, V2, V6, V7). Each lane segment V is classified as a Center line (e.g., V2-center), Right lane (e.g., V2-right), Left lane (e.g., V2-left), and its respective characteristics are defined.

[0060] Figure 8 (right) shows examples of "color types" and "shape types" for the Right and Left lanes. Each type is assigned an identification number. There are three color types: 0: White, 1: Yellow, and 2: Non-visible, with yellow lines in particular indicating that lane changes may be restricted. There are also three shape types: 0: Solid, 1: Dashed, and 2: Non-visible, which are used to determine whether lane changes are permitted.

[0061] Figure 9 shows a table of Lane topology information 430 according to this embodiment. As shown in Figure 9, the table of Lane topology information 430 numerically represents the connection relationships of each lane segment V (V1, V2, ..., V8).

[0062] In the table shown in Figure 9, 0 indicates no connection, and ±1 indicates a series connection. Furthermore, ±2 indicates a parallel connection (same direction of travel), and ±3 indicates a parallel connection (opposite direction of travel), defining the relative relationship between lane segments.

[0063] For example, the relationship between V1 and V5 is -1, which means that V5 is connected in series with V1 in the opposite direction of travel. Similarly, the relationship between V2 and V6 is -1, which also means that V6 is connected in series with V2 in the opposite direction of travel. On the other hand, the relationship between V3 and V7 is +1, which means that V7 is connected in series with V3 in the direction of travel.

[0064] Furthermore, the relationship between V2 and V1 is -2, which indicates that V2 and V1 are connected in parallel with each other and travel in the same direction. Similarly, the relationship between V3 and V4 is -2, which also means that they are connected in parallel with each other and travel in the same direction. On the other hand, the relationship between V6 and V7 is +3, which means that V7 is connected in parallel with V6 but travels in the opposite direction.

[0065] Figure 10 is a diagram showing the lane change topology according to this embodiment. As shown in Figure 10, the possibility of changing lanes between each lane segment V (V1, V2, ..., V8) is expressed numerically.

[0066] In the table shown in Figure 10, 0 indicates "lane change not permitted," 1 indicates "caution advised when changing lanes," and 2 indicates "lane change permitted." For example, changing lanes from V1 to V2 is rated 1, indicating that a lane change is possible but requires caution. On the other hand, changing lanes from V3 to V4 and from V5 to V6 is rated 2, indicating that a lane change is permitted. Conversely, many lane changes, such as from V1 to V2 and from V6 to V7, are rated 0, indicating that lane changes are not permitted.

[0067] Next, with reference to Figures 11 to 14, a specific example of the inference result of whether or not a lane change is permissible by the recognition processing unit 210 will be explained.

[0068] Figure 11 shows an example of the driving environment and lane information for vehicle 10. As shown in Figure 11, the road on which vehicle 10 is traveling and the adjacent road are composed of multiple lane segments V (V1, V2, V3, V4), and the line segments (Center line, Right lane, and Left lane) included in each lane segment V are indicated by arrows that show the direction of travel.

[0069] The driving environment shown in Figure 11 assumes a two-lane road in each direction, consisting of roadway R1 and roadway R2. As shown in Figure 11, vehicle 10 is traveling in the right-hand roadway R2. Between roadway R1 and roadway R2, a white dotted line and a yellow solid line are drawn on the road. Between roadway R1 and roadway R2, the white dotted line is on the left and the yellow solid line is on the right, so according to traffic laws, the driving environment allows lane changes only from roadway R1 to roadway R2.

[0070] In the driving environment shown in Figure 11, the AI ​​model 212 recognizes information for each lane segment V (V1, V2, V3, V4) as lane information. For example, for V1, the characteristics of the right lane of V1 (color: White, type: Dashed) and the characteristics of the left lane of V1 (color: White, type: Solid) are obtained.

[0071] Figure 12 shows an example of lane topology information and lane change topology information output by the AI ​​model 212 from the driving environment shown in Figure 11. As shown in Figure 12, each table numerically represents the relationship between lanes and whether or not a lane change is possible.

[0072] In the lane topology table shown in the upper part of Figure 12, the connection relationships between each lane segment V (V1, V2, V3, V4) are shown numerically. The meaning of the numbers is the same as in Figure 9. Therefore, for example, there is a "2" between V1 and V3, and between V2 and V4, indicating that they are parallel connections with the same direction of travel.

[0073] The lane change topology table shown in the lower part of Figure 12 indicates the feasibility of lane changes using numerical values. The meaning of these values ​​is the same as in Figure 10. Therefore, for example, it can be seen that lane changes from V1 to V3 and from V2 to V4 are possible. On the other hand, it can be seen that lane changes from V3 to V1 and from V4 to V2 are not possible.

[0074] Based on the lane change topology information shown in the lower part of Figure 12, the display control unit 220 can notify the driver of vehicle 10 via a screen display that it is not possible to change lanes to the left-hand roadway R1 (see Figure 11).

[0075] Figure 13 shows another example of the driving environment and lane information for vehicle 10. The driving environment in Figure 13 assumes that there is an intersection ahead in the direction of travel. As shown in Figure 13, vehicle 10 is traveling on the right-hand lane R4 of the two roadways R3 and R4. On the road, a solid white line is drawn between roadways R3 and R4, perpendicular to the stop line drawn before entering the intersection, and a dashed white line is drawn further before that. In the case of solid white lines and dashed white lines, if they are lane boundaries, traffic laws allow for lane changes and overtaking by crossing into either lane. However, the solid white line before the intersection prohibits lane changes for overtaking.

[0076] In the driving environment shown in Figure 13, the AI ​​model 212 recognizes the information of each lane segment V (V1, V2, V3, V4) as lane information. For example, for V1, the characteristics of the right lane of V1 (color: white, type: solid) and the characteristics of the left lane of V1 (color: white, type: solid) are obtained. Similarly, for V2, the characteristics of the right lane of V2 (color: white, type: dashed) and the characteristics of the left lane of V2 (color: white, type: solid) are obtained.

[0077] Figure 14 shows tables of lane topology and lane change topology output by the AI ​​model 212 from the driving environment shown in Figure 13. As shown in Figure 14, each table numerically represents the relationship between lanes and whether or not a lane change is possible.

[0078] In the lane topology table shown in the upper part of Figure 14, the connection relationships between each lane segment V (V1, V2, V3, V4) are shown numerically. The meaning of the numbers is the same as in Figure 9. Therefore, for example, there is a "2" between V1 and V3, and between V2 and V4, indicating that they are parallel connections in the same direction as the direction of travel.

[0079] The lane change topology table shown in the lower part of Figure 14 indicates the feasibility of lane changes using numerical values. The meaning of these values ​​is the same as in Figure 10. Therefore, for example, it can be seen that lane changes from V2 to V4 and from V4 to V2 are possible. On the other hand, it can be seen that caution is required when changing lanes from V1 to V3 and from V3 to V1.

[0080] As a result, the driver of vehicle 10 is notified via a screen display that caution is required when changing lanes from V3 to V1 when changing lanes onto the left-hand roadway R1.

[0081] <<2-2. Rule-Based Inference of Lane Change Feasibility>> The rule-based inference of lane change feasibility according to this embodiment will be explained in detail. The rule-based inference of lane change feasibility is realized by the rule-based determination unit 230 shown in Figure 2.

[0082] Figure 15 is a diagram illustrating the generation of rule-based lane change topology information 442 according to this embodiment.

[0083] The recognition processing unit 210 inputs the input data 410 to the AI ​​model 212 and generates multiple outputs. Specifically, the AI ​​model 212 analyzes the input data 410 and outputs Lane segment information 420 and Lane topology information 430.

[0084] Next, the rule-based decision unit 230 refers to the lane change permission table 232 based on the lane segment information 420 and lane topology information 430, and generates rule-based lane change topology information 442. The lane change permission table 232 stores the rules for lane changes based on traffic laws.

[0085] In this embodiment, by making a rule-based decision based on the output of the AI ​​model, it becomes possible to make a more accurate determination of whether or not a lane change is permissible in accordance with legal norms.

[0086] The rule-based lane change topology information 442 is output to the display control unit 220, which uses it to generate a display screen indicating whether or not a lane change is permitted.

[0087] (Operational Processing) Next, we will explain the operational processing for inferring whether or not a lane change is permissible based on the rules.

[0088] Figure 16 is a flowchart showing an example of the flow of the rule-based lane change topology generation process according to this embodiment.

[0089] As shown in Figure 16, first, the rule-based decision unit 230 acquires Lane segment information and Lane topology information (S203). The Lane segment information and Lane topology information may be obtained using an AI model.

[0090] Next, the rule-based decision unit 230 sets the color and shape information of the lane segment as a key (S206). Specifically, the rule-based decision unit 230 uses the color and shape information of the boundary lines of each lane as a key for rule-based decision-making.

[0091] Next, the rule-based decision unit 230 uses the key to refer to the lane change feasibility table 232 for lane segments that are in a parallel relationship in the same direction in the lane topology, and obtains whether or not a lane change is permitted (S209). As a result, whether or not a lane change is permitted between parallel lanes (specifically, between lane segments) is determined based on predefined rules. The feasibility of the change includes permitted, caution required (permitted but requires caution), and prohibited.

[0092] Finally, the rule-based decision unit 230 stores the feasibility of lane changes in matrix format and outputs it as rule-based lane change topology information (S212).

[0093] Figure 17 is a diagram illustrating the rule-based reasoning for whether or not a lane change is possible according to this embodiment. The diagram describes how to infer (recognize) whether or not a lane change from V1 to V2 is possible in a road environment consisting of multiple lane segments (V1, V2, V3, V4, V5, V6, V7, V8) as shown in Figure 17.

[0094] The rule-based decision unit 230 obtains information on V1 and V2 from the Lane segment information 420. Since pairs of parallel Lane segments share a boundary, the color information and shape information of the boundary line at that boundary are obtained from each Lane segment. In the case of V1 and V2, the Right lane of V1 and the Left lane of V2 are the boundary lines at the boundary between them.

[0095] For example, the rule-based decision unit 230 obtains from the lane segment information 420 that the right lane of V1 has a color of 1 (e.g., yellow) and a shape of 0 (e.g., a solid line), and the left lane of V2 has a color of 2 (e.g., white) and a shape of 1 (e.g., a dashed line). Here, we assume that there is not a single boundary line, but a double line for the right lane of V1 and the left lane of V2. The rule-based decision unit 230 also obtains from the lane topology information 430 that the relationship between V1 and V2 is "2 (parallel, left and right)". The rule-based decision unit 230 uses the obtained information as key 4410 and refers to the lane change permission table 232.

[0096] Specifically, the rule-based decision unit 230, when changing lanes from V1 to V2, refers to the lane change feasibility table 232 based on the color and shape of the lane to be crossed (lane boundary line) among the keys 4410. Since the lane change feasibility is based on the boundary line of the lane being changed, the rule-based decision unit 230 obtains a reference result 4412 in which the lane change topology from V1 to V2 corresponds to the values ​​in the table "color: 1" and "shape: 0".

[0097] The lane change permit / failure table 232 includes information on traffic laws and regulations and serves to correct the inference results of the AI ​​model. Figure 18 shows an example of the lane change permit / failure table 232 according to this embodiment. As shown in Figure 18, the lane change permit / failure table 232 determines whether a lane change is permitted based on the combination of lane color and shape.

[0098] In the lane change permission table 232, the vertical axis represents the lane color, where 0 is White, 1 is Yellow, and 2 is Non-visible. The horizontal axis represents the lane shape, where 0 is Solid and 1 is Dashed.

[0099] For example, in the case of a white line (0: White), a solid line (0: Solid) means that lane changes are permitted (2), while a dashed line (1: Dashed) means that lane changes require caution (1). On the other hand, in the case of a yellow line (1: Yellow), lane changes are not permitted (0) whether the line is solid or dashed. Also, if the line is not visible (2: Non-visible), lane changes are also deemed not permitted (0).

[0100] Therefore, the reference result 4412 shown in Figure 17 corresponds to the value "0" (lane change not permitted), as shown in Figure 18, where "color: 1" and "shape: 0".

[0101] <<2-3. Inference of Lane Change Feasibility by Integrated Processing>> Next, the inference of lane change feasibility by integrated processing according to this embodiment will be explained in detail. The inference of lane change feasibility by integrated processing is realized by the integrated processing unit 240 shown in Figure 2.

[0102] Figure 19 is a diagram illustrating the generation of lane change topology information 450 based on the integration according to this embodiment. As shown in Figure 19, this system generates final lane change topology information by integrating the inference results of an AI model and rule-based decisions.

[0103] First, the recognition processing unit 210 analyzes the input data using the AI ​​model 212 and outputs Lane segment information 420, Lane topology information 430, and AI-based lane change topology information 440. Next, the rule-based decision unit 230 refers to the lane change feasibility table 232 based on the Lane segment information 420 and Lane topology information 430 output from the AI ​​model 212 and generates rule-based lane change topology information 442.

[0104] Next, the integration processing unit 240 integrates the AI-based lane change topology information 440 and the rule-based lane change topology information 442. The integration processing unit 240 compares the AI-based inference result (AI-based lane change topology information 440) with the rule-based judgment (rule-based lane change topology information 442) and generates lane change topology information 450 based on more stringent criteria. This final lane change topology information 450 corrects the AI-based inference result and provides a highly reliable indication of whether or not a lane change is possible.

[0105] Figure 20 shows an example of the integration of AI-based lane change topology information 440a and rule-based lane change topology information 442a according to this embodiment. The integration processing unit 240 compares the AI-based and rule-based values ​​and performs a min process to adopt the stricter (smaller) value. For example, a change from V3 to V4 is "2" in the rule-based system and "0" in the AI-based system. Therefore, "0" is adopted in the integrated lane change topology information 450a, and a lane change from V3 to V4 becomes impossible.

[0106] In this way, integrated processing combines the flexible inference results of the AI ​​model with strict rule-based judgment, enabling safer and more appropriate lane change decisions.

[0107] (Operation Processing) Figure 21 is a flowchart showing an example of the process flow for generating lane change topology information based on the integration according to this embodiment.

[0108] As shown in Figure 21, first, the recognition processing unit 210 uses data obtained from sensors and the like as input to an AI model to infer Lane segment information, Lane topology information, and AI-based lane change topology information (S303).

[0109] Next, the rule-based decision unit 230 acquires rule-based lane change topology information using lane segment information, lane topology information, and a lane change feasibility table (S306).

[0110] The integration processing unit 240 then compares each element of the AI-based lane change topology information and the rule-based lane change topology information and adopts the smaller value as the topology of the integration result (S309). This generates lane change topology information that integrates the AI-based lane change topology information and the rule-based lane change topology information.

[0111] <3. Training of AI Model 212> Next, we will explain the training of AI Model 212 (construction of the recognition model). It is assumed that the training of AI Model 212 will be performed on a computer cluster or the like before deployment to an actual vehicle. Here, we will explain AI model training using the information processing device 50 as an example.

[0112] <<3-1. Configuration>> Figure 22 is a block diagram showing an example of the configuration of the information processing device 50 according to this embodiment. As shown in Figure 22, the information processing device 50 includes a communication unit 510, a control unit 520, an operation input unit 530, a display unit 540, and a storage unit 550.

[0113] (Communication Unit 510) The communication unit 510 has a transmitting unit that transmits data to an external device and a receiving unit that receives data from the external device. The communication unit 510 according to this embodiment may communicate with an external device or the Internet using, for example, a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile communication network (LTE (Long Term Evolution), 4G (fourth-generation mobile communication system), 5G (fifth-generation mobile communication system)), etc.

[0114] (Control Unit 520) The control unit 520 functions as an arithmetic processing unit and control unit, and controls the overall operation within the information processing unit 50 according to various programs. The control unit 520 is implemented by electronic circuits such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or a microprocessor. The control unit 520 may also include a ROM (Read Only Memory) for storing programs and arithmetic parameters used, and a RAM (Random Access Memory) for temporarily storing parameters that change as needed.

[0115] In this embodiment, the control unit 520 also functions as a learning data generation unit 522 and a learning unit 524. The learning data generation unit 522 generates correct answer data as learning data. The learning unit 524 trains the AI ​​model 212, which infers lane segment, lane topology, and AI-based lane change topology, based on the learning data (correct answer data).

[0116] (Operation input unit 530 and display unit 540) The operation input unit 530 receives operation input to the information processing device 50 and outputs the operation input information to the control unit 520. The display unit 540 displays various display screens. The operation input unit 530 and the display unit 540 may be configured as separate units or may be integrated.

[0117] (Storage Unit 550) The storage unit 550 is implemented by a ROM that stores programs and calculation parameters used in the processing of the control unit 520, and a RAM that temporarily stores parameters that change as needed.

[0118] <<3-2. Preparation of Training Data>> The preparation of training data by the training data generation unit 522 will be explained. In this embodiment, supervised learning is assumed, and correct answer data (teacher data) is prepared as training data.

[0119] Figure 23 is a diagram illustrating the generation of lane change topology information used as learning data according to this embodiment. The learning data generation unit 522 first acquires manually annotated Lane segment information 610 (an example of correct lane information), Lane topology information (without parallelism) 611 (an example of correct lane connection information), and manual base lane change topology information 631 (information about provisionally set lane changes). Lane topology information (without parallelism) 611 is data annotated only with serial connection information (serial connection information). The learning data generation unit 522 may acquire such data from an external device, such as a database on a network, via the communication unit 510, or it may generate it according to operation input (annotation work) from the operation input unit 530. Note that the data to be prepared is not limited to manually annotated data.

[0120] The learning data generation unit 522 creates Lane topology information (including parallel) 612, which includes connection information in the parallel direction, based on Lane segment information 610 and Lane topology information (without parallel) 611, to clarify the relationships between lanes in more detail. The learning data generation unit 522 can function as a first correct answer data acquisition unit. The learning data generation unit 522 can determine the overlap of boundary lines for each lane segment from the Lane segment information 610 and automatically add connection information in the parallel direction. Note that the addition of connection information in the parallel direction may be done manually. That is, the learning data generation unit 522 may acquire Lane topology information annotated with serial direction connection information and parallel direction connection information.

[0121] Next, the learning data generation unit 522 uses the rule-based decision unit 222a to refer to the lane change permission table 620 and make a decision on whether or not to change lanes based on the rules. The lane change permission table 620 stores the rules for changing lanes based on traffic laws. The rule-based decision unit 222a has the same functions as the rule-based decision unit 230. The lane change permission table 620 may be the same as the lane change permission table 232.

[0122] The rule-based decision unit 222a refers to the lane change feasibility table 620 based on the lane segment information 610 and lane topology information (including parallel) 612, and generates rule-based lane change topology information 621 (an example of a decision result).

[0123] Next, the learning data generation unit 522 integrates the manual-based lane change topology information 631 (an example of information about provisionally set lane changes) and the rule-based lane change topology information 621 (an example of a judgment result) using the integration processing unit 522b. The integration processing unit 522b has the same functions as the integration processing unit 240. More specifically, the integration processing unit 522b compares the manual-based and rule-based values ​​and performs a min process to adopt the stricter (smaller) value.

[0124] Finally, the integrated processing unit 522b generates the final lane change topology information 641 (an example of a correct recognition result). The integrated processing unit 522b can function as a second correct data acquisition unit. In this way, the lane change topology information 641 filtered by the rule base is used as one of the correct data for learning, which will be described later.

[0125] (Processing) Figure 24 is a flowchart showing an example of the process flow for generating lane change topology information used as learning data according to this embodiment.

[0126] First, the learning data generation unit 522 acquires manually annotated lane segment information, lane topology information (without parallelism), and manual base lane change topology information (S403).

[0127] Next, the learning data generation unit 522 adds parallel information to the lane topology information (no parallelism) based on the overlap information of the right lane and left lane of the lane segment (S406). Figure 25 is a diagram showing an example of a lane structure according to this embodiment. In Figure 25, each Center line, Right lane, and Left lane included in the multiple lane segments that make up the road are indicated by arrows pointing in the direction of travel. As shown in Figure 25, for example, let's assume that a dashed white line is drawn between V5 and V6. In this case, the learning data generation unit 522 can determine that V5-left and V6-right overlap in area D (position of the dashed white line) based on the lane segment information, and can determine that they are parallel. As a result, the learning data generation unit 522 can add parallel information to the lane topology information (no parallelism).

[0128] Next, the learning data generation unit 522 creates a key from the lane segment information and lane topology information (including parallel), and generates rule-based lane change topology information by referring to the lane change feasibility table 620 (S409).

[0129] Then, the integrated processing unit 522b integrates the rule-based lane change topology information with the manual-based lane change topology information to generate higher-quality lane change topology information as the final correct answer data (S412). By integrating with rules based on traffic laws, errors in manual annotation can be corrected, and highly accurate data for AI learning can be generated.

[0130] <<3-3. Learning Process>> Figure 26 is a diagram illustrating the learning process of the AI ​​model 212 according to this embodiment.

[0131] The learning unit 524 compares the Lane segment information 420, Lane topology information 430, and AI-based lane change topology information 440 obtained by inputting the input data 410 into the AI ​​model 212 with the Lane segment information 610, Lane topology information (including parallel) 612, and lane change topology information 641, which have been prepared as correct answer data, and performs processing to optimize the AI ​​model 212 so that losses are reduced.

[0132] (Operation Process) Figure 27 is a flowchart showing an example of the learning process flow according to this embodiment.

[0133] As shown in Figure 27, first, the learning unit 524 takes sensor data and the like as input to the AI ​​model 212 and outputs Lane segment information 420, Lane topology information 430, and AI-based lane change topology information 440 (S433).

[0134] Next, the learning unit 524 uses the correct data Lane segment information 610, Lane topology information (including parallel) 612, and Lane change topology information 641 to train the AI ​​model (S436).

[0135] While there are no specific limitations on the AI ​​model's learning method, one example is described below.

[0136] The learning unit 524 may utilize a convolutional neural network (CNN) for learning the Lane segment, for example, similar to Lane Seg Net: Map Learning with Lane Segment Perception for Autonomous Driving.

[0137] Furthermore, with regard to learning Lane topology and lane change topology, the learning unit 524 calculates a loss function (error) between the correct label and the output of the AI ​​model, as described below, and adjusts the parameters of the AI ​​model in a direction that minimizes the loss function.

[0138] Figure 28 shows an example of a formula for calculating the loss function according to this embodiment. In the formula shown in Figure 28, C is the number of categories that Lane topology and lane change topology can each take. The learning unit 524 can calculate the loss function (Loss) using the formula shown in Figure 28.

[0139] As a result, the accuracy of AI model 212 can be improved.

[0140] <4. Hardware Configuration Example> An embodiment of the present disclosure has been described above. Next, with reference to Figure 29, an example of a hardware configuration used in a vehicle control system 11 or information processing device 50 according to an embodiment of the present disclosure will be described.

[0141] Figure 29 is a block diagram showing an example of the hardware configuration of an information processing device 900 according to one embodiment of the present disclosure. The information processing device 900 is an example of a hardware configuration applied to the vehicle control system 11 or information processing device 50 according to this embodiment. Note that the information processing device 900 does not necessarily have all of the hardware configurations shown in Figure 29.

[0142] As shown in Figure 29, the information processing device 900 includes a processing circuit 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.

[0143] The processing circuit 901 functions as an arithmetic processing unit and control unit, and controls the overall operation or a part of the operation within the information processing unit 900 according to various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 927. The ROM 902 stores programs and calculation parameters used by the processing circuit 901. The RAM 903 temporarily stores programs used in the execution of the processing circuit 901 and parameters that change as appropriate during its execution. The processing circuit 901, ROM 902, and RAM 903 are interconnected by a host bus 907, which is composed of an internal bus. Furthermore, the host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 909.

[0144] The input device 915 is a device operated by the user, such as a button. The input device 915 may also include a mouse, keyboard, touch panel, switch, and lever. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an external connection device 929 such as a mobile phone that is compatible with the operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the processing circuit 901. By operating this input device 915, the user inputs various data to the information processing device 900 or instructs it to perform processing operations.

[0145] The input device 915 may also include an imaging device and sensors. The imaging device is a device that captures real space and generates an image using various components such as an image sensor, such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor), and a lens for controlling the imaging of a subject onto the image sensor. The imaging device may capture still images or motion images. The sensors are various types of sensors, such as distance sensors, acceleration sensors, gyro sensors, geomagnetic sensors, vibration sensors, light sensors, and sound sensors. The sensors acquire information about the state of the information processing device 900 itself, such as the orientation of the housing of the information processing device 900, and information about the surrounding environment of the information processing device 900, such as the brightness and noise around the information processing device 900. The sensors may also include a GPS sensor that receives GPS (Global Positioning System) signals and measures the latitude, longitude, and altitude of the device.

[0146] The output device 917 is comprised of a device capable of visually or audibly notifying the user of the acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, etc. The output device 917 outputs the results obtained from the processing of the information processing device 900 as images such as text or pictures, or as sound such as voice or sound. The output device 917 may also include a lighting device that brightens the surroundings.

[0147] The storage device 919 is a data storage device configured as an example of the storage unit of the information processing device 900. The storage device 919 is composed of, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores programs and various data executed by the processing circuit 901, as well as various data acquired from external sources.

[0148] The drive 921 is a reader / writer for removable recording media 927 such as magnetic disks, optical disks, magneto-optical disks, or semiconductor memory, and is either built into or external to the information processing device 900. The drive 921 reads information recorded on the installed removable recording media 927 and outputs it to the RAM 905. The drive 921 also writes data to the installed removable recording media 927.

[0149] The connection port 923 is a port for directly connecting equipment to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. Alternatively, the connection port 923 may be an RS-232C port, an optical audio terminal, or an HDMI (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0150] The communication device 925 is a communication interface, for example, consisting of a communication device for connecting to an external network 70. The communication device 925 may be, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth®, Wi-Fi®, or WUSB (Wireless USB). Alternatively, the communication device 925 may be a router for optical communication, an ADSL (Asymmetric Digital Subscriber Line) router, or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, to the Internet or other communication devices using a predetermined protocol such as TCP / IP. The external network 70 connected to the communication device 925 is a network connected by wire or wireless, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0151] For example, when the information processing device 900 functions as a vehicle control system 11 or information processing device 50 according to an embodiment of this disclosure, the processing circuit 901 of the information processing device 900 functions as a control unit 20 or control unit 520 by executing a program loaded on the RAM 903. The storage device 919 stores the information processing program according to this disclosure and various data stored in the storage unit 27 or storage unit 550. The processing circuit 901 reads and executes the program data from the storage device 919, but as another example, these programs may be obtained from other devices via an external network 70. In other words, the storage device 919 is not limited to being inside the information processing device 900, but may be located outside the information processing device 900. The processing circuit 901 is an example of an integrated circuit, and CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), APU (Accelerated Processing Unit), ASIC (Application Specific Integrated Circuit), and FPGA (Field Programmable Gate Array) can all be considered integrated circuits.

[0152] Furthermore, when the information processing device 900 functions as a vehicle control system 11 or information processing device 50 according to the embodiment of this disclosure, the communication device 925 of the information processing device 900 corresponds to the communication unit 21 or the communication unit 510. The input device 915 corresponds to the external sensor 22, the position information acquisition unit 23, the input unit 24, the in-vehicle sensor 25, the vehicle sensor 26, or the operation input unit 530. The output device 917 corresponds to the display unit 28, the speaker 29, or the display unit 540.

[0153] <5. Supplementary Information> Although preferred embodiments of the present disclosure have been described in detail above with reference to the attached drawings, the present technology is not limited to such examples. It is clear that a person with ordinary skill in the art of the present disclosure may conceive of various modifications or alterations within the scope of the technical idea described in the claims, and these will naturally be understood to fall within the technical scope of the present disclosure.

[0154] Each component of the illustrated device is a functional concept and does not necessarily have to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0155] The information processing system described herein may be a vehicle control system 11 or an information processing device 50 consisting of a single device, or the vehicle control system 11 or information processing device 50 may be configured with multiple devices.

[0156] Furthermore, the embodiments of this disclosure described above can be combined as appropriate in areas where the processing content is not contradictory. Also, the order of each step shown in the sequence diagram or flowchart of this embodiment can be changed as appropriate. For example, each step may be processed chronologically, repeatedly, or partially in parallel.

[0157] Furthermore, one or more computer programs can be created for the hardware, such as the CPU, ROM, and RAM, built into the vehicle control system 11 or the information processing device 50, to enable the vehicle control system 11 or the information processing device 50 to perform its functions. A computer-readable storage medium storing such one or more computer programs is also provided.

[0158] Furthermore, the effects described herein are merely descriptive or illustrative and not limiting. In other words, the technology relating to this disclosure may produce other effects that will be apparent to those skilled in the art from the description herein, in addition to or in lieu of the effects described herein.

[0159] Furthermore, this technology can also be configured as follows: (1) An information processing system comprising: an acquisition unit that acquires sensing results of the surrounding environment of a vehicle; and a recognition processing unit that takes the sensing results as input and outputs lane information, which is information for each of two or more lanes, and a recognition result regarding lane changes relating to the lane in which the vehicle is traveling, from a recognition model, from which the lane information and the recognition result are obtained. (2) The information processing system according to (1), wherein the recognition model takes the sensing results as input and further outputs lane connection information indicating the connection relationship between each lane, and the recognition processing unit further obtains the lane connection information from the recognition model. (3) The information processing system according to (1) or (2), wherein the information processing system further comprises a display control unit that generates a display screen showing guidance regarding lane changes after the recognition result has been obtained by the recognition processing unit. (4) The information processing system according to (3), wherein the display control unit generates the display screen by superimposing a display showing guidance regarding lane changes onto an image of the surrounding area of ​​the vehicle. (5) The information processing system according to any one of (2) to (4) above, wherein the lane connection information includes information indicating whether the relationship between each lane is parallel, adjacent, and in the same direction. (6) The information processing system according to (5) above, wherein the lane information includes information indicating the color or shape of the lane. (7) The information processing system according to (6) above, further comprising the lane information and the lane connection information, and a rule-based determination unit that obtains a determination result regarding a lane change with respect to the lane in which the vehicle is traveling, based on pre-set rules. (8) The information processing system according to (7) above, further comprising an integration processing unit that integrates the recognition result obtained by the recognition processing unit and the determination result obtained by the rule-based determination unit. (9) The information processing system according to (8) above, wherein if the recognition result and the determination result are different, the integration processing unit adopts the result that indicates that a lane change is not possible, or the result that is closer to the result that indicates that a lane change is not possible, from among the recognition result and the determination result.(10) The information processing system according to any one of (2) to (9), further comprising a learning unit that trains the recognition model using the sensing results and training data consisting of correct lane information, correct lane connection information, and correct recognition results. (11) The information processing system according to (10), wherein the correct lane connection information includes information indicating whether the connection relationship between each lane is adjacent and in the same direction and is parallel, and the information processing system further comprises a first correct data acquisition unit that acquires the correct lane connection information based on the correct lane information and serial connection information between each lane. (12) The information processing system according to (11), further comprising: a rule-based judgment unit that obtains a judgment result regarding a lane change relating to the lane in which the vehicle is traveling, based on the correct lane information and the correct lane connection information and a pre-set rule; and a second correct data acquisition unit that obtains the correct recognition result by integrating the judgment result and the provisionally set lane change information. (13) The information processing system according to (12), wherein the correct lane information, the connection information between each lane, and the provisionally set lane change information are manually created information. (14) An information processing method comprising: a processor obtaining sensing results of the surrounding environment of a vehicle; and obtaining the lane information and the recognition result from a recognition model that takes the sensing results as input and outputs lane information which is information for each of two or more lanes, and a recognition result regarding a lane change relating to the lane in which the vehicle is traveling. (15) The information processing method according to (14), wherein the recognition model takes the sensing result as input and outputs lane connection information indicating the connection relationship between each lane, and the processor further includes obtaining the lane connection information from the recognition model. (16) The information processing method according to (14) or (15), wherein the processor further includes generating a display screen indicating guidance regarding lane changes after obtaining the recognition result.(17) The information processing method according to (15) or (16), further comprising the processor obtaining a decision result regarding a lane change in the lane in which the vehicle is traveling, based on the lane information and the lane connection information and a pre-set rule. (18) The information processing method according to (17), further comprising the processor integrating the recognition result and the decision result. (19) The information processing method according to (18), wherein if the recognition result and the decision result are different, the processor adopts and integrates the result that is closer to the result that a lane change is not possible or that a lane change is not possible from among the recognition result and the decision result. (20) The information processing method according to any one of (15) to (19), further comprising the processor training the recognition model using the sensing result and training data consisting of correct lane information, correct lane connection information and correct recognition results.

[0160] 10 Vehicle 11 Vehicle control system 20 Control unit 210 Recognition processing unit 220 Display control unit 230 Rule-based decision unit 240 Integrated processing unit 50 Information processing device 510 Communication unit 520 Control unit 522 Learning data generation unit 522a Rule-based decision unit 522b Integrated processing unit 524 Learning unit 530 Operation input unit 540 Display unit 550 Storage unit

Claims

1. An information processing system comprising: an acquisition unit that acquires sensing results of the surrounding environment of a vehicle; and a recognition processing unit that takes the sensing results as input and obtains the lane information and the recognition results from a recognition model that outputs lane information, which is information for each of two or more lanes, and a recognition result regarding a lane change related to the lane in which the vehicle is traveling.

2. The information processing system according to claim 1, wherein the recognition model takes the sensing result as input and outputs lane connection information indicating the connection relationship between each lane, and the recognition processing unit further obtains the lane connection information from the recognition model.

3. The information processing system according to claim 1, further comprising a display control unit that generates a display screen showing guidance regarding lane changes after the recognition result has been obtained by the recognition processing unit.

4. The information processing system according to claim 3, wherein the display control unit generates the display screen by superimposing a display indicating guidance regarding the lane change onto an image of the area around the vehicle.

5. The information processing system according to claim 2, wherein the lane connection information includes information indicating whether the relationship between each lane is parallel, adjacent, and in the same direction.

6. The information processing system according to claim 5, wherein the lane information includes information indicating the color or shape of the lane.

7. The information processing system according to claim 6, further comprising a rule-based determination unit that obtains a determination result regarding a lane change in the lane in which the vehicle is traveling, based on the lane information and the lane connection information and pre-set rules.

8. The information processing system according to claim 7, further comprising an integration processing unit that integrates the recognition result obtained by the recognition processing unit and the judgment result obtained by the rule-based judgment unit.

9. The information processing system according to claim 8, wherein, if the recognition result and the judgment result differ, the integrated processing unit adopts the result that indicates lane changes are not permitted, or the result that is closer to the result that lane changes are not permitted, from among the recognition result and the judgment result.

10. The information processing system according to claim 2, further comprising a learning unit that trains the recognition model using the sensing results and training data consisting of correct lane information, correct lane connection information, and correct recognition results.

11. The information processing system according to claim 10, wherein the correct lane connection information includes information indicating whether the connection relationship between each lane is parallel, adjacent and in the same direction, and the information processing system further comprises a first correct data acquisition unit that acquires the correct lane connection information based on the correct lane information and the serial connection information between each lane.

12. The information processing system according to claim 11, further comprising: a rule-based determination unit that obtains a determination result regarding a lane change relating to the lane in which the vehicle is traveling, based on the correct lane information and the correct lane connection information and pre-set rules; and a second correct data acquisition unit that obtains a correct recognition result by integrating the determination result with the provisionally set information regarding the lane change.

13. The information processing system according to claim 12, wherein the correct lane information, the connection information between each lane, and the provisionally set lane change information are manually created information.

14. An information processing method comprising: a processor acquiring sensing results of the surrounding environment of a vehicle; and obtaining lane information and recognition results from a recognition model that takes the sensing results as input and outputs lane information, which is information for each of two or more lanes, and recognition results regarding lane changes related to the lane in which the vehicle is traveling.

15. The information processing method according to claim 14, wherein the recognition model takes the sensing result as input and outputs lane connection information indicating the connection relationship between each lane, and the processor further includes obtaining the lane connection information from the recognition model.

16. The information processing method according to claim 14, further comprising the processor generating a display screen showing guidance regarding lane changes after obtaining the recognition result.

17. The information processing method according to claim 15, further comprising the processor obtaining a decision result regarding a lane change in the lane in which the vehicle is traveling, based on the lane information and the lane connection information and a pre-set rule.

18. The information processing method according to claim 17, further comprising the processor integrating the recognition result and the judgment result.

19. The information processing method according to claim 18, wherein, if the recognition result and the judgment result are different, the processor adopts and integrates the result that is closer to the result that lane changes are not permitted or that lane changes are not permitted.

20. The information processing method according to claim 15, further comprising the processor training the recognition model using the sensing results and training data consisting of correct lane information, correct lane connection information, and correct recognition results.

Images