Route prediction device and learning method of route prediction ai

Abstract

Provided is a route prediction device capable of: efficiently improving the quality of route prediction AI by generating a wide variety of traffic scenarios; and relearning on the basis of the wide variety of traffic scenarios while maintaining key points of the scenarios contributing to AI learning. The route prediction device comprises: a training unit that generates route prediction AI; a prediction unit that predicts future routes of nearby vehicles; a scenario selection unit that selects an abstract traffic scenario; a parameter setting unit that sets parameters in the selected abstract traffic scenario; a simulation traffic scenario generation unit that generates a simulation traffic scenario; and a simulation unit that generates route prediction data observed by a vehicle sensor of a host vehicle by simulating the behaviors of the host vehicle and the nearby vehicles on the basis of the simulation traffic scenario. The training unit updates the route prediction AI by learning the route prediction data generated by the simulation unit.

Description

Route prediction device and learning method for route prediction AI

[0001] The present invention relates to a route prediction device that generates a route prediction AI for predicting the surrounding vehicle routes, and a method for learning the route prediction AI.

[0002] In recent years, vehicles equipped with advanced driver-assistance systems (ADAS) and autonomous driving (AD) functions have become increasingly common. Furthermore, vehicles equipped with artificial intelligence (AI) that predicts the paths of surrounding vehicles, which is a prerequisite for realizing advanced ADAS and AD, are also becoming more widespread.

[0003] In-vehicle route prediction AI is typically generated by training a dedicated route prediction device with a large amount of traffic scenarios (training data), and then porting that AI to the vehicle's system. Therefore, to improve the quality of in-vehicle route prediction AI, it is necessary to train the dedicated route prediction device with a wide variety of traffic scenarios.

[0004] However, since it is difficult to obtain a wide variety of traffic scenarios from real-world environments, a technique has been proposed to generate other traffic scenarios by changing the parameters of traffic scenarios obtained from real-world environments.

[0005] Patent Document 1 is known as a document disclosing this type of technology, and claims 1, 7, and 8 of the said document contain the following: (Claim 1) A method for updating a target driving policy of a vehicle at a target location, comprising the steps of: acquiring vehicle driving data at the target location by the vehicle; transmitting the acquired vehicle driving data and the current target driving policy for the target location to a data center by the vehicle; having the data center perform a traffic simulation for the target location using the vehicle driving data in order to acquire an updated target driving policy; and having the data center transmit the updated target driving policy to the vehicle. (Claim 7) The method according to any one of claims 1 to 6, further comprising the step of generating a different traffic scenario by modifying an initial traffic scenario acquired from the vehicle driving data, wherein the traffic simulation for the target location is performed using the generated different traffic scenario. (Claim 8) The method according to claim 7, wherein the modification of the initial traffic scenario comprises at least one of: increasing the number of agents in the traffic scenario; modifying the speed of the agents in the traffic scenario; modifying the initial position and / or direction of the agents in the traffic scenario; and modifying the trajectory of the agents in the traffic scenario.

[0006] Special Publication No. 2024-510880

[0007] As described above, claim 7 of Patent Document 1 modifies the initial traffic scenario obtained from vehicle driving data to generate a different traffic scenario. However, if the initial traffic scenario is modified haphazardly using any of the methods in claim 8 of the same document, it is possible that a traffic scenario deviating from the initial traffic scenario (for example, a scenario in which surrounding vehicles cut in front of the vehicle) may be generated (for example, a scenario in which surrounding vehicles do not cut in front of the vehicle).

[0008] Therefore, while learning based on initial traffic scenarios may improve the quality of route prediction AI, learning based on generated traffic scenarios may not always improve the quality of route prediction AI. This shows that expanding the range of traffic scenarios does not always contribute to improving the quality of route prediction AI.

[0009] Therefore, the present invention aims to provide a route prediction device and a method for learning a route prediction AI that can efficiently improve the quality of the route prediction AI by generating a wide variety of traffic scenarios while maintaining the essential elements of scenarios that contribute to AI learning (for example, a situation in which a surrounding vehicle cuts in front of the vehicle itself), and by retraining based on these wide variety of traffic scenarios.

[0010] To solve the above problems, the present invention provides a route prediction device comprising: a learning unit that generates a route prediction AI for predicting the future routes of surrounding vehicles by learning route prediction data; a prediction unit that predicts the future routes of surrounding vehicles using desired route prediction data and the route prediction AI; a scenario selection unit that selects any abstract traffic scenario stored in an abstract traffic scenario group storage unit; a parameter setting unit that sets parameters for the own vehicle and surrounding vehicles within the selected abstract traffic scenario; a simulation traffic scenario generation unit that generates a simulation traffic scenario by adding traffic conditions other than surrounding vehicles to the parameterized abstract traffic scenario; and a simulation unit that generates route prediction data observed by the vehicle's sensors by simulating the behavior of the own vehicle and surrounding vehicles based on the simulation traffic scenario, wherein the learning unit updates the route prediction AI by learning the route prediction data generated by the simulation unit.

[0011] According to the route prediction device and route prediction AI learning method of the present invention, it is possible to efficiently improve the quality of the route prediction AI by generating a wide variety of traffic scenarios while maintaining the essential elements of the scenarios that contribute to AI learning, and by retraining based on these wide variety of traffic scenarios.

[0012] Conceptual diagram of a route prediction system in one embodiment. Functional block diagram of a route prediction device in one embodiment. Example of a combination of road shape and vehicle behavior, which are elements of an abstract traffic scenario in one embodiment. Example of the position and behavior of other vehicles, which are elements of an abstract traffic scenario in one embodiment. Example of an abstract traffic scenario in one embodiment. Flowchart showing the method for generating route prediction data in one embodiment. Example of a traffic scenario for simulation in one embodiment. Flowchart showing the method for retraining the route prediction AI in one embodiment.

[0013] The following describes an embodiment of the path prediction device 1 of the present invention with reference to the drawings.

[0014] Figure 1 is a conceptual diagram of the route prediction system 100 of this embodiment. The route prediction system 100 shown here is a system comprising a route prediction device 1 that generates route prediction AI, a wireless communication network 2 such as a mobile phone network, and an on-board device 3 mounted on each vehicle V. Each on-board device 3 is equipped with the route prediction AI generated by the route prediction device 1. Therefore, each on-board device 3 can predict the surrounding vehicle routes based on the output of its own vehicle's on-board sensors (camera, radar, LiDAR, etc.) and realize appropriate ADAS and AD functions accordingly.

[0015] Figure 2 is a functional block diagram of the route prediction device 1. As shown here, the route prediction device 1 includes a learning unit 11, a prediction unit 12, a verification unit 13, a scenario selection unit 14, a parameter setting unit 15, a simulation traffic scenario generation unit 16, a simulation unit 17, a route prediction data storage unit DB1, and an abstract traffic scenario group storage unit DB2. Specifically, the route prediction device 1 is a computer such as a server on the cloud, equipped with hardware such as a computing device like a CPU, a main memory device like semiconductor memory, an auxiliary storage device like a hard disk or SSD, and a communication device. The computing device executes a program while referring to the database to realize each functional unit such as the learning unit 11. In the following, the details of each functional unit will be explained while appropriately omitting such well-known technologies.

[0016] <Learning Unit 11> The learning unit 11 is a functional unit that generates a path prediction AI by learning from path prediction data (training data) obtained from the path prediction data storage unit DB1, and registers the generated path prediction AI with the prediction unit 12.

[0017] Two types of route prediction data are stored in the route prediction data storage unit DB1. One type of route prediction data is data measured by the vehicle's onboard sensors (camera, radar, LiDAR, etc.) (video data of the area around the vehicle, distance data, point cloud data, etc.) and transmitted to the route prediction device 1 via the wireless communication network 2. The other type of route prediction data is data in the same format as the measured data (video data, distance data, point cloud data, etc.) generated by the route prediction device 1 using the method described later.

[0018] <Prediction Unit 12> The prediction unit 12 uses the route prediction AI generated by the learning unit 11 to predict the vehicle's environment under the conditions indicated by predetermined route prediction data obtained from the route prediction data storage unit DB1. ０ This is a functional unit that predicts the future paths of surrounding vehicles. This prediction unit 12 can predict multiple future paths for surrounding vehicles and can also predict the probability of each path.

[0019] The route prediction AI registered in the prediction unit 12 is transferred to the in-vehicle device 3 at a desired timing. When transferring the route prediction AI from the route prediction device 1 to the in-vehicle device 3, wireless transmission via the wireless communication network 2 may be used, or a removable storage medium may be used.

[0020] <Verification Unit 13> The verification unit 13 is a functional unit that verifies the accuracy of the prediction by comparing the surrounding vehicle route predicted by the prediction unit 12 based on the route prediction data with the surrounding vehicle route after the prediction time in the route prediction data. Therefore, by statistically processing the verification results by the verification unit 13, it is possible to determine whether the route prediction AI generated by the learning unit 11 and registered with the prediction unit 12 is of good or bad quality.

[0021] <Scenario Selection Unit 14> The scenario selection unit 14 is a functional unit that selects an appropriate abstract traffic scenario from a plurality of abstract traffic scenarios stored in the abstract traffic scenario group storage unit DB2.

[0022] Here, the abstract traffic scenarios stored in the abstract traffic scenario group storage unit DB2 will be described. An abstract traffic scenario is an abstract scenario generated by combining three types of information: (a) road shape, (b) the behavior of the host vehicle, and (c) the position and behavior of surrounding vehicles. Therefore, in the abstract traffic scenario, specific numerical values of various parameters such as the speed of the host vehicle V ０ are not registered.

[0023] FIG. 3 is a diagram illustrating a combination of (a) road shape and (b) the behavior of the host vehicle, which are elements of the abstract traffic scenario. As shown here, (a) road shapes (Road geometry) include single roads (non-intersection), merging roads (Marge zone), branching roads (Branch zone), intersections (Intersection), etc. Also, (b) the behavior of the host vehicle (Ego-vehicle behavior) includes going straight (Going straight), lane change (Lane change), turning (Turning), etc. Therefore, eight types of abstract traffic scenarios shown in the figure are generated by the combination of (a) road shape and (b) the behavior of the host vehicle.

[0024] Further, FIG. 4 is a diagram illustrating (c) the position and behavior of surrounding vehicles, which are elements of the abstract traffic scenario. As illustrated here, when focusing on the relative relationship with the host vehicle V ０ the surrounding vehicles moving in the same direction as the host vehicle V ０ are classified into surrounding vehicles V １ to V ６ , the surrounding vehicles V ０ approaching from the side of the host vehicle V ７ to V ８ , and the surrounding vehicles V ０ moving in the opposite direction to the host vehicle V ９ to V １１ . Also, as the behavior of the surrounding vehicles V ０ that may affect the safety of the host vehicle V １ to V １１ , the behaviors indicated by the arrows in the figure are included.

[0025] Figure 5 is an example of an abstract traffic scenario generated by combining three types of information: (a) road shape, (b) behavior of the host vehicle, and (c) positions and behaviors of surrounding vehicles, and shows an example of a comprehensive group of abstract scenarios based on FIGS. 3 and 4.

[0026] <Parameter setting unit 15> The parameter setting unit 15 is a functional unit that sets various parameters for the abstract traffic scenarios acquired from the abstract traffic scenario group storage unit DB2. The parameters set here are, for example, the speed of the host vehicle V ０ , the speed of surrounding vehicles, the traveling direction distance between the host vehicle V ０ and surrounding vehicles, the lateral distance between the host vehicle V ０ and surrounding vehicles, the lateral relative speed between the host vehicle V ０ and surrounding vehicles, and so on.

[0027] <Traffic scenario generation unit 16 for simulation> The traffic scenario generation unit 16 for simulation is a functional unit that generates a traffic scenario for simulation by adding traffic situations other than surrounding vehicles to the abstract traffic scenario in which the parameter setting unit 15 has set parameters such as the speed of the host vehicle. Note that traffic situations other than surrounding vehicles are, for example, other vehicles other than surrounding vehicles, various signs, road shapes in the distance, traffic signal devices, and the like.

[0028] <Simulation unit 17> The simulation unit 17 is a functional unit that obtains, by simulation, data (video data, distance data, point cloud data, etc.) that the in-vehicle sensors of the host vehicle V ０ would observe under the traffic scenario generated by the traffic scenario generation unit 16 for simulation.

[0029] <Method for generating route prediction data> Here, using the flowchart of FIG. 6, a method for generating various types of route prediction data by the route prediction device 1 of the present embodiment will be described.

[0030] In step S1, the route prediction AI of the prediction unit 12 predicts the routes of surrounding vehicles based on arbitrary route prediction data acquired from the route prediction data storage unit DB1.

[0031] In step S2, the verification unit 13 compares the predicted surrounding vehicle route in step S1 with the surrounding vehicle route after the prediction time in the acquired route prediction data.

[0032] In step S3, the verification unit 13 determines whether the two routes match, that is, determines the correctness of the predicted route by the route prediction AI. If the predicted route is correct, the process returns to step S1, and the route of the surrounding vehicle is predicted based on other route prediction data. On the other hand, if the prediction result is incorrect, the process proceeds to step S4. Note that the verification unit 13 determines the correctness of the predicted route based on the deviation of the predicted route from the actual route, but the actual route and the predicted route may be displayed on a display connected to the route prediction device 1, and the person may be allowed to input the determination result of the correctness.

[0033] In step S4, the scenario selection unit 14 acquires an abstract traffic scenario corresponding to the situation where the route prediction AI of the prediction unit 12 made an incorrect prediction from the abstract traffic scenario group storage unit DB2. Note that hereinafter, the situation where the prediction was incorrect is described as a situation where a surrounding vehicle traveling in the overtaking lane in front of the host vehicle V traveling in the one-way two-lane driving lane ０ cut in at low speed.

[0034] [[ID=·11]] In step S5, the parameter setting unit 15 sets a plurality of patterns of parameters such as the speeds of the host vehicle V and the surrounding vehicles in the abstract traffic scenario selected in step S4. Note that the parameter setting unit 15 sets various parameters within a range where the essence of the abstract traffic scenario selected in step S4 (for example, the situation of cut-in) is not lost. ０

[0035]

[0035] In step S6, the simulation traffic scenario generation unit 16 generates a plurality of simulation traffic scenarios by adding the positions and behaviors of other vehicles outside the abstract traffic scenario and roads and signs outside the abstract traffic scenario to the abstract traffic scenario whose parameters were set in step S5.

[0036] FIG. 7 is an example of a simulation traffic scenario generated in step S6. The simulation traffic scenario illustrated here is the surrounding vehicle that cuts in and the host vehicle V０ For an abstract traffic scenario showing the general relationship between the following, the parameter setting unit 15 sets the vehicle speed = 60 km / h, surrounding vehicle speed = 40 km / h, and the vehicle speed V ０ Distance from surrounding vehicles in the direction of travel = 30m, vehicle V ０ Lateral distance to surrounding vehicles = 3m, vehicle V ０ This model was created by setting the parameter of lateral relative speed of surrounding vehicles to 2 m / s, and then adding other vehicles traveling at 60 km / h in front of the surrounding vehicles, as well as various signs, intersections, traffic lights, etc.

[0037] In step S7, the simulation unit 17 simulates the environment of the multiple traffic scenarios generated in step S6, and the vehicle V ０ The system uses simulations to determine the data (video data, distance data, point cloud data, etc.) that the vehicle's onboard sensors (cameras, radar, LiDAR, etc.) will observe, and this data is stored in the route prediction data storage unit DB1 as route prediction data.

[0038] Through the above processing, it is possible to efficiently generate a wide variety of route prediction data similar to situations where the current route prediction AI fails to predict the routes of surrounding vehicles.

[0039] <Method for Retraining the Path Prediction AI> Next, the method for retraining the path prediction AI using the path prediction device 1 of this embodiment will be explained using the flowchart in Figure 8.

[0040] In step S11, the learning unit 11 learns from the path prediction data set added in step S7 and registers the updated path prediction AI with the prediction unit 12.

[0041] In step S12, the updated route prediction AI registered in the prediction unit 12 retrieves from the route prediction data storage unit DB1 the route prediction data that caused the misjudgment in step S3 by the route prediction AI before the update, or the route prediction data added in step S7, and predicts the routes of surrounding vehicles.

[0042] In step S13, the verification unit 13 compares the route of the surrounding vehicles predicted in step S12 with the route of the surrounding vehicles from the prediction point onward, as shown by the acquired route prediction data.

[0043] In step S14, the verification unit 13 determines whether the two paths match, that is, whether the predicted path by the updated path prediction AI is correct or incorrect. If the predicted path is incorrect, the process returns to step S11 and further updates the path prediction AI by learning from other path prediction data sets. On the other hand, if the prediction result is correct, the process proceeds to step S15.

[0044] In step S15, the verification unit 13 transfers the updated route prediction AI, whose quality improvement has been confirmed, to the in-vehicle device 3. For the sake of simplicity, Figure 8 illustrates an example where, if a "correct" judgment is made in step S14, the updated route prediction AI is immediately transferred to the in-vehicle device 3. However, the correct / incorrect judgment in step S14 may be repeated multiple times, and the process in step S15 may be executed only if the probability of a "correct" judgment is significantly higher than the probability of an "incorrect" judgment.

[0045] <Vehicle control using the updated route prediction AI> Next, the vehicle V equipped with the pre-update route prediction AI ０ And, the vehicle V equipped with the updated route prediction AI ０ This explains the differences in their behavior.

[0046] Vehicle V equipped with the pre-update route prediction AI ０ Under similar circumstances to the route prediction data that caused the misjudgment in step S3, it is highly likely that this will estimate an inappropriate route for surrounding vehicles. Therefore, this vehicle V ０ In ADAS and AD systems, there is a possibility that inappropriate acceleration, deceleration, and steering may be performed.

[0047] In response, the vehicle V equipped with the updated route prediction AI ０ Even under similar circumstances to the route prediction data that caused the misjudgment in step S3, this system has a high probability of estimating an appropriate surrounding vehicle route. Therefore, this vehicle V ０ With ADAS and AD, there is a high probability that appropriate acceleration, deceleration, and steering will be performed, which is expected to improve occupant comfort and safety.

[0048] <Effects of this embodiment> As described above, the route prediction device 1 of this embodiment can efficiently improve the quality of the route prediction AI by generating a wide variety of traffic scenarios while maintaining the essential elements of the scenarios that contribute to AI learning, and by retraining based on these wide variety of traffic scenarios.

[0049] 100...Route prediction system, 1...Route prediction device, 11...Learning unit, 12...Prediction unit, 13...Verification unit, 14...Scenario selection unit, 15...Parameter setting unit, 16...Traffic scenario generation unit for simulation, 17...Simulation unit, DB1...Route prediction data storage unit, DB2...Abstract traffic scenario group storage unit, 2...Wireless communication network, 3...In-vehicle device, V...Vehicle

Claims

1. A route prediction device comprising: a learning unit that generates a route prediction AI for predicting the future routes of surrounding vehicles by learning route prediction data; a prediction unit that predicts the future routes of surrounding vehicles using desired route prediction data and the route prediction AI; a scenario selection unit that selects any abstract traffic scenario stored in an abstract traffic scenario group storage unit; a parameter setting unit that sets parameters for the own vehicle and surrounding vehicles within the selected abstract traffic scenario; a simulation traffic scenario generation unit that generates a simulation traffic scenario by adding traffic conditions other than surrounding vehicles to the parameterized abstract traffic scenario; and a simulation unit that generates route prediction data observed by the vehicle's sensors by simulating the behavior of the own vehicle and surrounding vehicles based on the simulation traffic scenario, wherein the learning unit updates the route prediction AI by learning the route prediction data generated by the simulation unit.

2. A route prediction device according to claim 1, further comprising a verification unit that verifies the accuracy of the prediction by comparing the future route of surrounding vehicles predicted by the prediction unit with the route of surrounding vehicles after the prediction time in the route prediction data, wherein the scenario selection unit selects an abstract traffic scenario corresponding to the route prediction data that the verification unit has determined to have made an incorrect prediction.

3. A route prediction device according to claim 1 or claim 2, characterized in that the abstract traffic scenario is a scenario generated by combining three types of information: road shape, the behavior of the vehicle itself, and the position and behavior of surrounding vehicles.

4. A route prediction device according to claim 3, characterized in that the road shape is any of the shapes of a single road, a merging road, a branching road, or an intersecting road.

5. A route prediction device according to claim 1 or claim 2, characterized in that the parameters set by the parameter setting unit are the vehicle speed, surrounding vehicle speed, the distance between the vehicle and surrounding vehicles in the direction of travel, the lateral distance between the vehicle and surrounding vehicles, and the lateral relative speed between the vehicle and surrounding vehicles.

6. A route prediction device according to claim 1 or claim 2, characterized in that the traffic conditions other than surrounding vehicles added by the simulation traffic scenario generation unit are other vehicles other than surrounding vehicles, signs, distant road shapes, or traffic signals.

7. A method for learning a route prediction AI, comprising: a learning step of generating a route prediction AI that predicts the future routes of surrounding vehicles by learning route prediction data; a prediction step of predicting the future routes of surrounding vehicles using desired route prediction data and the route prediction AI; a scenario selection step of selecting any abstract traffic scenario stored in an abstract traffic scenario group storage unit; a parameter setting step of setting parameters for the own vehicle and surrounding vehicles within the selected abstract traffic scenario; a simulation traffic scenario generation step of generating a simulation traffic scenario by adding traffic conditions other than surrounding vehicles to the parameterized abstract traffic scenario; a simulation step of generating route prediction data observed by the vehicle's sensors by simulating the behavior of the own vehicle and surrounding vehicles based on the simulation traffic scenario; and a retraining step of updating the route prediction AI by learning the route prediction data generated in the simulation step.

Images