Autonomous Vehicle Planner Evaluation Using Reference Plan Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing shadow mode testing for autonomous vehicles does not provide a reliable indicator of how an autonomous driving system (ADS) would perform in real scenarios, as it lacks insight into the ADS's long-term planning and reasons for discrepancies with human driving, and fails to account for the different perspectives between human and ADS observations.
Innovation Solution
A computer-implemented method evaluates the performance of a target planner by computing ego plans and reference plans in real or simulated scenarios, allowing for a systematic comparison and providing insights into the planner's decision-making processes through incremental planning steps, using a reference planner as a benchmark.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If shadow mode testing is used to compare ADS decisions with human driving, then testing can be performed without actual autonomous driving, but the testing does not provide reliable insight into how the ADS would actually perform in real scenarios
Solution Approach 1:
The system performs preliminary planning by computing both an ego plan (what the ADS would do) and a reference plan (what a human expert would do) before actual execution. This allows comparison of planning decisions in advance, enabling reliable assessment of ADS performance without requiring actual autonomous driving in real scenarios.
Solution Approach 2:
The system creates a virtual copy of the human driver's decision-making process through the reference planner, which replicates expert human driving behavior. This copy can be compared against the ADS's ego planner decisions, providing reliable performance assessment while avoiding the need for actual human-in-the-loop testing.
2Device complexity
If shadow mode operation records autonomous decisions without implementation, then data collection is simplified, but the ADS does not experience consequences of its decisions or develop long-term planning capabilities
Solution Approach 1:
The system introduces a reference planner as an intermediary that simulates expert human driving behavior. This intermediary provides the missing planning insights and long-term decision-making perspective that simple shadow mode recording lacks, without requiring actual human drivers in the loop.
Solution Approach 2:
The reference planner performs preliminary computation of expert human-like trajectories before comparison with the ego planner. This preliminary action provides the ADS with insight into long-term planning and decision-making reasons without requiring actual autonomous driving consequences.
3Adaptability or versatility
If human driving behavior is used as the benchmark, then real-world driving context is captured, but the perspective difference between human senses and ADS sensors creates comparison difficulties
Solution Approach 1:
The reference planner creates a computational copy of expert human driving behavior that operates in the same sensor and decision-making framework as the ADS. This copy eliminates perspective differences by replicating human-like processing of ADS sensor data, while maintaining real-world driving context and applicability.
Data Source
Figure 1
Figure 1A
Figure 2
AI summary
A computer-implemented method of evaluating the performance of a target planner for an ego robot in a real or simulated scenario, the method comprising: receiving evaluation data for evaluating the performance of the target planner in the scenario, the evaluation data generated by applying the target planner at incrementing planning steps, in order to compute a series of ego plans that respond to changes in the scenario, the series of ego plans being implemented in the scenario to cause changes in an ego state the evaluation data comprising: the ego plan computed by the target planner at one of the planning steps, and a scenario state at a time instant of the scenario, wherein the evaluation data is used to evaluate the target planner by: computing a reference plan for said time instant based on the scenario state, the scenario state including the ego state at that time instant as caused by implementing one or more preceding ego plans of the series of ego plans computed by the target planner, and computing at least one evaluation score for comparing the ego plan with the reference plan.