Autonomous Vehicle Planner Performance Testing via Trajectory Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing shadow mode testing for autonomous vehicle planners lacks reliability in assessing how an autonomous driving system (ADS) would perform in real scenarios and fails to provide insights into the reasons for discrepancies between human and autonomous behavior, offering limited long-term planning insights for human drivers.
Innovation Solution
A computer-implemented method evaluates the performance of a target planner by comparing its ego trajectories with those generated by a reference planner, identifying juncture points where differences occur, and determining their significance using threshold values, allowing for improved analysis of performance metrics and trajectory divergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If shadow mode testing is used to compare ADS decisions with human driving behavior, then some discrepancy between human and autonomous behavior can be demonstrated, but the reliability of assessing how the ADS would have actually performed is poor
Solution Approach 1:
The patent creates virtual copies of human driving scenarios through detailed scenario descriptions including sensor data, road rules, and environmental conditions. These copied scenarios are then executed by both the ADS and human drivers in a controlled simulation environment, enabling reliable comparison of actual performance rather than just post-hoc analysis of recorded decisions.
Solution Approach 2:
The patent introduces a simulation environment as an intermediary between the ADS and real-world testing. This intermediary allows both human drivers and ADS to operate under identical controlled conditions, eliminating the uncertainties of real-world variability while maintaining ecological validity through realistic scenario design.
2Loss of information
If shadow mode operation records autonomous decisions for later comparison, then some insight into instantaneous reasoning can be provided, but no insight is given as to how the ADS would actually perform over the duration of the scenario
Solution Approach 1:
The patent requires human drivers to complete full scenario executions before analysis begins. This preliminary action ensures that complete long-term planning information is captured, including any changes in driver intent or strategy over time. The scenario descriptions record the entire trajectory and decision-making process, not just instantaneous decisions.
Solution Approach 2:
The patent implements a feedback loop where scenario outcomes are analyzed to provide insights about both human and ADS performance. The comparison reveals whether humans and ADS made different long-term plans, and why, by analyzing the recorded scenario descriptions that include timing, position, and decision rationale throughout the entire scenario duration.
3Ease of operation
If human driver behavior is used as the benchmark for testing, then real-world driving context is maintained, but structured insights into the reasons for decisions and long-term planning are lost
Solution Approach 1:
The patent replaces the mechanical limitation of human self-reporting with an automated data capture system. Sensors, GPS, and scenario management software automatically record position, timing, sensor data, and environmental conditions throughout the scenario. This substitution captures objective planning information without relying on human ability to articulate or recall decision-making processes.
Solution Approach 2:
The patent creates detailed digital copies of the driving scenario including all sensor inputs, road rules, environmental conditions, and the human driver's actual trajectory and decisions. This copied data preserves the realism of human decision-making while enabling structured analysis of the planning process through automated comparison with ADS performance.
4Reliability
If more comprehensive scenario data is collected to improve assessment reliability, then better performance evaluation is achieved, but the complexity and resource requirements of testing increase
Solution Approach 1:
The patent segments the testing system into distinct modular components: scenario management module, simulation environment, data collection module, and analysis module. Each component handles specific aspects of the testing process, allowing comprehensive data collection without overwhelming system complexity. The segmentation enables independent development, testing, and maintenance of each module.
Data Source
AI summary
A method of evaluating the performance of a target planner for an ego robot in a scenario. Evaluation data is generated by applying the target planner from an initial scenario state to generate an actual ego trajectory taken by the ego robot. The actual ego trajectory is defined by a target trajectory parameter. Comparison data is generated by applying a comparison planner from the same initial scenario state to generate a comparison ego trajectory for a comparison ego robot, the comparison ego trajectory comprising at least one comparison trajectory parameter. A juncture point at which the comparison trajectory parameter differs from the actual trajectory parameter is determined and a difference between the actual trajectory parameter and the comparison trajectory parameter at the juncture point is determined. A comparison between the determined difference and a threshold value indicates whether the juncture point is significant.