Autonomous Vehicle Planner Evaluation With Reference Plan Benchmarking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing shadow mode testing for autonomous vehicles does not provide a reliable indicator of how the autonomous driving system (ADS) would perform in real scenarios, as it lacks insight into the ADS's long-term decision-making and fails to account for the discrepancies between human and autonomous behaviors due to differing perspectives and lack of consequence implementation.
Innovation Solution
A computer-implemented method for evaluating the performance of a target planner by computing ego plans and reference plans in real or simulated scenarios, allowing for a systematic comparison and providing a meaningful benchmark through evaluation scores, with the reference planner operating from the same ego state and obstacle data as the target planner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If shadow mode operation is used to test autonomous vehicle planners, then testing can be performed without implementing autonomous decisions, but the testing does not provide reliable insight into how the ADS would actually perform in real scenarios
Solution Approach 1:
The patent creates a simulated environment that copies real-world driving scenarios, allowing the ADS to be tested in a virtual replica of actual conditions. This enables reliable performance assessment without requiring extensive real-world testing, as the simulation accurately reproduces the physical environment, sensor inputs, and decision-making contexts that the ADS would encounter in reality.
Solution Approach 2:
The patent implements preliminary action by having the ADS make decisions in the simulation before they would be implemented in real vehicles. The system pre-tests planning algorithms in controlled simulated scenarios, allowing developers to identify and correct issues before deploying to actual autonomous vehicles, thereby improving reliability while reducing the need for extensive real-world test miles.
2Object-affected harmful factors
If the ADS is not in control of the vehicle during testing, then autonomous decisions have no consequences, but the ADS is not required to address consequences of its decisions as the scenario progresses
Solution Approach 1:
The patent introduces a simulated environment as an intermediary between the ADS and the real world. The simulation acts as a safe mediator that allows the ADS to make decisions and experience consequences without risking actual safety. The virtual environment faithfully reproduces physical laws, vehicle dynamics, and environmental responses, so the ADS learns to address consequences of its decisions in a safe intermediate space before deploying to real vehicles.
Solution Approach 2:
The patent implements dynamics by allowing the simulated scenario to evolve in response to the ADS's decisions, just as a real scenario would. The simulation dynamically updates the environment, other agents, and consequences based on the ADS's actions, creating a realistic feedback loop. This dynamic interaction ensures the ADS must address consequences of its decisions in real-time, improving the accuracy of performance prediction while maintaining safety through the virtual medium.
Data Source
AI summary
A computer-implemented method of evaluating the performance of a target planner for an ego robot comprises receiving evaluation data for evaluating the performance of the target planner in the scenario, generated by applying the target planner at incrementing planning steps, to compute a series of ego plans that respond to changes in the scenario and are implemented in the scenario to cause changes in an ego state. The evaluation data includes the ego plan computed by the target planner at one of the planning steps, and a scenario state at a time instant of the scenario. The evaluation data is used to evaluate the target planner by computing a reference plan for said time instant based on the scenario state, the scenario state including the ego state at that time instant, and computing at least one evaluation score for comparing the ego plan with the reference plan.