Autonomous Vehicle Planner Evaluation via Trajectory Divergence

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing shadow mode testing for autonomous vehicles lacks reliability in assessing how an autonomous driving system (ADS) would perform in real scenarios and fails to provide insights into the reasons for discrepancies between human and autonomous behavior, offering limited long-term planning insights for human drivers.

Innovation Solution

A method involving a reference planner for systematic comparison with a target planner, using graphical user interfaces to visualize and analyze juncture points where trajectories diverge, enabling performance evaluation and insight into why certain road rules are failed or succeeded, and providing a scalable solution for simulated scenarios.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If shadow mode operation is used to compare ADS decisions with human driving behavior, then testing can be performed using real-world data, but the reliability of assessing how the ADS would have actually performed is insufficient

Engineering Contradiction:

Improvereliability of ADS performance assessmentVSAvoidcomplexity of testing system

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces a reference planner as an intermediary system that generates reference trajectories serving as an objective benchmark. This reference planner acts as a mediator between the target ADS and the evaluation process, providing reliable counterfactual trajectories that show what an optimal ADS would have done, thereby resolving the reliability issue without requiring complex human behavior analysis

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent creates a virtual copy of the target ADS by running it in simulation alongside the reference planner. This copying approach allows the system to evaluate what the target ADS would have done in counterfactual scenarios without actually implementing those decisions in the real world, providing reliable performance assessment while maintaining system simplicity

Inventive Principle:

Principle #26Copying

2Loss of information

If shadow mode operation is used to identify discrepancies between human and autonomous behavior, then some insights into behavioral differences can be obtained, but insight into the reasons for discrepancies and long-term planning is limited

Engineering Contradiction:

Improveloss of insight into decision-making reasonsVSAvoidefficiency of testing process

Core Design Contradiction:

Loss of informationVSProductivity

Solution Approach 1:

The patent implements feedback by systematically comparing target ADS trajectories with reference planner trajectories and identifying juncture points where they diverge. This feedback mechanism provides detailed insights into why discrepancies occur by showing exactly where and how the target ADS deviated from optimal behavior, enabling deep understanding of decision-making reasons while maintaining testing efficiency through automated comparison

Inventive Principle:

Principle #23Feedback

3Loss of information

If human driving behavior is used as a benchmark for assessing autonomous decisions, then real-world driving context is captured, but structured insights into long-term planning and decision reasons are not available

Engineering Contradiction:

Improveloss of structured planning insightsVSAvoidprecision of performance measurement

Core Design Contradiction:

Loss of informationVSMeasurement precision

Solution Approach 1:

The patent inverts the traditional approach by using an automated reference planner instead of human drivers as the benchmark. This inversion provides structured, consistent, and precise performance measurements because the reference planner generates objectively evaluable trajectories based on defined criteria, eliminating the information loss associated with human behavior analysis while maintaining real-world relevance through simulation

Inventive Principle:

Principle #13The other way round (Inversion)

4Speed

If instantaneous ego trajectories with short planning horizons are used, then real-time planning capability is demonstrated, but insight into long-term performance and strategic planning is reduced

Engineering Contradiction:

Improvespeed of planning responseVSAvoidduration of planning horizon

Core Design Contradiction:

SpeedVSDuration of action of moving object

Solution Approach 1:

The patent applies preliminary action by having the reference planner pre-compute reference trajectories for the entire scenario duration before evaluation. This allows the system to maintain short instantaneous planning horizons for real-time responsiveness while the pre-computed reference trajectories provide the long-term strategic context needed for comprehensive performance assessment

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20240256415A1Tools for performance testing autonomous vehicle planners

Publication Date: 2024.08.01 FIVE AI LTD

AI summary

A computer implemented method of evaluating the performance of a target planner for an ego robot in a scenario, the method comprising: rendering on a display of a graphical user interface of a computer device, a dynamic visualisation of an ego robot moving along a first path in accordance with a first planned trajectory from the target planner and of a comparison ego robot moving along a second path in accordance with a second planned trajectory from a comparison planner; detecting a juncture point at which the first and second trajectories diverge; rendering the ego robot and the comparison ego robot as a single visual object in motion along a common path shared by the first and second paths prior to the juncture point; and rendering the ego robot and the comparison ego robot as separate visual objects on the display along the respective first and second paths from the juncture point.