Autonomous Driving Simulator Metrics for Realism Evaluation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional methods for evaluating the realism of driving scenario simulators for autonomous vehicles are computationally infeasible due to their high-dimensionality, requiring large amounts of training data and failing to accurately measure overall simulator realism by focusing on single low-level properties.

Innovation Solution

The system computes a high-level metric as a combination of multiple low-level metrics to evaluate simulator realism, allowing for efficient and accurate assessment of simulator performance using less data, while prioritizing critical properties like collisions and road departures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional methods exactly evaluate simulator realism by estimating joint likelihoods of low-level properties, then measurement precision is improved, but computational complexity becomes infeasible and data requirements increase

Engineering Contradiction:

Improvesimulator realism evaluation accuracyVSAvoidcomputational complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the evaluation process into two distinct levels: (1) computing individual low-level metrics for specific properties (collisions, road departures, agent trajectories) and (2) combining these into a high-level realism metric. This segmentation allows efficient computation of individual components while maintaining comprehensive evaluation, resolving the contradiction between precision and complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent applies partial action by selectively evaluating only the most critical low-level properties (collisions, road departures, agent trajectories) rather than attempting to evaluate all possible properties. This partial evaluation approach maintains measurement precision for safety-critical aspects while significantly reducing computational complexity and data requirements.

Inventive Principle:

Principle #16Partial or excessive action

2Measurement precision

If conventional methods evaluate all low-level properties jointly, then measurement precision is improved, but data requirements increase

Engineering Contradiction:

Improveoverall realism measurement accuracyVSAvoidtraining data volume

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The evaluation is segmented into independent low-level metric computations that can be performed with smaller, targeted datasets. Each low-level metric (collisions, road departures, trajectories) can be evaluated separately with appropriate data samples, then combined into the overall realism assessment, reducing the total data volume required while maintaining precision.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent focuses computational resources on evaluating only the most critical properties (collisions, road departures) with high precision using targeted data samples, rather than attempting to evaluate all properties equally. This partial focus reduces overall data requirements while maintaining measurement precision for safety-critical aspects.

Inventive Principle:

Principle #16Partial or excessive action

3Productivity

If conventional methods focus on single low-level properties, then computational efficiency is improved, but measurement precision of overall realism deteriorates

Engineering Contradiction:

Improveevaluation computational efficiencyVSAvoidoverall simulator realism accuracy

Core Design Contradiction:

ProductivityVSMeasurement precision

Solution Approach 1:

The patent merges multiple individually evaluated low-level metrics (collisions, road departures, agent trajectories) into a unified high-level realism metric. Each low-level metric is computed efficiently separately, then combined through a weighted aggregation process that preserves the overall realism assessment accuracy while maintaining computational efficiency of individual components.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent applies partial action by selectively combining only the most critical low-level metrics into the overall realism assessment, rather than attempting to aggregate all possible properties. This selective combination maintains computational efficiency while ensuring that the most important aspects of simulator realism are accurately measured.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20240311269A1Simulator metrics for autonomous driving

Publication Date: 2024.09.19 WAYMO LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for selecting simulators for evaluating control software for autonomous vehicles. In one aspect, a system comprises: receiving data specifying a driving scenario in an environment; receiving an actual value of a low-level statistic measuring a corresponding property of the driving scenario; generating simulations of the driving scenario using a simulator; determining, for each simulation, a respective predicted value of the low-level statistic that measures the corresponding property of the simulation; determining, from the respective predicted values for the simulations, a likelihood assigned to the actual value of the low-level statistic by the simulations; and determining, from the likelihood, a low-level metric for the simulator and for the driving scenario that measures a realism of the simulator with respect to the corresponding property of the driving scenario.