Modular Autonomous Driving Evaluation Framework

Solution Overview

Problem

Developers and engineers face the cumbersome and time-consuming task of writing evaluation scripts from scratch for various driving scenarios in autonomous driving, with conventional evaluation approaches being too rigid and non-modular, limiting their ability to efficiently evaluate vehicle behavior across different components of the autonomy stack.

Innovation Solution

An evaluation framework is introduced that utilizes modular and configurable calculators and validators, allowing for flexible metric calculation and validation, enabling code reuse and decoupling metric calculation from validation, thus facilitating quick and modular evaluations across different scenarios and releases of the autonomy stack.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If conventional evaluation approaches are used, then evaluation can be performed, but the process is cumbersome and time-consuming requiring scripts to be written from scratch for each scenario

Engineering Contradiction:

Improveevaluation efficiencyVSAvoidtime to write evaluation scripts

Core Design Contradiction:

ProductivityVSLoss of time

Solution Approach 1:

The evaluation system is divided into independent, modular components including metric calculators, validators, and configuration files. Each component handles a specific aspect of evaluation (e.g., distance calculation, speed measurement, scenario validation), allowing developers to reuse these modules across different scenarios rather than writing complete scripts from scratch.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The framework creates universal evaluation components that can be applied across multiple driving scenarios and autonomy stack versions. The metric calculators and validators are designed to be scenario-agnostic, working with various data formats and evaluation criteria through configuration files, thereby reducing the need to create new evaluation scripts for each scenario.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Adaptability or versatility

If conventional evaluation approaches are used, then evaluation can be performed, but the system is rigid and non-modular limiting flexibility

Engineering Contradiction:

Improveflexibility in evaluationVSAvoidsystem modularity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The evaluation system is divided into independent, modular components including metric calculators, validators, and configuration files. Each component handles a specific aspect of evaluation (e.g., distance calculation, speed measurement, scenario validation), allowing developers to reuse these modules across different scenarios rather than writing complete scripts from scratch.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The framework enables dynamic configuration of evaluation parameters and scenarios through configuration files rather than hard-coded logic. This allows the system to adapt to different evaluation needs by simply changing configuration parameters, making the system both modular and flexible without increasing structural complexity.

Inventive Principle:

Principle #15Dynamics

3Reliability

If individual evaluation scripts are written for each scenario, then specific evaluation requirements can be met, but the burden on developers increases significantly

Engineering Contradiction:

Improveevaluation accuracyVSAvoidnumber of evaluation scripts

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The framework creates universal evaluation components that can be applied across multiple driving scenarios and autonomy stack versions. The metric calculators and validators are designed to be scenario-agnostic, working with various data formats and evaluation criteria through configuration files, thereby reducing the need to create new evaluation scripts for each scenario.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

Instead of creating unique evaluation scripts for each scenario, the framework uses template-based configuration files that can be copied and adapted for different scenarios. The same core evaluation logic is reused across multiple scenarios by simply changing the configuration parameters, reducing the number of scripts needed while maintaining scenario-specific accuracy.

Inventive Principle:

Principle #26Copying

Data Source

PatentUS11853292B2Evaluating driving data with a modular and configurable evaluation framework

Publication Date: 2023.12.26 WOVEN BY TOYOTA U S INC

AI summary

In one embodiment, a method includes receiving a request for evaluating driving data included in a data log, accessing, based on the request, an evaluation configuration file that includes a metric-calculation configuration specifying one or more metric calculators configured to generate one or more output metrics from the driving data and a validation configuration configured to validate the one or more output metrics, instantiating the one or more metric calculators specified by the metric-calculation configuration included in the evaluation configuration file, determining particular driving data from the driving data included in the data log based on the one or more instantiated metric calculators, and generating the one or more output metrics for the particular driving data by using the instantiated one or more metric calculators.