Autonomous Vehicle Test Parameter Validation via Simulation Noise Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Autonomous vehicle (AV) test parameter configuration changes need to be validated to ensure they do not introduce noise into the test suite, affecting the repeatability of simulation outputs, which is crucial for ensuring safety and performance in real-world settings.

Innovation Solution

The system runs multiple simulations with both pre-updated and post-updated test parameter configurations, comparing the resulting distributions using statistical characteristics and a Kolmogorov-Smirnov test to determine if the configuration change has introduced noise, thereby validating the update by assessing the similarity in simulation outputs across scenarios.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If configuration changes are made to test parameters, then the test suite can be updated to reflect new requirements or conditions, but the repeatability of simulation outputs may be affected due to introduced noise

Engineering Contradiction:

Improvetest parameter configuration flexibilityVSAvoidsimulation output repeatability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system performs preliminary validation by running simulations with the updated configuration before fully deploying the change. It compares the distribution of simulation outputs against a baseline to predict whether the configuration change will introduce unacceptable noise, allowing corrective action to be taken beforehand

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system implements feedback by continuously monitoring the distribution of simulation outputs and comparing them against predefined thresholds. When the noise level exceeds the threshold, the system provides feedback to alert users and allows for configuration adjustment, ensuring that only validated changes are deployed

Inventive Principle:

Principle #23Feedback

2Reliability

If multiple simulations are run with updated configurations to validate changes, then the reliability of test results is improved, but the time required for validation increases

Engineering Contradiction:

Improvetest result validityVSAvoidvalidation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system changes parameters by adjusting the number of simulations run based on the complexity of the configuration change and the desired confidence level. It dynamically sets validation thresholds and selects key statistical metrics to compare, reducing the number of required simulations while maintaining adequate reliability

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The system applies partial action by running a subset of simulations focused on critical test scenarios rather than exhaustively testing all possible conditions. It prioritizes validation of high-impact parameters and uses statistical sampling to infer results from representative cases, reducing total validation time while maintaining reliability

Inventive Principle:

Principle #16Partial or excessive action

3Measurement precision

If statistical comparisons are performed to detect noise in simulation outputs, then the precision of noise detection is improved, but the complexity of the validation process increases

Engineering Contradiction:

Improvenoise detection accuracyVSAvoidvalidation process complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The system extracts and focuses on key statistical characteristics (such as mean, standard deviation, and percentile values) from the simulation outputs rather than performing comprehensive complex analysis on all data. It isolates the most informative metrics that indicate noise levels, simplifying the validation process while maintaining detection precision

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system uses simple, pre-defined statistical thresholds and comparison metrics that are computationally inexpensive to calculate and interpret. Rather than employing complex machine learning models or sophisticated analysis algorithms, it relies on straightforward statistical tests that can be quickly executed and understood

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS20240219264A1Validation of a configuration change of autonomous vehicle test parameters

Publication Date: 2024.07.04 GM CRUISE HOLDINGS LLC

AI summary

The disclosed technology provides solutions for validating a configuration change of autonomous vehicle (AV) test parameter(s) by determining an impact of the configuration change on a simulation test suite. A process of the disclosed technology can include steps for receiving a first configuration of a test parameter associated with testing an AV, receiving a set of simulation scenarios from a test suite, running a predetermined number of simulations of the AV using the set of simulation scenarios, determining a first distribution with respect to repeatability based on simulation output, updating a configuration of the test parameter to generate a second configuration, running the predetermined number of simulations, determining a second distribution with respect to repeatability based on simulation output, and validating the configuration update based on a comparison between the first distribution and the second distribution. Systems and machine-readable media are also provided.