Robotic Simulation Search for Salient Safety Scenarios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current simulation-based testing methods for autonomous vehicles are inefficient and require a large number of simulations to achieve a desired safety level, as they often focus on worst-case failures rather than salient scenarios, and do not effectively handle undefined performance outcomes.
Innovation Solution
A directed search method using a hierarchical Gaussian process model to predict performance scores, incorporating 'Not a Number' (NaN) outcomes, to efficiently identify salient scenarios and improve the accuracy of performance evaluation in simulated environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a high-fidelity simulation is used to test autonomous vehicle safety, then the accuracy of safety evaluation is improved, but the computational resources and time required increase significantly
Solution Approach 1:
The patent applies preliminary action by using a scoring function to pre-evaluate scenario instances before full simulation execution. The scoring function computes a score based on scenario parameters to identify instances likely to cause failure, allowing the system to prioritize or skip full simulation of low-priority instances, thereby reducing overall simulation time while maintaining evaluation accuracy.
Solution Approach 2:
The patent implements self-service through iterative refinement where the system uses results from previous simulation iterations to improve the scoring function. The scoring function is retrained or adjusted based on accumulated data, enabling the system to automatically optimize its own evaluation process without external intervention, progressively improving efficiency while maintaining high-fidelity assessment.
2Reliability
If the number of simulation instances is increased to achieve a minimum safety level of 10^-9 severe accidents per hour, then the reliability of safety guarantee is improved, but the computational cost and time required increase exponentially
Solution Approach 1:
The patent applies local quality by focusing computational resources on specific regions of the parameter space that are most likely to reveal failures. The scoring function identifies local areas in the scenario parameter space where failure conditions are more probable, allowing the system to concentrate simulation instances in these critical regions rather than uniformly distributing them, thereby achieving high reliability with fewer total instances.
Solution Approach 2:
The patent utilizes parameter changes by varying scenario parameters systematically to explore different regions of the parameter space. The scoring function evaluates combinations of parameters (such as weather conditions, traffic density, road geometry) to identify configurations that maximize failure probability, enabling targeted exploration that achieves comprehensive safety validation more efficiently than exhaustive sampling.
3Productivity
If a directed search method using hierarchical Gaussian process model is used to identify salient scenarios, then the productivity of testing is improved, but the device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the complex evaluation task into two hierarchical levels: a scoring function that performs initial filtering and ranking of scenario instances, and a full simulation system that executes detailed testing on selected instances. This segmentation allows the complex Gaussian process model to operate only on a reduced subset of scenarios, maintaining high productivity while managing computational complexity through functional decomposition.
Solution Approach 2:
The patent introduces an intermediary scoring function that acts as a mediator between the scenario parameter space and the full simulation system. The scoring function, based on hierarchical Gaussian processes, transforms complex multi-parameter scenario descriptions into simplified scores that guide simulation selection, thereby reducing the complexity burden on the overall system while maintaining directed search capabilities.
Data Source
AI summary
A directed search method is applied to a parameter space of a scenario for testing the performance of a robotic system in simulation. The directed search method is applied based on at least one performance evaluation rule that returns a performance score that can be numerical or non-numerical. A hierarchical score prediction model is constructed as follows. A score classification model is trained to probabilistically predict whether a point in the parameter space will result in a numerical or non-numerical outcome. A score regression model is trained to probabilistically predict a performance score for a given point, given that the score is numerical. The score classification and regression models are used to guide a directed search of the parameter space towards the most salient instances of the scenario.