Prediction Error Scenario Mining for Autonomous Vehicle ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional simulators fail to effectively test and identify error-prone scenarios for autonomous vehicles, leading to inadequate training of machine learning models, which can result in unsafe navigation in rare and edge-case situations.
Innovation Solution
A prediction error scenario mining framework that searches a scenario database using SQL queries to identify and mine error-prone scenarios based on prediction errors, allowing for dynamic and comprehensive training of machine learning models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional simulators are used to test autonomous vehicle scenarios, then the testing process is simple and fast, but the ability to identify and test error-prone scenarios is insufficient
Solution Approach 1:
The system uses prediction errors from machine learning models as feedback to automatically query and retrieve error-prone scenarios from a database. This feedback loop enables the system to iteratively improve scenario selection based on actual model performance, transforming conventional simulators into adaptive testing platforms that focus on problematic scenarios.
Solution Approach 2:
A scenario database serves as an intermediary between the machine learning model and the testing process. The database stores historical scenarios and prediction errors, acting as a mediator that enables systematic retrieval and analysis of error-prone situations without requiring complex real-time simulation infrastructure.
2Adaptability or versatility
If all driving scenarios are tested in real-world environments, then comprehensive coverage is achieved, but the testing process becomes dangerous and unfeasible
Solution Approach 1:
The system creates virtual copies of real-world driving scenarios stored in a database, allowing comprehensive scenario testing without physical risk. These digital replicas enable repeated analysis of edge cases and error-prone situations while maintaining safety, as no actual vehicles or pedestrians are involved in the testing process.
Solution Approach 2:
Scenarios are pre-identified and stored in a database before actual testing occurs. By using prediction errors to query and retrieve relevant scenarios in advance, the system prepares comprehensive test cases beforehand, enabling thorough coverage of driving scenarios while avoiding dangerous real-world experimentation.
3Productivity
If machine learning models are trained on conventional simulator data, then training is efficient, but the models fail to handle rare and edge-case situations
Solution Approach 1:
Instead of uniformly training on all scenarios, the system identifies and focuses specifically on error-prone and rare scenarios using prediction error analysis. This local quality approach ensures that training resources are concentrated on the most critical scenarios that need improvement, rather than treating all scenarios equally.
Solution Approach 2:
Prediction errors from machine learning models during initial training serve as feedback to identify which scenarios caused the most errors. These error-prone scenarios are then retrieved from the database and used to refine and retrain the model, creating an iterative process that continuously improves performance on rare and edge-case situations.
Data Source
AI summary
Provided are methods for prediction error scenario mining for machine learning methods, which can include determining a prediction error indicative of a difference between a planned decision of an autonomous vehicle and an ideal decision of the autonomous vehicle. The prediction error is associated with an error-prone scenario for which a machine learning model of an autonomous vehicle is to make planned movements. The method includes searching a scenario database for the error-prone scenario based on the prediction error. The scenario database includes a plurality of datasets representative of data received from an autonomous vehicle sensor system in which the plurality of datasets is marked with at least one attribute of the set of attributes. The method further includes obtaining the error-prone scenario from the scenario database for inputting into the machine learning model for training the machine learning model. Systems and computer program products are also provided.