AV Model Evaluation Using Sliced Environmental Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in effectively evaluating and selecting machine learning models for autonomous vehicle operations due to variability in model performance across different environmental conditions, making it difficult to identify the best-performing models for controlling AV behaviors.
Innovation Solution
A model evaluation platform that slices datasets into sub-datasets based on critical environmental features, evaluates AV control models using these sub-datasets, and selects models based on performance and confidence calibration to ensure optimal control model deployment in autonomous vehicles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are evaluated using complete datasets without slicing, then the evaluation covers all environmental conditions, but the performance variability across different conditions makes it difficult to identify the best-performing models for specific conditions
Solution Approach 1:
The patent divides the complete dataset into multiple sub-datasets based on environmental conditions (e.g., weather, location, time of day). Each sub-dataset is used to evaluate model performance under specific conditions, enabling precise identification of best-performing models for particular environmental contexts rather than averaging performance across all conditions.
2Reliability
If multiple machine learning models are trained and evaluated to find the best-performing model, then model performance improves, but the computational resources and time required for training and evaluation increase
Solution Approach 1:
The patent performs preliminary slicing of the dataset into environmental condition-based sub-datasets before model evaluation. This preprocessing step allows for targeted evaluation of multiple models under specific conditions, reducing the overall evaluation time by focusing computational resources on relevant environmental contexts rather than evaluating all models against the entire dataset.
3Ease of operation
If machine learning models are selected based on overall performance metrics, then the selection process is simple, but the models may not perform optimally under specific environmental conditions
Solution Approach 1:
The patent applies local quality by selecting different machine learning models optimized for specific environmental conditions. Instead of choosing a single model based on overall average performance, the system identifies and selects the best-performing model for each environmental condition (e.g., different models for rainy vs. sunny conditions), thereby improving adaptability while maintaining operational simplicity through automated selection.
Data Source
AI summary
Machine learning models for controlling AV operations may be evaluated based on sliced data. A feature category critical for an environmental condition may be identified. The environmental condition is a condition in an environment where an AV performs an operation. A sub-dataset, which comprises sensor data capturing the environmental feature dataset, may be extracted from a dataset that comprises sensor data capturing the environment. The sub-dataset may be input into machine learning models that can classify the feature category. Performances of the machine learning models can be evaluated based on their outputs, which are generated based on the sub-dataset, and a ground-truth classification of the environmental feature. The output of each machine learning model comprises a classification of the environmental feature. A machine learning model may be selected based on the evaluated performances of the machine learning models and may be used to control driverless operations of AVs.