Autonomous Vehicle Evaluation Using Weighted Disengagement Categories
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional approaches to measuring autonomous vehicle performance, such as miles per intervention (MPI), are inadequate as they fail to accurately reflect safety and encourage undesirable behaviors like minimizing disengagements, leading to potential safety risks and gaming of the system.
Innovation Solution
The development of improved performance metrics that categorize disengagements based on potential outcomes, allowing for the exclusion of disengagements that would not have resulted in negative events, and the use of weighted road segment and scenario calibrations to provide a more accurate assessment of autonomous vehicle performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional performance metrics like miles per intervention (MPI) are used to measure autonomous vehicle performance, then the measurement is simple and easy to calculate, but it fails to accurately reflect safety and encourages undesirable behaviors like minimizing disengagements
Solution Approach 1:
The patent segments disengagements into different categories (planned vs. unplanned, and further subdivisions of unplanned disengagements based on potential outcomes). This segmentation allows the performance metric to distinguish between disengagements that indicate safety issues and those that are operational necessities, thereby improving measurement precision without requiring an overly complex system. Each category is assigned different weights in the overall performance calculation.
Solution Approach 2:
The patent changes the parameters used to measure performance from a simple count of disengagements to a multi-dimensional classification system that considers the context, cause, and potential outcome of each disengagement. This parameter change transforms the metric from a crude safety indicator to a nuanced performance evaluation tool that accurately reflects autonomous vehicle safety while maintaining computational feasibility.
2Reliability
If all disengagements are counted equally in performance metrics, then the calculation is straightforward, but it penalizes frequent disengagements that may be necessary for safety and encourages gaming of the system
Solution Approach 1:
The patent applies preliminary classification to each disengagement event, categorizing it as planned or unplanned before incorporating it into the performance metric. For unplanned disengagements, further preliminary analysis determines the potential outcome (negative event, potential negative event, or no negative event). This preliminary action ensures reliable safety assessment by distinguishing between meaningful safety indicators and operational necessities, while the structured classification process maintains ease of calculation through systematic procedures.
Solution Approach 2:
The patent implements a feedback mechanism where disengagement data is continuously collected, categorized, and used to update performance metrics. The system provides feedback on autonomous vehicle performance through these categorized metrics, which in turn informs operational decisions and system improvements. This feedback loop enhances reliability by continuously refining the safety assessment while maintaining operational simplicity through automated data processing.
3Measurement precision
If performance metrics do not account for different road segments and scenarios, then the metric system is simple, but it cannot provide accurate comparisons across different operating conditions
Solution Approach 1:
The patent applies local quality by creating specific performance calibrations for different road segments and operational scenarios. Each road segment or scenario type (e.g., urban environments, highways, adverse weather conditions) has its own calibration factors that account for the unique challenges and disengagement patterns associated with that location or condition. This allows accurate performance comparisons by evaluating autonomous vehicle behavior in the context of local operating conditions rather than using a single aggregate metric.
Solution Approach 2:
The patent adds dimensional complexity to the performance metric system by incorporating spatial (road segments) and contextual (scenarios) dimensions. Instead of a single-dimensional measure of disengagements per mile, the system creates a multi-dimensional performance landscape that evaluates safety across geography, environment, and operational context. This dimensional expansion enables precise performance comparison across different conditions while the modular calibration approach manages the associated complexity.
Data Source
AI summary
Systems, methods, and non-transitory computer-readable media can determine a first utility metric associated with a region and first autonomous vehicle eligibility criteria, wherein the first utility metric is determined based on a first plurality of rides and a subset of the first plurality of rides that can be successfully executed within the region based on the first autonomous vehicle eligibility criteria. A second utility metric associated with the region and second autonomous vehicle eligibility criteria can be determined, wherein the second utility metric is determined based on a second plurality of rides and a subset of the second plurality of rides that can be successfully executed within the region based on the second autonomous vehicle eligibility criteria. An autonomous vehicle associated with the first autonomous vehicle eligibility criteria can be selected to drive in the region based on a comparison of the first utility metric and the second utility metric.