Autonomous Vehicle Test Relevance Using Dynamic Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicle (AV) tests become obsolete over time due to changes in AV software and environmental conditions, making it difficult to determine which tests remain valid and which need updating.
Innovation Solution
Utilize pre-defined test metrics to evaluate AV performance against predefined intent paths, and employ machine-learning models to identify outdated tests that can be updated by modifying evaluation windows.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If AV tests are continuously updated to reflect new software and environmental conditions, then test relevance and reliability improve, but the complexity and resources required for maintenance increase
Solution Approach 1:
The patent implements dynamic test metric selection that adapts to different AV software versions and environmental conditions. The system automatically determines which test metrics are relevant based on current AV configurations, allowing tests to remain reliable without manual updates to all test parameters. This dynamic adaptation resolves the contradiction by maintaining test relevance through automation rather than complex manual maintenance.
Solution Approach 2:
The system performs self-service by automatically identifying outdated tests and generating updated test versions without requiring extensive human intervention. The automated test maintenance process evaluates test results, identifies obsolete metrics, and updates tests autonomously, reducing the complexity burden while maintaining high test relevance through continuous self-updating.
2Measurement precision
If comprehensive test metrics are collected to ensure thorough AV evaluation, then measurement precision improves, but the quantity of data and processing requirements increase
Solution Approach 1:
The patent extracts only the necessary test metrics from the complete set of available metrics based on current AV software versions and test objectives. By selectively removing irrelevant metrics while retaining essential ones, the system maintains high measurement precision for critical performance aspects while reducing the overall data quantity and processing burden associated with comprehensive metric collection.
3Manufacturing precision
If manual review and updating of each AV test is performed, then test accuracy is maintained, but the time and labor resources required increase significantly
Solution Approach 1:
The system implements automated feedback loops that continuously monitor test results and AV software changes. When discrepancies or obsolescence are detected, the system automatically triggers test updates and validates them against accuracy criteria. This feedback-driven automation maintains high test accuracy comparable to manual review while dramatically reducing the time and labor resources required by eliminating repetitive manual checking and updating processes.
Data Source
AI summary
The disclosed technology provides solutions for maintaining autonomous vehicle (AV) tests and, provides methods for evaluating test relevance and for determining how to fix outdated AV tests. In some aspects, a process of the disclosed technology includes steps for associating a set of test metrics with an AV test, monitoring operation of an AV to identify one or more behaviors performed by the AV in the simulated environment and determining a validity of the AV test with respect to the simulated environment based on the AV behaviors and the set of test metrics. Systems and machine-readable media are also provided.