Autonomous Vehicle Software Regression Detection via Simulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicle control software performance regressions are difficult to detect, as changes can cause inadvertent system drains, and existing methods lack reliability in identifying the affected functions.
Innovation Solution
A method involving simulations to track CPU and memory usage of software functions, comparing these metrics across different versions, and flagging anomalies based on standard deviation thresholds to detect performance regressions, ensuring the simulations mimic real-world vehicle conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If simulations are run to detect performance regressions, then reliability of detection is improved, but time consumption increases
Solution Approach 1:
The system performs preliminary actions by running multiple baseline simulations with different software versions before comparing the test simulation. This establishes a reference distribution of CPU time values that enables reliable anomaly detection while limiting the time required for the actual regression detection process.
Solution Approach 2:
The system uses a threshold-based approach where only simulations exceeding a predetermined number of standard deviations from the mean are flagged as anomalies. This partial action principle allows the system to process multiple simulations efficiently by focusing attention only on the extreme cases that likely represent genuine performance regressions.
2Measurement precision
If CPU time sampling is performed for each function, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The system segments the CPU time measurement process by function, sampling and recording CPU time for each individual function separately. This segmentation enables precise measurement of which specific functions are causing performance regressions, rather than only measuring overall system performance.
Solution Approach 2:
The system introduces an intermediary statistical analysis layer that compares sampled CPU time values against a predetermined distribution and calculates standard deviations. This intermediary processing step translates raw measurement data into actionable anomaly detections without requiring direct complex intervention in the software execution.
3Measurement precision
If multiple simulations are run for comparison, then detection accuracy is improved, but productivity decreases
Solution Approach 1:
The system implements self-service through automated anomaly detection that compares simulation results against predetermined thresholds and standard deviations. This automation eliminates the need for manual analysis of each simulation result, allowing multiple simulations to be processed efficiently while maintaining high detection accuracy.
Solution Approach 2:
The system uses feedback from baseline simulations to dynamically establish what constitutes normal performance variation. By continuously comparing new simulations against this feedback-based baseline and flagging only significant deviations, the system achieves high accuracy without requiring an excessive number of simulations.
Data Source
AI summary
The disclosure relate to detecting performance regressions in software used to control autonomous vehicles. For instance, a simulation may be run using a first version of the software. While the simulation is running, CPU and memory usage by one or more functions of the first version of the software may be sampled. The sampled CPU and memory usage may be compared to CPU or memory usage by each of the one or more functions in a plurality of simulations each running a corresponding second version of the software. Based on the comparisons, an anomaly corresponding to a performance regression in the first version of the software relating to one of the one or more functions may be identified. In response to detecting the anomaly, the first version of the software and the one of the one or more functions may be flagged for review.