Software Version Progression Monitoring with Control Charts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing software testing methods fail to effectively detect anomalies and unusual patterns in high-complexity tests, particularly in autonomous vehicle simulations, due to the reliance on pass/fail percentages that mask important test results and the inefficiency of manual triaging large volumes of simulation test data.
Innovation Solution
Implementing a system that generates control charts from test reports to automatically identify anomalies by monitoring software characteristics over multiple versions, using univariate and multivariate control charts to detect outliers and trends, and generating alerts for anomalous behavior.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If pass/fail percentages are used to measure software quality, then the overall quality metric is simplified and easy to calculate, but important testing situations and results go undetected
Solution Approach 1:
The patent segments the overall test results into multiple test groups (e.g., functional tests, performance tests, security tests) and monitors each group separately. This allows detailed analysis of specific test areas while maintaining an overall quality view, preventing important results from being masked by aggregate percentages.
Solution Approach 2:
The patent introduces additional dimensions beyond simple pass/fail percentages, including test coverage metrics, severity levels, and trend analysis over time. This multi-dimensional approach provides richer information about software quality while maintaining ease of interpretation through visual dashboards.
2Measurement precision
If manual review of all test results is performed, then detailed analysis of each test case is possible, but the process becomes inefficient and time-consuming
Solution Approach 1:
The patent replaces manual review processes with automated anomaly detection algorithms that use statistical methods and machine learning to identify problematic test results. This automation maintains detailed analysis capabilities while dramatically improving testing efficiency and reducing human effort.
Solution Approach 2:
The system enables self-service anomaly detection by automatically monitoring test results, comparing them against historical data, and generating alerts for unusual patterns. This eliminates the need for manual triaging while providing continuous, detailed analysis of test outcomes.
3Productivity
If simple pass/fail percentages are used, then the testing process remains simple and quick, but anomalies and unusual patterns are not detected
Solution Approach 1:
The patent implements continuous feedback loops where test results are automatically compared against historical data and control limits. When anomalies are detected, the system generates alerts and can trigger automated responses, maintaining fast testing cycles while improving reliability through statistical process control.
Solution Approach 2:
The system establishes control charts and baseline metrics before testing begins, enabling proactive anomaly detection. By preparing detection thresholds and historical comparisons in advance, the system can quickly identify deviations without slowing down the testing process.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for detecting anomalies during software testing. The methods include generating a plurality of test reports for the software program by executing one or more test cases on a plurality of versions of the software program, generating a control chart based on the plurality of test reports, generating an alert when at least one testing characteristic includes an anomaly over the plurality of versions of the software program as determined based on the control chart. The control chart includes a plot associated with at least one testing characteristic of the software program, and a historical context associated with execution of the one or more test cases on the plurality of versions of the software program.