Software Build Performance Regression Testing with Noise Confidence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software performance regression testing is hindered by noise in test results, leading to inconsistent and unreliable identification of problematic builds, as existing methods fail to accurately determine the build causing performance degradation due to inaccuracies in measurement and insufficient test time.
Innovation Solution
A method and system that calculate performance metrics for each build, determine differences between consecutive builds, identify the largest performance drop, and assess its confidence level to reliably pinpoint the problematic build by executing additional performance tests if the confidence level is below a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the bisect method is used to identify problematic builds, then the testing process can be divided into manageable steps, but the identification becomes unreliable when test results contain noise
Solution Approach 1:
The patent performs preliminary actions by calculating performance metrics for all builds before attempting to identify the problematic one. It pre-computes confidence levels for each build based on test result consistency, allowing the system to anticipate which builds are likely problematic rather than relying solely on sequential bisect testing. This preliminary analysis of test result quality enables more reliable identification even when noise is present.
Solution Approach 2:
The patent introduces feedback mechanisms by continuously evaluating the confidence level associated with each build's performance metric. When test results show inconsistency or noise, the system uses this feedback to adjust its identification process, potentially performing additional tests on specific builds to increase confidence before making a final determination. This feedback loop improves reliability by adapting to the quality of test results.
2Loss of time
If performance tests are executed with insufficient test time, then testing resources are conserved, but measurement accuracy deteriorates due to noise
Solution Approach 1:
The patent applies partial action by selectively performing additional performance tests only on builds that show suspicious performance drops with low confidence levels, rather than re-testing all builds. This approach consumes additional testing resources only where necessary to resolve uncertainty, balancing the trade-off between testing duration and measurement precision by focusing efforts on the most ambiguous cases.
Solution Approach 2:
The patent changes the parameter of test confidence by using statistical analysis to evaluate the reliability of performance measurements. When initial tests produce noisy or inconsistent results, the system adjusts the confidence parameter and determines whether additional tests are needed to achieve a satisfactory confidence level, thereby dynamically balancing testing time against measurement accuracy based on the observed data quality.
3Measurement precision
If comprehensive performance testing is conducted on all builds, then measurement precision is improved, but the complexity of analyzing results increases
Solution Approach 1:
The patent extracts and focuses attention on the most critical builds for analysis by calculating confidence levels for each build's performance metric. Instead of analyzing all test results equally, it identifies and extracts the builds with the lowest confidence levels or largest performance drops for further investigation. This extraction approach reduces analysis complexity by filtering out builds that are clearly acceptable or clearly problematic, concentrating analytical efforts only on the ambiguous cases.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of confidence level calculations that mediate between raw performance test results and final build identification. This intermediary layer processes the comprehensive test data, applying statistical analysis to evaluate result consistency and reliability. The confidence level metric serves as a mediator that simplifies the interpretation of comprehensive test results, transforming complex multi-dimensional test data into a single actionable indicator for each build.
Data Source
AI summary
A method and system includes calculating a performance metric for each of a plurality of builds of a software application in view of a respective performance test result associated with each of the plurality of builds, calculating a respective difference in performance metrics for each pair of consecutive builds of the plurality of builds, determining a largest performance drop in view of respective differences in the performance metrics among the pairs of consecutive builds of the plurality of builds, wherein the largest performance drop is associated with a first pair of consecutive builds comprising a first build and a second build, determining, by a processing device, a confidence level associated with the largest performance drop in view of performance test results associated with the first build and the second build, in response to determining that the confidence level is greater than or equal to a threshold, identifying one of the first build or the second build as a problematic build of the software application.