False Alarm Detection via Test Step Failure Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
False test alarms during system and integration testing disrupt the development process by requiring manual investigation and delaying code integration, despite providing no insights into product quality, and existing methods lack efficient automated solutions to differentiate between code defects and infrastructure issues.
Innovation Solution
A system utilizing association rule mining to identify patterns in test step failures, creating association rules that classify test failures as false alarms, and a comparator to automatically distinguish between true test failures and false alarms, thereby reducing manual inspection and speeding up development.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If system and integration tests are run to detect code defects, then product quality is improved, but false test alarms increase requiring manual investigation
Solution Approach 1:
The patent introduces an intermediary classification system that acts as a mediator between test execution and manual investigation. The system analyzes test failure characteristics and compares them against historical false alarm patterns to automatically classify failures as either 'likely false alarm' or 'requires investigation'. This intermediary classification layer filters out false alarms before they reach developers, reducing manual investigation time while maintaining product quality assurance.
2Reliability
If all test failures are investigated manually, then code defects are detected, but development speed decreases
Solution Approach 1:
The patent applies partial action by selectively investigating only those test failures that are classified as 'requires investigation' rather than manually examining all failures. The system performs full manual investigation only when necessary (when classification indicates potential code defect), while accepting partial automated classification for false alarm cases. This partial investigation approach maintains code defect detection capability while significantly improving development speed by avoiding unnecessary manual reviews.
3Loss of time
If automated classification of test failures is implemented, then manual investigation time is reduced, but false positives may occur
Solution Approach 1:
The patent implements feedback mechanisms where test failure classification results are continuously refined based on historical data and outcomes. The system learns from past classifications and actual investigation results, adjusting its classification criteria to improve accuracy over time. This feedback loop reduces false positives by continuously improving the classification algorithm's ability to distinguish between false alarms and genuine code defects, thereby maintaining low manual investigation time while improving classification precision.
Data Source
AI summary
Identifying false test alarms to a developer. A code build is executed in a test system that includes computing functionality and computing infrastructure that is able to execute the build. Executing the code build includes running a plurality of system and integration tests on the code build. As a result of executing the code build, a system and integration test failure is identified. One or more characteristics of the system and integration test failure are identified. The characteristics of the system and integration test failure are compared to characteristics of a set of historical previous known false test alarms. False test alarms are failures caused by a factor other than a factor for which a test is being run. Based on the act of comparing, information is provided to a developer with respect to if the system and integration test failure is potentially a false test alarm.