Automated Test Failure Clustering for Defect Attribution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated program code testing environments often fail to accurately distinguish between defects in the program code and defects in the automated test code, leading to inefficiencies and lower adoption rates due to the difficulty in measuring the effectiveness of automated testing, which can result in increased costs and resource wastage.
Innovation Solution
Implementing a method to winnow and cluster automated test failures, identifying those due to program code defects separately from those due to automated test code defects, and comparing the results to manual testing to assess the value and effectiveness of automated testing, thereby allowing for targeted modifications to improve testing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If automated testing is implemented to reduce manual testing efforts, then productivity is improved, but measurement precision deteriorates due to inability to accurately distinguish between program code defects and test code defects
Solution Approach 1:
The patent segments the defect identification process by creating separate clustering mechanisms for program code defects and test code defects. Automated test failures are divided into distinct groups based on their root causes, allowing independent analysis and measurement of each defect type's effectiveness separately
Solution Approach 2:
The patent introduces an intermediary analysis layer that sits between automated test execution and defect identification. This intermediary process clusters failures and determines whether they stem from program code or test code, providing accurate attribution without requiring manual intervention for each failure
2Loss of information
If automated testing effectiveness is difficult to measure, then loss of information increases regarding testing value, but implementing measurement mechanisms increases device complexity
Solution Approach 1:
The patent implements feedback mechanisms that automatically collect and analyze test failure data, clustering results, and defect attribution information. This feedback loop continuously measures testing effectiveness by comparing expected versus actual outcomes, providing actionable insights without manual assessment
Solution Approach 2:
The measurement system is designed to be self-service, automatically clustering failures and attributing defects without requiring external manual analysis. The system serves its own measurement needs by autonomously processing test data and generating effectiveness metrics
3Loss of time
If automated test failures are not properly analyzed, then loss of time increases due to unresolved defects, but implementing analysis processes increases device complexity
Solution Approach 1:
The patent applies preliminary action by clustering and categorizing test failures immediately upon occurrence, rather than waiting for manual analysis. Defects are pre-grouped and attributed to their source (program code or test code) before resolution begins, reducing the time required for subsequent debugging and fixing
Data Source
AI summary
Automated test failures that result from automated testing of program code under development are windows to include just the automated test failures occurring for a first time and that are due to automated test code defects or program code defects. The automated test failures that remain after winnowing are clustered into automated test failure clusters that each individually corresponding to a different automated test code defect or a different program code defect. The automated test failure clusters are window to include just the automated test failure clusters that each individually correspond to a different program code defect. The automated test failure clusters that remain after winnowing are output.