Test Case Failure Root Cause Isolation via State Similarity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Software testing environments face challenges in identifying the root cause of test case failures, particularly in distinguishing between actual software bugs and false positives, due to the complexity of large software products and varying system states during testing.
Innovation Solution
A method and system that execute a test case twice, capturing system states and results both times, and analyze the similarity score to categorize results as false positives based on state similarity and result differences, helping to determine if the test case or system changes caused the unexpected result.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If test cases are executed multiple times to determine root cause, then measurement precision of test results is improved, but loss of time increases
Solution Approach 1:
The system captures system states at the beginning of test execution before the failure occurs. This preliminary capture of state information allows for later analysis without requiring extensive re-execution time, as the baseline state is already recorded for comparison
Solution Approach 2:
The system creates copies of system states (snapshots) during test execution. These state copies are stored and can be analyzed without re-executing the entire test case, thus maintaining measurement precision while reducing the time penalty of multiple executions
2Reliability
If system state analysis is performed to distinguish false positives from actual bugs, then reliability of test results is improved, but device complexity increases
Solution Approach 1:
The system extracts and isolates specific state information relevant to the failure from the overall system state. By focusing analysis on extracted relevant features rather than the entire system state, the complexity of the analysis system is reduced while maintaining reliability
Solution Approach 2:
The system applies different analysis methods to different aspects of the system state. Rather than uniformly analyzing all system parameters, it focuses computational resources on local state features that are most relevant to distinguishing false positives from actual bugs
3Manufacturing precision
If the number of test cases is increased to cover larger software products, then manufacturing precision of software quality is improved, but difficulty of detecting and measuring increases
Solution Approach 1:
The system segments the analysis process into distinct phases: state capture during execution, state comparison after execution, and failure classification. This segmentation makes the overall complex task of root cause detection more manageable and systematic
Solution Approach 2:
The system introduces state similarity scoring as an intermediary mechanism between test execution and failure analysis. This intermediary metric automatically compares system states and provides objective evidence to guide root cause detection, reducing the difficulty of analyzing large numbers of test cases
Data Source
AI summary
Aspects of the invention include executing a first test case on a system, capturing a first state of the system during execution of the first test case, determining a first result of the first test case, performing a second execution of the first test case on the system in response to the first result of the first test case being an unexpected result, capturing a second state of the system during the second execution of the first test case, determining a second result of the first test case, analyzing the first state and the second state to determine a state similarity score, and categorizing the first result as a false positive based at least in part on the state similarity score being above a first threshold and a determination that the second result is a different result than the first result.