Root Cause Analysis for Flaky Software Tests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Non-deterministic software tests, which intermittently pass or fail without code changes, hinder effective codebase evolution and debugging due to false success results and unpredictability, making it difficult to identify the root cause of flakiness.
Innovation Solution
A system and method that involves call trace collection during test runs, construction of dynamic call trees for passing and failing traces, comparison to identify unique subtree patterns, and ranking these patterns to determine the root cause of non-deterministic tests, optionally including class renaming, initialization separation, and noise reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If non-deterministic tests are executed to validate code changes, then code functionality can be verified, but false success results occur and debugging becomes difficult
Solution Approach 1:
The system performs preliminary actions by collecting call traces during test executions before analyzing them. It proactively gathers execution data from both passing and failing test runs, then processes this data to identify root causes of non-determinism, preventing the loss of debugging information before it disappears.
Solution Approach 2:
The system introduces call trace data as an intermediary element between test execution and debugging. By capturing and analyzing call traces, the system creates a mediator that preserves execution information, allowing developers to understand why non-deterministic tests fail without losing critical debugging information.
2Difficulty of detecting and measuring
If call trace collection is enabled to record execution details, then root cause analysis becomes possible, but system complexity and overhead increase
Solution Approach 1:
The system segments the complex task of root cause analysis into manageable components: collecting call traces during execution, constructing call trees from traces, comparing passing and failing call trees, and identifying divergent subtrees. This segmentation makes the overall system more manageable and less complex.
Solution Approach 2:
The system extracts only the necessary information from test executions by focusing on call trace data and constructing call trees. It takes out and analyzes only the relevant portions of execution data needed for root cause identification, avoiding the complexity of processing entire test execution environments.
3Reliability
If multiple test runs are executed to identify non-deterministic behavior, then flakiness can be detected, but execution time increases
Solution Approach 1:
The system performs preliminary action by collecting call traces during each test run execution. By proactively gathering this data during normal test execution rather than requiring separate analysis runs, it detects non-determinism without significant additional time overhead.
Solution Approach 2:
The system maintains continuity of useful action by collecting call traces during every test execution. This continuous data collection allows the system to detect non-deterministic behavior across multiple runs efficiently, maintaining test consistency while minimizing idle time between executions.
Data Source
AI summary
A system and method are disclosed for determining the root cause of non-deterministic tests. An exemplary system may receive a non-deterministic test and run the non-deterministic test with call trace collection enabled. For each trace, the system may determine whether the trace was associated with a passing or failing test. Each trace may be labeled according to the trace's association. Then, dynamic call trees may be constructed, one for the traces associated with passing tests and one for traces associated with failing tests. The dynamic call trees may be compared to determine subtree patterns that occur in one tree, but not the other. The subtree patterns may then be ranked based on the number of times the subtrees occur in the traces in order to determine the root cause of the non-deterministic test.