Automated Root Cause Analysis for Non-Deterministic Test Failures

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Concurrent applications are difficult to test due to their non-deterministic nature, leading to unpredictable outcomes and requiring extensive manual intervention to identify root causes of failures, as traditional troubleshooting methods are inefficient and labor-intensive.

Innovation Solution

An automated root cause analysis system that reruns non-deterministic tests, captures data during reruns, and analyzes it to determine the root cause of failures by comparing passed and failed executions, using instrumentation code to monitor and record execution flows and environment data.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If traditional manual troubleshooting methods are used for non-deterministic tests, then developers can identify errors through manual log review, but the process requires extensive developer involvement and significant manual intervention

Engineering Contradiction:

Improveerror identification accuracyVSAvoidmanual intervention effort

Core Design Contradiction:

Measurement precisionVSEase of operation

Solution Approach 1:

The system performs self-service by automatically analyzing test execution data, comparing passed and failed test runs, and identifying potential root causes without requiring developer intervention. The automated root cause analysis engine processes execution flows, thread data, and test results to generate diagnostic information independently.

Inventive Principle:

Principle #25Self-service

Solution Approach 2:

The patent replaces the mechanical manual review process with an automated computational system. Instead of developers manually examining logs and execution traces, an automated engine uses data processing and pattern recognition to identify root causes, substituting human analytical work with machine-based analysis.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If non-deterministic tests are executed multiple times to identify failures, then more data is available for analysis, but the execution time and resources increase significantly

Engineering Contradiction:

Improvetest result reliabilityVSAvoidtest execution time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary action by capturing and storing execution data during test runs, including thread execution flows, variable states, and environment information. This pre-captured data is then efficiently reused during root cause analysis, avoiding the need to re-execute tests solely for data collection purposes.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent creates copies of execution data from multiple test runs, storing execution flows, thread states, and test results in a structured format. These data copies are then analyzed to identify patterns and root causes, allowing the system to derive insights from multiple executions without requiring continuous re-running of tests.

Inventive Principle:

Principle #26Copying

3Loss of information

If extensive data is collected during test reruns to analyze execution paths, then comprehensive information is available for root cause identification, but the data processing complexity increases

Engineering Contradiction:

Improveexecution data completenessVSAvoiddata analysis complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The system extracts only the relevant and necessary data elements from the complete execution data set for analysis. The root cause analysis engine identifies and extracts key execution flows, thread interactions, and state changes that are most indicative of test failures, filtering out redundant information to simplify the analysis process.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent segments the complex execution data into organized components such as individual thread execution flows, variable state changes, and temporal event sequences. This segmentation allows the analysis system to process data in manageable units, comparing specific segments between passed and failed tests to identify root causes systematically.

Inventive Principle:

Principle #1Segmentation

4Productivity

If automated root cause analysis is implemented, then manual intervention is reduced and efficiency improves, but the system complexity and instrumentation requirements increase

Engineering Contradiction:

Improvetroubleshooting efficiencyVSAvoidsystem instrumentation complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The system implements a multi-functional automated root cause analysis engine that performs multiple tasks: collecting execution data, comparing test results, identifying patterns, and generating diagnostic reports. This universal engine handles various aspects of test analysis through a single integrated system, reducing the need for multiple separate tools and reducing overall system complexity.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS10474565B2Root cause analysis of non-deterministic tests

Publication Date: 2019.11.12 MICRO FOCUS LLC

AI summary

Example implementations relate to root cause analysis of non-deterministic tests. In response to detecting a non-deterministic test of an application under test, some implementations may cause rerun executions of the non-deterministic test. Some implementations may also capture data during the rerun executions. The data may include application data, test data, and environment data. Some implementations may also analyze the captured data to determine a root cause of a failure of the non-deterministic test. The analysis may be based on a comparison between failed rerun executions and passed rerun executions.