Root Cause Analysis for Software Test Failures

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In software development, identifying the root cause of program failures is challenging, especially in large and complex code bases maintained by distributed teams, as existing techniques focus on optimizing test suites rather than tracing failures to their source.

Innovation Solution

A method that inserts lightweight instrumentation hooks into the program to capture runtime behaviors, flattens program states and commands along a timeline, and uses backtrack analysis to detect causal correlations between variables, states, and commands, determining joint conditional probabilities to identify candidate root causes.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If traditional test failure analysis methods are used in large distributed code bases, then manual effort and time are required to trace failures, but the complexity and difficulty of identifying root causes increases significantly

Engineering Contradiction:

Improveroot cause identification accuracyVSAvoidcode base complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent segments the complex program execution into discrete states, commands, and tests that can be individually analyzed. By dividing the monolithic code base execution into state transitions with associated commands and test results, the system enables granular analysis of failure paths without being overwhelmed by the overall system complexity.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces an intermediary analysis layer that sits between the raw program execution and the root cause identification. This intermediary system captures execution states, commands, and test results, then processes them through statistical analysis to identify causal relationships, thereby mediating between the complex code base and the simplified root cause output.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Measurement precision

If comprehensive program analysis is performed to identify root causes, then accurate root cause identification is achieved, but the time and computational resources required increase

Engineering Contradiction:

Improvefailure cause detection accuracyVSAvoidanalysis time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary actions by capturing and storing program states, commands, and test results during normal execution before failure analysis is needed. This pre-capture of execution data eliminates the need for time-consuming re-instrumentation and re-execution when failures occur, significantly reducing analysis time while maintaining comprehensive analysis capability.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces manual mechanical analysis processes with automated statistical analysis. Instead of manually tracing through code and analyzing execution paths, the system uses automated algorithms to compute joint conditional probabilities and identify causal relationships, dramatically reducing the time and human effort required while improving consistency and accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Loss of information

If detailed instrumentation is added to capture runtime behaviors, then precise failure tracing is enabled, but the program complexity and overhead increase

Engineering Contradiction:

Improveruntime behavior information captureVSAvoidprogram instrumentation complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent creates a universal instrumentation framework that serves multiple functions: capturing program states, recording commands, tracking test results, and enabling statistical analysis. This multi-functional instrumentation system eliminates the need for separate specialized instrumentation for each analysis goal, reducing overall complexity while comprehensively capturing runtime behavior information.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS9946629B2System, method and apparatus for deriving root cause for software test failure

Publication Date: 2018.04.17 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

For a program of interest represented as a sequence of states comprising variables and flags wherein controls transition the program from state to state, and wherein the program includes tests at prescribed locations in the program that are applied upon specific states and generate observables, a method includes inserting at different points in the program hooks that capture program runtime behaviors constituting classification features. The method further includes flattening the program states, commands and tests along a program run-time timeline, and identifying a root cause of a program failure by backtrack analyzing a stream of input states and commands and output consequent states, commands and test results to detect causal correlations between variables, states and commands. The step of backtrack analyzing includes determining joint conditional probabilities and identifying candidate failure root causes that maximize conditional probabilities.