Test Case Priority Selection via Runtime Similarity Analysis

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In product development and testing, the challenge lies in efficiently identifying and prioritizing test cases within a large regression bucket to quickly detect and localize faults, as existing methods require extensive execution times, leading to inefficiencies and resource wastage.

Innovation Solution

A computer-implemented method for test case priority selection that identifies failing test cases, assigns priority values based on similarity, and reorganizes the execution order of test cases at runtime, using Combinatorial Test Design (CTD) techniques and inverse combinatorics to optimize the execution of test cases.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If all test cases in a large regression bucket are executed to ensure comprehensive fault detection, then testing completeness is improved, but execution time and resource consumption increase significantly

Engineering Contradiction:

Improvetesting completenessVSAvoidexecution time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-calculating and storing similarity metrics between test cases before execution. When a test case fails, the system can immediately retrieve and execute similar test cases without performing complex similarity calculations at runtime, thus reducing execution time while maintaining comprehensive fault detection

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent implements dynamics by adaptively adjusting the test case execution order based on runtime information. When a failure is detected, the system dynamically reorders the regression bucket to prioritize similar test cases, transforming the static execution sequence into a dynamic one that responds to actual test outcomes

Inventive Principle:

Principle #15Dynamics

2Ease of operation

If test cases are executed in a fixed order to simplify test management, then ease of operation is improved, but fault localization efficiency deteriorates due to inability to prioritize critical test cases

Engineering Contradiction:

Improvetest management simplicityVSAvoidfault localization efficiency

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The system pre-calculates similarity metrics and stores them in advance, so that when fault localization is needed, the test cases can be quickly reordered based on pre-computed data without complex real-time analysis, maintaining both simplicity and efficiency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent replaces the manual or fixed mechanical ordering of test cases with an automated system that uses similarity metrics and machine learning algorithms to dynamically determine execution order, substituting rigid mechanical processes with intelligent automated decision-making

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS11663113B2Real time fault localization using combinatorial test design techniques and test case priority selection

Publication Date: 2023.05.30 INTERNATIONAL BUSINESS MACHINE CORPORATION

AI summary

A computer-implemented method for test case priority selection includes initiating execution of a set of test cases associated with a system under test. Further, the set of test cases are executed according to a first order of execution, and during the execution, at runtime a failing test case, from the set of test cases is determined. Further, at runtime, from the set of test cases, a subset of test cases that are not yet executed is identified. Further, at runtime, a priority value is assigned to each test case from the subset of test cases, the priority value based on a similarity of the test case with the failing test case. Further, at runtime, the set of test cases is reorganized to execute the subset of test cases in a second order of execution based on priority values that are assigned.