Combinatorial Test Design for Hidden Fault Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current testing infrastructures struggle to identify and address hidden variables and attributes that cause soft failures in computer systems, leading to incomplete testing and increased costs due to the complexity of managing test environments and dependencies.
Innovation Solution
The method involves modeling inputs to a system under test as attribute-value pairs, generating test vectors for complete n-wise coverage, executing testcases, and using combinatorial test design (CTD) to detect and localize faults, including the identification of hidden variables through pseudo-Boolean algebra and graphical user interface visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional testing infrastructure is used, then testing can be performed, but hidden variables and attributes causing soft failures cannot be identified
Solution Approach 1:
The patent segments the testing infrastructure into modular components: test vector generation module, testcase execution module, result analysis module, and visualization module. This segmentation allows each component to handle specific aspects of testing independently, improving fault detection capability while managing complexity through modular design.
Solution Approach 2:
The patent introduces an intermediary testing framework that sits between the system under test and the traditional testing infrastructure. This intermediary layer captures execution results, analyzes success rates, and identifies hidden variables, thereby enhancing measurement precision without directly modifying the complex test infrastructure.
2Reliability
If complete testing coverage is pursued, then more faults can be detected, but testing time and resources increase significantly
Solution Approach 1:
The patent implements feedback mechanisms where execution results are continuously analyzed and used to update success rates for attribute-value pairs. This feedback loop enables the system to identify problematic areas and focus testing efforts there, achieving comprehensive coverage efficiently by adapting test execution based on accumulated evidence.
Solution Approach 2:
The patent applies partial action by focusing testing resources on attribute-value pairs with success rates below the threshold. Instead of uniformly testing all possibilities, the system concentrates efforts on areas showing signs of failure, thereby achieving effective testing completeness with reduced time and resources.
3Ease of operation
If traditional binary success/failure results are used, then simple evaluation is possible, but hidden attributes and soft failures cannot be detected
Solution Approach 1:
The patent transforms the evaluation from simple binary outcomes to a multi-parameter analysis system. It introduces success rates as a continuous parameter, tracks execution results across multiple runs, and uses threshold-based classification. This parameter transformation maintains ease of operation through automated calculations while dramatically improving measurement precision in detecting soft failures.
4Reliability
If more testcases are executed to improve coverage, then more faults are detected, but infrastructure complexity and cost increase
Solution Approach 1:
The patent creates a universal testing framework that handles multiple functions: test vector generation, testcase execution, result collection, success rate calculation, and visualization. This multi-functional system improves fault detection capability while managing infrastructure complexity by consolidating diverse testing operations into a single integrated platform.
Data Source
AI summary
Inputs to a system under test (SUT) are modeled as a collection of attribute-value pairs. A set of testcases is executed using an initial set of test vectors that provides complete n-wise coverage of the attribute-value pairs. For each execution of the testcases, for each attribute-value pair, a non-binary success rate (SAV) is computed based on the binary execution results. in response to a success rate of an attribute-value pair being below a predetermined threshold, a subset of testcases that use the attribute-value pair is identified. Further, sets of code paths for the subset of testcases are identified, each set of code path respectively corresponding to a testcase from the subset of testcases. Further, an intersection of the sets of code paths is determined. Code paths of the SUT that are in the intersection, are highlighted to represent a soft failure with the SUT.