Soft Failure Detection via Execution Time Monitoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing testing methods fail to consistently identify and diagnose soft failures in computer systems, which are errors that occur due to specific environmental or timing circumstances, leading to inefficiencies and redundancy in regression testing.
Innovation Solution
A method involving combinatorial test designing (CTD) and inverse CTD is employed to execute a minimal set of test cases on a system under test (SUT), using a fingerprint repository to eliminate redundant tests and identify code paths associated with soft failures by monitoring execution times and comparing fingerprints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If comprehensive testing is performed to ensure high quality software, then reliability is improved, but testing time and complexity increase
Solution Approach 1:
The patent segments the testing process by dividing test cases into different categories based on their execution time characteristics (normal vs. slow tests). This segmentation allows the system to apply different handling strategies to different test categories, enabling comprehensive testing while managing time consumption through selective execution and prioritization of critical test cases.
Solution Approach 2:
The patent implements preliminary action by pre-identifying and marking slow test cases before execution. The system analyzes historical execution data to determine which tests are likely to take excessive time, and prepares alternative strategies (such as using test doubles, reducing test data, or executing only critical path tests) in advance. This allows the system to quickly respond when slow tests are encountered without derailing the overall testing timeline.
2Measurement precision
If extensive test cases are executed to cover all code paths, then measurement precision is improved, but device complexity increases
Solution Approach 1:
The patent applies partial action by executing only the necessary subset of test cases required to achieve adequate code coverage. Rather than blindly running all possible tests, the system intelligently selects test cases based on their expected value, execution time, and coverage contribution. This allows the system to achieve sufficient measurement precision without the overwhelming complexity of exhaustive testing.
Solution Approach 2:
The patent uses copying by creating simplified versions or proxies of complex test scenarios. When a slow or complex test is identified, the system may create a test double, mock object, or simplified test case that replicates the essential behavior without the full complexity. This maintains code coverage measurement accuracy while reducing the operational complexity of the testing system.
3Reliability
If regression testing is performed frequently to detect soft failures, then reliability is improved, but productivity decreases
Solution Approach 1:
The patent implements periodic action by scheduling regression tests at strategic intervals rather than continuously or after every minor change. The system identifies which test cases should be executed periodically based on risk assessment, change frequency, and historical failure data. This periodic execution maintains reliable failure detection capability while avoiding the productivity impact of overly frequent comprehensive regression testing.
Solution Approach 2:
The patent applies preliminary action by preparing and pre-configuring regression test suites in advance. The system pre-identifies the subset of tests that are most critical for detecting soft failures in regression scenarios, and pre-configures them for rapid execution. When regression testing is triggered, these pre-prepared test cases can be executed quickly, maintaining reliability while minimizing the productivity impact on software delivery.
Data Source
AI summary
A method for testing a system under test (SUT) in an active environment includes executing, by the testing system, on the SUT, a test from a set of tests. The method further includes, monitoring a first execution time to complete the test on the SUT in the active environment. Based on the first execution time being different than a second execution time of the test, marking, by the testing system, a code path associated with the test. The second execution time is a duration to complete execution of the test on the SUT in a clean execution environment. The method further includes communicating, by the testing system, the code path for analyzing a soft failure of the SUT in the active environment, wherein the soft failure occurs in the active environment during execution of the SUT based at least on a parameter of the active environment.