Emulation-Based Test Verification Decoupling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional software testing is limited by the coupling of test traversal steps with expected behavior and checks, leading to missed edge cases and potential programming errors, as it cannot effectively separate stimulus from checks, resulting in incomplete system verification.
Innovation Solution
The use of context objects, such as Actual and Model context objects, to separate stimulus and checking, along with a Weighted Constraint Driver for randomizing test flow and data, allowing for on-the-fly generation of varied and complex test scenarios that cover all possible system elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional software testing uses handcrafted test cases with coupled traversal steps and checks, then the test structure is simple and maintainable, but the testing coverage is incomplete and misses edge cases
Solution Approach 1:
The patent segments the testing system into two independent components: a test flow generator that creates randomized traversal steps and a check generator that creates verification steps. This segmentation allows each component to be developed and maintained separately while achieving comprehensive testing coverage through randomized combination of test elements.
Solution Approach 2:
The patent introduces dynamic randomness into the test generation process, where test flows and checks are generated on-the-fly rather than being statically predefined. This dynamic approach enables the system to adaptively cover edge cases and unusual scenarios that deterministic handcrafted tests would miss.
2Reliability
If software testing separates stimulus from checks using emulation, then testing coverage and efficacy are significantly increased, but the device complexity and implementation difficulty increase
Solution Approach 1:
The patent introduces an intermediary component called the check generator that acts as a mediator between the test flow execution and the verification process. This intermediary automatically generates appropriate checks based on the current test state, eliminating the need for manual coupling of stimulus and verification while simplifying the overall implementation.
Solution Approach 2:
The testing system implements self-service through automated test flow and check generation. The system generates its own test cases, executes them, and performs verification without requiring manual intervention for each test scenario. This self-service capability increases reliability while managing implementation complexity through automation.
3Reliability
If randomized testing generates test scenarios on-the-fly, then all possible system elements are checked and edge cases are covered, but the test execution time and resource consumption increase
Solution Approach 1:
The patent utilizes parameter changes by varying the randomness seed and test generation parameters to create diverse test scenarios. By changing these parameters dynamically, the system achieves comprehensive coverage of system elements and edge cases without requiring an exhaustive fixed set of test cases, thereby managing execution time more effectively.
Data Source
AI summary
Techniques for using emulation to disassociate verification from stimulus in functional test are described. In one approach, a computer stores first data representing an initial state of an application and second data representing the same initial state of a model application, wherein the model application models expected behavior of the application. The computer selects actions for the application to perform and causes both the application and the model application to perform the actions. The computer updates the first and second data to represent the state of both the application and the model application after performing the actions. The computer then compares the first and second data to determine whether both refer to the same state. In response to a determination that the first data and the second data do not refer to the same state, the computer stores data indicating a test failure.