Scenario Runners for Automated Software Validation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing complexity of software systems makes conventional testing challenging, as the number of possible scenarios expands exponentially, leading to difficulties in ensuring software functions as intended, especially in complex multi-node systems.
Innovation Solution
The use of scenario runners and a controller to manage and validate test cases within a predetermined time frame, allowing for predictable validation of test cases through event collection and evaluation, with the ability to adjust parameters like time, number of runners, and scenarios based on feedback for conclusive results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the number of scenarios is increased to cover software complexity, then testing coverage is improved, but testing time and resources increase exponentially
Solution Approach 1:
The patent segments the testing process into multiple independent validation passes, each executing a subset of scenarios. The test suite is divided into groups that can be run independently and in parallel, transforming one large sequential testing task into multiple smaller concurrent tasks, thereby reducing total testing time while maintaining comprehensive coverage.
Solution Approach 2:
The patent introduces a new dimension of parallel execution by running multiple scenario runners simultaneously. Instead of executing scenarios sequentially in a single thread, the system creates multiple parallel validation passes that can execute concurrently, effectively adding a temporal parallelism dimension to the testing process.
2Measurement precision
If more scenarios are run to ensure comprehensive testing, then validation accuracy is improved, but predictability of validation time deteriorates
Solution Approach 1:
The patent implements dynamic control of validation passes through a feedback mechanism. The system monitors the number of events collected during each pass and dynamically determines whether to continue with additional passes or terminate early. This dynamic approach allows the testing to adapt to actual system behavior, maintaining high validation accuracy while improving time predictability by avoiding unnecessary passes.
Solution Approach 2:
The patent employs a feedback mechanism where the validator monitors events collected from scenario runners and uses this information to control the continuation or termination of validation passes. The feedback loop enables the system to adjust testing duration based on actual validation needs, ensuring comprehensive coverage when necessary while allowing early termination when sufficient evidence is gathered.
3Productivity
If the number of scenario runners is increased to run more scenarios in parallel, then productivity is improved, but system complexity increases
Solution Approach 1:
The patent creates a universal controller that manages multiple scenario runners with identical interfaces and behaviors. Each scenario runner is a standardized component that can execute any scenario from the test suite, making the system multi-functional without increasing individual component complexity. The controller provides a unified management layer that handles the complexity of coordinating multiple runners.
Solution Approach 2:
The patent introduces a controller as an intermediary between the test suite and multiple scenario runners. This mediator component manages the complexity of coordinating parallel executions, handling scenario assignment, event collection, and validation logic. By placing the coordination logic in the intermediary controller rather than in each individual runner, the system achieves high productivity while keeping individual component complexity low.
Data Source
AI summary
The automated validation of one or more test cases in a validation pass. The validation pass is enabled using a plurality of scenario runners, which are executable components that respond to instructions to run one or more scenarios of a validation pass by beginning to run through such scenarios during the validation pass. The validation pass may be associated with a maximum time such that upper limits on how long the scenario runners may run is set. This likewise limits the time it takes to validate the results of the scenario. The scenario runners provide events that occur during the running of their respective scenarios into an event collector. The events of the validation pass may then be evaluated against one or more test cases to determine whether or not the events have a state indicative of a success or failure of each test cases.