Phased Combinatorial Testing for Data Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional testing methods for complex software products like distributed data storage systems, such as hardcoded tests and fuzz tests, fail to adequately test modern data storage systems, leading to inadequate results, high memory and processor usage, and inability to rapidly focus tests, especially in systems with many variable attributes.
Innovation Solution
The implementation of combinatorial testing, which systematically generates and executes test cases, leveraging synchronization points to manage parallelization and reduce memory consumption, allowing for efficient use of computing resources and human-readable results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional hardcoded tests or fuzz tests are used, then testing can be implemented, but testing adequacy is insufficient and computing resource usage (memory and processor) becomes excessive
Solution Approach 1:
The patent segments the testing process into distinct phases (setup phase, test execution phase, teardown phase) with synchronization points between them. This allows the test runner to manage parallel test execution in a controlled manner, dividing the complex testing task into manageable segments that can be executed efficiently with reduced memory footprint while maintaining comprehensive test coverage
Solution Approach 2:
The patent changes the parameter of test execution from sequential or uncontrolled parallel to phased parallel execution. By introducing phase boundaries and synchronization points, the system transforms how tests are executed, allowing multiple tests to run in parallel within phases while controlling resource usage through structured phase transitions, thereby improving testing adequacy without excessive resource consumption
2Reliability
If comprehensive combinatorial testing is performed, then testing coverage is improved, but memory consumption increases
Solution Approach 1:
The patent divides the combinatorial testing process into phases with synchronization points, where tests are executed in controlled batches rather than all simultaneously. This segmentation allows comprehensive test coverage to be achieved while managing memory consumption by limiting the number of active test cases in memory at any given time, as tests proceed through phases sequentially with parallel execution only within each phase
Solution Approach 2:
The patent implements periodic action through phased test execution with synchronization points. Tests are executed in periodic phases rather than continuously in uncontrolled parallel, allowing memory to be freed and reused between phases. This periodic structure enables comprehensive coverage while controlling peak memory consumption by resetting and reusing resources at each phase boundary
Data Source
AI summary
Phased parameterized combinatoric testing for a data storage system is disclosed. A testing recipe can be performed according to different input arguments. Combinatoric testing of the data storage system can be based on different combinations of operations and arguments. The disclosed testing can employ a consistent integer index for arguments passed into the sequenced operations of the recipe. The recipe can be employed to generate a phased test tree that can enable testing based on a phase rather than loading an entire test suite into memory. The consistent integer index can be used to identify failed test cases such that the entire test can be reconstituted from stored failed test information. Distribution of test cases to worker process can based on the phased test tree to facilitate interning an operation. Stored failed test information can include human-readable failure information.