Shadow Mode Grid Testing for HPC Job Replay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Testing high-performance computing (HPC) grids and clusters is challenging due to the need for manual creation and updating of tests for framework applications, cluster managers, and APIs, and capturing performance metrics across shared nodes, which can lead to inconsistencies and inefficiencies in ensuring that updates do not alter computing results.
Innovation Solution
Implementing a 'shadow mode' approach where jobs launched on HPC grids are recorded and replayed on new builds to compare reference and candidate results, using a job submission service to capture and store job definitions, inputs, and outputs, allowing for scalable and accurate testing without manual test composition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual creation and updating of tests is performed for framework applications, cluster managers, and APIs, then testing coverage can be achieved, but the complexity and time consumption of the testing process increases significantly
Solution Approach 1:
The patent captures actual production jobs and creates copies for replay during testing. Instead of manually creating test cases, the system automatically copies real job definitions, inputs, and outputs from the production environment and replays them against new builds, eliminating manual test composition while maintaining comprehensive testing coverage
Solution Approach 2:
The system performs preliminary capture and storage of job definitions, inputs, and outputs from production jobs before testing is needed. This pre-captured data is then replayed during testing, allowing tests to be executed without manual intervention at test creation time, thus reducing time consumption while ensuring reliability
2Reliability
If manual creation and updating of tests is performed for framework applications, cluster managers, and APIs, then testing coverage can be achieved, but the complexity of the testing process increases
Solution Approach 1:
The testing system serves itself by automatically capturing production jobs, storing them, and replaying them without external manual intervention. The system autonomously generates its own test cases from real production data, eliminating the need for manual test creation and updating, thus reducing testing process complexity while maintaining comprehensive coverage
Solution Approach 2:
The job capture and replay mechanism serves multiple functions: it captures production jobs for future replay, stores job definitions and inputs, and automatically generates test cases. This multi-functional approach replaces multiple manual processes with a single automated system, reducing overall testing process complexity
3Measurement precision
If performance metrics are captured across shared nodes, then comprehensive monitoring is achieved, but inconsistencies and inefficiencies occur in ensuring that updates do not alter computing results
Solution Approach 1:
The system captures job outputs from production runs and uses them as expected results for comparison during replay. This feedback mechanism automatically verifies whether new builds produce consistent results with production, providing precise measurement of computing result consistency without manual intervention and eliminating inconsistencies that would otherwise occur
Data Source
AI summary
Techniques are disclosed for testing applications and systems used to provide a high performance computing (HPC) system, such as a cluster or grid. A grid computing system is tested using a “shadow mode” approach, where information regarding a parallel computing job launched on a first instance of a computing grid is recorded, stored, and used to replay the parallel computing job on a second instance of the computing grid. For example, the parallel computing job may be executed using a production build of grid applications and replayed on a candidate build of grid applications. Output from executing the parallel computing job is also captured in order to provide a reference result which may be compared with candidate results generated by replaying the parallel computing job. The parallel computing job may specify a set of inputs and execution logic along with any other requirements or dependencies needed to accurately replay the parallel computing job.