Dynamic Checkpoint Adjustment for Accelerator Devices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing checkpoint methods for high-performance computers (HPCs) are limited in universality and lead to high checkpoint loads due to performance differences across applications, as they do not account for load changes during execution and are inefficient in data synchronization, especially with accelerator devices like GPUs.
Innovation Solution
A method and apparatus for adjusting checkpoints by monitoring API calls and determining a next checkpoint based on API execution logic and cycles, using a checkpoint adjustment strategy that considers mean time to failure and checkpoint cost, allowing for dynamic adjustment of checkpoint intervals based on API execution patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data synchronization is performed frequently to ensure data consistency in accelerator devices, then data consistency is improved, but the time cost and checkpoint load increase significantly
Solution Approach 1:
The checkpoint interval is dynamically adjusted based on the accelerator device's operational state. When the device is in a low-load state with fewer API calls, the checkpoint interval is extended. When the device is in a high-load state with frequent API calls, the checkpoint interval is shortened. This dynamic adjustment resolves the contradiction by adapting the synchronization frequency to actual needs rather than using a fixed interval.
Solution Approach 2:
The system changes the parameter of checkpoint interval based on monitored API execution patterns. By detecting the number and frequency of API calls, the system modifies the checkpoint timing parameter to optimize between data consistency and time cost, avoiding unnecessary frequent checkpoints when data changes are minimal.
2Reliability
If checkpoint interval is shortened to improve fault tolerance, then reliability is improved, but checkpoint load and system overhead increase
Solution Approach 1:
The checkpoint interval is made dynamic rather than fixed. The system monitors API execution patterns and adjusts the checkpoint interval in real-time. During periods of low API activity, the interval is extended to reduce checkpoint load. During periods of high API activity or when critical data changes occur, the interval is shortened to maintain fault tolerance. This resolves the contradiction by making checkpoint frequency adaptive to actual system state.
Solution Approach 2:
The system automatically monitors its own operational state through API call tracking and self-adjusts the checkpoint interval without external intervention. The monitoring component detects API execution patterns and the checkpoint control component responds by adjusting intervals, creating a self-regulating system that balances reliability and productivity based on real-time conditions.
3Device complexity
If fixed checkpoint intervals are used to simplify implementation, then device complexity is reduced, but adaptability to different application scenarios is limited
Solution Approach 1:
The checkpoint system transitions from a static fixed-interval approach to a dynamic adaptive-interval approach. The system monitors API execution patterns specific to different applications and automatically adjusts checkpoint timing accordingly. This maintains low complexity by using a standardized monitoring and adjustment mechanism while achieving high adaptability through automatic adaptation to various application workloads and patterns.
4Measurement precision
If checkpoint is performed at every API call to ensure data accuracy, then measurement precision is improved, but loss of time and productivity decrease
Solution Approach 1:
The system changes the checkpoint timing parameter based on API execution patterns rather than using a fixed rule. By analyzing the frequency and type of API calls, the system determines optimal checkpoint moments that ensure data accuracy without requiring checkpoints at every single API call. This reduces unnecessary checkpoint executions and associated time losses while maintaining data integrity.
Data Source
AI summary
A method and apparatus for adjusting a checkpoint are provided. The method includes monitoring calls of an application program interface (API) that are called when an accelerator device executes an application, and by the monitoring, checking an API execution logic and a current API execution cycle of the application with respect to the accelerator device; and determining a next checkpoint according to a checkpoint adjustment strategy that determines the next checkpoint based on the API execution logic and based on the current API execution cycle of the application, wherein the checkpoint adjustment strategy corresponds to at least one API execution logic among plural API execution logics.