Offload Programming Model for Local Checkpoints in HPC
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing (HPC) systems face significant challenges with exponential failure rates of compute entities, leading to inefficient checkpoint-restart schemes that consume substantial storage resources and processing bandwidth, making large-scale HPC jobs unsustainable.
Innovation Solution
An offload programming model is employed for local checkpoints, which automatically defines checkpoints, reduces synchronization overhead, and decomposes state saving operations throughout job execution, allowing for smaller storage footprints and improved restart success rates without suspending compute entities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional checkpoint-restart scheme is used with frequent synchronized checkpoints, then reliability is improved, but storage consumption and processing bandwidth increase substantially
Solution Approach 1:
The system segments the checkpoint operation into two independent parts: (1) compute entities continue processing without suspension, and (2) state information is asynchronously written to storage. This segmentation eliminates the need to halt computation for checkpointing, thereby maintaining reliability while reducing storage bandwidth consumption.
Solution Approach 2:
The system performs preliminary actions by pre-allocating storage resources and preparing checkpoint infrastructure before failures occur. State information is continuously tracked and prepared for checkpointing, allowing rapid recovery without requiring substantial storage bandwidth during actual checkpoint operations.
2Reliability
If conventional checkpoint-restart scheme is used with frequent synchronized checkpoints, then reliability is improved, but processing bandwidth is wasted significantly
Solution Approach 1:
The system maintains continuity of useful action by allowing compute entities to process workloads continuously without suspension for checkpointing. State information is captured asynchronously in the background, ensuring that processing bandwidth is fully utilized for productive computation rather than being wasted on synchronized checkpoint operations.
Solution Approach 2:
The system introduces an intermediary mechanism that decouples the checkpointing function from the computation function. A separate state management layer handles state capture and storage operations independently, allowing computation to proceed without interruption while still maintaining reliability through continuous state tracking.
3Device complexity
If checkpoint groups involve large numbers of entities with coarse granularity, then synchronization overhead is reduced, but restart capability becomes less efficient
Solution Approach 1:
The system applies local quality by allowing different granularity levels for different checkpoint operations. Individual compute entities or small groups can maintain fine-grained state information locally, enabling efficient restarts of specific entities without requiring global synchronization across all entities. This localized approach maintains restart efficiency while reducing overall synchronization overhead.
Data Source
AI summary
Methods, apparatus, and systems for leveraging an offload programming model for local checkpoints. Compute entities in a computing environment are implemented as one or more sources and a larger number of sinks. A job dispatcher dispatches jobs comprising executable code to the source(s), and the execution of the job code is managed by the source(s). Code sections in the job code designated for offload are offloaded to the sinks by creating offload context information. In conjunction with each offload, an offload object is generated and written to storage. The offloaded code sections are executed by the sinks, which return result data to the source, e.g., via a direct write to a memory buffer specified in the offload context information. The health of the sinks is monitored to detect failures, and upon a failure the source retrieves the offload object corresponding to the code section offloaded to the failed sink, regenerates the offload context information for the code section and sends this to another sink for execution.