Adaptive Multi-Level Checkpointing for HPC Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing systems face increasing checkpointing overhead as they scale, necessitating lower-overhead checkpointing schemes to manage growing data volumes and maintain system resilience without excessive input/output bandwidth utilization.
Innovation Solution
Adaptive multi-level checkpointing determines optimal times and locations for checkpointing by monitoring system performance, utilizing a hierarchy of storage tiers including high-bandwidth and high-capacity storage, and dynamically deciding between node-local storage and parallel file systems to minimize overhead and maximize resilience.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If checkpointing is performed frequently to maintain system resilience, then reliability is improved, but input/output overhead increases
Solution Approach 1:
The patent segments checkpointing into multiple levels: application-level checkpoints stored in node-local storage and system-level checkpoints stored in parallel file system. This segmentation allows frequent application-level checkpoints with minimal I/O overhead while maintaining reliability through periodic system-level checkpoints.
Solution Approach 2:
The patent implements local quality by storing different types of checkpoints in different storage locations tailored to their specific needs. Application-level checkpoints are stored locally in node-local storage for fast access and frequent updates, while system-level checkpoints are stored in parallel file system for long-term persistence and recovery.
2Speed
If checkpoint data is stored in node-local storage, then checkpointing speed is improved, but storage capacity is limited
Solution Approach 1:
The patent merges two storage systems with different characteristics: node-local storage provides high speed but limited capacity, while parallel file system provides large capacity but lower speed. By combining both in a multi-level architecture, the system achieves both fast checkpointing for small data and large capacity for comprehensive system states.
Solution Approach 2:
The patent adds a new dimension to the storage hierarchy by introducing multiple levels of storage with different performance and capacity characteristics. Instead of relying on a single storage dimension, the system uses vertical stratification of storage tiers to simultaneously satisfy speed and capacity requirements.
3Reliability
If all system data is checkpointed to ensure complete recovery, then reliability is improved, but input/output bandwidth utilization increases
Solution Approach 1:
The patent extracts and stores only the essential system-level state information in system-level checkpoints, separate from the detailed application-level checkpoints. This extraction approach ensures complete recoverability by capturing critical system state while minimizing the volume of data that needs to be transferred and stored, thus reducing I/O bandwidth utilization.
Data Source
AI summary
In some examples, with respect to adaptive multi-level checkpointing, a transfer parameter associated with transfer of checkpoint data from a node-local storage to a parallel file system may be ascertained for the checkpoint data stored in the node-local storage. The transfer parameter may be compared to a specified transfer parameter threshold. A determination may be made, based on the comparison of the transfer parameter to the specified transfer parameter threshold, as to whether to transfer the checkpoint data from the node-local storage to the parallel file system.