Low Latency Checkpointing in HPC Clusters via RDMA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional High Performance Computing (HPC) systems face increased latency and reduced checkpoint frequency as cluster sizes grow, due to the need for frequent checkpoints and the inefficiencies in data transfer between compute nodes and storage arrays, which hinders scalability and fault tolerance.
Innovation Solution
A multi-step checkpoint process utilizing Remote Direct Memory Access (RDMA) technology to minimize latency by transferring data directly from application memory to network, allowing compute nodes to continue processing while IO nodes handle data storage, and using non-volatile memory to reduce the burden on the host processor, thereby enabling more frequent checkpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional checkpoint methods are used in large HPC clusters, then fault tolerance is maintained, but checkpoint latency increases and checkpoint frequency decreases
Solution Approach 1:
The system segments the checkpoint process into two independent phases: (1) compute nodes copy checkpoint data to IO nodes' memory, and (2) IO nodes write data to storage arrays. This segmentation allows compute nodes to continue processing while IO nodes handle storage operations, reducing checkpoint latency without compromising fault tolerance.
Solution Approach 2:
The compute nodes perform preliminary action by copying checkpoint data to IO nodes' memory before the actual storage operation. This preliminary copying enables the compute nodes to resume processing immediately while the IO nodes complete the storage write in the background, effectively reducing the perceived checkpoint latency.
2Productivity
If cluster size increases, then computing power scales, but checkpoint frequency must increase which drives up latency requirements
Solution Approach 1:
By segmenting the checkpoint workflow into compute and IO phases executed by different node types, the system enables larger clusters to perform frequent checkpoints without proportionally increasing latency. Compute nodes can continue productive work while dedicated IO nodes handle the storage overhead.
Solution Approach 2:
IO nodes act as intermediaries between compute nodes and storage arrays. They receive checkpoint data from compute nodes via RDMA and handle the storage write operations, shielding compute nodes from storage latency and enabling faster checkpoint frequencies in large-scale clusters.
3Reliability
If data is transferred between compute nodes and storage arrays, then checkpoint data is saved, but transfer inefficiency hinders scalability
Solution Approach 1:
IO nodes serve as intermediary buffers between compute nodes and storage arrays. Compute nodes transfer data to IO nodes' memory using fast RDMA, and IO nodes subsequently write to storage arrays. This intermediary approach decouples the transfer speeds, allowing compute nodes to operate at high speed without being bottlenecked by storage array write speeds.
Solution Approach 2:
The system replaces traditional block-based storage transfer with RDMA (Remote Direct Memory Access) technology, which enables direct memory-to-memory transfer between compute nodes and IO nodes without involving operating system file systems. This substitution dramatically increases data transfer speed and reduces CPU overhead.
4Reliability
If host processors handle checkpoint data transfer, then checkpointing is performed, but processing burden increases latency
Solution Approach 1:
The system extracts the checkpoint data transfer burden from host processors and assigns it to dedicated IO nodes. Compute nodes only perform the initial data copy to IO nodes' memory via RDMA, then immediately resume processing. The IO nodes handle the time-consuming storage write operations, effectively removing this burden from the critical compute path and reducing processing latency.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An embodiment includes a low-latency mechanism for performing a checkpoint on a distributed application. More specifically, an embodiment of the invention includes processing a first application on a compute node, which is included in a cluster, to produce first computed data and then storing the first computed data in volatile memory included locally in the compute node; halting the processing of the first application, based on an initiated checkpoint, and storing first state data corresponding to the halted first application in the volatile memory; storing the first state information and the first computed data in non-volatile memory included locally in the compute node; and resuming processing of the halted first application and then continuing the processing the first application to produce second computed data while simultaneously pulling the first state information and the first computed data from the non-volatile memory to an input/output (IO) node.