Local Checkpoint Recovery for GPU Memory Error Containment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance computing systems face challenges in efficiently recovering from transient memory errors, particularly due to the limitations of global checkpoint and recovery methods, which lead to significant throughput losses and potential corruption of data across nodes.
Innovation Solution
Implementing fine-grained local checkpoint and recovery within GPUs, utilizing the Compute Instruction Level Preemption (CILP) feature to stall execution at instruction boundaries, store microarchitecture states, and prevent data propagation of errors, thereby isolating and correcting memory errors within affected memory clients.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If global checkpoint and recovery is implemented, then system reliability is improved, but system throughput deteriorates due to all nodes being halted and restored simultaneously
Solution Approach 1:
The patent divides the monolithic global checkpoint system into independent local checkpoint systems for each memory client. When an error occurs, only the affected memory client performs local checkpoint and recovery, while other memory clients continue executing without interruption. This segmentation isolates error impact and maintains overall system throughput.
Solution Approach 2:
The patent implements differentiated error handling where each memory client has its own local checkpoint capability. The affected memory client receives special treatment (local recovery) while other clients maintain normal operation. This local quality approach ensures reliability for the affected component without compromising overall system productivity.
2Productivity
If local checkpoint and recovery is implemented, then system throughput is improved, but error propagation control deteriorates without proper containment mechanisms
Solution Approach 1:
The patent extracts the error-prone memory client from the overall system execution by stalling it upon error detection. The affected memory client is taken out of the active execution flow, prevented from propagating corrupt data, while other memory clients continue operating. This extraction isolates the harmful effect without stopping the entire system.
Solution Approach 2:
The patent applies preliminary anti-action by preemptively stalling the affected memory client immediately upon error detection, before corrupt data can be propagated to other memory clients or system components. This preliminary containment prevents the harmful effect from spreading while maintaining system throughput.
3Measurement precision
If fine-grained local checkpoint is implemented at instruction boundaries, then error detection precision is improved, but device complexity increases due to CILP feature requirements
Solution Approach 1:
The patent uses Compute Instruction Level Preemption (CILP) to preempt execution at instruction boundaries before corrupt data can be used. This preliminary action captures the microarchitecture state at precise points, enabling accurate error detection and recovery. The preemption mechanism is built into the GPU architecture to manage the complexity.
Solution Approach 2:
The patent introduces CILP as an intermediary mechanism between the executing instructions and the checkpointing process. CILP mediates the preemption and state capture at instruction boundaries, providing fine-grained error detection precision while managing the underlying complexity through a standardized interface.
4Reliability
If memory clients are stalled upon error detection, then error containment is improved, but loss of time increases during recovery
Solution Approach 1:
The patent segments the recovery process so that only the affected memory client is stalled and performs local checkpoint recovery. Other memory clients continue executing without stalling, significantly reducing the overall time loss compared to global checkpoint where all nodes are halted. The containment is achieved with minimal time penalty.
Data Source
AI summary
Various embodiments include a parallel processing computer system that detects memory errors as a memory client loads data from memory and disables the memory client from storing data to memory, thereby reducing the likelihood that the memory error propagates to other memory clients. The memory client initiates a stall sequence, while other memory clients continue to execute instructions and the memory continues to service memory load and store operations. When a memory error is detected, a specific bit pattern is stored in conjunction with the data associated with the memory error. When the data is copied from one memory to another memory, the specific bit pattern is also copied, in order to identify the data as having a memory error.