GPU Checkpointing and Recovery via Non-Volatile Memory Persistence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computing systems lack effective mechanisms for checkpointing and recovering the state of graphics processing units (GPUs), including application data and kernel execution state, which is critical for ensuring crash recoverability in computing environments, especially in datacenters and supercomputers where GPU failures can occur frequently.
Innovation Solution
The implementation of a method that checkpoints GPU application data and kernel execution state, synchronizes CPU and GPU states, and persists this data on non-volatile memory, using APIs such as 'register_data_structure', 'create_checkpoint', and 'restore_checkpoint' to ensure efficient recovery, allowing for debugging and execution migration without modifying application kernels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If GPU application data and kernel execution state are checkpointed and persisted on non-volatile memory, then crash recoverability is improved, but device complexity and overhead increase
Solution Approach 1:
The system performs preliminary checkpointing of GPU application data and kernel execution state to non-volatile memory before crashes occur. This advance preparation ensures that recovery can be performed quickly by simply restoring from the pre-saved checkpoint, rather than attempting complex recovery procedures after a failure.
Solution Approach 2:
The invention creates copies of GPU application data and kernel execution state and stores them on non-volatile memory. This copying mechanism allows the original GPU state to be preserved while maintaining a backup that can be restored if needed, enabling reliable crash recovery without modifying the original execution environment.
2Reliability
If checkpointing is implemented for GPU states, then data integrity is improved, but processing time and overhead worsen
Solution Approach 1:
The checkpointing mechanism is designed to operate autonomously with minimal interference to the main GPU processing workflow. The system self-manages the checkpointing operations, automatically saving state information to non-volatile memory without requiring external intervention or significant disruption to ongoing computations.
Solution Approach 2:
The system implements selective checkpointing that focuses on critical GPU state information rather than attempting to save every possible data element. This partial action approach captures the essential information needed for recovery while minimizing the overhead associated with checkpointing operations.
3Reliability
If CPU and GPU states are synchronized during checkpointing, then recovery consistency is improved, but execution time increases
Solution Approach 1:
The invention introduces an intermediary synchronization mechanism that coordinates between CPU and GPU states during checkpointing. This mediator ensures that both processing units are in a consistent state when the checkpoint is taken, allowing for reliable recovery without requiring extensive coordination overhead during actual execution.
Data Source
AI summary
Embodiments for crash recoverability for graphics processing units (GPUs) by a processor. GPU application data and kernel execution state of one or more GPUs may be checkpointed. The checkpointed GPU application data and the kernel execution state may be recovered. The checkpointed GPU application data and the kernel execution state may be persisted on non-volatile memory.