GPU Error Detection via Redundant Computation and Checkpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High performance computing applications, such as financial and scientific modeling, are sensitive to bit errors but many GPUs lack hardware support for error checking and correction, leaving them vulnerable to errors caused by manufacturing defects, environmental factors, and other anomalies.
Innovation Solution
Implementing software-based error detection and recovery mechanisms by duplicating computations on multiple hardware resources, comparing results, and using checkpointing to roll back to a previous valid state in case of errors, with the option to scale up redundancy for critical computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If software-based error detection and recovery mechanisms are implemented by duplicating computations on multiple hardware resources, then reliability is improved, but device complexity increases
Solution Approach 1:
The patent implements error detection by creating duplicate copies of the computation on different hardware resources (first and second hardware resources). These copies execute the same computational task independently, allowing for comparison of results to detect errors. This copying approach provides reliability through redundancy while maintaining the same computational logic.
Solution Approach 2:
The computational task is segmented into multiple independent execution instances that run on separate hardware resources. By dividing the computation into separable copies that can be executed independently and then compared, the system achieves error detection capability. The segmentation allows each copy to be verified against the others.
2Reliability
If computations are duplicated on multiple hardware resources for error detection, then reliability is improved, but productivity decreases
Solution Approach 1:
The system creates duplicate computation copies on different hardware resources to enable error detection through result comparison. This copying mechanism ensures computation correctness by verifying that independent executions produce matching results, thereby improving reliability at the cost of additional computational overhead.
Solution Approach 2:
The patent merges the results from multiple duplicate computations by comparing them. When the results match, the system confidently accepts the outcome. This merging approach through comparison provides a efficient verification mechanism that confirms correctness without requiring complex validation procedures.
3Reliability
If checkpointing is implemented to enable rollback to previous valid state, then reliability is improved, but loss of time increases
Solution Approach 1:
The system performs preliminary actions by creating and storing checkpoint data at intermediate points during computation execution. These checkpoints capture the state of the computation at specific moments, enabling the system to rollback to a known good state if errors are detected, rather than restarting from the beginning.
Solution Approach 2:
The patent implements beforehand cushioning by pre-establishing checkpoint data that serves as a safety buffer. When errors are detected through result comparison, the system can revert to previously saved checkpoint states, providing protection against computation failure and reducing the impact of errors on overall execution time.
Data Source
AI summary
A service provider system may implement ECC-like features when executing computations on GPUs that do not include sufficient error detection and recovery for computations that are sensitive to bit errors. During execution of critical computations on behalf of customers, the system may automatically instrument program instructions received from the customers to cause each computation to be executed using multiple sets of hardware resources (e.g., different host machines, processor cores, or internal hardware resources). The service may provide APIs with which customers may instrument their code for execution using redundant resource instances, or specify parameters for applying the ECC-like features. The service or customer may instrument code to perform (or cause the system to perform) checkpointing operations at particular points in the code, and to compare intermediate results produced by different hardware resources. If the intermediate results do not match, the computation may be restarted from a checkpointed state.