AI Processor Error Recovery via Redundant Model Checkpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In artificial intelligence processing systems, when one node of a multiprocessor system experiences an error, existing methods often require restarting computations, leading to the need to recompute large amounts of data, which is time-consuming and inefficient, especially in large-scale clusters where frequent errors can severely impact performance.
Innovation Solution
Implementing a method where each worker group in a multi-processor computing environment maintains a redundant copy of the AI model, allowing for quick recovery by reloading the model from a previous iteration, thereby reducing the need to revert to a global checkpoint and enabling faster error recovery without interrupting other processors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If computations are restarted after an error in a multiprocessor system, then the system can recover from the error, but large amounts of data must be recomputed which is time-consuming
Solution Approach 1:
The patent applies preliminary action by maintaining redundant copies of the AI model at multiple checkpoints during processing. When an error occurs, the system can immediately restore from the most recent checkpoint without waiting for full recomputation, thus resolving the contradiction between error recovery reliability and time loss.
Solution Approach 2:
The patent uses copying by creating and maintaining redundant copies of the AI model state at different iteration checkpoints. These copies are stored in memory or storage, allowing rapid restoration when errors occur, eliminating the need to recompute entire datasets and significantly reducing recovery time while maintaining system reliability.
2Reliability
If a global checkpoint is used for error recovery, then all processors can restore to a known good state, but processing must be interrupted across the entire system
Solution Approach 1:
The patent applies segmentation by allowing individual worker groups to independently restore from their own checkpoints when errors occur, rather than requiring a system-wide global checkpoint. This enables isolated error recovery where only affected processors interrupt processing, while other worker groups continue operating, thus maintaining system throughput while achieving reliable error recovery.
Solution Approach 2:
The patent implements local quality by enabling each worker group to maintain and restore from its own local checkpoints independently. This allows localized error recovery where only the specific worker group experiencing errors needs to restore, while other worker groups continue processing without interruption, preserving overall system productivity while ensuring reliable recovery for affected components.
3Loss of time
If redundant model copies are maintained for each worker group, then error recovery speed is improved, but memory and storage requirements increase
Solution Approach 1:
The patent applies partial action by maintaining redundant model copies only at critical iteration checkpoints rather than continuously throughout processing. This selective checkpointing approach provides sufficient recovery capability for most error scenarios while limiting the quantity of redundant data stored, thus balancing recovery speed improvement with controlled resource consumption.
Data Source
AI summary
Embodiments of the present disclosure include an error recovery method comprising detecting a computing error, restarting a first artificial intelligence processor of a plurality of artificial intelligence processors processing a data set, and loading a model in the artificial intelligence processor, wherein the model corresponds to a same model processed by the plurality of artificial intelligence processors during a previous processing iteration by the plurality of artificial intelligence processors on data from the data set.