Machine Learning Failure Recovery via Intermediate Model Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning failure during workload processing requires restarting from the beginning, leading to high resource usage and prolonged processing times, which is inefficient and costly.
Innovation Solution
A machine learning failure recovery apparatus and control method that involves matching and storing intermediate storage models at preset backup times, dynamically adjusting the backup time interval based on factors like data type, remaining data, and failure cause, and resuming learning from the closest intermediate model and location when a failure occurs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning is restarted from the beginning after a failure, then the learning process can be completed with full data, but the processing time and resource usage increase significantly
Solution Approach 1:
The system performs preliminary actions by saving intermediate storage models at preset backup time intervals during the machine learning process. When a failure occurs, these pre-saved models enable recovery without restarting from the beginning, thus reducing processing time while ensuring learning completion reliability.
Solution Approach 2:
The system discards the failed learning process and recovers by loading the most recent intermediate storage model from backup. This allows the learning to resume from the last successful checkpoint rather than restarting entirely, resolving the contradiction between reliability and time loss.
2Reliability
If machine learning is restarted from the beginning after a failure, then the learning process can be completed with full data, but the resource usage increases significantly
Solution Approach 1:
The system performs preliminary actions by saving intermediate storage models at preset backup time intervals during the machine learning process. When a failure occurs, these pre-saved models enable recovery without restarting from the beginning, thus reducing resource usage while ensuring learning completion reliability.
Solution Approach 2:
The system discards the failed learning process and recovers by loading the most recent intermediate storage model from backup. This allows the learning to resume from the last successful checkpoint rather than restarting entirely, resolving the contradiction between reliability and resource usage.
3Loss of time
If intermediate storage models are saved at frequent intervals, then the recovery point is closer to the failure point, but the storage requirements and processing overhead increase
Solution Approach 1:
The system dynamically adjusts the backup time interval based on the type of learning data, amount of remaining learning data, and type and shape of machine learning model. This dynamic adjustment optimizes the balance between recovery time and storage requirements, saving models more frequently when critical and less frequently when stable.
Solution Approach 2:
The system changes the parameter of backup time interval dynamically based on learning progress and data characteristics. By adjusting this parameter, the system optimizes both recovery efficiency and storage utilization, resolving the contradiction between recovery time and storage requirements.
Data Source
AI summary
A machine learning failure recovery apparatus and control method thereof are provided. The control method of a machine learning failure recovery apparatus according to the present invention t may include performing machine learning with learning data as an input, wherein the machine learning includes matching a learning data location where a learning has been completed with an intermediate storage model whenever a preset backup time arrives and storing a result of the matching; determining whether a failure occurs during the machine learning; extracting the intermediate storage model closest to a point in time when the failure occurred and a position of the learning data matched with the corresponding intermediate storage model when it is determined that the failure occurred; and resuming machine learning based on the extracted intermediate storage model and location of the learning data.