ML Model Recovery Using Parameter Journals and Backup Checkpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training machine learning models is resource-intensive and prone to corruption or loss, leading to inefficiencies and downtime due to the need to recreate models from scratch, and maintaining full copies of prior versions consumes significant storage resources.
Innovation Solution
Maintain different checkpoints for a machine learning model using a journal that stores only modified parameters during training, combined with a backup representation, allowing recovery of a target version by applying selected journal entries to the backup.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full copies of prior versions of the machine learning model are maintained for recovery, then the ability to recover the model is improved, but the storage resources consumed increase significantly
Solution Approach 1:
The patent extracts only the modified parameters from each training iteration and stores them in a journal, rather than storing complete model copies. This allows recovery of previous versions by applying stored parameter changes to a base model, dramatically reducing storage requirements while maintaining recovery capability.
Solution Approach 2:
The patent segments the model recovery process into two parts: a base model representation and a journal of parameter changes. Each training iteration is segmented into discrete parameter modifications that can be independently stored and applied, enabling efficient recovery without storing complete versions.
2Reliability
If the machine learning model is recreated from scratch after loss or corruption, then data integrity is restored, but the time and resources required increase
Solution Approach 1:
The patent performs preliminary actions by continuously logging parameter changes during training iterations. This preparation enables rapid recovery by simply applying the logged changes to a base model, rather than requiring complete recreation of the model from scratch after loss or corruption.
3Reliability
If multiple training iterations are performed to improve model accuracy, then the model performance is improved, but the risk of corruption or loss increases
Solution Approach 1:
The patent performs preliminary action by logging parameter changes during each training iteration before they are permanently applied. This creates a recovery trail that enables restoration of previous accurate versions if corruption or loss occurs during subsequent training iterations.
Solution Approach 2:
The patent provides beforehand cushioning by maintaining a journal of parameter changes that can be used to restore previous model states. This cushioning protects against the harmful effects of corruption or loss during training by enabling recovery to a known good state.
Data Source
AI summary
In some examples, a system replicates modified parameters of a machine learning model to a journal, where the modified parameters relate to elements of a graph structure of the machine learning model, and the modified parameters in the journal are to be applied to a backup representation of the machine learning model. Based on receipt of a query associated with recovering a version of the machine learning model, the system builds the version of the machine learning model by retrieving a selected modified parameter from among the modified parameters in the journal and merge the selected modified parameter with a copy of the machine learning model represented by the backup representation of the machine learning model.