Heterogeneous Continual Learning Architecture Progression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional continual learning techniques require machine learning models to maintain the same architecture when updated, preventing knowledge transfer to models with different architectures and resulting in computationally expensive and storage-intensive retraining from scratch.
Innovation Solution
The method involves generating a new data set using a previously trained model with a different architecture, allowing for training of a new model with a different architecture by optimizing an objective function that minimizes distance between outputs and ensures correct predictions, eliminating the need for storing original training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional continual learning techniques are used to update a previously trained machine learning model, then the model can be updated with new data, but the architecture of the model must remain the same and knowledge cannot be transferred to models with different architectures
Solution Approach 1:
The patent uses an intermediary representation layer that decouples the source model architecture from the target model architecture. This intermediary allows knowledge to be transferred without requiring the target model to have the same architecture as the source model, resolving the contradiction between adaptability and architectural constraint.
Solution Approach 2:
The patent segments the knowledge transfer process into distinct components: feature extraction from the source model, representation learning in an intermediary layer, and adaptation to the target model architecture. This segmentation allows each component to be optimized independently, enabling architecture flexibility while maintaining effective knowledge transfer.
2Reliability
If a second machine learning model is trained from scratch using new data and the data that was used to train the previously trained machine learning model, then the model can learn all necessary patterns, but computational cost and storage requirements increase significantly
Solution Approach 1:
The patent performs preliminary action by pre-training a source model on the original training data before knowledge transfer. This preliminary training extracts valuable patterns and features that can then be transferred to the target model, eliminating the need to re-process the original training data and significantly reducing computational costs while maintaining training completeness.
Solution Approach 2:
The patent creates a copy of knowledge representations from the source model and transfers them to the target model. Instead of re-training from scratch, the target model receives initialized representations that capture essential patterns, reducing computational requirements while maintaining the reliability of comprehensive training.
3Reliability
If a second machine learning model is trained from scratch using all training data including historical data, then the model achieves complete learning, but storage space is required to store all training data for long periods
Solution Approach 1:
The patent extracts essential knowledge and patterns from the historical training data through the source model during pre-training. By taking out only the critical learned representations rather than storing the entire original dataset, the system achieves complete learning reliability while dramatically reducing storage requirements.
Solution Approach 2:
The patent creates compressed copies of knowledge in the form of model parameters and representations rather than storing the original training data. These copied representations contain the essential learned information needed for complete training, eliminating the need to retain large volumes of original training data.
Data Source
AI summary
One embodiment of a method for training a first machine learning model having a different architecture than a second machine learning model includes receiving a first data set, performing one or more operations to generate a second data set based on the first data set and the second machine learning model, wherein the second data set includes at least one feature associated with one or more tasks that the second machine learning model was previously trained to perform, and performing one or more operations to train the first machine learning model based on the second data set and the second machine learning model.