Incremental Learning via Stored Representative Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Incremental learning in machine learning models often results in catastrophic forgetting, where models lose previously learned knowledge due to the unavailability of old class training data, especially in online learning scenarios where new class data becomes available sequentially, leading to memory inefficiencies and potential bias towards new data.
Innovation Solution
The technique involves storing representative features extracted during training and using them for both training and testing, combined with data augmentation techniques to modify these features, allowing the model to retain learned knowledge and adapt to new classes without requiring extensive retraining on old data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If incremental learning is performed with new class data becoming available sequentially, then the model can adapt to new classes over time, but the model loses previously learned knowledge from older classes due to unavailability of old class training data
Solution Approach 1:
The patent extracts representative features from old class training data and stores them separately. These extracted features capture the essential characteristics of old classes without requiring the full original training data to be retained, thus preventing catastrophic forgetting while enabling incremental learning of new classes.
Solution Approach 2:
The patent performs preliminary feature extraction and storage from old class data before incremental learning begins. By preparing these representative features in advance, the system ensures that old class knowledge is preserved and can be readily accessed during subsequent incremental learning steps to prevent forgetting.
2Reliability
If old class training data is stored to prevent catastrophic forgetting, then the model can retain previously learned knowledge, but the memory footprint increases significantly
Solution Approach 1:
Instead of storing complete old class training datasets, the patent extracts only the essential representative features from this data. This extraction process dramatically reduces the memory footprint while preserving the critical information needed to prevent catastrophic forgetting during incremental learning.
Solution Approach 2:
The patent applies local quality by storing only the most representative features from old class data rather than all training data. This selective storage approach focuses resources on preserving the most critical knowledge elements, achieving effective retention with minimal memory requirements.
3Reliability
If the model is retrained extensively on old data to prevent forgetting, then the model retains previously learned knowledge, but the training time and computational resources increase
Solution Approach 1:
The patent extracts representative features from old class data and stores them for efficient retrieval during incremental learning. This approach eliminates the need for extensive retraining on original old data, as the extracted features serve as a compact proxy that preserves old class knowledge while enabling fast adaptation to new classes.
Solution Approach 2:
By performing feature extraction from old class data in advance and storing these representations, the patent prepares the system for efficient incremental learning. This preliminary action eliminates the need for time-consuming retraining operations during subsequent incremental learning steps.
4Reliability
If representative features from old classes are stored and used during incremental learning, then catastrophic forgetting is prevented, but the device complexity increases
Solution Approach 1:
The patent extracts representative features from old class training data and stores them in a separate repository. This extraction creates a simple yet effective mechanism for preventing catastrophic forgetting by making old class knowledge accessible during incremental learning without requiring complex architectural modifications.
Solution Approach 2:
The stored representative features act as an intermediary between old class knowledge and the incremental learning process. This intermediary enables the model to access and utilize old class information during learning of new classes, preventing forgetting while maintaining a relatively simple system architecture.
Data Source
AI summary
Computerized techniques are provided for storing a set of representative features for use with a machine learning process. A first set of training images for a class are accessed and processed, using a feature extraction technique, to determine a set of features for the class. A set of representative features from the set of features for the class is determined using a selection technique. The set of representative features are stored for the class. A second set of images is accessed. The machine learning process is executed, using the stored set of representative features, including executing one or more of: a training phase to train a classifier using the second set of images and at least part of the stored set of representative features; and a testing phase to run the classifier using the second set of images and at least part of the stored set of representative features.