Incremental Classification Models Using Orthogonal Embedding Expansion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing classification and detection models face the challenge of catastrophic forgetting when new classes are introduced, leading to a deterioration in performance on existing classes due to retraining, which is often computationally difficult and time-consuming, especially when the base training dataset is not available.
Innovation Solution
The technique involves expanding the embedding space by adding orthogonal subspaces for new classes, freezing the base embedding space, and using a distillation process to maintain performance on existing classes, allowing incremental learning without forgetting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the model is retrained with new classes, then the model learns new classes, but performance on existing classes deteriorates due to catastrophic forgetting
Solution Approach 1:
The patent lifts the base embedding space to a higher-dimensional space by adding an orthogonal subspace. This dimensional expansion allows new classes to be represented in the added dimensions while preserving the original base classes in the lower-dimensional subspace, enabling the model to learn new classes without forgetting existing ones.
Solution Approach 2:
The embedding space is segmented into two distinct parts: the original base embedding space that preserves existing class information, and a newly added orthogonal subspace that accommodates new classes. This segmentation allows independent learning for new classes without interfering with the representation of base classes.
2Adaptability or versatility
If the model is retrained with new classes, then the model learns new classes, but the retraining process is computationally difficult and time-consuming
Solution Approach 1:
The base embedding space is lifted to a higher-dimensional space in advance, creating the orthogonal subspace structure beforehand. This preliminary action enables incremental learning of new classes without requiring computationally intensive retraining of the entire model, as the architectural framework is already in place to accommodate new classes.
3Adaptability or versatility
If the base embedding space is expanded to include new classes, then the model can detect new classes, but the dimensionality of the embedding vectors increases
Solution Approach 1:
Instead of expanding the existing embedding space uniformly, the patent adds a separate orthogonal subspace with dimensions that are perpendicular to the original base embedding space. This approach increases dimensionality only in the directions needed for new classes while preserving the original dimensional structure for base classes, making the complexity increase more manageable and structured.
Data Source
AI summary
A computing system, computer program product, and computer-implemented method for incremental learning without forgetting for a classification/detection model are provided. The method includes receiving, at a computing system, a classification/detection model including a base embedding space and corresponding base embedding vectors that are based on a base training dataset including base classes. The method also includes expanding the classification/detection model to account for a new training dataset including new classes by lifting the base embedding space to add an orthogonal subspace for the new classes, producing an expanded embedding space and corresponding expanded embedding vectors that are of a higher dimension than the base embedding vectors. In some embodiments, the method also includes further expanding the expanded classification/detection model to account for another new training dataset including additional new classes by lifting the expanded embedding space to add another orthogonal subspace for the additional new classes.