Few-Shot Classification Weights to Prevent Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing continual learning methods in convolutional neural networks (CNNs) face catastrophic forgetting, where knowledge of past tasks is lost when learning new tasks, especially in classification tasks with small datasets, leading to decreased performance for both old and new classes.
Innovation Solution
A classification apparatus and method that utilizes a feature quantity output unit trained through meta learning, including path removal or addition between neural network layers, and distillation with a supervisor model, to maintain classification accuracy for both base and novel classes using smaller novel class datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If continual learning is performed by using a small dataset, then the model can adapt to new tasks with limited data, but catastrophic forgetting occurs and classification accuracy deteriorates
Solution Approach 1:
The patent creates a supervisor model that copies the structure and knowledge from the student model before training. This copied model serves as a reference to preserve original classification abilities while the student model learns new classes, preventing catastrophic forgetting through the copying mechanism
Solution Approach 2:
The supervisor model acts as an intermediary between the student model and the training process. It mediates the learning by providing guidance signals that help the student model learn new classes without forgetting old ones, resolving the contradiction between adaptability and reliability
2Quantity of substance
If sample data is duplicated to increase dataset size, then more training data is available, but overfitting occurs with good local performance but poor generalization
Solution Approach 1:
Instead of duplicating sample data, the patent copies the neural network model itself to create a supervisor model. This approach provides sufficient training signals without duplicating input samples, avoiding overfitting while maintaining generalization performance
Solution Approach 2:
The patent replaces the mechanical approach of duplicating input data with a model-based approach. Rather than mechanically copying samples, it uses model distillation where the supervisor model generates soft labels, substituting data duplication with knowledge transfer
3Reliability
If NISPA is used to retain stable node paths, then catastrophic forgetting is avoided, but the method assumes use of large-scale big data
Solution Approach 1:
The patent applies the copying principle by creating a supervisor model that preserves base class knowledge. This model copy serves as a reference during continual learning, enabling the system to maintain base class performance without requiring large-scale datasets like NISPA
Solution Approach 2:
The patent changes the approach from path-based retention (NISPA) to model-based retention through distillation. By altering the fundamental parameter of how knowledge is preserved - from structural path modification to soft label guidance - the method works effectively with small datasets
Data Source
AI summary
A classification apparatus includes: a feature quantity output unit that outputs a feature quantity of input data; and a classification unit that retains, as a classification weight, a feature quantity obtained by averaging, per each class, the feature quantity output by the feature quantity output unit in response to a base class dataset and a novel class dataset with a smaller number of data items than the base class dataset and that outputs a result of classification of the input data by using the feature quantity of the input data and the classification weight.