Neural Network Node Partitioning for Continual Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks (CNNs) suffer from catastrophic forgetting, where precision for old tasks decreases as new tasks are learned, leading to the forgetting of previous knowledge during continual learning.
Innovation Solution
A classification apparatus and method that classifies nodes in a neural network into stable and plastic nodes, maintaining paths between adjacent layers to retain previous knowledge while adapting to new tasks, by connecting stable nodes in specific layers and adjusting connections to enhance information transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If CNN parameters are re-learned in response to the entirety of the dataset to adapt to changes in data distribution, then the network can learn new tasks, but the precision for old tasks decreases due to catastrophic forgetting
Solution Approach 1:
The patent applies local quality by differentiating between stable nodes and plastic nodes within the neural network. Stable nodes maintain their connection states to preserve old task knowledge, while plastic nodes are allowed to change connections to learn new tasks. This localized differentiation of node properties resolves the contradiction by allowing different parts of the network to have different adaptability characteristics.
Solution Approach 2:
The patent segments the neural network nodes into two distinct categories: stable nodes and plastic nodes. This segmentation allows the network to simultaneously maintain stability for old tasks and plasticity for new tasks, resolving the catastrophic forgetting problem by separating the functional roles of different nodes.
2Adaptability or versatility
If paths across nodes are removed or added during learning to emulate memory mechanisms, then new knowledge can be acquired, but the connection density changes which may affect network performance
Solution Approach 1:
The patent applies local quality by allowing connection changes only at plastic nodes while maintaining stable connections at stable nodes. This localized approach to connection modification enables new knowledge acquisition without disrupting the overall network structure and performance stability.
Solution Approach 2:
The patent implements dynamics by making the connection state of nodes adaptive - stable nodes maintain fixed connections while plastic nodes dynamically adjust their connections based on learning requirements. This dynamic approach allows the network to adapt to new knowledge while preserving reliable pathways for existing functions.
3Device complexity
If disconnection is performed between certain layers to reduce complexity, then the network structure is simplified, but the estimation accuracy is significantly reduced due to loss of coadaptation between layers
Solution Approach 1:
The patent applies local quality by selectively maintaining connections between specific layers where coadaptation is critical. Rather than uniformly disconnecting layers, the patent preserves connections in layers that have developed inter-layer dependencies, thereby maintaining estimation accuracy while still reducing overall complexity.
Data Source
AI summary
A classification apparatus includes: a feature extraction unit subjected to training, which includes removing or adding a path across nodes between adjacent layers in a neural network, and adapted to extract a feature quantity of input data; and a classification unit that retains a classification weight of each class and classifies the input data based on the feature quantity and the classification weight in response to the feature quantity as an input. Learning includes classifying a plurality of nodes in the neural network into a stable node and a plastic node having a lower activation than the stable node and connecting the stable node and the plastic node in the case that the stable node is present in a predetermined layer in the neural network and the plastic node is present in a layer next to the predetermined layer.