Information-Bottleneck Neural Network Updates Against Catastrophic Forgetting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional neural networks face the issue of 'Catastrophic Forgetting' when adapting to new tasks, leading to a loss of performance on previous tasks due to the 'Stability-Plasticity Dilemma', and existing solutions like Regularization-based and Replay-based methods fail to adequately address the 'representation drift' and storage space requirements.
Innovation Solution
A sparsity-promoting neural network model that maintains an active network part for previous tasks and adaptively expands capacity by reactivating inactive neurons and expanding layers when needed, using an information bottleneck to determine neuron activation and minimize storage requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the neural network is retrained based on new input data to adapt to new tasks, then the neural network can give a response to the new dataset, but the neural network may not give a response to the original dataset (performance on historical tasks deteriorates)
Solution Approach 1:
The patent segments the neural network into active regions and inactive regions based on information bottleneck analysis. The active region maintains stability for historical tasks while the inactive region adapts to new tasks, resolving the contradiction between adaptability and reliability through spatial segmentation of the network architecture.
Solution Approach 2:
The patent dynamically adjusts the network configuration by identifying and freezing active regions while allowing inactive regions to be updated. This dynamic approach enables the network to adapt to new tasks while preserving performance on historical tasks, balancing adaptability and reliability through runtime region identification and selective updating.
2Stability of the object's composition
If the neural network maintains performance on historical tasks while learning new knowledge, then the stability is improved, but the ability to learn new knowledge (plasticity) is restricted
Solution Approach 1:
The patent divides the neural network into stable active regions and plastic inactive regions. The active region maintains stability by freezing parameters, while the inactive region provides plasticity for learning new knowledge. This segmentation simultaneously achieves both stability and plasticity by assigning different functional roles to different network regions.
Solution Approach 2:
The patent applies different quality characteristics to different parts of the network: the active region has high stability with frozen parameters, while the inactive region has high plasticity with trainable parameters. This local differentiation of properties enables the network to exhibit both stability and plasticity in appropriate regions simultaneously.
3Adaptability or versatility
If the neural network expands capacity to learn new information, then the adaptability to new tasks is improved, but the storage space requirements increase significantly
Solution Approach 1:
The patent dynamically identifies inactive regions that can be updated with new information without requiring significant network expansion. By selectively activating and updating specific regions based on information bottleneck analysis, the network achieves adaptability to new tasks while minimizing storage space requirements through targeted rather than comprehensive updates.
Data Source
AI summary
A neural network update method, a classification method and an electronic device thereof are provided. The neural network update method includes obtaining input data of the neural network; identifying an active neuron and an inactive neuron from among at least one neuron included in the neural network; and based on an information bottleneck of the at least one neuron, training the neural network based on the input data while maintaining information bottleneck of the active neuron.