Neural Network Pathway Pruning for Model Size Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pruning neural networks to reduce model size is challenging when learning is unstable, especially in networks with multiple pathways, as it leads to unstable pruning behavior and inefficient size reduction.
Innovation Solution
A learning apparatus and method that trains neural networks with multiple pathways, performs pruning by calculating the number of remaining parameters, generates candidate models by deleting pathways below a threshold, and determines further deletions or re-training based on pruning history, allowing for iterative pruning and reconstruction to achieve efficient model size reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning is applied to a neural network with multiple pathways, then model size is reduced, but learning stability deteriorates and pruning behavior becomes unstable
Solution Approach 1:
The patent segments the neural network into multiple independent pathways (first pathway, second pathway, etc.) that can be processed separately. Each pathway is pruned independently based on its own parameter importance metrics, allowing stable optimization of each segment without the instability that arises from pruning the entire multi-pathway network as a single unit.
Solution Approach 2:
The patent applies partial pruning by selectively removing only certain parameters (weights, biases, or channels) within each pathway based on their importance scores, rather than pruning the entire network uniformly. This partial action allows the model to maintain stability while achieving size reduction in specific pathways.
2Productivity
If pruning removes parameters to reduce model size, then efficiency is improved, but learning stability deteriorates making efficient pruning difficult
Solution Approach 1:
The patent calculates parameter importance metrics and identifies parameters for removal before actually performing the pruning operation. By pre-evaluating which parameters to remove based on their contribution to the loss function and gradient norms, the system can efficiently plan the pruning strategy without causing instability during the actual pruning process.
Solution Approach 2:
The patent uses feedback from the training process by monitoring parameter importance metrics (such as gradient norms and loss contributions) to dynamically adjust which parameters are pruned. This feedback mechanism allows the system to maintain learning stability by making pruning decisions based on real-time performance indicators rather than static criteria.
Data Source
AI summary
According to one embodiment, a learning apparatus includes a processor. The processor trains a neural network model having a plurality of pathways and generate a trained model. The processor performs pruning on the trained model and calculate a number of remaining parameters of each of the pathways. The processor generates a candidate model for reconstruction, the candidate model for reconstruction being generated by deleting a pathway in which the number of parameters is equal to or less than a threshold. The processor determines whether or not deletion of a further pathway included in the candidate model for reconstruction is possible. If it is determined that deletion of the further pathway is possible, the candidate model for reconstruction is subjected to each of the training, the pruning, and the generating.