Neural Network Weight Pruning Using Update Count Signals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional pruning methods for deep neural networks (DNNs) often remove important weights with small initial values due to their decreased magnitude after many training iterations, leading to inefficient computational speed and model size reduction while maintaining accuracy.
Innovation Solution
A method and apparatus that prioritize weight removal based on the number of updates rather than magnitude, counting updates for each epoch and removing weights with fewer updates to maintain accuracy and reduce model size efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional weight magnitude-based pruning is used, then model size is reduced, but important weights with small initial values are incorrectly removed leading to accuracy loss
Solution Approach 1:
The patent changes the pruning criterion from weight magnitude to number of updates. Instead of using the absolute value of weights as the pruning metric, the invention tracks how many times each weight has been updated during training and uses this update count as the new criterion for determining which weights to remove. This parameter change resolves the contradiction by providing a more accurate indicator of weight importance that doesn't suffer from the initial value bias problem.
2Productivity
If weights with small magnitude are pruned first, then sparsity is achieved faster, but computational speed improvement is limited due to removal of important weights
Solution Approach 1:
The patent implements a feedback mechanism by tracking the number of updates for each weight throughout the training process. This feedback information about weight activity and importance is then used to guide the pruning decision, ensuring that weights critical to the network's function (those with higher update counts) are preserved while less important weights are removed, thereby maintaining accuracy while achieving sparsity.
3Reliability
If large number of training iterations are performed, then accuracy is improved, but weights with large initial values may be pruned prematurely due to magnitude-based criteria
Solution Approach 1:
The patent performs preliminary tracking of weight update counts during the training process before the actual pruning decision is made. By recording the number of updates for each weight throughout training, the system prepares the necessary information in advance to make accurate pruning decisions that preserve important weights, avoiding the need for extensive retraining after pruning.
Data Source
AI summary
Provided are a method and an apparatus for pruning based on the number of updates, the method comprising counting the number of updates of weights for each epoch and removing n weights for which the number of updates is small after training of a network is completed, wherein n represents the number of weights to be removed to satisfy a desired sparsity.