Neural Network Weight Pruning via Dynamic Soft Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for pruning neural networks, such as hard-masking and soft-masking, often result in accuracy degradation due to the inability to control the sparsity and pruning pace effectively, leading to inefficient reduction of unnecessary weight values.
Innovation Solution
A method for pruning neural networks that involves identifying weights to prune based on a sparsity target and weight threshold, applying a dynamic pruning factor over multiple training epochs, and removing weights that fall below a threshold value, allowing for user-controlled sparsity and pruning pace.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If hard-masking technique is used to prune weights by setting them to zero, then the network size is reduced, but accuracy degradation occurs
Solution Approach 1:
The patent applies dynamics by transitioning from static hard-masking to dynamic soft-masking, where the mask values evolve continuously during training rather than being fixed at zero. The soft mask allows weights to be gradually reduced while maintaining network functionality, and the dynamic update mechanism adapts the pruning intensity based on training progress, thereby reducing accuracy degradation while achieving compression.
Solution Approach 2:
The patent changes the parameter state of weight masking from discrete (hard mask at zero) to continuous (soft mask with varying values). By introducing soft mask values that can take any value between 0 and 1 and updating them dynamically during training, the system achieves a smoother transition that preserves more information while still reducing network size, thus improving accuracy retention.
2Reliability
If soft-masking technique is used to reduce weights toward zero, then accuracy degradation is reduced, but user control over sparsity and pruning pace is lost
Solution Approach 1:
The patent implements feedback mechanisms through loss functions that monitor both the sparsity target and pruning pace during training. The system continuously evaluates the current state of weight reduction and adjusts the soft mask updates based on whether the sparsity target is being met and whether the pruning pace is appropriate, giving users indirect control through configurable targets while maintaining the benefits of soft-masking.
3Device complexity
If weights are pruned without configurable sparsity target, then pruning process is simpler, but calibration difficulty increases
Solution Approach 1:
The patent applies preliminary action by establishing configurable sparsity targets and pruning pace parameters before the pruning process begins. Users can pre-specify the desired level of compression and the rate at which it should occur, allowing for proper calibration and planning before actual pruning starts, thus reducing calibration difficulty while maintaining reasonable process complexity.
4Device complexity
If weights are pruned without configurable pruning pace, then the method is simpler, but accuracy degradation increases
Solution Approach 1:
The patent applies dynamics by making the pruning pace configurable and adaptable during training. Users can specify the desired pruning pace, and the system dynamically adjusts the soft mask update rates to match this pace while monitoring accuracy. This allows the method to remain relatively simple in implementation while achieving better accuracy preservation through controlled, gradual pruning.
Data Source
AI summary
Various embodiments of the present disclosure relate to pruning the data of a neural network, and in particular, to removing the unnecessary weights from the various channels of the neural network. In one example embodiment, a technique for pruning the weights of a neural network is provided. The technique first includes identifying weights to prune from a channel of the neural network based on a sparsity target and a weight threshold. Once identified, the technique includes determining a pruning factor for pruning the identified weights based on a current training epoch, an initial training epoch, a final training epoch, and a desired pruning pace. Next, the technique includes, over multiple training epochs, reducing each of the identified weights by multiplying the identified weights by the pruning factor. Finally, the technique includes removing the identified weights from the channel that have been reduced to below a threshold value.