Stochastic Weight Perturbation for DNN Sparsity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require significant computational resources and storage, and existing methods like weight pruning and quantization are either time-consuming or inflexible, limiting their deployment efficiency.
Innovation Solution
Applying a stochastic weight perturbation technique during training using a distribution function, such as an exponential decay distribution, to increase the sparsity of weights, reducing the number of multiplications in convolution operations and enhancing model robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If weight pruning is applied to reduce computational resources, then the number of multiplications decreases, but the training time increases significantly
Solution Approach 1:
The patent changes the parameter update rule from continuous gradient descent to discrete updates based on weight sign and sampled magnitude. This parameter change enables the model to achieve sparsity without requiring extensive iterative pruning operations, thus reducing training time while maintaining computational efficiency.
Solution Approach 2:
The patent applies preliminary random initialization to weight magnitudes and uses sign-based update rules during training to naturally drive weights toward sparsity. This preliminary action eliminates the need for post-training pruning operations, reducing both training time and computational resources required.
2Quantity of substance
If quantization is applied to reduce storage requirements, then memory usage decreases, but model flexibility and precision are reduced
Solution Approach 1:
The patent implements dynamic sparsity where the sparse structure is learned during training through iterative optimization rather than being fixed beforehand. This dynamic approach allows the model to adapt its sparse structure to the specific task, maintaining flexibility while achieving storage efficiency through learned sparsity patterns.
3Quantity of substance
If compression techniques are applied to reduce model size, then deployment requirements are relaxed, but reconstruction accuracy may deteriorate
Solution Approach 1:
The patent enables the model to self-organize into a sparse structure during training through the proposed optimization algorithm. The model automatically identifies and retains important weights while pruning less important ones, achieving compression without external intervention or complex reconstruction processes, thus maintaining accuracy while reducing model size.
4Measurement precision
If extensive training is performed to improve accuracy, then prediction accuracy increases, but computational cost and time increase
Solution Approach 1:
The patent changes the optimization landscape by using sign-based update rules with sampled magnitudes, which creates a more efficient training dynamics. This parameter change allows the model to converge to accurate solutions with fewer iterations and less computational power compared to traditional continuous optimization methods.
Data Source
AI summary
A method includes: providing a deep neural networks (DNN) model comprising a plurality of layers, each layer of the plurality of layers includes a plurality of nodes; sampling a change of a weight for each of a plurality of weights based on a distribution function, each weight of the plurality of weights corresponds to each node of the plurality of nodes; updating the weight with the change of the weight multiplied by a sign of the weight; and training the DNN model by iterating the steps of sampling the change and updating the weight. The plurality of weights has a high rate of sparsity after the training.