Neural Network Weight Pruning with Cyclical Sparsity Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Dense weight tensors in neural networks require more computational resources and memory but can decrease accuracy, while sparser tensors increase error and reduce accuracy, necessitating a balance between resource efficiency and accuracy.
Innovation Solution
Implement cyclical sparsity by alternating between increasing and decreasing the sparsity of weight tensors through masking and training to achieve a desired level of accuracy with reduced resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If the weight tensor is made sparser to reduce computational resources and memory, then resource efficiency improves, but accuracy deteriorates
Solution Approach 1:
The patent applies periodic action by cycling between increasing sparsity (pruning phase) and decreasing sparsity (densifying phase) multiple times. This cyclical process allows the network to progressively achieve higher sparsity while maintaining accuracy through repeated refinement iterations, resolving the contradiction between resource efficiency and accuracy.
Solution Approach 2:
The patent changes the sparsity parameter dynamically throughout the training process. By adjusting the sparsity level in cycles and using different sparsity targets for different phases, the method enables the network to achieve high sparsity without permanently compromising accuracy, as the parameter evolves adaptively rather than being fixed.
2Quantity of substance
If the weight tensor is made sparser to reduce memory storage, then memory requirements decrease, but error increases
Solution Approach 1:
The cyclical pruning and densifying process allows the weight tensor to achieve high sparsity for memory efficiency while periodically recovering important weights. This periodic recovery mechanism reduces the error rate that would otherwise result from permanent sparsification, resolving the contradiction between memory storage and error rate.
Solution Approach 2:
The patent implements discarding and recovering by temporarily setting weights to zero (discarding) during pruning phases, then recovering important weights during densifying phases. This cyclic discarding and recovery process enables high sparsity for memory efficiency while minimizing permanent weight loss, thereby reducing error rate.
3Use of energy by moving object
If cyclical sparsity is applied to achieve high sparsity levels, then resource costs are reduced, but training complexity increases
Solution Approach 1:
The patent segments the training process into distinct phases (pruning phase and densifying phase) that are repeated cyclically. Each phase has a specific sparsity target and duration, making the complex training process more manageable and systematic. This segmentation resolves the contradiction by organizing complexity into structured, repeatable units.
Solution Approach 2:
The patent introduces dynamics by making the sparsity level and training configuration adaptive rather than static. The cyclical nature allows the system to dynamically adjust between high-sparsity and low-sparsity states, optimizing resource costs at different training stages while managing complexity through systematic variation rather than fixed complexity.
Data Source
AI summary
Various embodiments include methods and devices for neural network pruning. Embodiments may include receiving as an input a weight tensor for a neural network, increasing a level of sparsity of the weight tensor generating a sparse weight tensor, updating the neural network using the sparse weight tensor generating an updated weight tensor, decreasing a level of sparsity of the updated weight tensor generating a dense weight tensor, increasing the level of sparsity of the dense weight tensor the dense weight tensor generating a final sparse weight tensor, and using the neural network with the final sparse weight tensor to generate inferences. Some embodiments may include increasing a level of sparsity of a first sparse weight tensor generating a second sparse weight tensor, updating the neural network using the second sparse weight tensor generating a second updated weight tensor, and decreasing the level of sparsity the second updated weight tensor.