Neural Network Pruning via Layer-Specific Attention Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks (NNs) with complex configurations achieve high performance but require significant computational resources and memory, and existing pruning methods to reduce these requirements often degrade inference accuracy if excessively applied.
Innovation Solution
A method for pruning NNs by selectively reducing elements in attention mechanisms, specifically using L1 regularization to identify channels with small scaling factors in convolutional layers connected to Batch Normalization layers, while maintaining inference accuracy through careful determination of pruning rates for each layer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If pruning is applied to reduce the number of elements in neural network layers, then the size of memory and calculation durations are reduced, but inference accuracy degrades
Solution Approach 1:
The patent applies different pruning rates to different layers of the neural network based on their specific characteristics. The Q layer, K layer, and V layer in the attention mechanism are pruned at different rates optimized for each layer's function, rather than applying a uniform pruning rate across all layers. This localized approach maintains inference accuracy while achieving compression.
Solution Approach 2:
The patent prunes only a portion of the elements in each layer rather than removing excessive elements. By carefully determining pruning rates (e.g., 20% for Q layer, 30% for K layer, 50% for V layer) and using L1 regularization to identify which elements to remove, the method achieves compression without significantly degrading inference accuracy.
2Quantity of substance
If pruning is applied to reduce the data size of machine learning models, then the size of memory is reduced, but inference accuracy degrades
Solution Approach 1:
The patent determines appropriate pruning rates for each layer based on its specific characteristics and importance to the model's function. Different layers (Q, K, V layers) are pruned at different rates optimized for their respective roles in the attention mechanism, rather than applying a uniform pruning rate. This localized approach maintains inference accuracy while achieving compression.
Solution Approach 2:
The patent uses L1 regularization during the training phase to preliminarily identify which elements (channels in convolutional layers) should be pruned. By adding an L1 penalty term to the loss function, the method pre-determines the structure of the pruned model before actual pruning occurs, ensuring that only non-critical elements are removed while maintaining inference accuracy.
3Reliability
If complex configurations are used in neural networks, then inference accuracy is improved, but the number of times of calculation and memory size increase
Solution Approach 1:
The patent extracts and removes redundant or less important elements from the neural network's attention mechanism. By applying pruning to the Q, K, and V layers with optimized rates, the method removes unnecessary computational components while preserving the essential structure needed for high inference accuracy, thus reducing device complexity.
Solution Approach 2:
The patent changes the parameters of the neural network by applying different pruning rates to different layers. This parameter adjustment optimizes the balance between model complexity and inference accuracy, reducing the number of calculations and memory requirements while maintaining performance through carefully tuned compression ratios for each layer.
Data Source
AI summary
A method including for an element of each of a Q layer and a K layer respectively outputting a Query and a Key, the Query and the Key being a result of an arithmetic operating process on an input tensor in an attention mechanism in a trained machine learning model of a neural network, deleting an element included in at least one of a tensor QT and a tensor KT such that elements having a same index are left in the tensor QT and the tensor KT from among one or more elements included in the tensor QT included in a reduced Q layer in which one or more elements are reduced based on a first reduction ratio and one or more elements included in the tensor KT included in a reduced K layer in which one or more elements are reduced based on a second reduction ratio.