Neural Network Filter Pruning via Learnable Scaling Factors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network (NN) compression techniques fail to balance computational cost and accuracy, particularly in resource-constrained environments, as conventional pruning methods either compromise on computational power or inference performance.
Innovation Solution
A neural network block is designed with a learnable scaling factor and mask function, where the scaling factor is trained during backward pass to prune filters based on a regularization function, enabling structural sparsity and reducing computational costs without sacrificing accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If conventional neural network compression techniques are applied, then computational cost is reduced, but inference accuracy deteriorates
Solution Approach 1:
The patent applies preliminary action by performing filter pruning during the training phase rather than after training. The scaling factors are learned concurrently with network weights, allowing the network to adapt to the pruned structure in advance. This prevents the accuracy deterioration that occurs when pruning is applied post-training, as the network has already adjusted its remaining filters to compensate for removed filters.
Solution Approach 2:
The patent introduces scaling factors as additional learnable parameters that control filter importance. By learning these scaling factors alongside network weights, the system dynamically adjusts filter contributions during training. This parameter change enables continuous optimization of both computational efficiency (through pruning) and accuracy (through adaptive scaling), resolving the trade-off between the two.
2Manufacturing precision
If more filters are retained to maintain accuracy, then memory requirements increase
Solution Approach 1:
The patent transforms the discrete decision of whether to keep or remove a filter into a continuous optimization problem by introducing learnable scaling factors. These parameters allow the network to express filter importance continuously, enabling fine-grained control over the balance between memory usage and accuracy. The pruning threshold applied to these parameters provides a memory-efficient representation while maintaining accuracy through the learned parameter values.
3Manufacturing precision
If neural network depth is increased to improve accuracy, then computational complexity increases
Solution Approach 1:
The patent extracts and removes redundant filters from the neural network during training based on learned scaling factors. By identifying and removing these unnecessary computational elements, the network achieves the same or better accuracy with reduced computational complexity. This extraction process eliminates redundant operations while preserving the essential computational pathways needed for accurate inference.
Data Source
AI summary
A method and system for pruning a neural network (NN) block of a neural network during training, wherein the NN block comprises: a convolution operation configured to convolve an input feature map with a plurality of filters, each filter including a plurality of weights, to generate a plurality of filter outputs each corresponding to a respective filter; an activation operation configured to generate, for each of the filter outputs, a respective non-linearized output; a scaling operation configured to scale the non-linearized output generated in respect of each filter by multiplying the non-linearized output with a mask function and a respective scaling factor that corresponds to the filter. During training: for each scaling factor corresponding to a filter, learning the scaling factor by minimizing loss of a loss function including a first regularization function with respect to the scaling factor; and if a value of the scaling factor satisfies a predetermined criterion, selectively pruning the filter corresponding to the scaling factor by masking the filter from the convolution operation.