Neural Network Pruning via Layer-Specific L1 Regularization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for pruning neural networks (NNs) to reduce calculation and memory usage often compromise inference accuracy, especially when applied to convolutional layers without Batch Normalization (BN) layers or fully connected layers, leading to mismatched tensor dimensions and reduced pruning rates.
Innovation Solution
A machine learning program that adjusts reduction ratios for each layer in a neural network, using L1 regularization learning to identify pruning targets and ensure accurate tensor concatenation across layers, thereby maintaining inference accuracy while downsizing the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning is applied to convolutional layers without BN layers or fully connected layers using existing methods, then the model size is reduced, but tensor dimension mismatches occur and inference accuracy degrades
Solution Approach 1:
The patent applies different pruning strategies to different layer types within the neural network. Specifically, it uses L1 regularization learning to determine pruning targets for convolutional layers without BN layers and fully connected layers, while using other methods for layers with BN layers. This localized approach ensures that each layer type is pruned appropriately for its characteristics, preventing tensor dimension mismatches and maintaining inference accuracy.
Solution Approach 2:
The patent changes the pruning parameters and methods based on the specific layer type. By introducing layer-type-specific pruning parameters (such as different regularization strengths and pruning ratios), it optimizes the pruning process for each layer category, thereby reducing model size without compromising the overall inference accuracy of the network.
2Device complexity
If existing pruning methods are applied uniformly across all layers, then the pruning process is simplified, but tensor dimension mismatches occur at concat units
Solution Approach 1:
The patent segments the neural network into different layer types (convolutional layers with BN, convolutional layers without BN, fully connected layers) and applies specific pruning methods to each segment. This segmentation ensures that tensor dimension consistency is maintained at concat units by preventing dimension mismatches, while keeping the overall pruning process manageable through systematic categorization.
3Reliability
If layers input to concat units are excluded from pruning targets, then tensor dimension consistency is maintained, but the overall pruning rate of the model decreases
Solution Approach 1:
Instead of excluding layers input to concat units from pruning, the patent applies localized pruning control by determining pruning targets layer by layer using L1 regularization learning. This allows selective pruning of specific channels or filters within each layer while maintaining the overall tensor dimension consistency required for concat operations, thereby achieving both high pruning rates and concatenation accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A machine learning program for causing a computer to execute a process including: selecting a reduction ratio of each element of a plurality of layers in a trained model of a neural network including the plurality of layers; and adjusting, when the neural network includes a calculating process that outputs a tensor serving as a result of a given calculation on a tensor from a first layer and one or more tensors of one or more second layers preceding the first layer, a first reduction ratio and one or more second reduction ratios based on one or more elements to be reduced in the first layer at the first reduction ratio and one or more elements to be reduced in each of the one or more second layers at the one or more second reduction ratios.