Neural Network Pruning with Zero Padding for Tensor Dimension Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network (NN) pruning methods face challenges in maintaining inference accuracy while reducing model size and calculation times, especially when applied to convolutional layers without Batch Normalization (BN) layers or fully connected layers, and can mismatch tensor dimensions in concatenating operations, leading to incorrect results or reduced pruning rates.
Innovation Solution
A method that calculates a reduction ratio for each layer in a neural network, inserts a zero padding layer between preceding layers and the concatenating process to ensure tensor dimensions match after pruning, and uses L1 regularization to identify channels for pruning, ensuring accurate inference and efficient model downsizing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If pruning is applied to reduce model size and calculation duration, then the data size and calculation times are reduced, but inference accuracy degrades
Solution Approach 1:
The patent applies L1 regularization to modify the optimization parameters of the neural network, encouraging sparse weight matrices. This parameter change allows the model to achieve compression while maintaining inference accuracy by systematically identifying and pruning near-zero weights that contribute minimally to the output.
Solution Approach 2:
The patent performs preliminary identification of channels and layers for pruning using L1 regularization before actual pruning execution. This preliminary action involves calculating reduction ratios for each channel based on regularization weights, allowing the system to plan the pruning strategy in advance and maintain accuracy by preserving critical channels.
2Quantity of substance
If excessive pruning is performed to reduce model size, then the data size is reduced, but inference accuracy significantly degrades
Solution Approach 1:
The patent implements a feedback mechanism where L1 regularization weights are continuously monitored and used to determine pruning thresholds. The system calculates reduction ratios based on regularization weights and adjusts the pruning strategy accordingly, providing feedback control to prevent excessive pruning that would degrade inference accuracy below acceptable levels.
Solution Approach 2:
The patent applies different pruning strategies to different parts of the neural network based on their specific characteristics. Convolutional layers with Batch Normalization use one approach, while other layers use alternative methods. This local quality approach ensures that each layer is pruned appropriately without uniformly degrading the overall model accuracy.
3Quantity of substance
If pruning is applied to convolutional layers without Batch Normalization or fully connected layers, then model size is reduced, but tensor dimension mismatches occur in concatenating operations
Solution Approach 1:
The patent segments the neural network into different layer types and applies specific pruning strategies to each segment. Concatenation layers are identified and handled separately, with the system calculating reduction ratios for preceding layers while accounting for the concatenation operation's dimension requirements. This segmentation prevents dimension mismatches by ensuring compatible pruning across concatenated tensors.
Solution Approach 2:
The patent introduces an intermediary calculation step that determines reduction ratios for layers preceding concatenation operations. This intermediary process ensures that the pruning of preceding layers maintains dimension compatibility with the concatenation layer, acting as a mediator between the pruning objective and the structural requirements of the network.
4Reliability
If layers inputting to concatenation units are excluded from pruning targets, then dimension mismatches are avoided, but the overall pruning rate of the model is lowered
Solution Approach 1:
The patent creates a universal pruning framework that handles multiple layer types uniformly through L1 regularization. Instead of excluding certain layers from pruning, the system applies a unified approach that calculates reduction ratios for all layers including those preceding concatenation units, maintaining both calculation correctness and high pruning rates through the regularization-based methodology.
Data Source
AI summary
A method including: obtaining a reduction ratio of each element of layers in a trained model of a neural network; when the neural network includes a process that outputs a tensor as a result of a given calculation on tensors and when tensors from first layers preceding the process are inputted, inserting a second layer that performs a zero padding between the first layers and the process, the first layers including a preceding layer of the process and including one or more layers preceding the preceding layer and being shortcut-connected to the process; and padding tensors inputted into second layers associated one with each first layer with one or more zero matrices such that a number of elements of each tensor inputted into the process from the first layers after reducing of elements of each first layer in accordance with the reduction ratio comes to be a first number.