Neural Network Pruning via Channel Length-Based Weight Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural network models are difficult to use in edge computing environments due to their large capacity and operation requirements, and current pruning techniques face challenges in sharing weights between layers and modifying already trained models without additional learning.
Innovation Solution
The method involves learning a pruning factor for each 1×1 pruning unit based on its channel length, determining which units to remove, and allowing weight sharing between layers, enabling minimal cost modification of the model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning is applied to reduce model capacity and operation amount, then the model becomes suitable for edge computing, but the pruned model has a sparse matrix structure requiring separate hardware or additional programming
Solution Approach 1:
The patent extracts and removes specific weight elements from the neural network model based on their importance, creating a pruned model that retains only essential weights. This extraction approach reduces the number of weights while maintaining model functionality, directly addressing the contradiction by removing unnecessary components rather than redesigning the entire system.
Solution Approach 2:
The patent applies different pruning strategies to different parts of the model (channel-wise pruning vs. element-wise pruning), allowing local optimization of the pruning process. By treating different regions of the model with different pruning intensities and methods, the system achieves better compression ratios while maintaining performance, reducing the need for complex hardware accommodations.
2Productivity
If weighted pruning is used to significantly reduce the number of weights, then inference operation amount is reduced, but separate hardware or additional programming is required to benefit from the sparse matrix structure
Solution Approach 1:
The patent merges the pruning process with the existing model training and deployment pipeline, integrating sparsity handling into standard neural network operations. By combining pruning with retraining and using pruning masks that can be applied during inference without separate hardware, the system achieves fast inference while avoiding additional programming complexity.
Solution Approach 2:
The patent introduces pruning masks as intermediary structures that mediate between the pruned weight matrix and the computation process. These masks enable the sparse matrix to be processed efficiently using standard dense matrix multiplication operations, acting as an intermediary that translates sparsity into computational efficiency without requiring specialized hardware or complex programming.
3Device complexity
If structured pruning by kernel or channel unit is applied, then hardware complexity is reduced, but it is difficult to share weights between layers and additional learning is required to modify the model
Solution Approach 1:
The patent segments the pruning process into independent channel-wise operations, where each channel can be pruned separately while maintaining the ability to share weights across layers. This segmentation allows flexible weight sharing between layers because the pruning decisions for each channel are independent, enabling the same pruned channel to be reused across multiple layers without requiring additional learning.
Solution Approach 2:
The patent implements dynamic pruning masks that can be adjusted and shared across different layers and training iterations. The pruning masks are not fixed but can be dynamically applied and reused, allowing the model to adapt and share weights between layers efficiently. This dynamic approach enables weight sharing while maintaining the benefits of structured pruning for hardware efficiency.
4Quantity of substance
If additional learning is performed to modify an already trained model, then the pruning ratio can be increased, but training time and computational cost increase
Solution Approach 1:
The patent performs preliminary pruning during the model training phase, integrating the pruning process into the training workflow. By applying pruning masks during training and performing lightweight updates rather than full retraining, the system achieves high pruning ratios without significant additional training time. The preliminary action of pruning during training avoids the need for extensive post-training learning.
Solution Approach 2:
The patent changes the pruning parameters (such as pruning ratio thresholds and mask generation methods) to optimize the balance between pruning ratio and training time. By adjusting these parameters, the system can achieve high pruning ratios with minimal additional learning, as the parameter changes enable more aggressive pruning without requiring proportional increases in training time.
Data Source
AI summary
A method for light-weighting an artificial neural network model, the method comprising is provided. The method includes the steps of: learning, on the basis of a channel length of a 1×1 pruning unit applied in a channel direction to each of a plurality of kernels included in an artificial neural network model, a pruning factor for each of the pruning units and a weight for each of the pruning units; and determining, on the basis of at least one of the learned pruning factors and weights, which of the pruning units is to be removed from the artificial neural network model, wherein the channel length of the pruning unit is shorter than a channel length of at least a part of the plurality of kernels.