Hybrid Neural Network Pruning for Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network pruning methods are single-type, which limits their ability to achieve optimal compression and accuracy, as they fail to effectively reduce the computational complexity and parameter amount of neural networks.
Innovation Solution
A hybrid pruning approach that sequentially applies multiple pruning algorithms, such as channel pruning via geometric median and ThiNet, followed by retraining, to identify and remove redundant parameters, thereby achieving a balance between compression and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If single-type pruning algorithm is used, then the pruning process is simple, but the compression rate is limited and accuracy loss increases
Solution Approach 1:
The patent combines multiple different pruning algorithms (e.g., L1 norm pruning, L2 norm pruning, magnitude pruning) into a hybrid pruning framework. This merging of diverse pruning techniques allows the system to achieve higher compression rates by leveraging the complementary strengths of each algorithm, while maintaining model accuracy through their coordinated application.
Solution Approach 2:
The patent creates a composite pruning approach by integrating multiple pruning algorithms with different characteristics. Similar to composite materials in engineering, this hybrid pruning strategy combines algorithms with different properties (sparsity patterns, computational requirements, pruning ratios) to achieve superior overall performance that none of the individual algorithms could attain alone.
2Use of energy by moving object
If single-type pruning algorithm is used, then the computational cost is low, but the accuracy maintenance capability is insufficient
Solution Approach 1:
The patent implements a dynamic hybrid pruning process where multiple pruning algorithms are applied sequentially with different pruning ratios and strategies at different stages. The system adaptively adjusts which pruning algorithm to apply and at what intensity, allowing it to maintain accuracy by using gentler pruning methods when needed and more aggressive methods when appropriate, rather than relying on a single static pruning approach.
3Quantity of substance
If multiple pruning algorithms are applied sequentially, then the compression rate increases, but the training time increases
Solution Approach 1:
The patent performs preliminary pruning using less computationally intensive algorithms before applying more rigorous pruning methods. By conducting initial pruning actions that remove obviously redundant parameters, the system reduces the overall training time required for subsequent more intensive pruning operations, as the model size and complexity are already reduced from the preliminary pruning stage.
Solution Approach 2:
The patent applies pruning algorithms with varying degrees of aggressiveness in a staged manner. Rather than applying one extremely aggressive pruning method that would require extensive retraining, the system uses multiple partial pruning actions with moderate intensity, each requiring relatively little retraining time, but collectively achieving high compression rates through their cumulative effect.
Data Source
AI summary
Embodiments of the disclosure provide an optimizing method and a computer system for a neural network, and a computer-readable storage medium. In the method, the neural network is pruned sequentially using two different pruning algorithms. The pruned neural network is retrained in response to each pruning algorithm pruning the neural network. Thereby, the computation amount and the parameter amount of the neural network are reduced.