Neural Network Model Compression via Structural Channel Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural network model compression methods, particularly parameter pruning and sharing, result in low compression ratios and require additional decompression modules, leading to increased complexity and resource demands in embedded systems.
Innovation Solution
The method involves recording input and output parameters of each network layer, dividing the network into small networks, setting pruning flag bits, training each small network using a channel pruning algorithm, and performing decomposition calculations based on pruned channel indices to reduce model size and calculation load.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If non-structural pruning algorithm is used, then the model size is reduced through data indexing, but the actual size of the model is essentially unchanged and requires additional decompression modules
Solution Approach 1:
The patent extracts and removes redundant convolutional layers from the neural network structure through structural pruning. By identifying and eliminating unnecessary layers based on channel importance analysis, the model achieves genuine size reduction without requiring decompression modules, directly resolving the contradiction between model size reduction and complexity increase
Solution Approach 2:
The patent segments the neural network into multiple convolutional layers and analyzes each layer's channel importance independently. This segmentation allows selective pruning of specific layers while preserving critical functionality, achieving efficient size reduction without compromising model performance or requiring additional decompression mechanisms
2Quantity of substance
If structural pruning algorithm is used, then the model compression ratio is improved, but convolutional layers are excessively pruned and require additional convolutional operations
Solution Approach 1:
The patent applies local quality analysis by evaluating the importance of individual channels within each convolutional layer using gradient-based metrics. This allows selective pruning of only the least important channels while preserving critical features, achieving high compression ratios without excessively damaging model performance or requiring complex compensation operations
Solution Approach 2:
The patent implements feedback mechanisms through iterative pruning processes where model performance is continuously monitored and evaluated. Based on performance feedback, the pruning process can be adjusted to prevent over-pruning of critical layers, maintaining a balance between compression ratio and model functionality without requiring additional complex operations
3Reliability
If deep neural network model is expanded to improve performance, then the calculation power and model volume increase, but the dependence on high-power consumption calculation platforms increases
Solution Approach 1:
The patent changes the parameters of the neural network by pruning redundant channels and layers while maintaining the essential computational functionality. This parameter transformation allows the model to achieve high performance with reduced computational complexity, enabling deployment on lower-power platforms without sacrificing reliability
Solution Approach 2:
The patent applies partial pruning strategies that remove only the necessary minimal portions of the network structure to achieve the desired performance level. By avoiding excessive pruning, the model maintains sufficient computational capacity for high-performance tasks while significantly reducing power consumption compared to full-scale models
Data Source
AI summary
A method and device for model compression of a neural network. The method comprises: recording input and output parameters of each layer of network in a network structure; dividing the network structure into several small networks according to the input and output parameters; setting a pruning flag bit of a first convolutional layer in each small network to be zero to obtain a pruned small network; training each pruned small network to obtain a network weight and a weight mask; recording a pruned channel index number of each convolutional layer of a pruned small network with the weight mask of zero; and carrying out decomposition calculation on each pruned small network according to the pruned channel index number. According to the method, a calculation amount and the size of a model is reduced, and during network deployment, the model can be loaded with one click, thus reducing usage difficulty.