Skip-Layer CNN Pruning for Lower FLOPs Without Accuracy Loss
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing convolutional neural network (CNN) pruning methods, such as Iterative Model Pruning (IMP) and Whole Model Pruning (WMP) greedy and independent strategies, are inefficient, time-consuming, and difficult to implement in real-time applications, especially for deeper networks, due to high computational complexity and accuracy loss during fine-tuning.
Innovation Solution
The One-shot Skip Layer Pruning (OSLP) method, which involves selecting non-contiguous layers for pruning based on L2-norm ensembling and using a higher pruning threshold, allows for faster convergence and scalability by reducing floating-point operations (FLOPs) while maintaining accuracy, employing structured pruning without error recovery phases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If Iterative Model Pruning (IMP) or Whole Model Pruning (WMP) greedy strategies are used, then pruning accuracy can be maintained, but computational complexity and time consumption increase significantly
Solution Approach 1:
The patent segments the CNN into non-contiguous layers for selective pruning, rather than pruning all layers uniformly. This segmentation allows the method to focus computational resources on specific layers that benefit most from pruning, reducing overall computational complexity while maintaining pruning accuracy through targeted layer-wise optimization
Solution Approach 2:
The patent introduces a layer-wise pruning threshold parameter that can be adjusted for different layers. By changing the pruning threshold parameter selectively across different layers rather than applying a uniform threshold, the method achieves accurate pruning results with reduced computational overhead through adaptive parameter tuning
2Ease of manufacture
If Whole Model Pruning (WMP) independent strategies are used, then implementation simplicity is improved, but accuracy loss increases during fine-tuning
Solution Approach 1:
The patent performs preliminary layer selection and pruning threshold determination before actual pruning execution. By pre-identifying which non-contiguous layers to prune and determining optimal pruning thresholds for each layer in advance, the method simplifies the implementation process while maintaining accuracy through carefully planned pruning strategies that account for layer interdependencies
Solution Approach 2:
The patent incorporates feedback mechanisms through layer-wise evaluation where the impact of pruning on subsequent layers is considered. The method uses feedback from layer performance metrics to adjust pruning decisions, ensuring that pruning in one layer does not negatively impact the overall model accuracy while maintaining implementation simplicity through systematic evaluation
3Adaptability or versatility
If deeper CNN networks are pruned using existing methods, then model capacity is improved, but fine-tuning time and computational overhead increase
Solution Approach 1:
The patent segments deeper CNN networks into non-contiguous layers for selective pruning, focusing on layers that can be pruned without significant accuracy loss. This segmentation approach enables the method to handle deeper networks by pruning only specific segments rather than the entire network, reducing fine-tuning time while maintaining model capacity through targeted layer optimization
Solution Approach 2:
The patent applies partial pruning action by selecting only certain non-contiguous layers for pruning rather than pruning all layers in deeper networks. This partial action approach maintains sufficient model capacity by preserving critical layers while reducing computational overhead and fine-tuning time through selective pruning of less critical layers
Data Source
AI summary
Systems, methods, and devices for pruning a convolutional neural network (CNN). A subset of layers of the CNN is chosen, and for each layer of the subset of layers, how salient each filter in the layer is to an output of the CNN is determined, a subset of the filters in the layer is determined based on the salience of each filter in the layer, and the subset of filters in the layer is pruned. In some implementations, the layers of the subset of layers of the CNN are non-contiguous. In some implementations, the subset of layers includes odd numbered layers of the CNN and excludes even numbered layers of the CNN. In some implementations, the subset of layers includes even numbered layers of the CNN and excludes odd numbered layers of the CNN.