Binary Neural Network Channel Pruning by Weight Flipping Frequency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing pruning methods for binary neural networks (BNNs) are inadequate due to the non-trivial nature of 0s and 1s, and unstructured pruning introduces memory overhead without significant savings, while existing full-precision network pruning methods are not applicable.
Innovation Solution
A pruning method for BNNs that uses weight flipping frequency as an indicator to identify insensitive weights, reducing the network size by shrinking channels in each layer and retraining the pruned model, avoiding unstructured pruning and memory overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If existing pruning methods for full-precision networks are applied to BNNs, then pruning can be performed, but the methods are inadequate because both 0s and 1s are non-trivial in BNNs and existing methods interpret 0s as trivial
Solution Approach 1:
The patent changes the parameter used for identifying prunable weights from the conventional magnitude-based metric (suitable for full-precision networks where 0s are trivial) to weight flipping frequency (suitable for BNNs where both 0s and 1s are non-trivial). This parameter change allows existing pruning methodologies to be adapted to BNNs while maintaining accuracy identification reliability.
2Quantity of substance
If unstructured pruning is applied to BNNs, then weight reduction can be achieved, but memory overhead is introduced to label the prunable weights
Solution Approach 1:
The patent applies structured pruning by segmenting the network into channels and groups, pruning entire channels or groups rather than individual weights. This segmentation approach achieves weight reduction while avoiding the need for per-weight labels, thereby eliminating the memory overhead associated with unstructured pruning.
Solution Approach 2:
The patent introduces channel-wise and group-wise pruning strategies that apply different pruning ratios to different channels or groups based on their importance. This local quality approach allows selective pruning of less important channels while preserving important ones, achieving compression without requiring detailed per-weight labels.
3Quantity of substance
If more weights are pruned to reduce network size, then compression ratio increases, but accuracy maintenance becomes more difficult
Solution Approach 1:
The patent employs iterative pruning with accuracy feedback, where weights are pruned in multiple stages and the network accuracy is evaluated after each stage. The pruning process adjusts the pruning ratio based on accuracy feedback, allowing aggressive pruning when accuracy is maintained and conservative pruning when accuracy degrades. This feedback mechanism enables achieving high compression ratios while maintaining acceptable accuracy levels.
Solution Approach 2:
The patent performs preliminary identification of important channels and groups using weight flipping frequency analysis before actual pruning. This preliminary action allows the method to prioritize which channels to prune first, ensuring that less important channels are removed while preserving important ones, thereby maintaining accuracy even with significant weight reduction.
Data Source
AI summary
Various embodiments of a system and method for pruning binary neural networks by analyzing weight flipping frequency and pruning the binary neural network based on the weight flipping frequency associated with each channel of the binary neural network are disclosed herein.