Neural Network Pruning with Learnable Masks for Merge Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning architectures face challenges in parameter reduction of neural networks through group-based pruning, as they fail to account for the varying importance of layers within groups, leading to performance deterioration.
Innovation Solution
An electronic device and method for pruning neural networks by applying learnable masks to individual groups, updating them based on computational graphs, and adjusting importance between groups to optimize pruning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If group-based pruning is applied to reduce parameter size, then the number of parameters is reduced, but performance deteriorates due to equal treatment of all layers
Solution Approach 1:
The patent applies different pruning rates to different layers within the same group based on their individual importance scores. Instead of uniformly pruning all layers equally, the method calculates importance metrics for each layer and selectively prunes less important layers while preserving more important ones, thereby maintaining performance while achieving parameter reduction.
Solution Approach 2:
The patent dynamically adjusts the pruning rate parameter for each layer based on importance calculations. The pruning rate is not fixed but varies across layers according to their contribution to network performance, allowing the system to optimize between parameter reduction and performance maintenance by changing the pruning parameter adaptively.
2Quantity of substance
If pruning is applied without considering layer importance, then parameter reduction is achieved, but more important weights are incorrectly omitted
Solution Approach 1:
The patent evaluates and treats each layer individually within groups by calculating importance scores that reflect their specific contribution to network performance. This localized assessment ensures that pruning decisions are made based on actual layer importance rather than uniform treatment, preventing misidentification of important weights.
Solution Approach 2:
The patent employs a feedback mechanism where importance scores are calculated based on network performance metrics, and these scores guide subsequent pruning decisions. The system continuously monitors performance impact and adjusts pruning strategies accordingly, ensuring that weights critical for performance are preserved while less important weights are pruned.
Data Source
AI summary
An electronic device includes a memory storing computer-readable instructions and at least one processor that coupled to the memory and configured to execute the computer-readable instructions. The at least one processor is configured to identify one or more merge layers included in a pruning target model of a neural network and generate a target group including a target merge layer among the one or more merge layers and a sub-layer logically connected with the target merge layer. The processor is configured to apply a learnable mask to the target group and update the learnable mask, through propagation of the pruning target model. The processor is also configured to perform pruning of the pruning target model based on the updated learnable mask.