Neural Network Group Pruning with Layer-Specific Importance Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning architectures face challenges in maintaining performance while reducing neural network parameters through group-based pruning, as they fail to account for the varying importance of layers within groups, leading to potential network deterioration.
Innovation Solution
An electronic device and method that individually applies importance to each layer within a group during pruning, using a learnable parameter to update and optimize the neural network, allowing for more efficient weight reduction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If group-based pruning is applied to reduce neural network parameters, then the parameter size is reduced, but the performance deteriorates due to equal treatment of all layers in the group
Solution Approach 1:
The patent applies local quality by introducing a learnable parameter that is specifically assigned to each layer within a group, allowing each layer to have its own pruning importance rather than treating all layers equally. This enables differentiated pruning ratios for different layers based on their individual importance, thus maintaining performance while achieving parameter reduction.
Solution Approach 2:
The patent segments the pruning process by dividing layers into groups and then further differentiating within each group using layer-specific learnable parameters. This segmentation allows the system to apply different pruning strategies to different layers within the same group, resolving the contradiction between parameter reduction and performance maintenance.
2Productivity
If pruning is performed without considering layer importance, then the pruning process is simple and fast, but the network performance deteriorates
Solution Approach 1:
The patent implements self-service by making the pruning importance a learnable parameter that the model itself learns during training. The model automatically determines the importance of each layer through the learnable parameter without requiring external intervention or complex manual assessment, thus maintaining pruning efficiency while improving performance.
Solution Approach 2:
The patent changes the parameter representation by introducing a learnable parameter that dynamically represents layer importance. This parameter is updated during training through propagation, allowing the pruning strategy to adapt and optimize based on the model's learned features, thereby maintaining both efficiency and performance.
3Reliability
If individual layer importance is applied in group-based pruning, then performance is maintained, but the complexity of the pruning process increases
Solution Approach 1:
The patent merges the concept of layer importance into the existing group-based pruning framework by introducing a learnable parameter. This integration allows the system to maintain the benefits of group-based pruning while adding individual layer differentiation, achieving performance maintenance without excessive complexity increase.
Data Source
AI summary
An electronic device includes a memory storing computer-executable instructions and at least one processor coupled to the memory and configured to execute the computer-readable instructions. The at least one processor is configured to identify a merge layer included in a pruning target model of a neural network to determine a target group including layers, including the merge layer and a sub-layer logically connected with the merge layer. The at least one processor is configured to apply a learnable parameter to each of the layers included in the target group. The at least one processor is configured updates the learnable parameter through propagation of the pruning target model. The at least one processor is configured to perform pruning of the pruning target model based on the updated learnable parameter.