Adaptive Sparsity Regularization for Deep Neural Network Slimming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks face challenges in achieving high accuracy with minimal computation, particularly in mobile and IoT scenarios, due to redundancy and inefficiencies in sparsity regularization methods like L1 loss, which uniformly reduce scale terms across batch normalization layers.
Innovation Solution
The implementation of adaptive regularization based on a target pruning ratio and scale term during learning in deep neural networks, using sparsity regularization to minimize task loss and optimize network performance by adaptively determining parameters, performing pruning, and fine-tuning the results.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If L1 loss is used for sparsity regularization, then the number of non-zero terms is reduced, but all scale terms are reduced with the same gradient causing uniform pruning that may not optimize task performance
Solution Approach 1:
The patent applies local quality by making the regularization strength adaptive and location-specific. Instead of uniform L1 regularization applied equally to all scale terms, the method dynamically adjusts the regularization parameter λ based on the current pruning ratio and task loss, allowing different parts of the network to be pruned at different rates according to their importance and current sparsity level.
Solution Approach 2:
The patent implements dynamics by making the regularization parameter adaptive rather than static. The parameter λ is dynamically adjusted during training based on the current pruning ratio and task loss, enabling the pruning process to evolve and adapt as the network structure changes, rather than applying a fixed uniform reduction.
2Productivity
If aggressive pruning is performed to reduce network size, then computation is reduced, but task loss increases due to performance degradation
Solution Approach 1:
The patent applies feedback by using task loss as a feedback signal to adjust the regularization parameter. The task loss information flows back into the regularization mechanism, allowing the system to automatically adjust the pruning aggressiveness based on performance impact, preventing excessive pruning that would degrade task performance.
Solution Approach 2:
The patent implements self-service by making the pruning process self-regulating through adaptive parameter adjustment. The system automatically adjusts its own pruning behavior based on task loss feedback without requiring external intervention or manual tuning, enabling it to self-optimize the balance between compression and performance.
3Ease of manufacture
If uniform regularization is applied to all batch normalization layers, then implementation is simple, but it cannot adapt to different pruning ratios and scale terms
Solution Approach 1:
The patent applies parameter changes by making the regularization parameter λ dynamic and adaptive. Instead of using a fixed uniform regularization strength, the method changes the parameter based on the current pruning ratio and task loss, allowing the system to adapt to different pruning stages and network states while maintaining a relatively simple implementation framework.
Data Source
AI summary
Disclosed are a deep neural network lightweight device based on batch normalization, and a method thereof. The deep neural network lightweight device based on batch normalization includes a memory that stores at least one data and at least one processor that executes a network lightweight module. When executing the network lightweight module, the processor performs learning on an input neural network based on sparsity regularization to adaptively determine at least one parameter of the sparsity regularization, performs pruning on the learning result, and performs fine tuning on the pruning result.