DNN Surgery Pruning for Deep Neural Network Parameter Density Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks (DNNs) are dense and resource-intensive, making them impractical for deployment on embedded/mobile devices due to high memory and computation costs, with previous methods often resulting in accuracy loss when attempting to reduce parameter density.
Innovation Solution
The DNN Surgery method employs a Joint Feed-forward and Backward Propagation Approximation (JFBPA) optimization to progressively prune less important parameters in each layer, integrated with retraining to maintain accuracy, constructing a highly-sparse DNN model efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If DNN models are made deep and dense to improve accuracy, then model accuracy is improved, but memory and computation cost increase significantly
Solution Approach 1:
The patent extracts and removes redundant parameters from DNN layers through pruning techniques. Specifically, it prunes columns in weight matrices that correspond to redundant features, thereby reducing parameter density while maintaining model accuracy. This extraction of unnecessary parameters directly addresses the contradiction by reducing quantity without sacrificing reliability.
Solution Approach 2:
The patent applies different pruning strategies to different layers and parameters based on their importance. Instead of uniform pruning, it identifies and prunes only the least important parameters in each layer, preserving critical parameters that maintain accuracy. This localized approach allows selective reduction of parameter density where it matters least, maintaining overall model reliability.
2Quantity of substance
If conventional compression methods (truncated SVD, VQ, Hashing) are applied to reduce parameter density, then storage requirements are reduced, but model accuracy drops significantly
Solution Approach 1:
The patent performs preliminary feature selection and importance assessment before compression. By pre-identifying which parameters are most important through methods like L1 regularization or sensitivity analysis, it preserves critical parameters during compression operations. This preliminary action ensures that compression reduces parameter density without significantly impacting model accuracy, as the most important parameters are protected from removal.
Solution Approach 2:
The patent changes the approach from compressing all parameters uniformly to selectively preserving parameters based on their importance metrics. It modifies the compression process to maintain parameters with high importance scores while removing those with low scores, thereby changing the outcome from accuracy loss to accuracy preservation while achieving parameter density reduction.
3Quantity of substance
If naive pruning methods are used to delete parameters smaller than thresholds, then storage requirements are reduced, but the method lacks theoretical guidance and may not maintain accuracy
Solution Approach 1:
The patent implements feedback mechanisms where the pruning process is guided by continuous monitoring of model performance and parameter importance. Importance metrics are updated based on feedback from forward and backward propagation errors, allowing the pruning process to adapt to the specific characteristics of each DNN model. This feedback-driven approach provides theoretical guidance while maintaining accuracy, overcoming the limitations of naive threshold-based methods.
Solution Approach 2:
The patent creates a universal pruning framework that can be applied to different DNN architectures and tasks through a unified importance assessment mechanism. The same theoretical framework handles various pruning scenarios by adjusting importance metrics, providing broad applicability and theoretical guidance across different models without requiring architecture-specific methods.
4Quantity of substance
If Binarization compression methods are applied to reduce parameter density, then storage requirements are reduced, but model accuracy drops significantly
Solution Approach 1:
Instead of applying aggressive binarization that removes too much information, the patent applies partial compression that retains enough parameter precision to maintain accuracy. It uses intermediate compression levels between full precision and binarization, selectively applying compression only to less important parameters while maintaining full precision for critical parameters, thereby achieving parameter density reduction without significant accuracy loss.
Data Source
AI summary
An apparatus and method are described for reducing the parameter density of a deep neural network (DNN). A layer-wise pruning module to prune a specified set of parameters from each layer of a reference dense neural network model to generate a second neural network model having a relatively higher sparsity rate than the reference neural network model; a retraining module to retrain the second neural network model in accordance with a set of training data to generate a retrained second neural network model; and the retraining module to output the retrained second neural network model as a final neural network model if a target sparsity rate has been reached or to provide the retrained second neural network model to the layer-wise pruning model for additional pruning if the target sparsity rate has not been reached.