Data-Aware Neural Network Pruning via Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) require significant computational resources for both training and inference, despite their high accuracy, and existing techniques for reducing computational costs, such as sparsity, may not be sufficient.
Innovation Solution
The implementation of data-aware model pruning techniques for neural networks, where a computing system uses training data to calculate entropy values for activation matrices, allowing for the pruning of unnecessary weights and connections based on these values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep neural networks are made larger to achieve better accuracy, then accuracy is improved, but computational cost increases
Solution Approach 1:
The patent extracts and removes unnecessary weights and connections from the neural network through pruning techniques. By identifying and eliminating redundant parameters based on their impact on model accuracy, the system reduces computational cost while maintaining the essential functionality and accuracy of the network.
Solution Approach 2:
The patent changes the parameters of the neural network by applying pruning operations that modify the weight matrix. Through techniques like magnitude-based pruning and sensitivity-based pruning, the system transforms the network structure to reduce the number of active parameters, thereby lowering computational requirements while preserving accuracy.
2Use of energy by moving object
If sparsity techniques are applied to reduce computational costs, then computational cost is reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent applies local quality by differentiating between important and unimportant weights within the network. Instead of uniform pruning, the system identifies specific locations in the weight matrix that can be safely pruned based on their individual contribution to accuracy, allowing selective removal that preserves critical information while reducing computational load.
Solution Approach 2:
The patent employs feedback mechanisms during the pruning process where the model performance is continuously monitored and used to guide further pruning decisions. This feedback loop ensures that pruning operations do not compromise accuracy by allowing iterative adjustment based on observed model behavior.
3Device complexity
If more pruning is applied to reduce network size, then network size is reduced, but model performance may deteriorate
Solution Approach 1:
The patent applies partial action by implementing pruning at controlled levels rather than complete removal. Through techniques like L1/L2 ratio pruning and sensitivity-based pruning, the system removes only the necessary portion of weights to achieve optimal compression, avoiding excessive pruning that would harm model performance.
Data Source
AI summary
Embodiments of the present disclosure include systems and methods for performing data-aware model pruning for neural networks. During a training phase, a neural network is trained with a first set of data. During a validation phase, inference with the neural network is performed using a second set of data that causes the neural network to generate a first set of outputs at a layer in the neural network. During the validation phase, a plurality of mean values and a plurality of variance values are calculated based on the first set of outputs. A plurality of entropy values are calculated based on the plurality of mean values and the plurality of variance values. A second set of outputs are pruned based on the plurality of entropy values. The second set of outputs are generated by the layer of the neural network using a third set of data.