Neural Network Weight Pruning via Sensitivity Thresholds
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks require significant computation and memory, leading to increased complexity and the risk of over-fitting, which hinders miniaturization and commercialization, and there is a need for a compression method to reduce system costs while maintaining performance.
Innovation Solution
A method involving pruning of artificial neural networks by determining a percentile-based pruning threshold based on pruning sensitivity, which measures the change in behavior of the network, allowing for the removal of weights without affecting prediction accuracy, and applying a magnitude-based pruning threshold to kernels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the artificial neural network is trained with more data and complex connections to improve accuracy, then prediction accuracy improves, but device complexity and memory allocation increase excessively
Solution Approach 1:
The patent extracts and removes unnecessary weights and connections from the neural network through pruning operations. By identifying and eliminating redundant parameters that do not contribute significantly to prediction accuracy, the network complexity is reduced while maintaining the essential functional capabilities needed for accurate predictions.
Solution Approach 2:
The patent changes the parameters of the neural network by applying pruning thresholds to modify the weight distribution. Through percentile-based and magnitude-based pruning, the network parameters are transformed to remove insignificant weights, thereby reducing complexity while preserving accuracy-critical connections.
2Reliability
If the artificial neural network is trained with more data and complex connections to improve reliability, then prediction accuracy improves, but memory allocation increases excessively
Solution Approach 1:
The patent extracts and removes redundant weights from the neural network through systematic pruning. By eliminating unnecessary parameters that consume memory but contribute minimally to reliability, the network maintains its predictive reliability with significantly reduced memory allocation requirements.
Solution Approach 2:
The patent transforms the network parameters by applying pruning thresholds that selectively remove weights based on their contribution to reliability. This parameter modification reduces the quantity of stored weights while preserving those essential for maintaining reliable predictions.
3Device complexity
If pruning is applied to reduce network complexity and memory allocation, then device complexity reduces, but prediction accuracy may deteriorate
Solution Approach 1:
The patent employs feedback mechanisms to monitor prediction accuracy during the pruning process. By evaluating the network's performance at different pruning stages and adjusting pruning thresholds accordingly, the system ensures that accuracy deterioration is prevented while achieving the desired complexity reduction.
Solution Approach 2:
The patent carefully modifies pruning parameters such as percentile thresholds and magnitude thresholds to control the pruning intensity. By adjusting these parameters, the network achieves optimal compression while maintaining prediction accuracy within acceptable ranges, preventing excessive accuracy loss.
Data Source
AI summary
A method and apparatus for compressing an artificial neural network may acquire weights corresponding to an artificial neural network trained in advance, wherein the artificial neural network includes a plurality of layers, and a processor configured to generate data for acquiring a change of behavior of the artificial neural network due to pruning of the artificial neural network based on the weights, determine a pruning threshold for pruning of the artificial neural network based on the change of the behavior of the artificial neural network, and compress the neural network based on the pruning threshold.