Neural Network Pruning by Weight Thresholds to Limit Memory Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As neural networks become more complex, they face issues such as increased memory requirements, reduced efficiency, and overfitting, which hinder miniaturization and commercialization.
Innovation Solution
A method and apparatus for pruning neural networks by setting a weight threshold based on a determined weight distribution, predicting changes in inference accuracy, and selectively removing layers with minimal impact on performance, without retraining, using a pruning data set.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the neural network complexity is increased to handle more data and training operations, then the processing capability is improved, but the memory usage increases and device miniaturization becomes difficult
Solution Approach 1:
The patent extracts and removes redundant or less important connections (weights) from the neural network based on weight distribution analysis. By identifying and eliminating weights below certain thresholds, the network maintains essential processing capability while reducing memory requirements, directly resolving the contradiction between processing capability and memory usage.
Solution Approach 2:
The patent applies different pruning strategies to different layers and connections within the network based on their specific weight distributions and importance. Rather than uniform reduction, it selectively preserves critical weights while removing others, maintaining local processing quality in important areas while reducing overall memory usage.
2Measurement precision
If the neural network complexity is increased to improve training accuracy, then the accuracy on past training data is improved, but the reliability of prediction on new data decreases due to overfitting
Solution Approach 1:
The patent removes redundant connections that cause overfitting by pruning weights based on distribution analysis. This extraction of unnecessary weights reduces the network's memorization of training data patterns while preserving generalization capability, thereby improving prediction reliability on new data while maintaining acceptable training accuracy.
Solution Approach 2:
The patent changes the network structure parameters by dynamically adjusting which weights are retained based on weight distribution characteristics. This parameter optimization allows the network to achieve better generalization by modifying its complexity to match the actual information content in the data, resolving the overfitting issue.
3Device complexity
If the neural network is pruned to reduce complexity and memory usage, then the device miniaturization is improved, but the inference accuracy may decrease
Solution Approach 1:
The patent applies selective pruning that preserves weights in critical regions while removing weights in less important regions. By analyzing weight distributions locally across different layers and connections, it maintains inference accuracy in important processing paths while reducing overall network complexity and memory usage.
Solution Approach 2:
The patent optimizes the pruning thresholds and retention ratios based on weight distribution analysis to find the optimal balance between complexity reduction and accuracy maintenance. By dynamically adjusting retention parameters, it achieves maximum compression while minimizing accuracy loss.
Data Source
AI summary
A method and apparatus for the pruning of a neural network is provided. The method sets a weight threshold value based on a weight distribution of layers included in a neural network, predicts a change of inference accuracy of a neural network by pruning of each layer based on the weight threshold value, determines a current subject layer to be pruned with a weight threshold value among the layers included in the neural network, and prunes a determined current subject layer.