Channel-Wise Neural Network Pruning for Memory-Accuracy Balance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face issues of complexity and overfitting, leading to increased memory allocation and reduced accuracy for new data, hindering miniaturization and commercialization.
Innovation Solution
A neural network pruning method that adjusts weights between nodes of channels based on a preset learning weight and channel-by-channel pruning parameters, updating these parameters to enhance task accuracy, and repeatedly performing pruning and evaluation operations to optimize network complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the learning amount of the artificial neural network increases to process complex input data, then the computational capability improves, but the connectivity becomes complex and memory allocation amount excessively increases
Solution Approach 1:
The patent extracts and removes unnecessary connections (weights) from the neural network through pruning operations. By identifying and eliminating weights that contribute minimally to the output, the network maintains its computational capability while reducing connectivity complexity and memory requirements.
Solution Approach 2:
The patent changes the parameter of weight values by setting them to zero or removing them entirely through pruning. This parameter modification reduces the overall complexity of the network while preserving the essential computational functions needed for processing complex input data.
2Measurement precision
If the learning amount of the artificial neural network increases, then the accuracy of old learning data increases, but the confidence of an estimate value of new data degrades due to overfitting
Solution Approach 1:
The patent removes redundant and overfit connections from the neural network through pruning. By eliminating weights that have become overly specialized to training data, the network generalizes better to new data while maintaining accuracy on existing data.
Solution Approach 2:
The patent modifies weight parameters by setting them to zero or removing them, which prevents overfitting. This parameter change reduces the network's memorization of training data patterns and improves its ability to generalize to new, unseen data.
3Productivity
If the complexity of the artificial neural network increases, then the processing capability improves, but the memory allocation amount excessively increases which causes issues in miniaturization and commercialization
Solution Approach 1:
The patent extracts and removes unnecessary weights and connections from the neural network. This extraction reduces the memory allocation required to store network parameters while preserving the processing capability needed for complex data analysis.
Solution Approach 2:
The patent changes the state of weight parameters by setting them to zero or removing them entirely. This parameter modification reduces the memory footprint of the network, enabling miniaturization and making the technology more viable for commercial applications.
Data Source
AI summary
A neural network pruning method includes acquiring a first task accuracy of an inference task processed by a pretrained neural network, pruning, based on a channel unit, the neural network by adjusting weights between nodes of channels based on a preset learning weight and based on a channel-by-channel pruning parameter corresponding to a channel of each of a plurality of layers of the pretrained neural network, updating the learning weight based on the first task accuracy and a task accuracy of the pruned neural network, updating the channel-by-channel pruning parameter based on the updated learning weight and the task accuracy of the pruned neural network, and repruning, based on the channel unit, the pruned neural network based on the updated learning weight and based on the updated channel-by-channel pruning parameter.