Neural Network Pruning Mask Optimization for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face challenges in maintaining performance while reducing complexity to avoid overfitting and excessive memory allocation, which hinders miniaturization and commercialization.
Innovation Solution
A method and apparatus that determine a pruning mask to maximize weight importance in a trained neural network by using a processor to express weight importance as a pruning binary vector and apply constraints, optimizing weight pruning based on binary vector optimization algorithms to reduce complexity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the complexity of a neural network is increased to improve learning capacity and accuracy, then the accuracy of learned data increases, but confidence in estimated value of newly learned data decreases and overfitting occurs
Solution Approach 1:
The patent extracts and removes redundant or less important weights from the neural network through pruning operations. By identifying and eliminating unnecessary connections between neurons, the network complexity is reduced while maintaining the essential learning capacity, thus preventing overfitting and improving generalization to new data.
Solution Approach 2:
The patent changes the parameters of the neural network by adjusting the pruning threshold and pruning ratio. These parameter changes allow the network to transition between different complexity levels, optimizing the balance between accuracy and reliability for different application scenarios.
2Manufacturing precision
If the complexity of a neural network is increased to improve learning capacity, then the accuracy increases, but memory allocation excessively increases
Solution Approach 1:
The patent removes unnecessary weights and connections from the neural network through systematic pruning. This extraction process reduces the quantity of stored parameters while preserving the network's accuracy on training data, directly addressing the memory allocation issue.
Solution Approach 2:
The patent applies partial pruning by removing only a portion of the weights rather than all connections. By carefully selecting which weights to prune based on importance metrics, the network maintains sufficient complexity for accurate learning while reducing memory requirements to practical levels.
3Manufacturing precision
If the complexity of a neural network is increased to improve learning capacity, then the accuracy increases, but miniaturization and commercialization are hindered
Solution Approach 1:
The patent extracts redundant computational elements from the neural network, reducing the overall size and complexity. This enables the network to be implemented in smaller, more cost-effective hardware configurations suitable for miniaturization and commercial deployment.
Solution Approach 2:
The patent modifies the network architecture parameters through pruning to create a compact version that maintains accuracy while being suitable for smaller hardware platforms. This facilitates ease of manufacture and commercialization by adapting the network to practical deployment constraints.
Data Source
AI summary
A method and apparatus with pruning are disclosed. A method is performed by an apparatus including a processor, the method includes determining weight importance of a trained neural network, receiving a constraint condition related to an operation resource, and determining, in accordance with the constraint condition, a pruning mask for maximizing the weight importance of the trained neural network.