Joint Pruning and Quantization for Memory-Limited Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large neural networks are limited by their size, which restricts their storage and execution on devices with insufficient memory and processing resources, and existing methods of pruning and quantization often result in non-optimal performance when applied separately or sequentially.
Innovation Solution
A technique for jointly pruning and quantizing neural networks, integrating pruning and quantization processes to reduce the size and resource requirements of neural networks, involving iterative training with quantization logic simulation and bit width determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning and quantization are applied separately, then the neural network size is reduced, but the performance becomes non-optimal
Solution Approach 1:
The patent merges pruning and quantization into a single joint operation that simultaneously identifies redundant parameters and applies quantization. This integrated approach allows the system to optimize both network size and performance together, avoiding the suboptimal results of sequential application while achieving better compression ratios and maintaining higher accuracy.
Solution Approach 2:
The patent introduces trainable quantization parameters that are learned during the joint pruning and quantization process. By dynamically adjusting quantization parameters alongside pruning decisions, the system adapts the quantization strategy to preserve important features while removing redundant parameters, thereby maintaining performance while reducing network size.
2Reliability
If more trainable parameters are included in the neural network, then the network performance improves, but the memory requirements increase
Solution Approach 1:
The patent extracts and removes redundant trainable parameters through the joint pruning process, identifying and eliminating parameters that contribute minimally to network performance. This extraction of unnecessary parameters reduces memory consumption while preserving the essential parameters needed for maintaining network performance.
Solution Approach 2:
The patent applies different quantization precision levels to different parts of the network based on their importance. By applying higher precision to critical parameters and lower precision to less important parameters, the system optimizes the balance between memory consumption and performance, ensuring that memory is allocated efficiently according to local parameter importance.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A technique jointly prunes and quantizes an original neural network. This produces a final neural network having a smaller size than the original neural network. The technique involves converting the original neural network into a quantized-preprocessed neural network by adding weight-quantization logic that simulates effects of converting weight parameters used by the original neural network into quantized weight parameters, and/or adding activation-quantization logic that simulates effects of converting activations produced by the layers of the original neural network into quantized activation information. The technique then applies an iterative training process that includes: (a) identifying a prescribed number of groups of weight parameters in a set of original groups as redundant, a remainder of the original set of groups being to-be-retained groups, and (b) determining quantization parameters that will govern quantization used in levels of the final neural network.