Partial Tensor Correction for Neural Network Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The widespread adoption of neural networks in real-world applications is hindered by significant computational demands, including computational complexity, bandwidth management, energy consumption, and accuracy degradation due to quantization, especially in heterogenous computing systems.
Innovation Solution
Implement partial tensor correction and retention methods, where sensitive channels are retained in original precision and less sensitive channels are quantized to lower precision, utilizing a combination of CPUs and accelerators for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantization is applied to neural network weight tensors to reduce computational complexity and energy consumption, then processing speed and energy efficiency are improved, but accuracy degradation occurs
Solution Approach 1:
The patent applies different precision levels to different parts of the neural network based on their sensitivity to quantization. Sensitive channels are retained in original precision while less sensitive channels are quantized to lower precision, optimizing the balance between accuracy and computational efficiency for each specific component
Solution Approach 2:
The patent dynamically adjusts the precision parameter for different weight tensors and channels based on their quantization sensitivity. By changing the precision parameter selectively rather than uniformly across the entire network, the system achieves better accuracy preservation while maintaining computational benefits
2Use of energy by moving object
If quantization is applied to reduce energy consumption, then energy efficiency is improved, but computational accuracy deteriorates
Solution Approach 1:
The patent identifies and treats different channels with different precision requirements based on their energy consumption patterns and accuracy sensitivity. By applying quantization selectively to less sensitive channels while maintaining high precision for sensitive channels, the system optimizes overall energy efficiency without sacrificing critical computational accuracy
Solution Approach 2:
The patent applies quantization partially rather than uniformly across the entire neural network. By selectively quantizing only the less sensitive portions of the network while maintaining full precision for sensitive parts, the system achieves significant energy savings while preserving necessary computational accuracy
3Productivity
If heterogeneous computing systems are used to accelerate neural network computations, then processing performance is improved, but system complexity increases
Solution Approach 1:
The patent divides the neural network into different segments or channels based on their quantization sensitivity and computational requirements. This segmentation allows the system to allocate different processing resources (CPU vs. accelerator) to different parts of the network, improving performance while managing complexity through organized division of labor
4Measurement precision
If partial tensor retention is used to maintain accuracy for sensitive channels, then computational accuracy is improved, but computational overhead increases
Solution Approach 1:
The patent applies different processing treatments to different parts of the weight tensor based on channel sensitivity. By identifying sensitive channels and retaining them in original precision while quantizing less sensitive channels, the system achieves accuracy improvement for critical computations without incurring excessive overhead for the entire network
Data Source
AI summary
Systems, methods, and apparatus for partial tensor correction are disclosed. During quantization, weight tensors can be corrected for quantization errors in order to increase accuracy that is otherwise degraded as a result of quantization. To correct errors, the weight tensor is partially corrected using a data-free, non-iterative, per-input channel level technique to achieve accuracy improvement while using lower precision. Further, sensitive channels prone to accuracy degradation due to quantization are identified. Based on this identification, parts of weight tensor is retained for CPU computation and remaining parts of the weight tensor are offloaded for accelerator computation. The proposed partial tensor retention scheme achieves efficient heterogenous DNN computations with improved performance and accuracy on heterogenous systems. Furthermore, combining the partial tensor correction and partial tensor retention techniques allows for achieving improved performance and accuracy in a heterogenous computing environment while using low precision computations.