Dynamic Quantization for Deep Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing quantization techniques for deep neural networks lack flexibility in trading off output quality and inference performance, leading to significant accuracy loss and requiring extensive modifications to the network, which hinders their adoption in resource-constrained mobile and embedded systems.
Innovation Solution
A method for dynamically quantizing feature maps based on predicted maximum and minimum values, allowing for on-the-fly quantization of kernel weights and feature maps, utilizing a weighted average or predetermined formula to initialize these values, and re-quantizing outputs to reduce memory access and maintain accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If aggressive quantization is applied to reduce precision, then execution time and energy consumption are reduced, but output quality deteriorates significantly
Solution Approach 1:
The patent implements dynamic quantization where the quantization bit-width is adjusted based on the actual data distribution during inference. Instead of using a fixed low precision for all computations, the system dynamically determines the appropriate precision level for each operation, allowing it to use lower precision when possible (improving speed) while maintaining higher precision when accuracy is critical (preserving output quality).
Solution Approach 2:
The system changes the precision parameter dynamically during inference based on the characteristics of the input data and intermediate results. By monitoring data ranges and variability, the system adjusts the quantization parameters on-the-fly, enabling it to achieve faster execution when data allows while preventing accuracy degradation when data requires higher precision.
2Device complexity
If fixed precision quantization is used, then hardware implementation is simplified, but flexibility in trading off output quality and performance is lost
Solution Approach 1:
The patent introduces dynamic adaptability to the quantization process by implementing runtime decision-making about precision levels. The system can adapt to different input data characteristics, model layers, and performance requirements, providing flexibility without requiring complex hardware modifications. This is achieved through software-controlled dynamic precision adjustment rather than fixed hardware precision settings.
3Productivity
If existing quantization techniques are applied, then inference cost is reduced, but extensive modifications to the network are required
Solution Approach 1:
The patent implements a self-adaptive quantization system that automatically determines appropriate precision levels without requiring manual network redesign or extensive modifications. The system analyzes the data flow and computational requirements during inference and autonomously adjusts quantization parameters, making the process self-service rather than requiring external intervention or complex network restructuring.
4Quantity of substance
If low precision quantization is applied, then memory capacity requirement is reduced, but accuracy loss increases
Solution Approach 1:
The system dynamically changes the precision parameter based on the actual memory requirements and accuracy needs of different computation stages. By monitoring data ranges and the impact of quantization on intermediate results, the system adjusts the bit-width allocation dynamically, using lower precision when memory constraints are tight but accuracy can be maintained, and higher precision when accuracy requirements demand it.
Data Source
AI summary
A method for dynamically quantizing feature maps of a received image. The method includes convolving an image based on a predicted maximum value, a predicted minimum value, trained kernel weights and the image data. The input data is quantized based on the predicted minimum value and predicted maximum value. The output of the convolution is computed into an accumulator and re-quantized. The re-quantized value is output to an external memory. The predicted min value and the predicted max value are computed based on the previous max values and min values with a weighted average or a pre-determined formula. Initial min value and max value are computed based on known quantization methods and utilized for initializing the predicted min value and predicted max value in the quantization process.