Dynamic Quantization for Edge Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks deployed on edge devices face accuracy loss due to quantization, as existing methods like post training quantization and quantization aware training have limitations in handling dynamic data formats and constraints, leading to suboptimal inference performance.
Innovation Solution
Implementing dynamic quantization for neural networks by calculating statistics on output layers and dynamically adjusting data formats and weight kernels based on current image statistics, allowing for real-time adaptation and parallel processing in heterogeneous architectures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If post training quantization is used to minimize effort in porting neural networks to edge devices, then ease of operation is improved, but manufacturing precision deteriorates due to locked data formats and precision loss
Solution Approach 1:
The patent implements dynamic quantization where data formats are not locked but continuously adjusted based on input statistics. The system calculates statistics like min, max, and variance for each layer during inference and dynamically determines optimal data formats, allowing the quantization parameters to adapt to different input distributions while maintaining ease of deployment on edge devices.
Solution Approach 2:
The system changes quantization parameters (data formats, precision levels) based on calculated statistics from representative datasets. By computing layer-wise statistics and using them to determine optimal fixed-point representations, the system achieves better accuracy while maintaining the simplicity of post-training quantization deployment.
2Manufacturing precision
If quantization aware training is used to model quantization loss and improve accuracy, then manufacturing precision is improved, but device complexity increases due to additional training requirements and unmodeled constraints
Solution Approach 1:
The patent performs preliminary calculation of statistics (min, max, variance, histogram) on representative datasets before finalizing the quantization scheme. This preliminary action allows the system to determine optimal data formats in advance without requiring complex quantization-aware training, reducing training complexity while maintaining accuracy.
Solution Approach 2:
The system uses calculated statistics from representative datasets as feedback to determine optimal data formats for each layer. This feedback mechanism allows the system to iteratively refine quantization parameters based on actual performance metrics, achieving accuracy improvement without the complexity of quantization-aware training.
3Use of energy by moving object
If fixed point container format is used to save bandwidth and computation cost, then use of energy is improved, but manufacturing precision deteriorates due to trade-off between range and precision
Solution Approach 1:
The patent applies different data formats and precision levels to different layers based on their specific requirements. By calculating statistics for each layer individually and determining optimal fixed-point representations locally, the system achieves energy efficiency while minimizing precision loss in critical layers.
Solution Approach 2:
The system dynamically adjusts precision and range parameters of fixed-point representations based on layer-wise statistics. By changing these parameters optimally for each layer rather than using a uniform format, the system achieves the right balance between energy efficiency and precision.
Data Source
AI summary
A method of generating a quantized neural network comprises (i) receiving a pre-trained neural network model and (ii) modifying the pre-trained neural network model to calculate one or more statistics on an output of one or more layers of the pre-trained neural network model based on a current image and set up an output data format for one or more following layers of the pre-trained neural network model for one or more of the current image and a subsequent image dynamically based on the one or more statistics.