Neural Network Quantization-Aware Training Precision Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network quantization techniques face challenges in reducing memory and calculation costs while maintaining performance, especially when transitioning from floating-point to reduced precision representations, particularly in resource-constrained environments like embedded systems.
Innovation Solution
A learning method that configures a target calculation circuit to gradually reduce quantization precision during training, using techniques like Scale-Adjusted Training (SAT) and Non-uniform-to-Uniform Quantization (N2UQ), allowing for the adaptation of neural network parameters and activation values to lower bit representations, enabling efficient inference on limited resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantization precision is reduced from floating-point to reduced precision, then memory space and computational cost are reduced, but performance deteriorates
Solution Approach 1:
The patent applies preliminary action by performing quantization-aware training before deployment. During this training phase, the neural network learns optimal quantization parameters and adapts to reduced precision representations, ensuring that when the model is later deployed with reduced precision, it maintains performance while achieving the desired memory space reduction.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting quantization parameters during training. The system modifies precision levels, quantization steps, and scaling factors to find the optimal balance between memory efficiency and performance, allowing the model to adapt to different precision requirements while maintaining acceptable accuracy.
2Use of energy by moving object
If quantization precision is reduced from floating-point to reduced precision, then computational cost is reduced, but performance deteriorates
Solution Approach 1:
The patent applies preliminary action by performing quantization-aware training before deployment. During this training phase, the neural network learns optimal quantization parameters and adapts to reduced precision representations, ensuring that when the model is later deployed with reduced precision, it maintains performance while achieving the desired computational cost reduction.
Solution Approach 2:
The patent employs parameter changes by dynamically adjusting quantization parameters during training. The system modifies precision levels, quantization steps, and scaling factors to find the optimal balance between computational efficiency and performance, allowing the model to adapt to different precision requirements while maintaining acceptable accuracy.
3Quantity of substance
If training is performed with reduced precision, then memory and computational resources are reduced, but training time increases
Solution Approach 1:
The patent applies segmentation by dividing the training process into distinct phases: an initial training phase with higher precision to establish good weight initialization, followed by a quantization-aware training phase with reduced precision. This segmented approach allows the model to benefit from reduced memory resources during the majority of training while ensuring adequate performance through the initial high-precision phase.
Solution Approach 2:
The patent applies preliminary action by performing a preliminary training phase with higher precision before transitioning to reduced precision training. This preliminary high-precision training establishes good weight initialization and feature representations, which then serve as a foundation for the subsequent reduced precision training, reducing the overall training time compared to starting directly with reduced precision.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Learning method (700) for a neural network, the learning method being based on a quantization-sensitive technique, the quantization-sensitive technique implementing at least one quantization function, the quantization function enabling the quantization of at least one variable of a layer of neurons, the quantization function being characterized by a quantization precision (p(k)), the quantization precision being defined as the number of steps of the quantization function, the learning method being characterized in that the quantization precision of the quantization function is modified (770) progressively during the epochs of the learning method, to go from an initial quantization precision (2b+2) to a target quantization precision (2b), the initial and target quantization precisions being predefined,the initial quantification accuracy being strictly greater than the target quantification accuracy.