Neural Network Quantization via Logarithmic Parameter Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for converting neural network models from FP32 to INT8 format suffer from accuracy loss due to the statistical summarization of min and max values during training, which are not differentiable and thus cannot be learned, leading to instability in deep learning applications with non-normalized inputs.
Innovation Solution
The method involves formulating min and max values in an analytical function to clip values beyond a defined range, optimizing the quantization range through an L2 regularization method, and updating these values during training to ensure optimal quantization resolution and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If statistical summarization of min and max values is used during calibration or training, then the quantization process is simple to implement, but the model accuracy deteriorates due to loss of numerical precision and susceptibility to input data spikes
Solution Approach 1:
The patent changes the parameter representation from direct statistical min/max values to logarithmic parameters (alpha and beta) that are differentiable and can be learned during training. This transformation allows the quantization parameters to be optimized through gradient descent while maintaining the clipping functionality, thus improving model accuracy without significantly complicating implementation
Solution Approach 2:
The patent introduces an intermediary analytical function (clipping function with logarithmic parameters) that bridges the gap between statistical summarization and differentiable optimization. This intermediary layer allows gradients to flow through the quantization process while still enforcing the min/max value constraints, resolving the contradiction between simplicity and accuracy
2Adaptability or versatility
If min and max values are updated according to batch statistics during training, then the quantization range adapts to input data, but the quantization range increases dramatically due to sudden spikes in non-normalized inputs, negatively impacting training loss and accuracy
Solution Approach 1:
By transforming min/max values into logarithmic parameters (alpha and beta), the patent creates a bounded optimization space that is less sensitive to extreme values. The logarithmic transformation compresses the scale of large values, preventing sudden spikes from dominating the quantization range while still allowing adaptation to the overall data distribution
Solution Approach 2:
The patent implements feedback through gradient-based optimization where the clipping parameters are continuously updated based on the loss function. This feedback mechanism allows the system to learn optimal quantization ranges that balance adaptability to input distribution with protection against extreme values, improving accuracy by learning from training errors
3Measurement precision
If quantization range is minimized to optimize resolution, then the quantization precision improves, but the clipping function becomes more complex requiring analytical formulation and L2 regularization
Solution Approach 1:
The patent uses parameter transformation (logarithmic scaling) to simplify the optimization landscape. By working in the logarithmic domain, the optimization process naturally tends to find balanced solutions that minimize the quantization range while avoiding the need for complex explicit constraints, as the logarithmic parameters inherently bound the min/max values
Solution Approach 2:
The patent changes the state of the quantization parameters from direct linear values to logarithmic values, which transforms the optimization problem. This parameter change allows the use of L2 regularization to effectively control the quantization range, as the regularization term operates on the logarithmic scale to prevent extreme values while maintaining differentiability
Data Source
AI summary
An electronic apparatus performs a method of quantizing a neural network. The method includes: clipping a value used within the neural network beyond a range from a minimum value to a maximum value; simulating a quantization process using the clipped value; updating the minimum value and the maximum value during a training of the neural network to optimize the quantization process; and quantizing the value used within the neural network according to the updated minimum value and the maximum value. In some embodiments, the method of quantizing a neural network further includes minimizing the range during the training.