Neural Network Gradient Correction for Low-Bit Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training low-bit quantization neural networks focus solely on minimizing forward propagation quantization error, neglecting its impact on gradient calculation, which leads to inconsistency in forward and backward calculations, making them unsuitable for resource-constrained devices.
Innovation Solution
A training method that quantizes continuous real-valued network parameters, calculates and corrects the gradient based on quantization error in both magnitude and direction, and updates the network model using the corrected gradient to reduce inconsistency and enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If low-bit quantization is applied to reduce calculation amount, then device resource requirements are reduced, but gradient calculation accuracy deteriorates due to non-differentiable step functions
Solution Approach 1:
The patent introduces a gradient correction term as an intermediary to bridge the gap between forward propagation quantization error and backward propagation gradient calculation. This correction term compensates for the non-differentiable nature of quantization by adding an auxiliary gradient component that accounts for quantization effects, thereby maintaining gradient accuracy while preserving low-bit quantization benefits
Solution Approach 2:
The patent modifies the gradient calculation parameters by incorporating quantization error compensation terms. Specifically, it adjusts the gradient update rule to include correction factors derived from quantization characteristics, transforming the standard gradient descent into a quantization-aware gradient descent that maintains accuracy in low-bit representations
2Measurement precision
If quantization error minimization is focused in forward propagation, then quantization accuracy improves, but gradient calculation consistency deteriorates due to ignoring quantization error impact on backward propagation
Solution Approach 1:
The patent implements a feedback mechanism where quantization error information from forward propagation is fed back into the backward propagation gradient calculation. The quantization error and its derivatives are used to generate correction terms that adjust the gradient updates, creating a closed-loop system that maintains consistency between forward and backward calculations
Solution Approach 2:
The patent converts the harmful effect of quantization error into a beneficial correction signal. Instead of treating quantization error purely as noise to be minimized, it utilizes the error information to generate gradient correction terms that improve the consistency and accuracy of backward propagation, transforming a detrimental factor into a useful component
Data Source
AI summary
The present disclosure provides training and application methods and apparatuses for a neural network model, and a storage medium. The training method includes: quantizing, in a forward transfer process, a network parameter represented by a continuous real value, and calculating a quantization error; determining, in a backward transfer process, a gradient of a weight in the neural network model; correcting the gradient of the weight based on the calculated quantization error, wherein the correcting includes correcting a magnitude of the gradient and correcting a direction of the gradient; and updating the neural network model according to the corrected gradient.