Quantized Neural Network Normalization for Accuracy Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The accuracy of large-scale language models decreases significantly due to quantization errors when transitioning from 32-bit floating-point data to smaller bit formats for deployment on user terminals with limited computing resources.
Innovation Solution
A method to adjust normalization layers in a quantized neural network model by compensating for quantization errors in input and output tensors using error calculations and linear regression, allowing for accurate deployment on user terminals.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the neural network model is quantized to use smaller bit formats (16-bit floating-point, 16-bit integer, 8-bit integer) for deployment on user terminals, then the model can be deployed on devices with limited computing resources, but the accuracy of the neural network model significantly decreases due to quantization errors
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing correction values for normalization layers before deployment. The server端 calculates correction values based on quantization errors and stores them in a lookup table, which is then deployed to the client端. This allows the client to quickly correct quantization errors without performing complex real-time calculations, thus maintaining accuracy while enabling deployment on resource-constrained devices.
Solution Approach 2:
The patent introduces correction values as an intermediary element between the quantized model and the original high-precision model. These correction values act as a mediator that bridges the gap between the low-precision quantized representations and the high-precision original data, allowing the normalization layer to compensate for quantization errors without requiring the entire model to operate at high precision.
2Measurement precision
If normalization layers are adjusted to compensate for quantization errors, then model accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent applies this principle by using simple lookup tables stored in memory rather than complex real-time computational algorithms. The correction values are pre-computed and stored in a lookup table that can be quickly queried during inference. This replaces what would otherwise be complex iterative correction computations with simple table lookups, significantly reducing computational complexity while maintaining accuracy compensation.
Solution Approach 2:
The patent changes the parameters of the normalization layer by introducing correction values that adjust the mean and standard deviation calculations. Instead of performing complex error correction computations, the system modifies the normalization parameters using pre-calculated correction factors, which simplifies the computational process while maintaining the ability to compensate for quantization errors.
Data Source
AI summary
A neural network model normalization method may include selecting a first normalization layer included in a first model obtained by quantizing a second model; adjusting the first normalization layer; and providing the first model including the adjusted first normalization layer for deployment on an external device. The adjusting of the first normalization layer includes: adjusting a first input tensor of the first normalization layer to be normalized based on a first error between a second input tensor of a second normalization layer included in the second model and a third input tensor of a third normalization layer included in a third model obtained by dequantizing the first model; and adjusting an first output tensor of the first normalization layer to be corrected based on a second error between a second output tensor of the second normalization layer and a third output tensor of the third normalization layer.