LLM Weight Quantization Accuracy via Normalization Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face memory pressure and high computational demands during inference due to their high-precision weight data, which limits their execution on edge devices and results in accuracy issues when quantized to lower precision formats.
Innovation Solution
A post-training model optimization technique that quantizes the weights of linear layers in LLMs to lower precision formats like 4-bit integers, followed by an affine transformation parameter estimation to improve accuracy, and applies normalization correction to restore the model's original accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weight data is quantized to lower-precision format (e.g., 4-bit integer), then memory bandwidth and computational demands are reduced, but accuracy deteriorates
Solution Approach 1:
The patent changes the precision parameter of weight data from high-precision (FP32) to low-precision (INT4) format to reduce memory bandwidth requirements. This parameter change enables efficient deployment on edge devices while the system compensates for accuracy loss through specialized hardware design and calibration techniques.
Solution Approach 2:
The patent introduces calibration data and calibration processes as intermediaries between the quantized model and accurate inference. By using calibration datasets to adjust and fine-tune the quantized weights, the system mediates the accuracy degradation caused by low-precision representation, achieving a balance between memory efficiency and model performance.
2Measurement precision
If weight data is stored in high-precision format (e.g., 32-bit floating point), then accuracy is maintained, but memory pressure increases during inference
Solution Approach 1:
The patent fundamentally changes the storage precision parameter from 32-bit floating point to 4-bit integer format, reducing the memory footprint by approximately 8x. This parameter transformation enables the model to fit within limited on-device memory while maintaining acceptable accuracy through calibration and hardware optimization.
Solution Approach 2:
The patent employs calibration datasets as temporary, disposable resources that are used during the quantization calibration phase and then discarded. These calibration datasets enable accurate weight quantization without requiring permanent storage of high-precision reference data, thus reducing ongoing memory pressure.
3Quantity of substance
If conventional quantization is applied, then memory demands are reduced, but accuracy problems negate the benefits
Solution Approach 1:
The patent introduces calibration processes and calibration datasets as intermediary steps between conventional quantization and final model deployment. This intermediary calibration phase adjusts the quantized weights to minimize accuracy loss, ensuring that the model maintains reliable performance despite using low-precision representation.
Solution Approach 2:
The patent implements feedback mechanisms during calibration where the model's performance on calibration data is continuously monitored and used to adjust the quantization parameters and weights. This feedback loop ensures that accuracy requirements are met while maintaining the memory efficiency benefits of low-precision quantization.
Data Source
AI summary
Systems, apparatuses and methods may provide for technology that accesses a pre-trained artificial intelligence (AI) model, quantizes a plurality of weights of the pre-trained AI model to generate a compressed AI model, and applies normalization correction to the compressed AI model to generate an output AI model.