LLM Quantization via Clamping Function for Memory Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing quantization techniques for Large Language Models (LLMs) face challenges in preserving accuracy when reducing model parameters from 16-bit floating-point to 4-bit integers, leading to decreased performance quality, especially with larger model sizes.
Innovation Solution
A two-step quantization method is introduced, where floating-point numbers are first converted to a dynamic fixed-point representation with a shared scale, and then further quantized using a clamping function to reduce the number of bits, thereby reducing memory consumption without loss in quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantization is applied to reduce memory consumption and increase inference speed, then memory usage and computational efficiency are improved, but model accuracy and performance quality deteriorate
Solution Approach 1:
The patent applies different quantization strategies to different parts of the model based on their sensitivity to quantization. Weight quantization uses one approach while activation quantization uses another, with each tailored to the specific characteristics of that component. This local differentiation allows aggressive quantization where acceptable and conservative quantization where accuracy is critical.
Solution Approach 2:
The patent dynamically adjusts quantization parameters such as bit-width, scaling factors, and rounding modes based on the specific layer, operation type, and input characteristics. This parameter optimization enables the model to maintain accuracy in critical regions while achieving memory reduction in less sensitive areas.
2Quantity of substance
If quantization to 4-bit or lower precision is applied, then memory consumption is significantly reduced, but performance quality and generation accuracy decrease
Solution Approach 1:
The patent segments the quantization process into multiple stages: calibration phase where a subset of data is used to determine optimal quantization parameters, and execution phase where the pre-determined parameters are applied. This segmentation allows the system to invest computational resources in parameter optimization once, then achieve high compression with minimal runtime overhead.
Solution Approach 2:
The patent performs preliminary calibration and parameter optimization before the actual quantized inference. During this preliminary phase, the system analyzes the data distribution and determines optimal scaling factors, zero-points, and quantization levels. This advance preparation ensures that the subsequent 4-bit or lower precision computation maintains accuracy without requiring additional computational resources during inference.
3Quantity of substance
If aggressive quantization is applied to larger model sizes, then memory efficiency is improved, but the presence of outliers in activations causes performance degradation
Solution Approach 1:
The patent extracts and separately handles outlier values in activations by identifying them through statistical analysis during calibration, then applying specialized quantization strategies such as asymmetric scaling or separate quantization parameters for outlier-prone regions. This extraction allows the majority of normal values to be efficiently quantized while outliers receive targeted handling to prevent performance degradation.
Data Source
AI summary
A method and an apparatus for storing data points are provided. The method comprises: receiving the plurality of data points, each data point of the plurality of data points being represented in a floating-point representation; quantizing each one of the first plurality of data points, by: executing, during a first quantization phase: converting each data point of the plurality of data points into a corresponding first data point of a plurality of first data points; executing, during a second quantization phase: applying, to each first data point of the plurality of first data points a clamping function, thereby converting each first data point of the plurality of first data points into a corresponding second data point of a plurality of second data points; and storing the plurality of second points for further calculations instead of the first plurality of data points.