LLM Weight Matrix Mixed-Precision Quantization for GPU Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) require significant memory resources, making them difficult to upload to general-purpose GPUs due to high memory demands, and quantization methods to reduce bit precision often lead to performance degradation.
Innovation Solution
A mixed-precision scheme is employed to identify and separate weight columns based on sensitivity to quantization, preserving higher precision for sensitive columns and applying quantization to others, using a dedicated GPU kernel for efficient calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If quantization is applied to reduce bit precision of weights, then memory requirements are reduced, but model inference performance degrades
Solution Approach 1:
The patent applies different quantization strategies to different columns of the weight matrix based on their sensitivity characteristics. Columns identified as sensitive to quantization (through Hessian-based sensitivity analysis) are preserved at full precision, while less sensitive columns are quantized to lower bit precision. This local differentiation allows the system to reduce overall memory requirements while preserving performance-critical weight components.
Solution Approach 2:
The weight matrix is segmented into multiple columns that are processed differently according to their sensitivity to quantization. By dividing the weight matrix into sensitive and non-sensitive columns, the system can apply quantization selectively to portions of the model that are less critical for performance, thereby reducing memory footprint without significantly impacting inference quality.
2Reliability
If full precision weights are stored for all columns, then model inference performance is maintained, but memory requirements increase significantly
Solution Approach 1:
Instead of applying uniform precision to all weight columns, the system evaluates the sensitivity of each column to quantization errors and assigns appropriate precision levels accordingly. Columns with low sensitivity can use reduced precision, while high-sensitivity columns maintain full precision, optimizing the balance between memory efficiency and performance preservation.
Solution Approach 2:
The system changes the precision parameter of weight storage dynamically based on column sensitivity characteristics. By adjusting the bit precision parameter for different columns according to their Hessian-based sensitivity scores, the system achieves optimal memory utilization while maintaining inference performance for critical components.
3Device complexity
If uniform quantization is applied to all weight columns, then implementation is simple, but performance degradation increases due to loss of precision in sensitive columns
Solution Approach 1:
Before applying quantization, the system performs a preliminary sensitivity analysis using Hessian matrices to identify which columns are most sensitive to quantization errors. This preliminary characterization allows the system to plan and execute selective quantization strategies that preserve performance by protecting sensitive columns from quantization while still reducing precision for less critical columns.
Data Source
AI summary
The disclosure relates to a method and an apparatus for lightweighting of artificial intelligence models, and the method of lightweighting of artificial intelligence models includes identifying an outlier in an input vector of a layer, identifying at least one column corresponding to the outlier in a weight matrix, and quantizing weight values of columns which do not correspond to the outlier.