Low Rank Adapter Layers for LLM Quantization Error
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deploying large language models (LLMs) on edge devices is limited by memory pressure during inference due to high memory bandwidth requirements, and existing model compression techniques often compromise accuracy.
Innovation Solution
The proposed solution involves a low rank adaptation approach for model optimization, which selectively adapts the most problematic linear layers in LLMs using low rank adapter layers to reduce quantization error, thereby improving accuracy without significant overhead on model size or inference latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If weight quantization is applied to reduce model size, then model footprint is reduced, but accuracy deteriorates
Solution Approach 1:
The patent changes the parameter of weight representation from full precision to quantized low-bit formats, and introduces adapter layers with different precision parameters to compensate for the accuracy loss from quantization
Solution Approach 2:
The patent creates a composite model structure combining quantized weight layers and adapter layers, where each component has different precision characteristics that together achieve both compression and accuracy preservation
2Quantity of substance
If model size is reduced for edge deployment, then memory bandwidth requirements are reduced, but model accuracy deteriorates
Solution Approach 1:
The patent segments the model into quantized weight layers and separate adapter layers, allowing the main model to be highly compressed while adapter layers provide accuracy compensation with minimal additional size
3Manufacturing precision
If low rank adapter layers are added to compensate for quantization error, then accuracy is improved, but device complexity increases
Solution Approach 1:
The patent changes the rank parameter of adapter layers to be much smaller than the original weight dimensions, creating low-rank approximations that reduce complexity while maintaining accuracy compensation capability
Data Source
AI summary
Systems, apparatuses and methods may provide for technology that selects a subset of linear layers from a plurality of linear layers in a pre-trained artificial intelligence (AI) model, wherein a quantization error of the subset of linear layers exceeds an error threshold. For each linear layer in the subset of linear layers, the technology solves a singular value decomposition (SVD) approximation, generates a first adapter layer and a second adapter layer based on the SVD decomposition, wherein the first adapter layer and the second adapter layer include weight matrices having a first dimension that is less than a first rank threshold and a second dimension that is greater than a second rank threshold, and determines an inference output based on the linear layer, the first adapter layer and the second adapter layer.