LLM Compression and Domain Tuning With Pruning and Factorization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in resource efficiency, computational power, speed, and deployment due to their substantial size and computational requirements, which limit their use on devices with limited infrastructure and increase energy consumption and operational costs.
Innovation Solution
A method and system for compressing and tuning LLMs through dependency-wise pruning and rank-based factorization, followed by updating with additional layers based on factorized weights, to generate a compressed and fine-tuned model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If LLMs are trained with vast amounts of data to enhance computational power, then model accuracy and capability are improved, but model size and resource requirements increase significantly
Solution Approach 1:
The patent segments the large language model into multiple smaller models or modules, each handling specific tasks or aspects of language processing. This segmentation allows the system to maintain high accuracy through specialized sub-models while reducing the overall resource burden compared to a single monolithic large model.
Solution Approach 2:
The patent applies local quality by assigning different levels of model complexity to different parts of the system based on task requirements. High-accuracy models are deployed only where needed, while simpler models handle routine tasks, optimizing the balance between accuracy and resource consumption across the system.
2Adaptability or versatility
If LLMs are deployed on devices with limited infrastructure, then accessibility and deployment flexibility are improved, but computational power and processing speed deteriorate
Solution Approach 1:
The patent implements dynamic model deployment that adapts to the computational capabilities of the target device. The system can dynamically select between different model sizes, precision levels, or processing modes based on available resources, allowing flexible deployment across devices ranging from high-performance servers to resource-constrained mobile devices while maintaining acceptable performance.
3Reliability
If LLMs are trained extensively to improve computational power, then model capability is enhanced, but energy consumption and operational cost increase
Solution Approach 1:
The patent employs parameter changes by adjusting model precision (e.g., using lower-precision data types), reducing model depth or width in certain layers, and optimizing training parameters to achieve acceptable performance with reduced computational requirements. These parameter adjustments directly lower energy consumption while maintaining sufficient model capability for practical applications.
4Quantity of substance
If LLMs are pruned to reduce model size, then resource requirements are reduced, but computational power and accuracy deteriorate
Solution Approach 1:
The patent implements iterative pruning and recovery cycles where less important model components are discarded to reduce size, then the model is fine-tuned or augmented with knowledge distillation to recover lost capabilities. This cyclical process allows progressive model compression while minimizing accuracy degradation through systematic recovery phases.
Data Source
AI summary
A method and a system of compressing and tuning large language models is disclosed. A processor 104 receives an LLM, a pruning ratio, an initial rank, and a set of target layers from a plurality of layers of the LLM. A dependency-wise pruning is performed of the LLM based on the pruning ratio. A rank-based factorization of the LLM is performed based on the initial rank to generate factorized weights. A pruned LLM is determined based on the dependency-wise pruning. The pruned LLM is updated by injecting one or more additional layers to one or more corresponding layers of the pruned LLM to generate a compressed LLM. The compressed LLM is fine-tuned for a specific domain or for a specific task by fine-tuning the factorized weights for the additional layers of the compressed LLM based on the domain-specific training data or task-specific training data.