LLM Compression with Progressive Pruning and Low-Rank Adaptation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies for compressing large language models (LLMs) face challenges in reducing their size without significantly degrading performance, particularly in resource-constrained environments, due to the high computational requirements of existing pruning methods.
Innovation Solution
The approach involves low rank adaption modules, including trainable rank decomposition matrices, minimally removable structure identification, progressive structured pruning, and dynamic knowledge recovery using both pretraining and instruction datasets to preserve and restore knowledge.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional structured pruning is applied to compress LLMs, then model size is reduced, but performance is significantly degraded
Solution Approach 1:
The patent applies preliminary action by performing progressive structured pruning in multiple stages rather than a single step. The model is pruned incrementally with intermediate fine-tuning steps, allowing the model to adapt gradually to the reduced structure. This staged approach prevents catastrophic performance degradation that would occur with aggressive single-step pruning.
Solution Approach 2:
The patent changes parameters by dynamically adjusting the pruning ratio and fine-tuning hyperparameters across different pruning stages. The pruning ratio increases progressively while fine-tuning learning rates and batch sizes are adjusted to optimize knowledge recovery at each stage, allowing the model to maintain performance while achieving compression.
2Quantity of substance
If aggressive pruning is applied to achieve higher compression ratios, then model size is reduced further, but the amount of fine-tuning training data and GPU resources required increases significantly
Solution Approach 1:
The patent implements periodic action through alternating cycles of pruning and fine-tuning. Rather than continuously increasing pruning aggression, the method periodically pauses pruning to perform fine-tuning that recovers knowledge. This rhythmic alternation allows aggressive overall compression while managing computational resource requirements at each step.
Solution Approach 2:
The patent applies partial action by using moderate pruning ratios at each individual stage rather than attempting maximum compression in one step. While the cumulative compression ratio is high, each intermediate pruning step uses a manageable ratio that requires moderate computational resources, making the process feasible on constrained hardware.
3Quantity of substance
If low rank adaption is used during pruning and fine-tuning, then model compression is achieved, but model performance is significantly reduced
Solution Approach 1:
The patent applies preliminary action by performing extensive fine-tuning with low-rank adaption at intermediate stages before final pruning. This preliminary fine-tuning with the adapted architecture allows the model to recover most of its performance capability before the final compression is applied, minimizing performance loss.
Solution Approach 2:
The patent creates a composite structure by combining the pruned model architecture with low-rank adaption modules. This composite approach integrates the compression benefits of pruning with the performance-preserving capabilities of low-rank adaption, achieving both size reduction and performance maintenance simultaneously.
Data Source
AI summary
Technologies related to compressing a large language model (LLM) that has been fine-tuned using low rank adaption are described. Minimally removable structures in the LLM are identified, and node groups are constructed based upon the identified minimally removable structures. Progressive structured pruning is employed to remove prunable variables corresponding to prunable node groups of the LLM. The pruned LLM is then fine-tuned to recover knowledge lost during pruning.