LLM Compression Using Dual Distillation and Matrix Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) are computationally and memory-intensive, posing deployment challenges on resource-limited devices, and conventional compression techniques like structured pruning and knowledge distillation require extensive computations or induce significant accuracy drops.
Innovation Solution
A training-free method for compressing LLMs using dual knowledge distillation, which groups matrices into modules and applies joint matrix decomposition, reducing dimensions without extensive training or backward propagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Weight of stationary object
If knowledge distillation is used to compress LLMs, then model size is reduced, but extensive computations and training data are required
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing covariance matrices and their inverses during the training phase of the teacher model. These pre-computed components are then directly used in the student model initialization, eliminating the need for extensive post-training computations and data processing while achieving effective knowledge transfer and compression.
2Weight of stationary object
If structured model pruning is used to compress LLMs, then model size is reduced, but accuracy drops significantly
Solution Approach 1:
The patent employs copying by initializing the student model weights as a scaled version of the teacher model weights, specifically using the formula W_student = sqrt(r) * W_teacher where r is the compression ratio. This weight copying strategy preserves the essential patterns and relationships learned by the teacher model while reducing the overall model size, thereby maintaining high accuracy without the significant drops associated with structured pruning.
3Weight of stationary object
If matrix decomposition is applied to compress LLMs, then model size is reduced, but computing resources and training time are increased
Solution Approach 1:
The patent applies preliminary action by pre-computing covariance matrices and their inverses during the teacher model's training phase. These pre-computed components are then directly utilized for student model initialization, eliminating the need for time-consuming matrix decomposition operations during the compression process and significantly reducing training time while achieving effective model size reduction.
Data Source
AI summary
A method performed by an electronic device, includes: receiving a calibration dataset corresponding to a task relevant to a user; providing the calibration dataset as an input to a trained teacher model and an on-device model that are stored in the electronic device; updating the on-device model by, for each layer of the on-device model, determining covariances between the trained teacher model and the on-device model based on: differences between feature maps generated by the trained teacher model and the on-device model, and differences between weights in the trained teacher model and weights in the on-device model; and modifying the weights for the on-device model using matrix decomposition on weight matrices in the trained teacher model based on the determined covariances.