Language Model Compression via Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale language models face challenges in being applied to smart devices due to increasing computational complexity and difficulty in generalizing with small samples during compression.
Innovation Solution
A compression method and platform for pre-training language models using knowledge distillation, incorporating a feature mapping knowledge distillation module, self-attention cross knowledge distillation module, and a linear learning module based on Bernoulli probability distribution, which enables incremental learning between teacher and student models, improving learning quality and generalization ability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If one-way compression method is used to distill knowledge from teacher model to student model, then model size is reduced, but small samples are difficult to generalize
Solution Approach 1:
The patent combines multiple distillation strategies (feature mapping distillation, self-attention cross distillation, and linear transfer based on Bernoulli probability) into a unified compression framework. This merging of multiple knowledge transfer mechanisms enables the student model to learn from the teacher model more effectively, improving generalization ability while maintaining reduced model size.
Solution Approach 2:
The patent introduces intermediate layer features of the teacher model as mediators to guide the student model training. By using intermediate features as bridges between teacher and student models, the knowledge transfer becomes more gradual and effective, helping small samples generalize better in the compressed model.
2Ease of operation
If large-scale language model is compressed, then deployment on smart devices becomes feasible, but learning quality in early training stage deteriorates
Solution Approach 1:
The patent applies preliminary actions by pre-processing the knowledge transfer process through feature mapping distillation before main training. The intermediate layer features are extracted and used to guide early student model training, preparing the foundation for better learning quality before full-scale compression training begins.
Solution Approach 2:
The patent changes training parameters dynamically using Bernoulli probability distribution for linear transfer. This probabilistic approach allows the model to adaptively adjust knowledge transfer intensity during training, maintaining learning quality while enabling deployment on resource-constrained devices.
3Quantity of substance
If computational complexity is reduced for smart device deployment, then model size decreases, but reasoning speed improvement is limited
Solution Approach 1:
The patent extracts only the essential knowledge from the teacher model through selective feature mapping and self-attention distillation. By taking out only the critical knowledge components rather than compressing the entire model structure, the student model achieves better reasoning speed with appropriately reduced size for smart device deployment.
Data Source
AI summary
Provided is a method and a platform for compressing a pre-training language model based on knowledge distillation. According to the method, a universal knowledge distillation strategy of feature migration is firstly designed, and in the process of knowledge distillation from the teacher model to the student model, the feature mapping of each layer of the student model is approaching the teacher's features, focusing on the ability of small samples to express features in the intermediate layer of the teacher model, and guiding the student model by using these features; then, a knowledge distillation method based on self-attention cross is constructed; finally, a linear transfer strategy based on Bernoulli probability distribution is designed to gradually complete the knowledge transfer of feature mapping and self-attention distribution from teachers to students.