Multilevel Knowledge Distillation for Pre-trained Language Model Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing language model compression methods are task-specific and require time-consuming fine-tuning, making them inefficient for deployment on devices with limited memory and resources, and lack a general compression architecture independent of tasks.
Innovation Solution
A method and platform for pre-trained language model automatic compression using multilevel knowledge distillation, which involves constructing knowledge structures at different levels, meta-learning to generate a general compression architecture, and an evolutionary algorithm to search for an optimal compression structure, utilizing a meta-network and Bernoulli distribution sampling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If knowledge distillation is performed at multiple levels (self-attention units, hidden layer states, embedded layers), then the compression effectiveness and accuracy are improved, but the device complexity and computational cost increase
Solution Approach 1:
The patent segments the knowledge distillation process into three distinct levels: self-attention unit level, hidden layer state level, and embedded layer level. Each level processes and distills knowledge independently, allowing the system to capture different types of linguistic patterns and representations at appropriate granularity, thereby improving compression accuracy while managing complexity through modular organization
Solution Approach 2:
The patent designs a universal distillation structure that can simultaneously perform multiple functions across different levels of the language model. The same basic distillation framework is applied recursively at self-attention units, hidden layers, and embedded layers, creating a multi-functional system that handles various types of knowledge transfer in a unified manner
2Reliability
If existing task-specific compression methods are used, then the model can be compressed for a specific task, but the model requires time-consuming fine-tuning and high computing cost when facing other downstream tasks
Solution Approach 1:
The patent creates a task-agnostic compression framework that produces universally applicable compressed models. By performing knowledge distillation at multiple hierarchical levels without task-specific customization, the resulting compressed model retains adaptability to various downstream tasks, eliminating the need for extensive fine-tuning when deploying to new tasks while maintaining compression effectiveness
3Ease of manufacture
If artificially designed knowledge distillation strategies are used, then the distillation process can be controlled, but it is almost impossible to design all possible distillation structures and find an optimal structure
Solution Approach 1:
The patent transitions from static, manually designed distillation structures to dynamic, automatically generated distillation structures. The system adaptively determines the optimal distillation configuration at each level based on the input data characteristics, allowing the distillation process to be both controllable and adaptable to different scenarios without requiring exhaustive manual design
4Reliability
If large-scale pre-trained language models are deployed on devices with limited memory, then the model performance can be maintained, but the deployment faces huge challenges due to memory constraints
Solution Approach 1:
The patent extracts and distills essential knowledge from large-scale pre-trained language models at multiple levels (embedded layers, hidden states, self-attention mechanisms). By selectively extracting the most important linguistic patterns and representations across different hierarchical levels, the system creates a compressed model that fits within limited memory constraints while preserving essential language understanding capabilities
Data Source
AI summary
Disclosed are an automatic compression method and platform for a pre-trained language model based on multilevel knowledge distillation. The method includes the following steps: step 1, constructing multilevel knowledge distillation, and distilling a knowledge structure of a large model at three different levels: a self-attention unit, a hidden layer state and an embedded layer; step 2, training a knowledge distillation network of meta-learning to generate a general compression architecture of a plurality of pre-trained language models; and step 3, searching for an optimal compression structure based on an evolutionary algorithm. Firstly, the knowledge distillation based on meta-learning is studied to generate the general compression architecture of the plurality of pre-trained language models; and secondly, on the basis of a trained meta-learning network, the optimal compression structure is searched for via the evolutionary algorithm, so as to obtain an optimal general compression architecture of the pre-trained language model independent of tasks.