Dense Knowledge Distillation for Compact NLP Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High-performance neural network-based Natural Language Processing (NLP) models like BERT require significant computational resources and storage, making them impractical for resource-limited devices such as mobile devices, and existing knowledge distillation methods are not task-independent or efficient for transformer-based architectures like BERT.
Innovation Solution
A dense knowledge distillation approach is used to train a reduced-scale NLP model by leveraging a full-scale teacher model, incorporating both masked and unmasked tokens, and IsNext/NotNext predictions to generate multiple loss functions, allowing the student model to mimic the teacher's behavior efficiently, thus reducing the need for heuristics and labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-performance neural network-based NLP models like BERT are used, then NLP performance is improved, but computational resources and storage space requirements increase significantly
Solution Approach 1:
The patent creates a compressed student model that copies the essential knowledge and representations from the full-scale teacher model. The student model learns to generate the same hidden state representations as the teacher model would produce, thereby capturing the core NLP processing capabilities with significantly fewer parameters and lower computational requirements.
Solution Approach 2:
The patent extracts and transfers only the essential knowledge distillation components from the full-scale teacher model to the student model. By extracting the knowledge representations and using them to train the student model, the system separates the essential NLP processing functionality from the computational overhead of the full model architecture.
2Reliability
If high-performance neural network-based NLP models like BERT are used, then NLP performance is improved, but storage space requirements increase significantly
Solution Approach 1:
The patent creates a compressed student model that copies the essential knowledge and representations from the full-scale teacher model. The student model learns to generate the same hidden state representations as the teacher model would produce, thereby capturing the core NLP processing capabilities with significantly fewer parameters and lower computational requirements.
Solution Approach 2:
The patent changes the parameter scale from the full-scale teacher model (110 million parameters) to a compressed student model (fewer parameters). By adjusting the parameter dimensions and using knowledge distillation techniques, the system maintains essential NLP performance while reducing the storage footprint to make the model suitable for resource-limited devices.
3Device complexity
If existing knowledge distillation methods are used, then model size is reduced, but they are not task-independent or efficient for transformer-based architectures like BERT
Solution Approach 1:
The patent creates a universal student model that can perform multiple NLP tasks by learning the general knowledge representations from the teacher model. The student model is trained to generate appropriate hidden state representations for various tasks without requiring task-specific fine-tuning, making it adaptable to different NLP applications while maintaining a compact size.
Data Source
AI summary
Techniques for training a first neural network (NN) model using a pre-trained second NN model are disclosed. In an example, training data is input to the first and second models. The training data includes masked tokens and unmasked tokens. In response, the first model generates a first prediction associated with a masked token and a second prediction associated with an unmasked token, and the second model generates a third prediction associated with the masked token and a fourth prediction associated with the unmasked token. The first model is trained, based at least in part on the first, second, third, and fourth predictions. In another example, a prediction associated with a masked token, a prediction associated with an unmasked token, and a prediction associated with whether two sentences of training data are adjacent sentences are received from each of the first and second models. The first model is trained using the predictions.