Top-K Label Selection for NLP Model Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for fine-tuning pre-trained language models in NLP for text classification are hindered by high time and space costs due to complex coding structures and memory-intensive prediction label storage, especially in knowledge distillation for massive label text classification.
Innovation Solution
A method that processes annotated and unannotated datasets to select the maximum class probabilities and corresponding prediction labels, reducing memory usage and improving calculation speed by training a second model based on these selected labels, rather than all prediction labels, and using a weighted loss function for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If knowledge distillation is used to compress the pre-trained model, then model size and complexity are reduced, but memory space is still excessively consumed due to storing prediction labels for all classes
Solution Approach 1:
The patent extracts only the necessary information from the full prediction label vector by selecting top-K class probabilities and their corresponding labels. Instead of storing and processing all N class probabilities, the method extracts only the K most probable classes, significantly reducing memory consumption while preserving the essential knowledge from the teacher model.
Solution Approach 2:
The patent applies partial action by using only a subset (top-K) of the available class probabilities rather than all N classes. This partial approach is sufficient for achieving accurate knowledge distillation while dramatically reducing the memory burden associated with storing and processing the complete prediction label vector.
2Measurement precision
If all prediction labels are used in loss function calculation, then knowledge transfer accuracy is improved, but calculation speed decreases due to high vector latitudes
Solution Approach 1:
The patent extracts only the top-K class probabilities from the full N-dimensional prediction label vector for use in loss function calculation. This extraction reduces the vector latitude from N to K, maintaining the most important knowledge signals while enabling faster computation during the distillation process.
Solution Approach 2:
The method uses partial action by calculating the loss function over only the top-K predicted classes rather than all N classes. This partial computation preserves the critical knowledge transfer accuracy while significantly improving calculation speed by reducing the dimensional complexity of the vectors involved in loss computation.
3Measurement precision
If pre-trained language models are fine-tuned for text classification, then classification accuracy is improved, but time and space costs increase due to complex coding structure
Solution Approach 1:
The patent applies preliminary action by pre-processing the annotation data set using the teacher model to generate top-K prediction labels before the actual fine-tuning process. This preliminary step creates a simplified training target that accelerates the subsequent fine-tuning of the student model, reducing the overall time cost while maintaining classification accuracy.
Solution Approach 2:
The method extracts the essential classification knowledge by selecting only the top-K predicted classes and their probabilities from the teacher model's full prediction output. This extraction creates a streamlined training objective for the student model that reduces computational complexity and fine-tuning time while preserving the most important classification signals.
Data Source
Figure 1~2
Figure 3~4
Figure 5
AI summary
A method for training classification model is provided. The method includes: an annotated data set is processed based on a pre-trained first model, to obtain, for each of annotated sample data in the annotated data set, N first class probabilities, each being a probability that the annotated sample data is classified as a respective one of N classes; for each of the annotated sample data, maximum K first class probabilities are selected from the N first class probabilities, and K first prediction labels, each corresponding to a respective one of K first class probabilities, are determined; and a second model is trained based on the annotated data set, a real label of each of the annotated sample data and the K first prediction labels of each of the annotated sample data. A classification method and device for training classification model are also provided.