Distilled BERT Student Model for CPU Text Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks, such as BERT, are resource-intensive and require long training times, making them unsuitable for smaller applications with only CPU resources, and struggle with out-of-distribution (OOD) classification due to inaccurate probabilistic distributions and the difficulty in identifying inputs beyond pre-defined classes.
Innovation Solution
A smaller student model is trained using knowledge distillation from a BERT teacher model, supplemented with OOD training, and employs preprocessing with Gaussian noise, separation of reference class vectors, and sparsification to improve OOD identification accuracy, and generates OOD samples from in-distribution samples to enhance detection capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If BERT model is used for text classification, then classification accuracy is improved, but training time and hardware resource requirements increase significantly
Solution Approach 1:
The patent creates a distilled student model that copies the essential classification knowledge from the pre-trained BERT teacher model. The student model is trained using the teacher's predictions and probability distributions as training targets, enabling it to replicate BERT's classification accuracy while requiring significantly fewer computational resources and training time.
Solution Approach 2:
The patent extracts only the necessary classification knowledge from the full BERT model by using its predictions and probability distributions as training signals for a smaller student model. This extraction process removes unnecessary components and complexity while preserving the essential classification capabilities needed for the specific text classification task.
2Measurement precision
If BERT model is used for text classification, then classification accuracy is improved, but hardware resource requirements increase
Solution Approach 1:
The student model copies the classification functionality from BERT with much lower hardware requirements. By training a compact model to replicate BERT's behavior through knowledge distillation, the system achieves comparable accuracy on CPU-only workstations without needing GPU hardware.
Solution Approach 2:
The patent replaces the expensive, resource-intensive BERT model with a cheaper, lighter student model that can run on standard CPU hardware. This substitution makes text classification accessible on ordinary workstations without requiring specialized GPU infrastructure.
3Use of energy by moving object
If traditional classification models are used, then resource consumption is reduced, but ability to handle out-of-distribution inputs deteriorates
Solution Approach 1:
The student model copies not only the classification predictions but also the probability distributions from the BERT teacher model. This includes copying the teacher's ability to identify OOD inputs through its probability confidence scores, enabling the resource-efficient student model to detect out-of-distribution samples effectively.
Solution Approach 2:
The patent performs preliminary OOD detection by analyzing the probability distributions generated during the distillation process. The student model is trained to recognize patterns in probability confidence that indicate OOD inputs, allowing it to identify out-of-distribution samples before making final classification decisions.
Data Source
AI summary
Embodiments described herein provides a training mechanism that transfers the knowledge from a trained BERT model into a much smaller model to approximate the behavior of BERT. Specifically, the BERT model may be treated as a teacher model, and a much smaller student model may be trained using the same inputs to the teacher model and the output from the teacher model. In this way, the student model can be trained within a much shorter time than the BERT teacher model, but with comparable performance with BERT.