Data Augmentation for Intent Classification Using Pre-trained Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Constructing an intent classification dataset with a sufficient number of intent samples is challenging, leading to struggles in training encoder language models for accurate intent classification in natural language understanding applications.
Innovation Solution
A data augmentation system and method that uses a large, pre-trained autoregressive generative language model, such as GPT-3, to generate new intent samples from an initial intent classification dataset without fine-tuning, thereby creating a sufficiently developed dataset for training intent classifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual intent labeling is performed to create intent classification datasets, then intent classification accuracy can be improved, but the process becomes cumbersome and expensive
Solution Approach 1:
The patent uses a pre-trained autoregressive language model to generate synthetic intent samples that copy the structure and patterns of real intent data. The model learns from a small set of manually labeled examples and generates additional training samples by copying linguistic patterns, thereby reducing the need for extensive manual labeling while maintaining classification accuracy.
Solution Approach 2:
The patent applies preliminary action by pre-training the language model on general language data before fine-tuning it for intent classification. This preliminary pre-training phase allows the model to acquire general language understanding capabilities, which then transfer to the specific intent classification task, reducing the amount of task-specific labeled data needed.
2Measurement precision
If fine-tuning is applied to improve model performance on specific tasks, then task-specific accuracy improves, but the model loses generality
Solution Approach 1:
The patent implements a dynamic approach where the model can operate at different levels of specialization. The system allows the language model to be adapted to specific tasks when needed while maintaining the option to use the general pre-trained model for broader applications. This dynamic flexibility enables tuning the degree of specialization based on the specific requirements of each application scenario.
Solution Approach 2:
The patent achieves universality by designing a system where a single pre-trained language model can serve multiple purposes: general language understanding, intent classification, and generation of synthetic training data. The model's multi-functionality allows it to maintain generality while still providing task-specific performance when fine-tuned, as the underlying pre-trained knowledge serves multiple roles.
3Loss of time
If a small initial dataset is used for training, then data preparation time is reduced, but the intent classifier struggles to correctly classify intents
Solution Approach 1:
The patent applies self-service by enabling the language model to generate its own training data autonomously. The pre-trained model uses its internal knowledge to synthesize realistic intent samples without requiring extensive manual data preparation. This self-service capability allows the system to rapidly expand training datasets while maintaining data quality and classification reliability.
Solution Approach 2:
The patent utilizes parameter changes by adjusting the generation parameters of the language model to produce diverse synthetic intent samples. By modifying parameters such as temperature, top-k sampling, and prompt structures, the system generates varied training examples that improve classifier reliability without requiring proportional increases in manual data preparation time.
Data Source
AI summary
The present disclosure relates to a data augmentation system and method that uses a large pre-trained encoder language model to generate new, useful intent samples from existing intent samples without fine-tuning. In certain embodiments, for a given class (intent), a limited number of sample utterances of a seed intent classification dataset may be concatenated and provided as input to the encoder language model, which may generate new sample utterances for the given class (intent). Additionally, when the augmented dataset is used to fine-tune an encoder language model of an intent classifier, this technique improves the performance of the intent classifier.