Out-of-domain Data Augmentation for NLP Intent Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing chatbot systems struggle with accurately classifying user utterances into intents due to inadequate training data, leading to frequent misclassification of intents, especially when encountering out-of-domain (OOD) data.
Innovation Solution
The proposed solution involves augmenting the training data with OOD examples, filtering out examples with similar context to the in-domain data, determining a difficulty value for each OOD example, and generating augmented batches of utterances for training the machine-learning model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If training data is augmented with out-of-domain examples, then the model's ability to learn intent patterns improves, but the complexity of data processing increases
Solution Approach 1:
The patent applies preliminary action by filtering out-of-domain examples with similar context before training. The system pre-processes the data by removing examples that have contextual similarity to in-domain data, ensuring that only truly out-of-domain examples are used for augmentation. This preliminary filtering step simplifies the overall data processing pipeline while maintaining the benefits of OOD augmentation for improving intent classification accuracy.
2Quantity of substance
If all OOD examples are used for training, then data coverage increases, but noise from similar context examples reduces model performance
Solution Approach 1:
The patent applies the taking out principle by extracting and removing out-of-domain examples that have contextual similarity to in-domain data. The system identifies and extracts these similar examples from the OOD dataset and excludes them from training. This ensures that only pure out-of-domain examples with distinct contexts are used, preventing noise from similar-context examples from degrading model performance while maintaining comprehensive data coverage.
3Reliability
If difficult OOD examples are prioritized in training batches, then model resilience improves, but training time increases
Solution Approach 1:
The patent applies dynamics by implementing dynamic batch generation that adapts to the difficulty of out-of-domain examples. The system calculates difficulty values for each OOD example and dynamically adjusts the composition of training batches based on these values. This dynamic approach allows the model to learn from difficult examples more effectively, improving resilience to irrelevant context, while the automated batch generation process optimizes training efficiency by balancing difficulty distribution across batches.
Data Source
AI summary
A method includes receiving a training set of utterances for training a machine-learning model to identify one or more intents for one or more utterances, and augmenting the training set of utterances with out-of-domain (OOD) examples. The augmenting includes: generating a data set of OOD examples, filtering out OOD examples from the data set of OOD examples, determining a difficulty value for each OOD example remaining within the filtered data set of the OOD examples, and generating augmented batches of utterances including utterances from the training set of utterances and utterances from the filtered data set of the OOD based on the difficulty value for each OOD. Thereafter, the machine-learning model is trained using the augmented batches of utterances in accordance with a curriculum training protocol.