Intent Classifier Augmentation for Ambiguous Training Samples
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing natural language classifiers for chatbots face challenges in accurately interpreting and disambiguating user intents due to high ambiguity in training data, leading to reduced effectiveness in natural language communication.
Innovation Solution
A method for enhancing an intent classifier through training data augmentation by selecting target samples with low ambiguity levels, determining nearest neighboring samples, merging confidence scores, and modifying ambiguity levels to improve classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If more text examples are added to training data to increase classifier accuracy, then the accuracy of intent identification improves, but the complexity of data preparation and processing increases
Solution Approach 1:
The system performs preliminary actions by automatically generating synthetic training data samples before the actual classification task. It creates augmented training datasets by generating additional text examples with predefined intent labels, thereby preparing enhanced training data in advance without requiring manual data collection and annotation efforts
Solution Approach 2:
The system creates copies of existing training data samples to generate augmented datasets. It synthesizes new training examples by transforming and replicating existing samples, maintaining the original data distribution while increasing the overall volume and diversity of training data available for classifier training
2Measurement precision
If manual data annotation is performed to increase training data quality, then the classification accuracy improves, but the time and resources required increase
Solution Approach 1:
The system performs self-service by automatically generating and annotating training data samples without human intervention. It uses predefined intent schemas and automated processing to create labeled training data, thereby eliminating the need for manual annotation while maintaining consistent data quality and intent label accuracy
Solution Approach 2:
The system changes parameters by systematically varying text transformation parameters such as synonym replacement, back-translation languages, and augmentation intensity. These parameter adjustments enable automated generation of diverse training samples with different linguistic characteristics while maintaining correct intent labels, replacing manual annotation processes
3Adaptability or versatility
If the classifier is trained with more diverse intent variations to improve generalization, then the adaptability improves, but the ambiguity in training data increases
Solution Approach 1:
The system applies local quality by maintaining distinct, well-defined intent categories with clear boundary definitions. Each intent type is characterized by specific semantic features and criteria that differentiate it from other intents, ensuring that while the system handles diverse intent variations, each category maintains clear and unambiguous characteristics for accurate classification
Data Source
AI summary
A computer-implemented method, a computer system and a computer program product enhance an intent classifier through training data augmentation. The method includes selecting a target sample from a plurality of samples. The method also includes determining an ambiguity level for the target sample based on confidence scores of at least two intent labels associated with the target sample. The method further includes selecting a nearest neighboring sample from a group of neighboring samples when the ambiguity level is below a threshold. The nearest neighboring sample includes a confidence score associated with an intent label. The method also includes, for every intent label, merging the confidence scores of the two samples into an overall confidence score for the intent label and modifying the ambiguity level using the overall confidence score. Lastly, the method includes labeling the target sample with the intent label when the modified ambiguity level is above the threshold.