Natural Language Classifier Training Data Expansion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current natural language classifiers for chatbots face challenges in achieving high degrees of natural language comprehension due to the difficulty and resource-intensiveness of developing training data that covers variously expressed intents, leading to limited understanding and interpretation of user inputs.
Innovation Solution
A method is introduced to efficiently expand initial training data by generating an updated set of distinct, diversified, and expansive expressions corresponding to an intent, using a statistical language model and cognitive training data expansion module to improve natural language comprehension by training the natural language classifier.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training data is manually curated to cover variously expressed intents, then natural language comprehension accuracy is improved, but time and resource costs increase significantly
Solution Approach 1:
The system performs preliminary actions by automatically generating diverse expressions for each intent category before the actual training process. The expression generation module creates synthetic training data in advance, covering various ways users might express the same intent, thereby eliminating the need for time-consuming manual curation while ensuring comprehensive coverage of potential user inputs
Solution Approach 2:
The training data generation process is self-service in nature, where the system automatically generates its own training data without requiring external manual intervention. The expression generation module uses the intent category and sample expressions to autonomously create additional training examples, making the system self-sufficient in preparing training data
2Measurement precision
If training data is manually curated to cover variously expressed intents, then natural language comprehension accuracy is improved, but resource costs increase significantly
Solution Approach 1:
The system performs self-service by automatically generating diverse expressions for each intent category before the actual training process. The expression generation module creates synthetic training data in advance, covering various ways users might express the same intent, thereby eliminating the need for time-consuming manual curation while ensuring comprehensive coverage of potential user inputs
Solution Approach 2:
The system uses copying by generating synthetic expressions that replicate the structure and semantics of sample expressions. The expression generation module creates multiple variations by substituting synonyms and rephrasing, effectively copying the intent while diversifying the表达方式, thereby reducing the need for manual creation of unique training examples
3Productivity
If the natural language classifier is trained with limited expressions, then training speed is improved, but the degree of natural language comprehension is reduced
Solution Approach 1:
The system performs preliminary actions by automatically generating diverse expressions for each intent category before the actual training process. The expression generation module creates synthetic training data in advance, covering various ways users might express the same intent, thereby eliminating the need for time-consuming manual curation while ensuring comprehensive coverage of potential user inputs
Solution Approach 2:
The system applies parameter changes by transforming the sample expressions into diverse variations using synonym substitution and rephrasing techniques. The expression generation module modifies linguistic parameters such as word choice, sentence structure, and phrasing while preserving the core intent, thereby expanding the training data coverage without proportionally increasing training time
Data Source
AI summary
A computer-implemented method for training a natural language classifier associated with a chat interface of a computer system is provided. The method may include receiving a training dataset comprising an initial set of expressions corresponding to an intent. Additional expressions corresponding to the intent may be generated, wherein the additional expressions are generated based on the initial set of expressions corresponding to the intent. The natural language classifier may be trained based on the initial set of expressions and the additional expressions corresponding to the intent. The trained natural language classifier may be implemented to determine an intent expressed by a detected query based on the initial set of expressions and the additional expressions by which the natural language classifier was trained.