NLU Training Data Generation via Paraphrase Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current dialogue systems face challenges in generating accurate annotations for user inputs, leading to poor performance in understanding natural language, particularly in generating inaccurate annotations for conversational turns similar to previously encountered inputs, which requires extensive human intervention and computational resources for training data improvement.
Innovation Solution
The system identifies conversational turns with accurate annotations and generates paraphrases to create additional training data, associating the same correct annotation with each paraphrase, thereby improving the natural language understanding engine's performance by targeting areas where it performs poorly without the need for extensive human interaction or computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the natural language understanding engine is trained on more training data to improve annotation accuracy, then the annotation precision improves, but the time and computational resources required for training increase
Solution Approach 1:
The patent segments the training data into two categories: (1) training data covering conversational turns similar to those the engine already handles well, and (2) training data covering conversational turns where the engine performs poorly. By segmenting the training approach, the system focuses computational resources on the problematic segments rather than uniformly training on all possible conversational turns, thereby improving annotation accuracy for difficult cases without proportionally increasing overall training time.
Solution Approach 2:
The patent employs preliminary action by using an initial version of the NLU engine to process conversational turns during dialogue operations, collecting real-world performance data before final training. The system preliminarily identifies which types of conversational turns cause annotation errors, then uses this pre-analyzed information to guide subsequent targeted training, avoiding the need to train on all possible conversational scenarios and reducing overall training time while improving accuracy on problematic cases.
2Measurement precision
If the system collects and processes more conversational data to improve training quality, then the annotation accuracy improves, but the computational resources required increase
Solution Approach 1:
The patent applies local quality by making different parts of the training process have different priorities. Specifically, the system identifies specific conversational turn patterns where the NLU engine performs poorly and concentrates computational resources on generating and training with paraphrases of those specific patterns. Rather than uniformly processing all conversational data with equal computational effort, the system allocates computational resources locally to the areas that need improvement most, thereby improving annotation accuracy without proportionally increasing overall computational resource consumption.
3Adaptability or versatility
If the natural language understanding engine is trained on diverse conversational turns to improve versatility, then the adaptability improves, but the difficulty of detecting and measuring performance degradation increases
Solution Approach 1:
The patent implements feedback by continuously monitoring the NLU engine's annotation performance on real conversational turns during dialogue operations. The system collects feedback on which conversational turns result in annotation errors, then uses this feedback to identify patterns and generate targeted paraphrases for retraining. This closed-loop feedback mechanism enables the system to adapt to diverse conversational turns while maintaining simple performance monitoring, as the feedback directly indicates where improvements are needed without requiring complex overall performance evaluation.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating additional training data for a natural language understanding engine. One of the methods includes: obtaining data identifying (i) a first input conversational turn and (ii) a first annotation, determining that the first annotation accurately characterized the first input conversational turn, determining that the natural language understanding engine is likely to generate inaccurate annotations of other conversational turns that are similar to the first input conversational turn, in response to the determining, obtaining one or more first paraphrases of the first input conversational turn; and generating, for each of the one or more first paraphrases, a respective first training example that identifies the first annotation as the correct annotation for the first paraphrase; and training the natural language understanding engine on at least the first training examples.