NLU Model Generation via Interactive Utterance Labeling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The development of dialog applications is a time-consuming process due to the labor-intensive nature of generating natural language understanding (NLU) models, which requires manual labeling of utterances and creating annotation guides.
Innovation Solution
A system and method for rapidly generating NLU models by using interactive tools for user experience personnel to label transcribed data, grouping utterances into categories, and flagging positive and negative examples, which are then used to generate an annotation guide and subsequently an NLU model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling of utterances is used to generate NLU models, then the quality and accuracy of the model can be ensured, but the time and labor required for development increases significantly
Solution Approach 1:
The system performs preliminary clustering of utterances into call types before manual labeling. This preliminary organization reduces the scope of manual work required while maintaining model accuracy, as labelers only need to review and refine pre-clustered groups rather than label all utterances from scratch.
Solution Approach 2:
The system enables self-service model generation by allowing automatic clustering of utterances based on their content and characteristics. This automated preprocessing reduces dependency on extensive manual labeling while preserving the quality needed for accurate NLU models.
2Reliability
If manual labeling of all utterances is performed, then comprehensive coverage of speech data can be achieved, but the labor intensity and cost increase
Solution Approach 1:
The system segments the large corpus of utterances into smaller, manageable clusters based on call types and characteristics. This segmentation allows for more efficient processing and labeling while maintaining comprehensive coverage, as each cluster can be handled independently with reduced labor intensity.
Solution Approach 2:
The system introduces an intermediary clustering mechanism that automatically organizes utterances before manual labeling. This intermediary step reduces the direct labor intensity of comprehensive labeling while ensuring that all utterances are covered through systematic clustering and selective refinement.
3Measurement precision
If traditional NLU model generation processes are used, then thorough analysis of speech data can be performed, but the complexity and time consumption of the process increases
Solution Approach 1:
The complex NLU model generation process is segmented into distinct phases: automatic clustering, preliminary labeling, model generation, and iterative refinement. This segmentation reduces process complexity by breaking down the overwhelming task into manageable steps while maintaining thorough analysis through systematic progression through each phase.
Data Source
AI summary
A system and method is provided for rapidly generating a new spoken dialog application. In one embodiment, a user experience person labels the transcribed data (e.g., 3000 utterances) using a set of interactive tools. The labeled data is then stored in a processed data database. During the labeling process, the user experience person not only groups utterances in various call type categories, but also flags (e.g., 100-200) specific utterances as positive and negative examples for use in an annotation guide. The labeled data in the processed data database can also be used to generate an initial natural language understanding (NLU) model.