NLP Intent Hierarchy for Low Confidence Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Natural language processing systems face challenges in determining user intents from utterances with low confidence, often requiring human analysts for clarification or presenting repetitive lists of options, which can frustrate users and increase costs.
Innovation Solution
The system employs a domain hierarchy with leaf nodes representing specific intents and ancestor nodes for more general intents, using confidence scores to select ancestor nodes when specific intent classification is uncertain, and intelligently prompts users for more specific information without relying on human analysts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system queries a human intent analyst for clarification when confidence is low, then the accuracy of intent determination is improved, but the cost and time required increase
Solution Approach 1:
The system performs preliminary classification of utterances into confidence categories (high, medium, low) before determining the appropriate response path. This preliminary action allows the system to identify which utterances require human analyst intervention and which can be handled autonomously, optimizing the allocation of human resources and reducing unnecessary delays in high-confidence cases.
2Adaptability or versatility
If the system presents a long list of possible intents to the user, then the completeness of intent options is improved, but the user experience deteriorates due to frustration and repetitiveness
Solution Approach 1:
The system applies different response strategies based on the local characteristics of each utterance's confidence level. For high-confidence utterances, it provides direct responses; for medium-confidence utterances, it offers targeted clarification options; and for low-confidence utterances, it escalates to human analysts. This localized approach ensures comprehensive intent coverage while maintaining excellent user experience by avoiding generic long lists for most cases.
3Reliability
If the system uses a hierarchical intent ontology with ancestor nodes, then the robustness of intent classification is improved, but the system complexity increases
Solution Approach 1:
The intent ontology is segmented into a hierarchical structure with ancestor nodes representing general intent categories and descendant nodes representing specific intents. This segmentation allows the system to classify utterances at appropriate levels of specificity based on confidence thresholds, improving robustness by providing fallback categories while managing complexity through modular organization of the ontology.
4Measurement precision
If the system relies on human intent analysts for low-confidence utterances, then the accuracy of intent determination is improved, but the cost of operation increases
Solution Approach 1:
The system applies partial human intervention only when necessary - specifically for low-confidence utterances that fall below predetermined thresholds. For the majority of high and medium confidence utterances, the system handles classification autonomously. This partial action approach maintains high accuracy for critical cases while significantly reducing operational costs compared to full human analysis of all utterances.
Data Source
AI summary
A natural language processing system has a hierarchy of user intents related to a domain of interest, the hierarchy having specific intents corresponding to leaf nodes of the hierarchy, and more general intents corresponding to ancestor nodes of the leaf nodes. The system also has a trained understanding model that can classify natural language utterances according to user intent. When the understanding model cannot determine with sufficient confidence that a natural language utterance corresponds to one of the specific intents, the natural language processing system traverses the hierarchy of intents to find a more general user intent that is related to the most applicable specific intent of the utterance and for which there is sufficient confidence. The general intent can then be used to prompt the user with questions applicable to the general intent to obtain the missing information needed for a specific intent.