Automated Chatbot Knowledge Gap Evaluation via Entity Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing chatbot evaluation methods require manual expert approval, which is cumbersome and lacks confidence in knowledge extraction before deployment, especially when chatbots are automatically built from document corpora.
Innovation Solution
A method to automatically identify knowledge gaps in chatbots by classifying text portions, associating entities, and determining gap entities within a document corpus, allowing for automated chatbot approval based on threshold criteria.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual expert approval is used for chatbot deployment, then knowledge extraction confidence is improved, but evaluation process complexity and time consumption increase
Solution Approach 1:
The system performs preliminary automated evaluation of chatbot knowledge gaps before deployment using text classification and entity association. This preliminary action identifies potential knowledge deficiencies in advance, allowing experts to focus their review on specific gap areas rather than conducting comprehensive manual evaluation, thus maintaining reliability while reducing overall process complexity
Solution Approach 2:
An automated knowledge gap identification system acts as an intermediary between chatbot deployment and manual expert review. This intermediary component classifies text portions, associates entities, and generates gap reports that bridge the gap between automated chatbot generation and human expert validation, reducing both time consumption and expert workload while maintaining knowledge extraction confidence
2Productivity
If automated chatbot building from document corpora is implemented, then productivity is improved, but knowledge gap identification accuracy deteriorates
Solution Approach 1:
The system implements feedback mechanisms where classification results and entity associations are continuously refined based on identified knowledge gaps. The automated evaluation process provides feedback about missing knowledge areas, which can be used to improve subsequent chatbot training from document corpora, thereby maintaining productivity while progressively improving gap identification accuracy through iterative refinement
Solution Approach 2:
The evaluation process segments the document corpus into classified text portions and associates entities systematically. By breaking down the knowledge base into manageable classified segments and tracking entity associations, the system can efficiently evaluate automated chatbots at scale while maintaining accurate identification of knowledge gaps through structured analysis of segmented content
3Measurement precision
If comprehensive text classification and entity association are performed, then knowledge gap identification accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The system performs partial classification and entity association focused specifically on identifying knowledge gaps rather than comprehensive analysis of all text portions. By applying classification and entity association only where necessary to detect gaps, the system achieves sufficient accuracy for deployment decisions while significantly reducing processing time and computational resource requirements compared to exhaustive analysis
Data Source
AI summary
Knowledge gaps in a chatbot are identified with reference to a domain-specific document and a set of QA pairs of the chatbot. Entities and/or entity values associated with the document are compared to the entities and/or entity values of the QA pairs. Entities of the document not associated with the QA pairs are identified as knowledge gaps. The QA pairs and knowledge gaps are ranked by relevance to the domain.