Contextual Policy Chatbot for Extractive Answers and Clarification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing chatbots fail to efficiently ascertain user needs by requesting irrelevant information and require maintenance due to inflexible decision-tree systems that need updates for changing policies and user scenarios.
Innovation Solution
A two-staged approach using contextual search and large language models to identify relevant text and ask clarifying questions until a well-defined question is formed, eliminating the need for manual updates and optimizing computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a decision-tree system is used to handle user queries, then the chatbot can provide structured responses, but the system requires manual updates for changing policies and user scenarios
Solution Approach 1:
The patent replaces the mechanical decision-tree system with an AI-based system using large language models. The LLM autonomously processes user queries, searches relevant policies, and generates appropriate responses without requiring manual programming of decision paths. This substitution eliminates the need for continuous manual updates while maintaining structured response capabilities.
Solution Approach 2:
The system implements self-service through automated policy search and response generation. The LLM independently identifies relevant policies from the knowledge base, extracts necessary information, and formulates responses without human intervention. The completeness analyzer automatically determines when additional clarification is needed, enabling the system to adapt to policy changes autonomously.
2Reliability
If a chatbot asks follow-up questions to handle requests properly, then the service quality improves, but the interaction time increases
Solution Approach 1:
The system performs preliminary actions by proactively identifying missing information before the user completes their query. The completeness analyzer evaluates the user's input against the knowledge base structure and predicts what additional information would be needed, then asks clarifying questions in advance. This prevents unnecessary back-and-forth interactions and reduces overall time by addressing information gaps early.
Solution Approach 2:
The system implements feedback through the completeness analyzer, which continuously monitors the conversation state and determines when additional clarification is needed. The LLM receives feedback about the completeness of the user's query and dynamically adjusts its response strategy, asking targeted follow-up questions only when necessary. This feedback mechanism ensures high service quality while minimizing unnecessary interaction time.
3Productivity
If a chatbot provides answers without asking follow-up questions, then the interaction speed increases, but the answers may be incomplete or incorrect
Solution Approach 1:
The system performs preliminary analysis by using the completeness analyzer to evaluate whether the user's query contains all necessary information before generating a final response. The LLM proactively identifies potential gaps in the query and asks clarifying questions in advance, ensuring that the subsequent answer will be complete and accurate. This preliminary action prevents incorrect or incomplete responses while maintaining efficient interaction speed.
Solution Approach 2:
The system uses feedback from the completeness analyzer to determine the appropriate response strategy. The analyzer provides feedback about the sufficiency of the user's input, and the LLM adjusts its behavior accordingly - asking clarifying questions when information is missing, or providing direct answers when the query is complete. This feedback mechanism ensures high answer accuracy while maintaining productivity by avoiding unnecessary follow-up questions when the query is already sufficient.
4Reliability
If the chatbot searches through all documents to ensure complete information, then the answer completeness improves, but the computational resources increase
Solution Approach 1:
The system applies local quality by searching only the portions of the knowledge base that are relevant to the specific user query. The LLM and completeness analyzer work together to identify which policies and documents contain the necessary information, rather than processing the entire knowledge base. This targeted approach ensures complete and accurate answers while significantly reducing computational resources by focusing search efforts on locally relevant content.
Solution Approach 2:
The completeness analyzer acts as an intermediary between the user query and the full knowledge base. It evaluates the query against the knowledge base structure and identifies the specific subsets of documents that need to be searched. This intermediary function filters out irrelevant content before it reaches the LLM, reducing computational load while ensuring that all necessary information is retrieved for complete and accurate responses.
Data Source
AI summary
A chatbot system described herein uses a two-staged approach to answer a question. The first stage consists of a contextual search that takes in the question, searches a library of documents and finds a relevant piece of text. The second stage is to use the relevant piece of text, present it to a large language model, and have the model answer the question give the context of the text. The model in question formulates the answer by extracting the most relevant section of the text. When asked an ill-posed question, the chat bot will ask the user clarifying questions until a well-defined question is found.