Chatbot Node Performance Scoring via Dialog Flow Instrumentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current solutions lack a method for effectively measuring and improving chatbot performance, making it difficult for dialog flow designers to identify areas for improvement in conversational systems.
Innovation Solution
A system and method using an AI and machine learning information handling system to instrument and score individual nodes in a chatbot dialog flow based on user interactions, applying weighted awards or penalties to capture and analyze performance metrics such as time, access count, sentiment, and outcome, and assessing overall dialog flow performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If qualitative guidelines are used to build chatbots, then chatbot development becomes easier, but performance measurement capability deteriorates
Solution Approach 1:
The patent introduces a performance measurement intermediary system that sits between the chatbot dialog flow and the evaluation process. This intermediary captures metrics at each node (access count, time spent, sentiment, outcomes) and translates qualitative interactions into quantitative performance data, enabling precise measurement without changing the chatbot development approach
Solution Approach 2:
The patent replaces subjective qualitative evaluation with objective automated measurement systems. Instead of manual assessment of chatbot performance, the system uses automated tracking of performance metrics (response time, sentiment analysis, outcome tracking) to objectively measure and evaluate chatbot effectiveness
2Measurement precision
If performance metrics are tracked at each dialog node, then measurement precision improves, but device complexity increases
Solution Approach 1:
The patent divides the chatbot dialog flow into discrete nodes and instruments each node independently with performance tracking. By segmenting the measurement function across individual nodes rather than implementing a monolithic evaluation system, the complexity is distributed and manageable while achieving comprehensive measurement coverage
Solution Approach 2:
The patent implements self-service performance tracking where each dialog node automatically instruments and captures its own performance metrics (access count, time spent, sentiment, outcomes) without requiring external intervention. This automated self-measurement reduces the operational complexity of the overall measurement system
Data Source
AI summary
A method, apparatus, system and a computer program product are provided for scoring performance of a chatbot by capturing specified node performance metrics based on user interactions with one or more nodes in a dialog flow for the chatbot; assigning a score to each of the one or more nodes based on the specified node performance metrics; and assessing a performance measure for at least a portion of the chatbot by combining scores assigned to selected nodes in the dialog flow.