Chatbot Training Confidence Scoring and Feedback Loop

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Training intelligent chatbots with high-quality data is challenging due to the inefficiency of relying on end-user-provided inputs, which may not be sufficient for understanding sophisticated human conversations, especially in specific contexts.

Innovation Solution

A method and system for training chatbots by receiving user intent inputs, calculating confidence scores for predictions, and delivering training scores to users based on these scores, facilitating the improvement of chatbot responses through user interactions and a leaderboard system to rank user contributions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If training data is collected from end users, then the chatbot can be trained on real user inputs, but the quality of training data cannot be guaranteed as users may not provide sophisticated conversation examples

Engineering Contradiction:

Improveamount of training dataVSAvoidquality of training data

Core Design Contradiction:

Quantity of substanceVSManufacturing precision

Solution Approach 1:

The system implements a feedback mechanism where the chatbot provides confidence scores to users about its understanding of their inputs. When the confidence score is low, the user is prompted to refine their input, creating a feedback loop that improves data quality while collecting training examples.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system performs preliminary processing by having the chatbot analyze and evaluate user inputs before they become training data. The confidence score calculation acts as a preliminary filter to identify high-quality training examples that meet sophistication thresholds.

Inventive Principle:

Principle #10Preliminary action

2Ease of operation

If the chatbot attempts to interpret and respond to all user inputs, then the conversational experience is maintained, but the system cannot distinguish between high-quality training inputs and regular conversations

Engineering Contradiction:

Improveconversational experienceVSAvoididentification of training quality

Core Design Contradiction:

Ease of operationVSMeasurement precision

Solution Approach 1:

The system replaces manual quality assessment with an automated confidence score calculation mechanism. The chatbot's internal confidence metric serves as an automated measurement tool to distinguish training-quality inputs from regular conversations without disrupting user experience.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Ease of manufacture

If the chatbot is trained on public chat log datasets, then training can proceed with available data, but the data is insufficient for understanding deeper conversations in specific contexts

Engineering Contradiction:

Improveavailability of training dataVSAvoideffectiveness of chatbot understanding

Core Design Contradiction:

Ease of manufactureVSReliability

Solution Approach 1:

The system enables users to self-generate high-quality training data through guided interactions. When the chatbot detects low-confidence inputs, it prompts users to provide refinements, allowing the system to self-collect sophisticated training examples specific to its operational context.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10617959B2Method and system for training a chatbot

Publication Date: 2020.04.14 SERVICENOW INC

AI summary

A computer-implemented method for training a chatbot is provided. The method includes receiving a training input through a platform associated with the chatbot. The training input indicates user intent for interacting with the chatbot. The method includes calculating a confidence score associated with a prediction of the user intent identified by the chatbot. The method further includes providing a training score to the user providing the training input based on the confidence score.