Virtual Assistant Response Harmonization via Confidence Score Normalization

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Virtual assistants provide inconsistent responses due to different confidence score scales and domain expertise, leading to user frustration and increased processing resources in selecting accurate answers.

Innovation Solution

A method that normalizes confidence scores across multiple virtual assistants and assigns domain-specific weights to rank responses accurately, leveraging a learning-based approach to determine the relevant domain and adapt weights based on user inputs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If multiple virtual assistants are used to answer user queries, then the coverage of domains and expertise are improved, but the confidence scores from different assistants are on different scales making it difficult to select the best response

Engineering Contradiction:
Improvedomain coverageVSAvoidconfidence score comparison
Core Design Contradiction:
Adaptability or versatilityVSMeasurement precision

Solution Approach 1:

The patent applies parameter changes by transforming confidence scores from different scales into a normalized common scale. Each assistant's confidence score is adjusted using learned normalization parameters specific to their domain expertise, enabling direct comparison while preserving the original scale variations that reflect domain-specific confidence levels.

Inventive Principle:
Principle #35Parameter changes

Solution Approach 2:

The patent introduces an intermediary normalization layer between the multiple virtual assistants and the final response selection. This intermediary component learns domain-specific normalization parameters that mediate the comparison process, allowing confidence scores from different assistants to be fairly evaluated without directly comparing raw scores on different scales.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Reliability

If domain-specific weights are assigned to personal assistants, then the accuracy of response selection is improved, but the complexity of the system increases

Engineering Contradiction:
Improveresponse selection accuracyVSAvoidsystem complexity
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent applies preliminary action by pre-learning domain-specific weights and normalization parameters during a training phase before actual query processing. This preliminary learning phase creates a knowledge base that simplifies the runtime operation, where the system only needs to apply pre-determined weights rather than calculating them in real-time, thus reducing operational complexity while maintaining high accuracy.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The system performs self-service by automatically learning and adapting domain-specific weights through the training process without requiring manual configuration. The normalization parameters and domain weights are derived automatically from training data, allowing the system to self-optimize its response selection mechanism while managing its own complexity through automated learning rather than manual setup.

Inventive Principle:
Principle #25Self-service

3Ease of operation

If confidence scores are normalized to a common scale, then the selection process is simplified, but the domain-specific nuances of each assistant's confidence are lost

Engineering Contradiction:
Improveresponse selection processVSAvoiddomain-specific confidence nuances
Core Design Contradiction:
Ease of operationVSLoss of information

Solution Approach 1:

The patent applies local quality by implementing domain-specific normalization parameters that are tailored to each assistant's expertise area. Instead of applying a single universal normalization, each domain has its own normalization characteristics that preserve local confidence nuances while still enabling comparison. This allows the system to maintain domain-specific confidence characteristics while achieving operational simplicity through the unified normalization framework.

Inventive Principle:
Principle #3Local quality

Data Source

PatentUS11355118B2Virtual assistants harmonization
Publication Date: 2022.06.07 KYNDRYL INC
  • US11355118B2 patent drawing
  • US11355118B2 patent drawing
  • US11355118B2 patent drawing

AI summary

The present disclosure relates to a method of coordinating personal assistants. The method comprises: sending a request to a set of personal assistants based on a received query. Assistant responses may be received in response to the request from respective personal assistants of the set of personal assistants. Each assistant response of the assistant responses comprises a confidence score of the respective assistant response. The confidence scores may be adjusted to a common scale. The adjusted confidence scores may be weighted using domain-specific weights of the personal assistants in accordance with the domain of the query. One or more assistant responses of the received assistant responses may be selected based on the weighted scores and the selected assistant responses may be provided to the user.