GenAI OOS Detection for Digital Assistant Routing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current digital assistants face challenges in detecting and handling out-of-scope (OOS) and out-of-domain (OOD) utterances, which can lead to user frustration and inefficient routing of queries, as they lack effective methods to identify and respond appropriately to utterances that fall outside the trained intents or domain of the chatbot.

Innovation Solution

The proposed solution involves a computer-implemented method that uses a generative artificial intelligence (GenAI) component to route utterances to a skill bot, generating prompts that include scope-related elements, such as scenarios and negative few-shot examples, to determine if an utterance is OOS or OOD, and transitioning to a different state or workflow when such utterances are detected, ensuring proper handling and routing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a digital assistant uses simple predefined scripts for conversation, then the system complexity is low, but the ability to handle diverse linguistic nuances and context is limited

Engineering Contradiction:

Improveability to handle diverse linguistic nuancesVSAvoidsystem complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary OOS/OOD detection mechanism between the user input and the main conversation flow. This detector acts as a mediator that filters and categorizes utterances before they reach the skill bot, preventing inappropriate responses to out-of-scope queries while maintaining the simplicity of the underlying conversation framework.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent segments the conversation handling into distinct components: an OOS/OOD detection module, a skill bot for domain-specific tasks, and a fallback mechanism. This segmentation allows each component to have specialized functionality, improving overall adaptability while keeping individual modules relatively simple.

Inventive Principle:

Principle #1Segmentation

2Reliability

If a digital assistant attempts to handle all possible user queries, then the adaptability is high, but the reliability decreases when encountering out-of-scope utterances

Engineering Contradiction:

Improveresponse accuracy for in-scope queriesVSAvoidability to handle out-of-scope queries

Core Design Contradiction:

ReliabilityVSAdaptability or versatility

Solution Approach 1:

The patent converts the harmful effect of encountering out-of-scope queries into a beneficial detection opportunity. By training the GenAI model to identify OOS/OOD utterances, the system transforms potential errors into accurate classifications, improving reliability for in-scope queries while providing appropriate responses to out-of-scope queries through the fallback mechanism.

Inventive Principle:

Principle #22Blessing in disguise (Convert harm into benefit)

Solution Approach 2:

The patent implements feedback loops where the GenAI model continuously learns from interactions and the system monitors query outcomes. This feedback mechanism allows the system to improve its OOS/OOD detection accuracy over time, enhancing both reliability for in-scope queries and adaptability for handling diverse input.

Inventive Principle:

Principle #23Feedback

3Productivity

If a digital assistant uses GenAI components for processing utterances, then the productivity and response quality improve, but the computational resources and processing time increase

Engineering Contradiction:

Improvequery processing efficiencyVSAvoidcomputational resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent applies partial action by using GenAI components only for specific tasks where they provide the most value - namely OOS/OOD detection and generating responses for in-scope queries. The system avoids using GenAI for all processing uniformly, instead deploying it selectively to improve productivity for critical functions while reducing overall computational consumption.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20250094734A1Large language model handling out-of-scope and out-of-domain detection for digital assistant

Publication Date: 2025.03.20 ORACLE INT CORP

AI summary

Techniques for using a LLM to detect OOS and OOD utterances. In one aspect, a method includes routing an utterance to a skill bot. The skill bot is configured to execute an action for completing a task associated with the utterance, and a workflow associated with the action includes a GenAI component state configured to facilitate completion of at least part of the task. The method further includes inputting a prompt into a GenAI model for processing. The prompt includes the utterance and scope-related elements that teach the GenAI model to output an invalid input variable when the utterance is OOS or OOD. When the GenAI model determines the utterance is OOS or OOD as part of the processing, the response is generated to include the invalid input variable, and the GenAI component state is caused to transition to a different state or workflow based on the response.