Dialog System Using RNN Mixture of Language Models

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing dialog systems face challenges in generating domain-specific, natural language utterances that are both accurate and grammatically correct, particularly in knowledge domain-specific applications like customer support, where call center agents often lack expertise.

Innovation Solution

A dialog system that uses a combination of language models, integrated with a recurrent neural network (RNN), specifically a Long Short-Term Memory (LSTM) model, to construct natural language utterances word-by-word, leveraging a concatenation of dialogue history and current utterance context to generate domain-specific responses that adhere to proper vocabulary and grammar.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a single language model is used to generate domain-specific utterances, then the system can maintain simplicity in architecture, but the generated utterances may lack either domain accuracy or natural language correctness

Engineering Contradiction:

Improvedomain accuracyVSAvoidsystem architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent segments the language generation task into two specialized components: a domain-specific language model for accuracy and a general language model for natural language correctness. Each model handles a specific aspect of the generation task, allowing both to maintain simplicity while collectively achieving high reliability in both domain accuracy and linguistic quality.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent merges multiple language models into a unified generation system where outputs from domain-specific and general language models are combined. This merging allows the system to leverage the strengths of each model—domain knowledge from the specialized model and natural language fluency from the general model—achieving both high reliability and acceptable complexity through integrated operation.

Inventive Principle:

Principle #5Merging (Combining)

2Reliability

If multiple language models are combined to improve utterance quality, then domain accuracy and natural language correctness improve, but the system architecture becomes more complex

Engineering Contradiction:

Improveutterance qualityVSAvoidmodel integration complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary component that mediates between multiple language models and the final output generation. This intermediary manages the integration of domain-specific and general language models, coordinating their outputs and resolving conflicts without requiring complex direct interactions between models, thus improving utterance quality while controlling integration complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent utilizes parameter changes in the language models to control the balance between domain accuracy and natural language correctness. By adjusting model parameters such as temperature, top-k sampling, and blending ratios, the system can dynamically optimize utterance quality without fundamentally changing the architecture, thereby managing complexity while maintaining high reliability.

Inventive Principle:

Principle #35Parameter changes

3Reliability

If call center agents are trained to improve expertise, then domain knowledge improves, but training time and resources increase

Engineering Contradiction:

Improveagent expertiseVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces the mechanical system of human training with an automated dialog system that provides real-time support to agents. Instead of investing time in extensive training programs, the system embeds domain expertise directly into the dialog interface, allowing agents to access expert-level knowledge during interactions without requiring prolonged training periods, thus improving reliability while reducing time loss.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent enables agents to access domain expertise on-demand through the dialog system, which provides context-aware suggestions and information during customer interactions. This self-service approach allows agents to retrieve expert knowledge independently during work, eliminating the need for extensive pre-training while maintaining high domain accuracy in customer support.

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS10431205B2Dialog device with dialog support generated using a mixture of language models combined using a recurrent neural network

Publication Date: 2019.10.01 CONDUENT BUSINESS SERVICES LLC

AI summary

A dialog device comprises a natural language interfacing device (chat interface or a telephonic device), and a natural language output device (the chat interface, a display device, or a speech synthesizer outputting to the telephonic device). A computer stores natural language dialog conducted via the interfacing device and constructs a current utterance word-by-word. Each word is chosen by applying a plurality of language models to a context comprising concatenation of the stored dialog and the current utterance thus far. Each language model outputs a distribution over the words of a vocabulary. A recurrent neural network (RNN) is applied to the distributions to generate a mixture distribution. The next word is chosen using the mixture distribution. The output device outputs the current natural language utterance after it has been constructed by the computer.