Neural Conversation Model Biasing with Domain Knowledge

Solution Overview

Problem

Neural conversation models often fail to generalize well with small datasets due to capturing spurious features rather than semantic domain knowledge, requiring large dialogue example sets for effective training, which is costly and time-consuming.

Innovation Solution

Incorporating domain knowledge into neural conversation models through natural language rules, allowing for minimal dialogue examples and reducing the influence of spurious features by using a small set of rules to bias the model towards semantically robust features.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If large dialogue example sets are used for training, then model generalization and semantic understanding improve, but training cost and time increase significantly

Engineering Contradiction:

Improvemodel generalizationVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

Domain knowledge is extracted and encoded into rules beforehand, creating a knowledge base that guides the model during training. This preliminary preparation of structured knowledge allows the model to learn from smaller datasets by leveraging pre-processed domain expertise, reducing the need for extensive dialogue examples while maintaining generalization capability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A knowledge base acting as an intermediary component is introduced between the training data and the neural conversation model. This knowledge base stores domain-specific rules and facts, serving as a mediator that enhances model learning from limited examples by providing structured domain knowledge, thereby improving generalization without requiring large amounts of training data

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If large dialogue example sets are used for training, then model generalization and semantic understanding improve, but training cost increases

Engineering Contradiction:

Improvemodel generalizationVSAvoidtraining data volume

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

Domain knowledge is extracted and encoded into rules beforehand, creating a knowledge base that guides the model during training. This preliminary preparation of structured knowledge allows the model to learn from smaller datasets by leveraging pre-processed domain expertise, reducing the need for extensive dialogue examples while maintaining generalization capability

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

A knowledge base acting as an intermediary component is introduced between the training data and the neural conversation model. This knowledge base stores domain-specific rules and facts, serving as a mediator that enhances model learning from limited examples by providing structured domain knowledge, thereby improving generalization without requiring large amounts of training data

Inventive Principle:

Principle #24Intermediary (Mediator)

3Measurement precision

If formal language is used to represent domain knowledge, then precision improves, but system complexity and development cost increase

Engineering Contradiction:

Improvedomain knowledge representation precisionVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

Instead of using complex formal languages, the system copies the natural language structure of domain knowledge directly into the knowledge base. This approach preserves the semantic meaning and precision of domain expertise while avoiding the complexity of formalization, allowing experts to contribute knowledge in their natural language without requiring specialized formal language skills

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system uses simple, easily modifiable natural language rules rather than rigid formal language structures. These natural language rules can be quickly created, modified, and updated by domain experts without requiring complex formal language expertise, reducing both system complexity and development costs while maintaining adequate precision for practical applications

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS11037546B2Nudging neural conversational model with domain knowledge

Publication Date: 2021.06.15 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

Examples of the present disclosure describe systems and methods utilize domain knowledge to influence a selection of a candidate action template in a neural conversation model. More specifically, natural language rules may be provided to a natural language rule inferencer to bias a selection of a candidate action template. In some instances, the natural language rules may include a user input and a system action. In other instances, the natural language rules may include a previous system action and a next system action. A biasing vector may then influence a selection of a candidate action template of a set of candidate action templates to determine a most relevant candidate action template based on the natural language rules, the candidate action templates, and the user utterance or other system input.