Multi-Modal Dialog System Grammar Generation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing multi-modal dialog systems do not effectively assist users in using multiple modalities in a coordinated manner and fail to provide responses related to modality availability, making them difficult to use, especially for inexperienced users.

Innovation Solution

A multi-modal dialog system with an input processor that includes a Multi-Modal Input Fusion component, dialog manager, visual context manager, and multi-modal utterance generator, which interprets user questions and generates user-friendly multi-modal utterances based on modality capability information, current context, and task models, guiding users on the appropriate use of modalities.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If multiple modalities are used in a multi-modal dialog system, then the functionality and capability of the system is improved, but the complexity of operation and difficulty of use increases

Engineering Contradiction:

ImprovefunctionalityVSAvoidease of use

Core Design Contradiction:

Adaptability or versatilityVSEase of operation

Solution Approach 1:

The patent introduces a multi-modal grammar generation component that acts as an intermediary between the dialog manager and the user interface. This component generates context-specific grammar rules that guide users on which modalities to use and how to coordinate them, thereby mediating the complexity of multi-modal interaction without reducing the system's functional capabilities

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system implements feedback mechanisms by monitoring dialog context and modality availability, then dynamically adjusting the generated grammar to provide real-time guidance to users. This feedback loop helps users understand which modalities are currently available and appropriate to use, reducing operational complexity while maintaining versatility

Inventive Principle:

Principle #23Feedback

2Ease of operation

If modality availability information is integrated into the dialog system, then the responsiveness and helpfulness to user questions is improved, but the system complexity increases

Engineering Contradiction:

ImprovehelpfulnessVSAvoidsystem complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent merges modality availability information tracking with the existing dialog context management functionality. By combining these functions into the same processing pipeline that generates multi-modal grammar, the system provides enhanced responsiveness to user questions without adding separate complex subsystems

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The multi-modal grammar generation component serves multiple functions: it manages dialog context, tracks modality availability, generates context-specific grammar rules, and provides user guidance. This multi-functional approach improves helpfulness while avoiding the complexity increase that would result from separate dedicated components for each function

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS7548859B2Method and system for assisting users in interacting with multi-modal dialog systems

Publication Date: 2009.06.16 GOOGLE TECHNOLOGY HOLDINGS LLC

AI summary

A method and system for assisting a user in interacting with a multi-modal dialog system (104) is provided. The method includes interpreting a “What Can I Do? (WCID)” question from a user in a turn of the dialog. A multi-modal grammar (212) is generated, based on the current context of the dialog. One or more user multi-modal utterances are generated, based on the WCID question and the multi-modal grammar. One or more user multi-modal utterances are conveyed to the user.