Multi-modal Interaction Framework for Automated Assistants

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing automated assistants often require verbal free form natural language input for interaction with third-party computing services, limiting accessibility for users with disabilities and inefficiently utilizing computing resources, as they struggle to interpret and match user intents effectively across various input modalities.

Innovation Solution

Implementing a multi-modal interaction framework that allows users to interact with third-party computing services using verbal and non-verbal inputs, such as visual and tactile modalities, through a client-server architecture with a graphical user interface, enabling touchless interaction and efficient traversal of dialog state machines.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If automated assistants use only verbal free form natural language input for interaction with third-party computing services, then the system architecture remains simple, but accessibility for users with disabilities is limited and computing resources are inefficiently utilized

Engineering Contradiction:

Improveaccessibility for users with disabilitiesVSAvoidsystem architecture complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The automated assistant is enhanced to accept multiple input modalities (verbal, visual, tactile) beyond traditional voice commands. The system processes different types of user inputs through a unified architecture that routes them to appropriate handling mechanisms, enabling users with various disabilities to interact effectively while maintaining a single system structure

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

A dialog state machine acts as an intermediary layer between the multi-modal input interface and the third-party computing services. This mediator translates diverse input forms into standardized intent representations, managing the complexity of multi-modal processing while preserving system modularity and ease of integration

Inventive Principle:

Principle #24Intermediary (Mediator)

2Adaptability or versatility

If automated assistants process and match user intents across various input modalities, then accessibility is improved, but computing resources are consumed more heavily for interpretation and matching

Engineering Contradiction:

Improvemulti-modal input processingVSAvoidcomputing resource consumption

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The dialog state machine performs preliminary processing of user inputs by maintaining state information across interaction turns. By tracking the conversational context and user intent state, the system avoids redundant interpretation and matching operations, reducing computing resource consumption while supporting multi-modal inputs

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts processing parameters based on the input modality and dialog state. By changing the level of interpretation depth and matching complexity according to the current context, the system optimizes computing resource usage while maintaining accurate intent recognition across different input types

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If automated assistants support multiple input modalities including visual and tactile interfaces, then user interaction flexibility is enhanced, but the device complexity increases

Engineering Contradiction:

Improveuser interaction flexibilityVSAvoidinterface architecture complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The interface architecture is segmented into distinct modular components: a dialog state machine module, an intent recognition module, and an input handling module. Each modality (verbal, visual, tactile) has its own specialized handler that interfaces with the common dialog state machine, reducing overall complexity through functional decomposition

Inventive Principle:

Principle #1Segmentation

4Measurement precision

If automated assistants implement comprehensive intent matching across multiple modalities, then user intent resolution accuracy improves, but communication overhead and processing time increase

Engineering Contradiction:

Improveintent resolution accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The dialog state machine maintains its own internal state representation that automatically updates based on processed inputs. This self-managing state tracking eliminates the need for complex cross-modality intent matching in each interaction turn, resolving user intent accurately while minimizing processing time through efficient state transitions

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS11200893B2Multi-modal interaction between users, automated assistants, and other computing services

Publication Date: 2021.12.14 GOOGLE LLC

AI summary

Techniques are described herein for multi-modal interaction between users, automated assistants, and other computing services. In various implementations, a user may engage with the automated assistant in order to further engage with a third party computing service. In some implementations, the user may advance through dialog state machines associated with third party computing service using both verbal input modalities and input modalities other than verbal modalities, such as visual/tactile modalities.