Multimodal Dialog State Circulation via Protocol Buffers

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional mobile devices struggle with efficiently handling multimodal inputs, such as voice and non-verbal responses, in dialog systems, leading to increased latency and data transmission issues, especially when maintaining state consistency across different network connections.

Innovation Solution

Implementing a method where user devices receive and manage multiple protocol buffers (DialogTurnIntents) to handle voice and non-verbal inputs locally, updating the dialog state and interpreting voice inputs while reducing data transmission by bundling state information and sending it to the dialog system only when necessary.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the mobile device sends all dialog state information to the server system for processing, then the dialog system can maintain state consistency, but data transmission volume increases and latency increases

Engineering Contradiction:

Improvestate consistencyVSAvoidlatency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The dialog state information is segmented into two parts: critical state information that must be synchronized with the server, and local state information that can be processed and cached on the mobile device. This segmentation allows the device to operate with reduced latency while maintaining necessary state consistency through selective synchronization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The mobile device is empowered with local processing capabilities to handle and interpret dialog state information without requiring constant server communication. The device maintains a local copy of dialog state and can process non-verbal responses locally, reducing data transmission needs while preserving state consistency for critical operations.

Inventive Principle:

Principle #3Local quality

2Reliability

If the mobile device sends all dialog state information to the server system, then state consistency is maintained, but data transmission volume increases

Engineering Contradiction:

Improvestate consistencyVSAvoiddata transmission volume

Core Design Contradiction:

ReliabilityVSQuantity of substance

Solution Approach 1:

Essential state information is extracted and retained on the mobile device, while only necessary updates and critical state changes are transmitted to the server. This extraction approach reduces data transmission volume significantly while maintaining the state consistency required for proper dialog management.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The mobile device performs preliminary processing of dialog state information locally, preparing and filtering data before transmission to the server. This preliminary action reduces the volume of data that needs to be transmitted while ensuring that the server receives only the most relevant state information for maintaining consistency.

Inventive Principle:

Principle #10Preliminary action

3Reliability

If the dialog system processes all inputs centrally on the server, then state consistency is ensured, but network dependency increases and private data must be transmitted

Engineering Contradiction:

Improvestate consistencyVSAvoidnetwork dependency

Core Design Contradiction:

ReliabilityVSObject-affected harmful factors

Solution Approach 1:

Processing responsibilities are segmented between the mobile device and server system. The device handles local processing of non-verbal responses and maintains local state, while the server handles critical dialog management. This segmentation reduces network dependency while preserving necessary state consistency through selective synchronization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The mobile device is equipped with self-service capabilities to process and interpret dialog state information locally without requiring constant server intervention. The device can independently handle non-verbal responses and maintain local state consistency, reducing network dependency while ensuring critical state information is synchronized with the server.

Inventive Principle:

Principle #25Self-service

4Loss of time

If the device handles multimodal inputs locally with multiple protocol buffers, then latency is reduced and data transmission is minimized, but device complexity increases

Engineering Contradiction:

ImprovelatencyVSAvoidprotocol buffer management

Core Design Contradiction:

Loss of timeVSDevice complexity

Solution Approach 1:

The protocol buffer framework provides a universal, standardized mechanism for handling multiple types of dialog state information and non-verbal responses. This multi-functional approach allows the device to manage complex multimodal inputs through a unified interface, reducing the apparent complexity despite the sophisticated processing capabilities required.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS11379181B2Multimodal state circulation

Publication Date: 2022.07.05 GOOGLE LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for managing dialogs. In one aspect, a method includes receiving a request to perform a task from a user device; obtaining a dialog corresponding to the task; providing multiple protocol buffers to the user device; receiving a voice input and one or more annotated protocol buffers from the user device, the one or more annotated protocol buffers identifying corresponding non-verbal responses to content in the protocol buffers; and using the received protocol buffers to update a state of the dialog and to interpret the voice input.