Streaming Query Response Interface for Low-Latency Voice AI

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing techniques in human-to-computer dialog systems using generative models suffer from increased latency due to waiting for complete user query transcription before processing, leading to bottlenecks in generating responsive outputs.

Innovation Solution

Implementing a streaming multi-head attention layer in a generative model at a remote computing system to process tokenized instances of query text concurrently with local transcription at a client device, reducing the need for complete text transmission.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If the system waits for complete text representation generation before transmitting to remote computing system, then transcription accuracy is improved, but latency increases

Engineering Contradiction:

Improvetranscription accuracyVSAvoidlatency

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system transmits partial text representations to the remote computing system before the complete transcription is available. This preliminary action allows the generative model to start processing early, reducing overall latency while maintaining the ability to update or complete the transcription later without sacrificing accuracy

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The text representation is divided into segments or tokens that can be transmitted incrementally. The ASR model generates text in discrete units (words, subwords, or characters) that are sent to the remote system as they become available, enabling parallel processing without waiting for the complete transcription

Inventive Principle:

Principle #1Segmentation

2Productivity

If input side transcription batching is used, then processing efficiency is improved, but bottleneck in query text processing occurs

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidprocessing bottleneck

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

Instead of batching all transcriptions before processing, the system begins processing individual transcriptions as soon as they are generated. This preliminary processing action eliminates the batching bottleneck while maintaining efficient throughput by keeping the generative model continuously occupied

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system maintains continuous processing by the generative model without idle waiting periods. While the ASR model continues generating transcriptions, the remote computing system continuously processes received text representations, eliminating gaps in useful action and preventing bottlenecks

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS12620394B2Query response interface with server side generative model(s)

Publication Date: 2026.05.05 GOOGLE LLC

AI summary

Various implementations include processing, at a client device, an instance of audio data capturing a user voice query using an automatic speech recognition model to generate a sequence of instances of tokenizable query text. In many implementations, one or more instances of the sequence can be transmitted to a remote computing system prior to generating the entire sequence. In a variety of implementations, each instance in the sequence can be processed using a generative model which includes a streaming multi-head attention portion. Responsive output can be transmitted from the remote computing system to the client device, where the client device renders the responsive output to the user. In many implementations, the time between the user speaking the user query and the client device rendering the responsive output is reduced, thus decreasing latency in the system.