LLM-Modified Assistant Responses for Proactive Voice Dialog

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Automated assistants often fail to engage users in natural conversations, requiring additional user inputs to solicit information that could be proactively provided, wasting computational resources and disrupting the flow of dialog sessions.

Innovation Solution

Implement large language models (LLMs) to process assistant outputs and contextual signals, generating modified outputs that dynamically adapt to user queries and dialog contexts, incorporating diverse personalities and prosodic properties to enhance engagement.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If automated assistants use traditional turn-based dialog processing, then the system structure remains simple and predictable, but the conversational engagement quality deteriorates and users must provide additional inputs to solicit information that could be proactively provided

Engineering Contradiction:

Improveconversational engagement qualityVSAvoidsystem structure complexity

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The system performs preliminary analysis of user statements to identify implicit information needs and proactively retrieves relevant information before the user has to ask. For example, when a user mentions going to the beach, the system proactively looks up weather forecasts and beach activities without waiting for explicit user requests, thereby improving conversational engagement while managing complexity through structured information retrieval protocols.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces an intermediary processing layer that sits between the user input and the traditional ASR-NLU-fulfillment pipeline. This intermediary component analyzes user statements for implicit context, generates proactive information requests, and coordinates with existing system components, thereby enhancing engagement quality without completely redesigning the underlying system architecture.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If automated assistants proactively provide contextually relevant information, then conversational engagement improves and user inputs are reduced, but computational resources are consumed processing additional user inputs that could have been avoided

Engineering Contradiction:

Improvedialog session efficiencyVSAvoidcomputational resource consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary analysis of user statements to identify implicit information needs and proactively retrieves relevant information before the user has to ask. This reduces the total number of user inputs required, thereby improving dialog efficiency. The system balances computational resource usage by selectively applying proactive information retrieval only to statements with clear implicit context, rather than processing every user input through the complete pipeline.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system dynamically adjusts processing parameters based on the type and context of user input. For statements with clear implicit context (e.g., mentioning specific locations or activities), the system activates proactive information retrieval with higher processing depth. For straightforward queries, the system uses standard processing with reduced computational overhead, thereby optimizing the balance between dialog efficiency and resource consumption.

Inventive Principle:

Principle #35Parameter changes

3Ease of operation

If automated assistants use simple response generation, then computational resources are conserved and processing speed is maintained, but the naturalness and resonance of conversations deteriorates

Engineering Contradiction:

Improveconversation naturalnessVSAvoidprocessing speed

Core Design Contradiction:

Ease of operationVSProductivity

Solution Approach 1:

The system applies enhanced natural language generation selectively to specific portions of the dialog where context and personalization are most valuable. Rather than processing every response through complex naturalness-optimized generation, the system focuses computational resources on responses that require contextual understanding (e.g., proactive information, follow-up questions) while using streamlined generation for routine responses, thereby achieving natural conversation quality without sacrificing overall processing speed.

Inventive Principle:

Principle #3Local quality

Data Source

PatentEP4169015B1Using large language model(s) in generating automated assistant response(s)

Publication Date: 2025.12.31 GOOGLE LLC

Figure 1
Figure 2
Figure 3

AI summary

As part of a dialog session between a user and an automated assistant, implementations can receive a stream of audio data that captures a spoken utterance including an assistant query, determine, based on processing the stream of audio data, a set of assistant outputs that are each predicted to be responsive to the assistant query, process, using large language model (LLM) output(s), the assistant outputs and context of the dialog session to generate a set of modified assistant outputs, and cause given modified assistant output, from among the set of modified assistant outputs, to be provided for presentation to the user in response to the spoken utterance. In some implementations, the LLM output(s) can be generated in an offline manner for subsequent use in an online manner. In additional or alternative implementations, the LLM output(s) can be generated in an online manner when the spoken utterance is received.