Dynamic GUI Adaptation for Automated Assistant Spoken Input
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated assistants often fail to accurately interpret incomplete or ambiguous user commands in human-to-computer dialogs, leading to prolonged dialog sessions and wastage of computational resources due to the need for users to reiterate requests.
Innovation Solution
The automated assistant dynamically adapts graphical user interface (GUI) elements based on user input, using streaming automatic speech recognition (ASR) and natural language understanding (NLU) models to render generic container elements that are tailored to specific intents, providing suggestions and current state information to aid users in completing requests efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the automated assistant waits for complete command phrases before acting, then accuracy of interpretation is improved, but dialog session duration increases and user inputs increase
Solution Approach 1:
The system performs preliminary processing of incoming spoken utterances using streaming ASR and NLU models to identify candidate intents and relevant parameters before the user completes their speech. This allows the system to prepare potential responses and update GUI elements in advance, reducing the overall dialog session duration while maintaining accurate interpretation through iterative refinement as more speech input becomes available.
2Reliability
If the automated assistant requires complete slot values before acting, then reliability of action execution is improved, but productivity decreases due to repeated user invocations
Solution Approach 1:
The system provides visual feedback through dynamic GUI elements that display candidate intents, current slot values, and missing information in real-time as the user speaks. This feedback loop allows users to see what the system has understood so far and what additional information is needed, enabling them to complete their request more efficiently while ensuring the system has sufficient information for reliable action execution before proceeding.
3Productivity
If the automated assistant processes incomplete spoken utterances, then productivity is improved by reducing re-invocations, but measurement precision deteriorates due to ambiguity
Solution Approach 1:
The system dynamically adapts its processing approach based on the completeness and clarity of incoming speech input. As streaming ASR and NLU models process progressively more input, the system updates candidate intents and their associated probabilities dynamically. When sufficient confidence is achieved or the user pauses, the system proceeds with action execution, balancing productivity gains from processing incomplete utterances with maintained accuracy through dynamic confidence assessment.
4Adaptability or versatility
If the system renders different display interfaces for different intents, then adaptability to user needs is improved, but device complexity increases
Solution Approach 1:
The system uses a universal container-based GUI framework where a single display interface structure can represent multiple different intents and states. The container elements are dynamically populated and configured based on the identified intent and available slot values, allowing the same interface template to serve multiple functions. This reduces device complexity by avoiding the need for separate dedicated interfaces for each possible intent while maintaining high adaptability to different user requests.
Data Source
AI summary
Implementations described herein relate to an automated assistant that iteratively renders various GUI elements as a user iteratively provides a spoken utterance, or sequence of spoken utterances, corresponding to a request directed to the automated assistant. These various GUI elements can be dynamically adapted as the user iteratively provides the spoken utterance to assist the user with efficiently completing the request. In some implementations, a generic container graphical element associated with candidate intent(s) can be initially rendered at a display interface of a computing device and dynamically adapted with tailored container graphical elements as a particular intent is determined while the user iteratively provides the spoken utterance. In additional or alternative implementations, the tailored container graphical elements can include a current status of one or more settings associated with the computing device or additional computing device(s) such that the user can view the current status while completing the spoken utterance.