Condensed GUI Voice Control for Ambiguous Assistant Commands
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Interactions between users and intricate application graphical user interfaces (GUIs) are inefficient due to the need for multiple touch gestures and interruptions when using automated assistants, leading to repeated inputs and resource wastage.
Innovation Solution
An automated assistant mediates interactions by using content description data to identify and control GUI elements through condensed spoken utterances, allowing seamless transitions between user inputs and application actions without explicit invocation phrases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If users interact with automated assistant using spoken natural language input while concurrently interacting with application GUI, then user input efficiency improves, but form field inputs may be negated and repeated inputs are required
Solution Approach 1:
The patent introduces an intermediary mechanism that monitors and coordinates interactions between the automated assistant and application GUI. When the automated assistant is activated, it mediates the interaction by preventing interruptions to ongoing form field inputs, thereby maintaining input stability while still enabling spoken language control efficiency.
2Ease of operation
If users control intricate application GUIs using multiple touch gestures, then precise GUI element selection is achieved, but power consumption and processing bandwidth increase
Solution Approach 1:
The patent replaces the mechanical touch gesture system with an acoustic system (spoken natural language input). Instead of requiring multiple physical touch gestures to navigate and select GUI elements, users can speak commands that are converted to text and processed by the automated assistant, significantly reducing power consumption while maintaining or improving operational ease.
Solution Approach 2:
The patent changes the input modality parameter from tactile (touch gestures) to acoustic (spoken language). This parameter change allows the system to interpret user intent through speech recognition and natural language processing, reducing the number of interaction steps required and thereby lowering power consumption and processing bandwidth usage.
3Measurement precision
If users provide detailed spoken utterances to control specific GUI elements, then action accuracy improves, but utterance length and processing time increase
Solution Approach 1:
The patent implements a universal text-based interface that serves multiple functions: it can interpret both specific GUI element references and general commands, handle both detailed and concise inputs, and work across different applications. This universal interface allows the automated assistant to accurately identify GUI elements through various input formats, reducing the need for lengthy specific utterances while maintaining high identification accuracy.
Data Source
AI summary
Implementations set forth herein relate to an automated assistant that can control graphical user interface (GUI) elements via voice input using natural language understanding of GUI content in order to resolve ambiguity and allow for condensed GUI voice input requests. When a user is accessing an application that is rendering various GUI elements at a display interface, the automated assistant can operate to process actionable data corresponding to the GUI elements. The actionable data can be processed in order to determine a correspondence between GUI voice input requests to the automated assistant and at least one of the GUI elements rendered at the display interface. When a particular spoken utterance from the user is determined to correspond to multiple GUI elements, an indication of ambiguity can be rendered at the display interface in order to encourage the user to provide a more specific spoken utterance.