Condensed Voice Control for Intricate Application GUIs

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Interactions between users and intricate application graphical user interfaces (GUIs) are inefficient due to the need for multiple touch gestures and repeated inputs when using automated assistants, leading to interruptions and increased power consumption.

Innovation Solution

An automated assistant mediates interactions by using content description data to correlate spoken utterances with GUI elements, allowing users to control intricate application GUIs with condensed voice inputs, even without explicit invocation phrases.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If users interact with automated assistants while filling out application forms, then user assistance and task completion are improved, but form inputs are negated and users must repeat inputs

Engineering Contradiction:

Improveuser assistanceVSAvoidform input stability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The patent introduces an intermediary mechanism that detects and manages focus state between the automated assistant and application form fields. When the automated assistant is invoked, the system temporarily suspends form field editing mode, preventing the negation of form inputs. This mediator layer coordinates between the assistant and application to maintain input stability while preserving user assistance capabilities.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Ease of operation

If users control intricate application GUIs with multiple touch gestures, then precise control is achieved, but interaction time and power consumption increase

Engineering Contradiction:

ImproveGUI control precisionVSAvoidinteraction time

Core Design Contradiction:

Ease of operationVSLoss of time

Solution Approach 1:

The patent replaces mechanical touch gestures with voice-based commands for controlling application GUI elements. Instead of requiring multiple precise touch gestures to navigate and control complex GUI interfaces, users can speak natural language commands that the automated assistant interprets and executes, significantly reducing interaction time and physical effort while maintaining control precision.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces a new dimension of interaction by adding voice input modality to the traditional two-dimensional touch interface. This third dimension (acoustic space) allows users to control intricate GUI elements through spoken commands, bypassing the need for multiple sequential touch gestures and reducing interaction time.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Ease of operation

If users control intricate application GUIs with multiple touch gestures, then precise control is achieved, but power consumption increases

Engineering Contradiction:

ImproveGUI control precisionVSAvoidpower consumption

Core Design Contradiction:

Ease of operationVSUse of energy by moving object

Solution Approach 1:

The patent substitutes acoustic-based voice recognition for mechanical touch gesture processing. Voice commands require less processing power and energy consumption compared to interpreting multiple complex touch gesture sequences, while achieving the same level of precise GUI control. The automated assistant processes spoken natural language more efficiently than the system would process multiple gesture inputs.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

4Adaptability or versatility

If automated assistants process spoken natural language inputs, then user interaction flexibility is improved, but processing complexity and time increase

Engineering Contradiction:

Improveinteraction flexibilityVSAvoidprocessing complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements a universal natural language processing framework that handles multiple types of commands (assistant invocations, GUI control commands, form input commands) through a single unified processing system. This multi-functional approach allows the automated assistant to interpret various forms of input flexibly while managing processing complexity through shared processing pathways and common interpretation logic.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS12353798B2Condensed spoken utterances for automated assistant control of an intricate application GUI

Publication Date: 2025.07.08 GOOGLE LLC

AI summary

Implementations set forth herein relate to an automated assistant that can control graphical user interface (GUI) elements via voice input using natural language understanding of GUI content in order to resolve ambiguity and allow for condensed GUI voice input requests. When a user is accessing an application that is rendering various GUI elements at a display interface, the automated assistant can operate to process actionable data corresponding to the GUI elements. The actionable data can be processed in order to determine a correspondence between GUI voice input requests to the automated assistant and at least one of the GUI elements rendered at the display interface. When a particular spoken utterance from the user is determined to correspond to multiple GUI elements, an indication of ambiguity can be rendered at the display interface in order to encourage the user to provide a more specific spoken utterance.