Gesture Lexicon Expansion for Low-Latency Assistant Control

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Automated assistants often fail to recognize intended physical motion gestures, leading to resource wastage and latency due to processing of unrecognized gestures, as they are not explicitly mapped to any action, and users resort to alternative inputs to achieve desired actions.

Innovation Solution

A method to correlate available input gestures with newly provided application functions by determining frequently used and accurately performed gestures, prompting users for confirmation, and generating mapping data to enable gesture control of previously unresponsive actions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If the automated assistant processes physical motion gestures that are not explicitly mapped to actions, then the assistant attempts to identify and respond to gestures, but this results in resource wastage and latency since the gestures are unrecognized

Engineering Contradiction:

Improvegesture recognition capabilityVSAvoidresource consumption

Core Design Contradiction:

Adaptability or versatilityVSLoss of energy

Solution Approach 1:

The system performs preliminary actions by pre-defining and storing a comprehensive lexicon of physical motion gestures and their corresponding actions before user interaction. This allows the automated assistant to immediately recognize and execute gestures without attempting to process unrecognized gestures, thereby avoiding resource wastage while maintaining versatility.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system enables users to expand the gesture lexicon through self-service mechanisms where users can define new gestures and associate them with desired actions. This allows the system to adapt to user needs without requiring manual reconfiguration by developers, maintaining both resource efficiency and versatility.

Inventive Principle:

Principle #25Self-service

2Adaptability or versatility

If the automated assistant processes physical motion gestures that are not explicitly mapped to actions, then the assistant attempts to identify and respond to gestures, but this results in latency due to additional processing and user re-inputs

Engineering Contradiction:

Improvegesture recognition capabilityVSAvoidprocessing latency

Core Design Contradiction:

Adaptability or versatilityVSLoss of time

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system incorporates feedback mechanisms where users can provide feedback on gesture recognition accuracy and suggest new gesture mappings. This feedback loop enables continuous improvement of the gesture lexicon, reducing latency over time as the system becomes more adept at recognizing intended gestures.

Inventive Principle:

Principle #23Feedback

3Productivity

If the automated assistant uses a fixed set of predefined gestures, then processing is efficient, but the system cannot adapt to new application functions or user preferences

Engineering Contradiction:

Improveprocessing efficiencyVSAvoidgesture-action mapping flexibility

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The system transitions from a static, fixed gesture lexicon to a dynamic, evolving gesture dictionary that can automatically adapt to new application functions and user preferences. This dynamic structure allows the system to maintain processing efficiency for recognized gestures while gaining the flexibility to accommodate new mappings through user-defined extensions.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

Inventive Principle:

Principle #25Self-service

4Adaptability or versatility

If the system processes all physical motion gestures, then comprehensive gesture support is achieved, but unnecessary processing occurs for gestures that do not correspond to valid actions

Engineering Contradiction:

Improvegesture input supportVSAvoidgesture processing complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentEP4160363B1Expanding physical motion gesture lexicon for an automated assistant

Publication Date: 2026.03.18 GOOGLE LLC

Figure 1A
Figure 1B
Figure 1C

AI summary

Implementations provided herein relate to correlating available input gestures to recently created application functions, and adapting available input gestures, and/or user-created input gestures, to be correlated with existing application functions. Available input gestures (e.g., a hand wave) can be those that can be readily performed upon setup of a computing device. When a user installs an application that is not initially configured to handle the available input gestures, the available input gestures can be correlated to certain functions of the application. Furthermore, a user can create new gestures for application actions and/or modify existing gestures according to their own preferences and/or physical capabilities. When multiple users elect to modify an existing gesture in the same way, the modification can be made universal, with permission from the users, in order to eliminate latency when subsequently adapting to preferences of other users.