Gesture and Gaze Assistant Invocation for Resource-Efficient Sensing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants require explicit user invocation, such as spoken phrases or button presses, which can be inefficient and resource-intensive, especially when network and computing resources are utilized for every interaction.
Innovation Solution
Detecting a specific gesture and gaze directed at the device invokes dormant functions of the automated assistant, allowing selective transmission of sensor data and processing only when necessary, reducing network and computing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If explicit invocation methods (spoken phrases or button presses) are used to activate automated assistant, then the assistant can be reliably invoked, but network and computing resources are consumed for every interaction even when not needed
Solution Approach 1:
The system performs preliminary detection of user gaze direction and hand gestures before activating the automated assistant. Vision components continuously monitor for specific gestures (such as pointing) and gaze patterns, so that when both are detected, the assistant is invoked only when actually needed, avoiding unnecessary resource consumption while maintaining reliable activation
Solution Approach 2:
The patent introduces vision components and gesture recognition algorithms as intermediaries between the user and the automated assistant. These intermediaries interpret user intent through gaze and gesture detection, serving as a mediator that determines whether assistant activation is necessary, thereby reducing unnecessary resource usage while preserving reliable invocation
2Speed
If sensor data is transmitted continuously to remote components, then the automated assistant can respond quickly to user inputs, but network capacity and computing resources are wasted during periods of inactivity
Solution Approach 1:
Instead of continuous data transmission, the system implements periodic monitoring of gesture and gaze conditions. Sensor data is transmitted to remote components only when the periodic detection identifies both the target object and invocation gesture, creating a rhythm of transmission that maintains responsiveness while eliminating waste during inactivity periods
Solution Approach 2:
The patent applies different data transmission qualities based on local conditions: when no gesture or gaze is detected, minimal or no data is transmitted; when gaze is detected but gesture is absent, partial data may be transmitted; when both gaze and invocation gesture are detected, full sensor data transmission occurs. This localized quality adjustment optimizes both response speed and resource efficiency
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
Invoking one or more previously dormant functions of an automated assistant in response to detecting, based on processing of vision data from one or more vision components: (1) a particular gesture (e.g., of one or more "invocation gestures") of a user; and/or (2) detecting that a gaze of the user is directed at an assistant device that provides an automated assistant interface (graphical and/or audible) of the automated assistant. For example, the previously dormant function(s) can be invoked in response to detecting the particular gesture, detecting that the gaze of the user is directed at an assistant device for at least a threshold amount of time, and optionally that the particular gesture and the directed gaze of the user co-occur or occur within a threshold temporal proximity of one another.