Automated Assistant Visual-Audio Fusion for Hot Command Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants require explicit invocation using predefined hot words, which can be cumbersome and inefficient, especially in resource-constrained devices, and may lead to inadvertent activation.
Innovation Solution
Implementing techniques that analyze both audio and visual inputs on a client device to determine user intent for interaction without explicit invocation, using neural networks to fuse audio and visual features for confidence-based decision-making, allowing local processing to trigger responsive actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If the automated assistant operates in limited listening state with hot word detection only, then power consumption and computing resource usage are reduced, but user interaction convenience deteriorates due to requiring explicit invocation
Solution Approach 1:
The system dynamically transitions between limited listening state (hot word detection only) and full listening state (speech recognition enabled) based on detected user intent indicators. Visual indicators such as gaze direction toward the device, hand gestures pointing at the device, or approaching motion can trigger transition from low-power mode to high-power mode, allowing the assistant to adapt its resource consumption to actual user needs rather than maintaining a fixed state
Solution Approach 2:
The automated assistant autonomously determines when to activate full speech recognition capabilities by monitoring for visual indicators of user intent. Instead of requiring explicit hot word invocation, the system uses computer vision to detect when a user is looking at or gesturing toward the device, automatically transitioning to a state where it can process arbitrary speech inputs without continuous hot word monitoring
2Ease of operation
If the automated assistant operates in full listening state with speech-to-text processing, then user interaction convenience is improved, but power consumption and computing resource usage increase
Solution Approach 1:
Instead of continuously operating in full listening state, the system applies partial action by activating speech-to-text processing only when visual indicators suggest user intent. The visual indicator detection acts as a gatekeeper, allowing full speech recognition to run only when necessary, thereby reducing overall power consumption while maintaining interaction convenience when needed
Solution Approach 2:
The system performs preliminary visual analysis before activating resource-intensive speech-to-text processing. By first detecting visual indicators such as gaze direction or hand gestures pointing at the device, the system prepares for potential speech input only when user intent is indicated, avoiding unnecessary activation of full listening state and associated power consumption
3Reliability
If hot words are selected to be long tail words or phrases, then inadvertent invocation is reduced, but user interaction convenience deteriorates due to cumbersome invocation requirements
Solution Approach 1:
Visual indicators serve as an intermediary between the user and the automated assistant, replacing the need for explicit hot word invocation. The visual detection system mediates the activation process by detecting gaze direction, hand gestures, or approaching motion, thereby eliminating the requirement for users to utter specific long tail hot words while maintaining reliable invocation through multiple alternative signals
4Loss of information
If speech recognition processing is performed locally on the client device, then user privacy is preserved, but computing resource requirements increase
Solution Approach 1:
The system segments processing tasks between local visual indicator detection and potential cloud-based speech recognition. Visual processing (detecting gaze, gestures, or approach) is performed locally on the resource-constrained device, while speech-to-text processing may be offloaded to cloud services when activated. This segmentation allows privacy-preserving local visual analysis while managing computing resource requirements through selective cloud offloading
Data Source
AI summary
Techniques are described herein for detecting and/or enrolling (or commissioning) new “hot commands” that are useable to cause an automated assistant to perform responsive action(s) without having to be first explicitly invoked. In various implementations, an automated assistant may be transitioned from a limited listening state into a full speech recognition state in response to a trigger event. While in the full speech recognition state, the automated assistant may receive and perform speech recognition processing on a spoken command from a user to generate a textual command. The textual command may be determined to satisfy a frequency threshold in a corpus of textual commands. Consequently, data indicative of the textual command may be enrolled as a hot command. Subsequent utterance of another textual command that is semantically consistent with the textual command may trigger performance of a responsive action by the automated assistant, without requiring explicit invocation.