Selective On-Device Speech Recognition for Faster Assistant Responses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated assistants require explicit invocation, such as hot-words or user inputs, which prolongs the time to obtain responses and consumes unnecessary resources, and indiscriminate on-device processing can compromise security and resource efficiency.
Innovation Solution
Implement selective activation of on-device speech recognition, NLU, and fulfillment based on implicit and explicit cues, including directed speech, user presence, and sensor data, to reduce unnecessary processing and conserve resources while ensuring secure and efficient interaction with automated assistants.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If explicit invocation (hot-words or user inputs) is required to activate automated assistant, then user privacy and resource consumption are controlled, but interaction time is prolonged
Solution Approach 1:
The system performs preliminary voice activity detection and speaker verification before full assistant activation. The VAD module continuously monitors for voice presence, and the speaker verification module pre-validates the speaker identity, so that when a command is detected, the assistant can activate immediately without requiring explicit hot-words or user input gestures.
2Speed
If on-device speech recognition is continuously activated, then response time is reduced, but resource consumption increases
Solution Approach 1:
Instead of continuous full processing, the system uses periodic voice activity detection to determine when speech is present. The VAD module operates continuously at low power to detect voice presence, and only triggers the more resource-intensive on-device speech recognition and processing when voice activity is detected, creating a periodic activation pattern that balances speed and energy consumption.
3Loss of time
If on-device processing is indiscriminately performed, then response time is reduced, but security and resource efficiency are compromised
Solution Approach 1:
The system applies different processing qualities to different audio segments. Voice activity detection is performed on all audio segments at basic quality level. When voice is detected, speaker verification is applied to validate the speaker identity. Only after successful verification does the system activate full on-device speech recognition and processing. This layered approach ensures security through verification while enabling fast response when conditions are met.
4Productivity
If explicit invocation is required, then resource consumption is controlled, but productivity decreases
Solution Approach 1:
The system performs preliminary voice activity detection and speaker verification to prepare for fast activation. By having the VAD module continuously monitor and the speaker verification module pre-validate identities, the system eliminates the need for explicit hot-words or user input gestures, enabling immediate response when commands are detected while controlling resources through conditional activation based on verified voice presence.
Data Source
AI summary
Implementations can reduce the time required to obtain responses from an automated assistant by, for example, obviating the need to provide an explicit invocation to the automated assistant, such as by saying a hot-word/phrase or performing a specific user input, prior to speaking a command or query. In addition, the automated assistant can optionally receive, understand, and/or respond to the command or query without communicating with a server, thereby further reducing the time in which a response can be provided. Implementations only selectively initiate on-device speech recognition responsive to determining one or more condition(s) are satisfied. Further, in some implementations, on-device NLU, on-device fulfillment, and/or resulting execution occur only responsive to determining, based on recognized text form the on-device speech recognition, that such further processing should occur. Thus, through selective activation of on-device speech processing, and/or selective activation of on-device NLU and/or on-device fulfillment, various client device resources are conserved.