E2E Intended Query Detection for Continuous Assistant Conversations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech-enabled environments require users to repeatedly invoke a hotword to interact with digital assistants, disrupting the flow of continued conversations and placing a cognitive burden on users.
Innovation Solution
An end-to-end (E2E) automatic speech recognition (ASR) system integrated with an intended query (IQ) detection model, utilizing a recurrent neural network-transducer (RNN-T) model to detect intended queries by encoding audio data, generating label history representations, and determining query intentions at each time step, allowing for continuous interaction without repeated hotword invocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If users repeatedly invoke hotword to interact with digital assistant, then the digital assistant can accurately detect user intent, but the conversational flow is interrupted and user cognitive burden increases
Solution Approach 1:
The system performs preliminary action by detecting hotword in the background continuously before user speaks. The hotword detector is always active, monitoring audio input for hotword presence, so when user invokes hotword, the system is already prepared to process the following query without interruption.
Solution Approach 2:
The patent introduces an intermediary component - the hotword detector - that mediates between user speech and digital assistant processing. The detector acts as a trigger mechanism that separates the activation function from the processing function, allowing natural conversation flow while maintaining accurate intent detection.
2Reliability
If the system processes every spoken utterance as potential query, then no query is missed, but unintended queries are incorrectly processed increasing false positives
Solution Approach 1:
The system applies local quality by making the query processing capability context-dependent. Instead of uniformly processing all utterances, the system dynamically activates processing only in local contexts where hotword is detected, creating spatial and temporal selectivity in query handling.
Solution Approach 2:
The hotword detection performs preliminary filtering before full query processing occurs. By detecting hotword presence in advance, the system pre-identifies which utterances warrant further processing, eliminating the need to analyze every spoken phrase and thereby reducing false positives while maintaining completeness.
Data Source
AI summary
A method includes receiving, as input to a speech recognition model, audio data corresponding to a spoken utterance. The method also includes performing, using the speech recognition model, speech recognition on the audio data by, at each of a plurality of time steps, encoding, using an audio encoder, the audio data corresponding to the spoken utterance into a corresponding audio encoding, and decoding, using a speech recognition joint network, the corresponding audio encoding into a probability distribution over possible output labels. At each of the plurality of time steps, the method also includes determining, using an intended query (IQ) joint network configured to receive a label history representation associated with a sequence of non-blank symbols output by a final softmax layer, an intended query decision indicating whether or not the spoken utterance includes a query intended for a digital assistant.