Automated Assistant Streaming ASR-NLU for Natural Conversations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Turn-based dialog sessions with automated assistants are inefficient and resource-intensive, as they often require users to provide multiple utterances to convey a single thought, leading to prolonged interactions and computational waste due to incomplete processing.
Innovation Solution
Implement a streaming automatic speech recognition model to generate ASR output, process it with a natural language understanding model to generate NLU output, and determine a next interaction state based on audio-based characteristics, allowing the automated assistant to iteratively respond without waiting for complete user input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If the automated assistant processes each spoken utterance in isolation using turn-based dialog sessions, then the processing logic is simplified, but the dialog sessions are prolonged and computational resources are wasted
Solution Approach 1:
The patent merges multiple discrete spoken utterances into a single continuous transcription stream. Instead of processing each utterance separately as in traditional turn-based dialog, the system combines utterances provided during the rendering of assistant responses into one continuous processing flow, enabling the assistant to respond more naturally without prolonged wait times
Solution Approach 2:
The system performs preliminary transcription of audio data into text before full processing occurs. By converting audio to text in advance and maintaining a transcription stream, the assistant can begin processing information earlier and respond more quickly without waiting for complete user input sequences
2Measurement precision
If the automated assistant waits for complete user input before processing, then processing accuracy is improved, but computational resources are wasted due to prolonged idle time
Solution Approach 1:
The system performs partial processing on incomplete information. By analyzing the transcription stream as it develops rather than waiting for complete input, the assistant can begin formulating responses based on available information, reducing computational waste from prolonged idle time while maintaining acceptable accuracy through iterative refinement
Solution Approach 2:
The system uses feedback from the transcription stream to dynamically adjust processing. By continuously monitoring the developing transcription and using this feedback to guide when and how to process information, the assistant balances accuracy requirements with resource efficiency, processing only when sufficient information is available
3Adaptability or versatility
If the automated assistant processes multiple utterances simultaneously in a continuous stream, then natural conversation capability is improved, but the processing complexity increases
Solution Approach 1:
The patent segments the processing system into distinct functional components: an automatic speech recognition component that generates the transcription stream, a natural language understanding component that processes the stream, and a response generation component. This segmentation manages complexity by assigning specific functions to separate modules while enabling simultaneous processing of multiple utterances
Data Source
AI summary
As part of a dialog session between a user and an automated assistant, implementations can process, using a streaming ASR model, a stream of audio data to generate ASR output, process, using an NLU model, the ASR output to generate NLU output, and generate, based on the NLU output, a stream of fulfillment data. Further, implementations can further determine, based on processing the stream of audio data, audio-based characteristics associated with spoken utterance(s) captured in the stream of audio data. Based on a current state of the stream of NLU output, the stream of fulfillment data, and the audio-based characteristics, implementations can determine whether a next interaction state to be implemented is: (i) causing fulfillment output to be implemented; (ii) causing natural conversation output to be audibly rendered; or (iii) refrain from causing any interaction to be implemented, can cause the next interaction state to be implemented.