Automated Assistant Intent Routing via Multi-Modal Input Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Automated assistants struggle to differentiate between spoken utterances intended for transcription and those meant to perform actions, leading to manual editing and increased user inputs, wasting computational resources.
Innovation Solution
The system determines whether to incorporate recognized text into a transcription or perform an assistant command based on touch input, audio-based characteristics, and the state of the transcription, using multi-modal input processing to distinguish between dictation and command intents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the automated assistant processes all spoken utterances as transcription input, then the transcription completeness is improved, but the accuracy of distinguishing commands from dictation deteriorates
Solution Approach 1:
The system segments spoken utterances into different types (transcription vs. command) using multiple classification signals including touch input detection, audio characteristic analysis, and transcription state evaluation. Each utterance is independently analyzed and routed to appropriate processing paths based on the detected intent.
Solution Approach 2:
The system introduces an intermediary classification layer that analyzes multiple signals (touch input, audio characteristics, transcription state) before determining whether to process an utterance as transcription or command. This intermediary analysis prevents direct misclassification and enables accurate intent detection.
2Manufacturing precision
If the automated assistant performs manual editing of transcriptions, then the transcription accuracy is improved, but the user input quantity and processing time increase
Solution Approach 1:
The system performs preliminary classification of spoken utterances as either transcription or command intent before full processing. By detecting touch input, analyzing audio characteristics, and evaluating transcription state in advance, the system prevents incorrect processing and reduces the need for manual editing and re-processing.
Solution Approach 2:
The system uses self-service mechanisms by automatically detecting intent through multi-signal analysis and autonomously routing utterances to appropriate processing paths. This reduces reliance on manual user correction and automated re-processing, improving overall efficiency.
3Manufacturing precision
If the automated assistant repeats processing of already provided spoken utterances, then the transcription completeness is improved, but the computational resource consumption increases
Solution Approach 1:
The system performs preliminary intent classification using touch input detection, audio characteristic analysis, and transcription state evaluation before committing computational resources to full processing. This preliminary action prevents redundant processing of commands that should not be transcribed.
Solution Approach 2:
The system uses feedback from transcription state analysis and audio characteristic detection to determine appropriate processing paths. By continuously monitoring the transcription state and analyzing audio features, the system avoids redundant processing and optimizes computational resource utilization.
Data Source
AI summary
Systems and methods described herein relate to determining whether to incorporate recognized text, that corresponds to a spoken utterance of a user of a client device, into a transcription displayed at the client device, or to cause an assistant command, that is associated with the transcription and that is based on the recognized text, to be performed by an automated assistant implemented by the client device. The spoken utterance is received during a dictation session between the user and the automated assistant. Implementations can process, using automatic speech recognition model(s), audio data that captures the spoken utterance to generate the recognized text. Further, implementations can determine whether to incorporate the recognized text into the transcription or cause the assistant command to be performed based on touch input being directed to the transcription, a state of the transcription, and/or audio-based characteristic(s) of the spoken utterance.