Adaptive Voice Messaging With Confidence-Based Audio Fallback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current voice-based messaging systems face inefficiencies due to overuse of processing capacity by ASR systems, inaccuracies in modality selection, and inability to adapt to contextual factors, leading to errors and unnecessary power consumption.
Innovation Solution
An automated assistant selectively communicates audio recordings with text messages when speech-to-text confidence is low, and considers sender and recipient context to optimize processing and conserve resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If speech-to-text processing is always performed to ensure accurate transcription, then transcription accuracy is improved, but processing capacity and power consumption increase
Solution Approach 1:
The system dynamically changes the processing parameter (speech-to-text conversion) based on the confidence score threshold. When confidence is above the threshold, text messaging is used; when below, audio messaging is used, optimizing the balance between accuracy and energy consumption
Solution Approach 2:
The messaging modality is made dynamic rather than static. The system automatically switches between text and audio messaging based on real-time confidence score evaluation, adapting to varying transcription quality without manual intervention
2Reliability
If audio messages are always sent to ensure accurate communication, then communication accuracy is improved, but processing bandwidth and power consumption increase
Solution Approach 1:
The system changes the communication parameter (message format) based on confidence levels. High confidence messages are sent as text (low energy), while low confidence messages are sent as audio (high energy but higher accuracy), optimizing the energy-accuracy tradeoff
Solution Approach 2:
The system automatically determines the appropriate message format without user intervention. The automated assistant evaluates confidence scores and selects the optimal messaging modality (text or audio) independently, reducing user cognitive load and decision time
3Loss of information
If speech-to-text processing is performed for all messages, then transcription completeness is improved, but processing capacity is overused
Solution Approach 1:
The system extracts only the necessary processing step based on confidence levels. When confidence is high, speech-to-text processing is omitted entirely and text is used directly. When confidence is low, audio is attached to provide the missing information, avoiding unnecessary processing
Solution Approach 2:
The system applies speech-to-text processing partially rather than universally. It performs processing only when confidence scores indicate potential errors, avoiding excessive processing for high-confidence messages while ensuring completeness for low-confidence ones
Data Source
AI summary
Implementations set forth herein relate to an automated assistant that can selectively communicate audio data to a recipient when a user solicits the automated assistant to send a text message to the recipient. The audio data can include a snippet of audio that characterizes content of the text message, and the automated assistant can communicate the audio data to the recipient when score data for a speech recognition hypothesis does not satisfy a confidence threshold. The score data can correspond to an entirety of content of a text message and/or speech recognition hypothesis, and/or less than an entirety of the content. A recipient device can optionally re-process the audio data using a model that is associated with the recipient device. This can provide more accurate transcripts in some instances, thereby improving accuracy of communications and decreasing a number of corrective messages sent between users.