Preemptive Session Establishment for Multimodal Input Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Voice-based user interfaces in resource-constrained devices experience significant latency when switching from low latency inputs like text to high latency inputs like voice recognition due to the need to establish a voice-to-text conversion session, which can be impractical in environments with fluctuating wireless connectivity.
Innovation Solution
Preemptively establishing a voice-to-text conversion session when circumstances indicate a user is likely to switch from text to voice input, such as through evaluating text input criteria or device context, allowing immediate processing of voice input without the delay of session establishment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If a voice-to-text conversion session is established on-demand when user switches from text to voice input, then the system can process voice input using substantial computing resources, but the user experiences significant latency of 1-2 seconds or more while the session is established
Solution Approach 1:
The system preemptively establishes a voice-to-text conversion session before the user actually switches to voice input. When the system detects that a user is likely to switch from text to voice input (based on circumstances indicating such likelihood), it proactively creates the conversion session in advance. This preliminary action eliminates the 1-2 second latency that would otherwise occur when the session is established on-demand, while still utilizing substantial computing resources for voice processing.
2Adaptability or versatility
If a client-server architecture is used for voice-to-text conversion, then complex speech recognition and semantic analysis functionality can be implemented, but the client must maintain continuous connectivity with the online service
Solution Approach 1:
The system establishes the voice-to-text conversion session preemptively when circumstances indicate the user is likely to switch to voice input. This advance session establishment ensures that the client-server connection is already active and ready when needed, eliminating the requirement for continuous connectivity maintenance while still enabling complex speech recognition and semantic analysis through the online service.
3Reliability
If voice input processing is handled locally on resource-constrained devices, then connectivity requirements are reduced, but the device lacks sufficient computing resources for complex speech recognition
Solution Approach 1:
The system preemptively establishes a voice-to-text conversion session that leverages substantial computing resources at the online service. This approach allows resource-constrained client devices to maintain offline capability and connectivity flexibility while still accessing the powerful speech recognition and semantic analysis functionality of the cloud-based service when needed, effectively combining local autonomy with remote processing power.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, apparatus, and computer-readable media (transitory and non-transitory) are provided herein for reducing latency caused by switching input modalities. In various implementations, a first input such as text input may be received at a first modality of a multimodal interface provided by an electronic device. In response to determination that the first input satisfies one or more criteria, the electronic device may preemptively establish a session between the electronic device and a query processor configured to process input received at a second modality (e.g., voice input) of the multimodal interface. In various implementations, the electronic device may receive a second input (e.g., voice input) at the second modality of the multimodal interface, initiate processing of at least a portion of the second input at the query processor within the session, and build a complete query based on output from the query processor.