Multi-Modal Controller Synchronizing Visual and Voice Views
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional distributed techniques for multi-modal user interfaces in client-server systems face challenges in synchronizing the states of multiple modalities across a network, often requiring non-standard protocols and unconventional content authoring techniques, which are not readily embraced by application designers.
Innovation Solution
The implementation of distributed multi-modal applications using standard protocols and conventional content authoring techniques, where a client device communicates with an application server and a voice server to synchronize visual and voice modalities through asynchronous and synchronous request/response protocols, allowing consistent synchronization of user interface states.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If distributed techniques are used to shift processing to remote servers, then client device complexity is reduced, but synchronization of multiple modalities becomes difficult to achieve
Solution Approach 1:
The patent introduces a multi-modal controller as an intermediary component that coordinates between the visual modality (handled by the client device) and the voice modality (handled by the remote voice server). This mediator ensures that both modalities remain synchronized by managing the interaction between them, thus resolving the synchronization difficulty while maintaining the distributed architecture benefits.
2Adaptability or versatility
If non-standard protocols and unconventional content authoring techniques are used, then multi-modal user interface can be implemented, but ease of operation decreases
Solution Approach 1:
The patent enables a single multi-modal display element to function with multiple input modalities (visual and voice). By making the display element universal and capable of handling both modalities through standard protocols, the system achieves multi-modal capability without requiring separate specialized elements for each modality, thus maintaining ease of operation and content authoring.
3Adaptability or versatility
If all requisite hardware and software for multiple modalities are included in the client device, then multi-modal user interface is achieved, but device complexity and cost increase
Solution Approach 1:
The patent extracts the voice modality processing requirements from the client device and places them on a remote voice server. This extraction allows the client device to maintain only the visual modality components, significantly reducing device complexity and cost while still providing full multi-modal user interface capability through the distributed architecture.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments include methods and apparatus for synchronizing data and focus between visual and voice views associated with distributed multi-modal applications. An embodiment includes a client device adapted to render a visual display that includes at least one multi-modal display element for which input data is receivable though a visual modality and a voice modality. When the client detects a user utterance via the voice modality, the client sends uplink audio data representing the utterance to a speech recognizer. An application server receives a speech recognition result generated by the speech recognizer, and sends a voice event response to the client. The voice event response is sent as a response to an asynchronous HTTP voice event request previously sent to the application server by the client. The client may then send another voice event request to the application server in response to receiving the voice event response.