Multi-modal Fusion Engine for Mobile Input Contradictions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users face difficulties interacting with applications on mobile devices lacking traditional input modalities like physical keyboards and mice, as current multi-modal solutions do not allow tight integration of speech and other input modalities, limiting natural interaction and application usage on devices with limited input capabilities.
Innovation Solution
A modular architecture that combines uni-modal input processors, such as voice recognition and touch screen event processors, to detect and combine command and selecting inputs from different modalities, using a unique event repository and natural language understanding module to resolve references and produce application usable outputs, allowing for more natural interaction and enhanced use of applications on devices with limited input modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Weight of moving object
If mobile devices lack traditional input devices like physical keyboards and mice, then device compactness and mobility are improved, but ease of operation with applications is worsened
Solution Approach 1:
The patent combines multiple input modalities (touchscreen, voice, gestures, eye tracking) into a unified multi-modal input system. The fusion engine integrates signals from different input devices to generate coordinated commands, allowing users to interact with applications using any combination of available modalities, thereby maintaining ease of operation despite limited hardware
Solution Approach 2:
The system creates a universal input interface that can accept commands through multiple modalities (touch, voice, gestures, gaze). The fusion engine processes all modalities through a common framework, enabling the same application to be controlled by any combination of input methods, making the system adaptable to different user preferences and situations
2Adaptability or versatility
If applications are designed for traditional input environments, then application functionality is improved, but adaptability to mobile devices is worsened
Solution Approach 1:
The multi-modal fusion engine acts as an intermediary layer between the application and the input devices. It translates diverse input modalities into application-friendly commands, shielding applications from input device complexity while enabling them to work seamlessly on mobile devices. The fusion engine handles the complexity of modality integration, allowing applications to remain simple and focused on their core functionality
Solution Approach 2:
The system segments the input processing into separate modular components: touchscreen event processor, voice recognition processor, gesture recognition processor, and fusion engine. Each module handles a specific modality independently, then the fusion engine integrates them. This segmentation allows applications to interact with standardized interfaces without needing to understand the complexity of individual input modalities
Data Source
AI summary
An embodiment provides a method, including: detecting, at an electronic device, a command input using a first modality; detecting, at the electronic device, a selecting input using a different modality; combining, using a processor, the command input and the selecting input into an application formatted command; and providing the application formatted command to an application running on the electronic device for execution. Other aspects are described and claimed.