Multimodal HCI Intent Recognition via Input Fusion
Solution Overview
Problem
Conventional graphical user interfaces (GUIs) face challenges in mobile devices due to physical size constraints and hands-busy or eyes-busy scenarios, and speech recognition technology is prone to errors due to variability in speakers, speaking styles, and acoustic environments, limiting its effectiveness in human-computer interaction.
Innovation Solution
A multi-modal human-computer interface (HCI) system that combines various input modalities such as speech, text, pen, gestures, and gaze to enhance user intent recognition, leveraging data fusion and semantic parsing to improve accuracy and robustness by synthesizing inputs and employing extrinsic information to clarify user intent.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If speech recognition technology is used to enable natural human-computer communication, then ease of operation is improved, but reliability deteriorates due to errors from speaker variability, speaking style differences, and acoustic environment variations
Solution Approach 1:
The patent combines multiple input modalities (speech, text, pen, gestures, gaze) into a unified multimodal interface. By merging these different input methods, the system can cross-validate information and resolve ambiguities that would plague any single modality alone, thereby maintaining ease of operation while improving reliability through redundant verification channels.
Solution Approach 2:
The system implements feedback mechanisms where the computer provides responses to users about its interpretation of their intent. This allows users to confirm or correct the system's understanding, creating a loop that improves reliability by enabling error detection and correction while maintaining natural interaction.
2Ease of operation
If speech alone is used as input to computers, then ease of operation is improved, but reliability deteriorates due to ambiguity in spoken language and imperfection of speech recognition technology
Solution Approach 1:
The patent merges speech input with other modalities such as text, pen input, gestures, and gaze. When speech alone is ambiguous, the system can supplement it with gestures or text, and when speech recognition fails, it can fall back to alternative modalities, thereby maintaining ease of operation while significantly improving reliability through multiple confirmation channels.
3Ease of operation
If conventional graphical user interfaces are used in mobile devices, then ease of operation is improved through intuitive real-world metaphors, but device complexity increases due to physical size constraints and hands-busy or eyes-busy scenarios
Solution Approach 1:
The patent creates a universal interface that works across multiple device types and usage scenarios. By designing a multimodal system that can adapt to hands-busy or eyes-busy situations through alternative input modalities, the interface maintains ease of operation without requiring complex device-specific implementations, thereby reducing overall device complexity while preserving user-friendly interaction.
4Reliability
If multiple input modalities are combined to overcome speech technology limitations, then reliability is improved, but device complexity increases due to the need to fuse multiple input modalities
Solution Approach 1:
The patent segments the multimodal processing into distinct modules, each handling a specific modality (speech processing module, text processing module, gesture processing module, etc.). This segmentation allows each module to be optimized independently while maintaining a clear interface for integration, thereby improving reliability through comprehensive modal coverage while managing device complexity through modular architecture.
Solution Approach 2:
The patent introduces an intermediary layer that translates various modal inputs into a unified representation of user intent. This mediator component sits between the diverse input modalities and the core processing system, standardizing interfaces and simplifying integration, thereby enabling reliable multimodal operation without proportionally increasing device complexity.
Data Source
AI summary
A multi-modal human computer interface (HCI) receives a plurality of available information inputs concurrently, or serially, and employs a subset of the inputs to determine or infer user intent with respect to a communication or information goal. Received inputs are respectively parsed, and the parsed inputs are analyzed and optionally synthesized with respect to one or more of each other. In the event sufficient information is not available to determine user intent or goal, feedback can be provided to the user in order to facilitate clarifying, confirming, or augmenting the information inputs.