Multimodal Input Interpretation Using Context and User Preferences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer interface systems are unable to handle multimodal input that may need to change in real-time according to user needs and cannot accurately interpret user intent when multiple input modalities are used simultaneously.
Innovation Solution
A multimodal input system that processes data from various input modalities such as speech, touch, and 2D or 3D gestures, using a combined dictionary and preference data to interpret user intent, with a recommendation engine evaluating input events to determine the intended meaning and a personalization engine providing personalized information for accurate interpretation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a system supports multiple input modalities (speech, touch, gesture), then the system's versatility and ability to interpret user intent improve, but the system complexity and difficulty of processing multiple input types increase
Solution Approach 1:
The system divides the complex multimodal processing task into separate modules: an input processor that handles raw data from different devices, a recommendation engine that evaluates input events, and a personalization engine that provides customized dictionaries. Each module handles a specific aspect of processing, making the overall system more manageable despite handling multiple modalities.
Solution Approach 2:
The patent introduces intermediary components including a combined dictionary that maps multiple input types to unified meanings, and a personalization engine that acts as a mediator between user preferences and system interpretation. These intermediaries translate diverse input modalities into a common framework for interpretation.
2Measurement precision
If the system processes and combines multiple input modalities in real-time, then the accuracy of interpreting user intent improves, but the processing time and computational resources increase
Solution Approach 1:
The system performs preliminary processing by maintaining pre-configured dictionaries and preference data that are prepared in advance. The personalization engine pre-processes user preferences and environmental conditions to create customized interpretation frameworks before actual input processing occurs, reducing real-time computational burden.
Solution Approach 2:
The system uses feedback mechanisms where the recommendation engine evaluates input events and adjusts interpretations based on previous interactions and environmental context. This allows the system to refine its understanding of user intent iteratively, improving accuracy while managing processing time through learned patterns.
3Adaptability or versatility
If the system allows real-time adjustment of input modalities based on user preferences and environmental conditions, then the system's adaptability improves, but the complexity of managing dynamic configurations increases
Solution Approach 1:
The system implements dynamic configuration where the personalization engine continuously adapts the combined dictionary and processing parameters based on real-time environmental conditions and user preferences. The system transitions from static to dynamic operation, allowing modalities to be enabled, disabled, or reweighted based on current context without requiring manual reconfiguration.
Data Source
AI summary
The subject disclosure relates to user input into a computer system, and a technology by which one or more users interact with a computer system via a combination of input modalities. When the input data of two or more input modalities are related, they are combined to interpret an intended meaning of the input. For example, speech when combined with one input gesture has one intended meaning, e.g., convert the speech to verbatim text for consumption by a program, while the exact speech when combined with a different input gesture has a different meaning, e.g., convert the speech to a command that controls the operation of that same program.