Multimodal Interaction Using Discrete Gesture Values
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Human-machine interaction (HMI) is typically limited to unimodal or restricted multimodal interfaces, lacking the richness of natural human communication, which hinders accurate interpretation and accessibility, especially for individuals who require hands-free interaction or have limited dexterity.
Innovation Solution
The implementation of a multimodal finite state machine (FSM) that combines hand gestures with non-gesture user input data, such as text, voice, and tactile inputs, using discrete pose and motion values to define and recognize hand gestures, allowing for the creation of multimodal acts that trigger application functions without the need for machine learning or computer vision processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional unimodal or limited multimodal interfaces (switches, buttons, keyboard, pointing devices) are used for HMI, then device complexity is reduced and ease of operation is maintained, but the richness of natural human communication is lost, hindering accurate interpretation and accessibility
Solution Approach 1:
The patent segments the hand gesture recognition system into discrete pose values and discrete motion values, each representing specific hand configurations and movements. This segmentation allows complex gestures to be broken down into manageable atomic units that can be independently recognized and combined, reducing the overall complexity while maintaining rich interaction capabilities
Solution Approach 2:
The patent transforms continuous hand pose and motion data into discrete parameter values. By quantizing continuous measurements into discrete categories (e.g., specific pose angles, motion directions), the system simplifies processing requirements while preserving the essential information needed for accurate gesture recognition and interpretation
2Measurement precision
If machine learning or computer vision processing is used for hand gesture recognition, then measurement precision and accuracy are improved, but computational costs and processing time increase
Solution Approach 1:
Instead of using expensive and computationally intensive machine learning models, the patent employs simple discrete value comparisons that can be quickly evaluated. Each gesture is represented by a set of discrete pose and motion values that can be directly compared against predefined patterns, providing accurate recognition with minimal computational resources
Solution Approach 2:
The patent replaces complex computer vision processing and machine learning algorithms with a simplified discrete value-based recognition mechanism. By substituting heavy computational methods with straightforward discrete comparisons, the system achieves comparable accuracy with significantly reduced processing demands
3Adaptability or versatility
If hand gestures are defined using continuous pose and motion values, then the system can capture subtle variations in gestures, but the complexity of processing and recognizing gestures increases
Solution Approach 1:
The patent transforms continuous pose and motion parameters into discrete values, maintaining the ability to capture gesture variations through multiple discrete categories while dramatically simplifying processing. The discrete representation allows for efficient comparison and matching against predefined gesture patterns without the computational burden of continuous analysis
Data Source
AI summary
A system for associating between a computerized model of multimodal human interaction and application functions, comprising:(a) An interface for receiving instructions from a programmer defining one or more application functions.(b) A memory storing hand gestures each defined by a dataset of discrete pose values and discrete motion values.(c) A code store storing a code.(d) One or more processors coupled to the interface, the memory and the code store for executing the stored code which comprises:(1) Code instructions to define a logical sequence of user input per instructions of the programmer. The logical sequence combines hand gestures with non-gesture user input.(2) Code instructions to associate the logical sequence with the application function(s) for initiating an execution of the application function(s) during runtime of the application in response to detection of the logical sequence by analyzing a captured data depicting a user during runtime.