Hand Gesture Input for Wearable Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current augmented reality (AR), virtual reality (VR), and mixed reality (MR) systems face challenges in providing precise and intuitive user interactions, particularly in 3D environments, where traditional inputs like touchpads are limited by being two-dimensional, leading to high error rates and incorrect operations.
Innovation Solution
The use of hand gestures as inputs in AR/VR/MR environments, where keypoints on the user's hands are tracked to recognize gestures, and a multi-DOF controller is formed based on these gestures, allowing interaction with virtual objects through a registered interaction point and a control ray, reducing complexity and error by leveraging the expressive nature of hand movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional 2D touchpad inputs are used in AR/VR/MR systems, then device complexity is reduced, but interaction precision and reliability deteriorate due to limited dimensional expression
Solution Approach 1:
The patent transitions from 2D touchpad input to 3D hand gesture input by capturing spatial coordinates, orientation, and pose information from hand movements in three-dimensional space. This dimensional expansion enables precise interaction with virtual objects while maintaining intuitive natural gestures, resolving the contradiction between precision and complexity.
Solution Approach 2:
The patent replaces mechanical touchpad interfaces with optical/electromagnetic sensing systems that detect hand gestures through cameras or sensors. This substitution eliminates the need for physical contact interfaces while achieving higher interaction precision through spatial and orientational data from hand movements.
2Ease of operation
If hand gesture recognition is implemented, then interaction intuitiveness and precision are improved, but computational complexity and processing requirements increase
Solution Approach 1:
The patent segments the hand gesture recognition process into multiple independent components: keypoint detection, gesture classification, and interaction execution. By dividing the complex task into manageable segments, the system achieves high intuitiveness while managing computational complexity through modular processing.
Solution Approach 2:
The patent performs preliminary actions by pre-defining gesture templates and classification models that enable rapid recognition during operation. This preparation allows the system to achieve high intuitiveness through familiar gestures while reducing real-time computational burden through pattern matching rather than complex analysis.
3Measurement precision
If multiple degrees of freedom are used for controller formation, then interaction precision is improved, but device complexity increases
Solution Approach 1:
The patent enables the system to automatically form multi-DOF controllers from raw hand gesture data without requiring manual configuration or complex hardware assemblies. The controller is self-generated through computational processing of hand pose and gesture information, achieving high precision while minimizing physical device complexity.
Data Source
AI summary
Techniques are disclosed for allowing a user's hands to interact with virtual objects. An image of at least one hand may be received from an image capture devices. A plurality of keypoints associated with at least one hand may be detected. In response to determining that a hand is making or is transitioning into making a particular gesture, a subset of the plurality of keypoints may be selected. An interaction point may be registered to a particular location relative to the subset of the plurality of keypoints based on the particular gesture. A proximal point may be registered to a location along the user's body. A ray may be cast from the proximal point through the interaction point. A multi-DOF controller for interacting with the virtual object may be formed based on the ray.