Hand Gesture Tracking via Joint Extraction and Simulated D-Pad
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hand tracking systems in VR/AR/MR environments fail to accurately track the positions and precise movements of users' fingers and thumbs, leading to poor representation of hand movements within simulated environments.
Innovation Solution
A system comprising an imaging sensor to capture hand positions, a console to extract joint information, and determine gestures, which updates display instructions to generate simulated elements in response to detected hand movements, allowing users to interact with virtual objects through precise hand gestures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional hand tracking systems are used in VR/AR/MR headsets, then the system structure remains simple, but the tracking precision of finger and thumb positions deteriorates
Solution Approach 1:
The hand tracking system is segmented into multiple independent imaging sensors, each dedicated to tracking specific body parts (hands, eyes, head). This segmentation allows each sensor to optimize for its specific tracking target, improving overall measurement precision without requiring a single overly complex system
Solution Approach 2:
Multiple imaging sensors are nested within the headset structure, with sensors positioned to capture specific views of the user's hands and body. The nested arrangement allows compact integration of multiple tracking functions within the limited space of the headset, maintaining simplicity while enhancing precision
2Measurement precision
If multiple imaging sensors are added to track hand movements accurately, then tracking precision improves, but device complexity increases
Solution Approach 1:
The imaging sensors serve multiple functions simultaneously: tracking hand positions, detecting gestures, monitoring eye movements, and capturing head orientation. This multi-functionality allows the system to achieve high tracking precision across multiple parameters without proportionally increasing the number of sensors
Solution Approach 2:
Multiple tracking functions (hand tracking, eye tracking, head tracking) are merged into a unified system using the same imaging sensor array. The console processes all sensor data through a single gesture recognition engine, combining multiple functions into one integrated system that improves precision without linearly increasing complexity
3Adaptability or versatility
If the headset blocks the user's view of their own hands, then immersion in the virtual environment improves, but ease of operation deteriorates
Solution Approach 1:
The system creates accurate virtual copies of the user's hands in the simulated environment by tracking real hand positions with multiple imaging sensors. These virtual hands provide visual feedback to the user, allowing them to see and control their virtual representations even when their physical hands are obscured by the headset
Solution Approach 2:
The system provides continuous visual feedback by displaying the user's virtual hands in the VR/AR environment, which are updated in real-time based on sensor data. This feedback loop allows users to maintain awareness of their hand positions and gestures without needing to see their physical hands, preserving both immersion and ease of operation
Data Source
AI summary
A system includes an electronic display configured to display one or more simulated objects in accordance with display instructions, an imaging sensor configured to capture images of a user's hands, and a console. The console is configured to receive the captured images from the imaging sensor, extract joint information of the user's hands from the captured images, and determine one or more poses based on the extracted joint information. In response to the determined poses indicating the user's index finger positioned orthogonally to the user's thumb, and the thumb within a minimum distance to the index finger, the console detects a directional pad display gesture, and update the display instructions to cause the electronic display to generate a simulated directional pad adjacent to the user's thumb in a simulated environment that is presented to the user via the electronic display.