Pinch Gesture Recognition with Occlusion Locking in XR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Extended Reality (XR) systems face challenges in accurately tracking and recognizing hand gestures, particularly when the hand is occluded, leading to unreliable user input and unintended gestures.
Innovation Solution
A system that utilizes hand tracking networks and gesture determination frameworks to predict hand poses and intentionality, combining image and depth data with occlusion handling techniques to lock the gesture state before occlusion and adjust touch signals, ensuring accurate gesture recognition even when parts of the hand are obscured.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If hand tracking is performed in XR environments, then gesture recognition capability is improved, but reliability deteriorates when hand is occluded
Solution Approach 1:
The system performs preliminary actions by locking the gesture state before occlusion occurs. When a pinch gesture is detected, the system records the gesture state (e.g., pinch detected) and maintains this state even when the hand becomes occluded, ensuring continuous and reliable gesture recognition without requiring constant visual confirmation of the hand.
2Speed
If continuous hand tracking is performed, then gesture detection responsiveness is improved, but false inputs increase due to occlusion
Solution Approach 1:
The system applies preliminary anti-action by actively preventing false inputs through occlusion detection. Before a false gesture can be processed, the system detects hand occlusion and suspends or cancels the gesture detection, thereby offsetting the harmful effect of occlusion-induced false inputs while maintaining responsive gesture detection when the hand is visible.
3Reliability
If gesture state is locked before occlusion, then gesture recognition reliability is improved, but system complexity increases
Solution Approach 1:
The system segments the gesture detection process into distinct phases: detection phase (when hand is visible), locking phase (when occlusion is detected), and maintenance phase (during occlusion). This segmentation allows the system to handle occlusion systematically by separating the gesture detection logic from the occlusion handling logic, reducing overall system complexity while maintaining reliability.
Data Source
AI summary
Processing gesture input includes obtaining hand tracking data based on a set of camera frames, determining a hand pose based on the hand tracking data, and determining an intentionality classification for a gesture based on the hand pose. An input action corresponding to the gesture is enabled based on the hand pose and the intentionality classification. An occlusion classification is determined for the hand based on the hand pose and the input gesture can be determined based on the occlusion classification.