Hand Mesh Prediction for Real-Time XR Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing XR systems face challenges in dynamically occluding virtual objects with hands due to high computational demands and the need for fast, accurate hand motion updates, leading to unrealistic XR experiences.
Innovation Solution
A method for generating a hand mesh from live depth sensor data at a high frame rate, using key-points to represent hand segments and predicting hand poses for real-time occlusion processing, reducing computational burden.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hand mesh is generated from live depth sensor data at high frame rate, then occlusion accuracy is improved, but computational resources are consumed
Solution Approach 1:
The hand is segmented into multiple key-points representing different segments (fingers, palm, joints). This segmentation allows the system to represent the hand with a simplified set of discrete points rather than processing the entire hand surface, significantly reducing computational requirements while maintaining occlusion accuracy for virtual object rendering.
Solution Approach 2:
The system creates a simplified representation (copy) of the hand using key-points that capture essential hand geometry and pose information. This copy is then used for occlusion calculations instead of processing the complete hand mesh, enabling high frame rate updates with reduced computational burden.
2Speed
If hand mesh is updated in real time, then occlusion responsiveness is improved, but processing time increases
Solution Approach 1:
By dividing the hand into discrete key-points, the system can rapidly update only the positions of these points in response to hand motion. This segmented approach enables real-time occlusion updates at high frame rates without the processing time penalty of updating complete hand meshes, as only the essential key-point positions need to be recomputed and applied.
3Device complexity
If key-points are used to represent hand segments, then device complexity is reduced, but hand representation accuracy may worsen
Solution Approach 1:
The hand is represented by strategically placed key-points that segment the hand into functional regions (fingers, palm, joints). This segmentation provides sufficient geometric information for accurate occlusion rendering while dramatically simplifying the data structure compared to dense mesh representations, reducing system complexity without sacrificing the accuracy needed for virtual object occlusion.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An augmented reality/mixed reality system that provides an immersive user experience. That experience is provided with increased speed of update with low computational resources for occlusion data representing a user's hand that can be used to occlude virtual objects. The hand occlusion data are generated by computing a hand mesh from live depth sensor data. A small amount of data is processed by masking live depth data with a model in which a hand is simply represented by multiple segments identified from key-points. The hand occlusion data may be generated by predicting changes in a hand pose between a time of capture of the depth data and the time when the hand mesh will be used for occlusion processing. The hand mesh may be distorted to represent the hand in the predicted pose.