Hand Tracking via Head-Mounted and Hand-Held Camera Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hand tracking technologies in extended reality (XR) environments face challenges in accuracy when users wear hand-held controllers, as these devices can occlude parts of the hand, leading to incomplete image capture and reduced tracking precision.
Innovation Solution
A hand tracking method that utilizes a combination of images from a head-mounted device's camera and a hand-held device's camera, where the first camera captures a complete image of the hand and the second camera captures occluded parts, allowing for comprehensive hand pose determination and improved gesture recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If a hand-held device is worn on the hand, then the user can interact with XR environments, but the device occludes parts of the hand leading to incomplete image capture and reduced tracking precision
Solution Approach 1:
The system divides hand tracking into two segments: (1) head-mounted device captures images of visible hand parts, (2) hand-held device captures images of occluded hand parts. Each device independently processes its segment, and the results are merged to form a complete hand tracking solution, resolving the contradiction between wearing the device and tracking accuracy.
Solution Approach 2:
The hand-held device acts as an intermediary that captures images of the occluded hand parts and transmits them to the host system. This intermediary function allows the system to overcome the occlusion problem created by the device itself, maintaining tracking precision while enabling hand interaction.
2Device complexity
If only a head-mounted device is used for hand tracking, then the system is simpler, but parts of the hand occluded by the hand-held device cannot be captured
Solution Approach 1:
The imaging task is segmented between two devices: the head-mounted device captures visible hand portions while the hand-held device captures occluded portions. This segmentation ensures complete hand information is gathered without requiring one device to be overly complex.
Solution Approach 2:
The hand-held device serves multiple functions: it acts as a controller for XR interaction and simultaneously functions as an imaging device for capturing occluded hand parts. This multi-functionality reduces the need for additional dedicated devices, managing system complexity while preventing information loss.
3Measurement precision
If multiple cameras are used to capture complete hand images, then tracking accuracy improves, but the system complexity and data processing requirements increase
Solution Approach 1:
The camera system is segmented across two devices with distinct roles: the head-mounted camera captures the overall hand scene while the hand-held camera captures specific occluded regions. This segmentation allows accurate hand pose determination without requiring a single complex multi-camera system.
Solution Approach 2:
The hand-held device performs self-service by using its own camera to capture images of the hand parts that it occludes. This self-capturing approach simplifies the overall system configuration compared to using multiple external cameras, as the device serves its own imaging needs.
Data Source
AI summary
The embodiments of the disclosure provide a method for hand tracking. The method includes following steps. A first image of a hand is obtained through a head-mounted device. A first pose of a first part of the hand is determined through a processor based on the first image. A second image of the hand is obtained through a hand-held device. A second pose of a second part of the hand is determined through a processor based on the first image. The first part and the second part complementarily form an entirety of the hand. A gesture of the hand is determined through the processor based on the first pose and the second pose.