Handheld Controller Pose Tracking Under Illuminator Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing extended reality (XR) systems face challenges in accurately tracking handheld controllers when illuminators or other features become occluded, leading to inaccuracies in determining the controller's position and orientation.
Innovation Solution
A technique that combines hand tracking data with motion data from the controller, such as IMU data, to infer the pose of the controller when illuminator-based detection is unreliable, using a combined network to jointly predict hand and controller poses and adjust the reliance on visibility of illuminators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If illuminator-based detection is used to track the controller, then tracking accuracy is improved when illuminators are visible, but tracking reliability deteriorates when illuminators are occluded
Solution Approach 1:
The patent introduces hand tracking data as an intermediary to bridge the gap when illuminator-based detection fails. The system uses a combined network that processes both image data and hand tracking data, allowing continuous controller pose estimation even when illuminators are occluded by the user's hand or other objects.
Solution Approach 2:
The patent merges multiple tracking approaches by combining illuminator-based detection with hand tracking data through a combined network. This fusion of multiple data sources (image data from illuminators and hand tracking data) allows the system to maintain tracking accuracy and reliability across different visibility conditions.
2Reliability
If hand tracking data is combined with motion data to infer controller pose, then tracking reliability is improved when illuminators are occluded, but device complexity increases
Solution Approach 1:
The combined network serves multiple functions: it processes illuminator-based tracking when available, switches to hand tracking data when illuminators are occluded, and continuously provides controller pose estimation. This multi-functionality allows a single system to handle various tracking scenarios without requiring separate dedicated systems for each condition.
3Measurement precision
If a combined network is used to jointly predict hand and controller poses, then tracking accuracy is maintained under occlusion, but computational requirements increase
Solution Approach 1:
The system dynamically adjusts its processing approach based on illuminator visibility. When illuminators are visible, the system uses the simpler illuminator-based detection. When occlusion is detected, the system transitions to using hand tracking data through the combined network. This dynamic adaptation allows the system to maintain accuracy while minimizing computational energy expenditure by only using the more complex processing when necessary.
Data Source
AI summary
Hand-held controllers continue to be tracked when illuminators on the controllers are occluded. Image data is captured of a hand holding a physical controller with illuminators, and motion sensor data is received from the controller. A determination is made as to whether illuminator-based pose detection is reliable based on the visibility of the illuminators. When the illuminator-based pose detection is not considered reliable, the controller's pose is determined using hand-tracking data for the hand holding the controller. Tracking information for the controller is determined by considering the spatial relationship between the hand and controller in previous frames and adjusting parameters based on a visibility metric. This facilitates generating virtual content that corresponds with the physical controller's current pose.