Wearable Controller Tracking via Hand Grip and IMU Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR/VR devices face challenges in accurately and cost-effectively tracking controllers due to the need for infrared LEDs, which increase manufacturing costs and can be interfered with by occlusions or other light sources, while hand tracking methods are less accurate.
Innovation Solution
A method that estimates the grip of a user's hand based on feature-tracking from captured images and adjusts this grip using inertial measurement unit (IMU) data from the controller, allowing for accurate tracking without the need for LEDs on the controller.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If infrared LEDs are used on the controller for tracking, then tracking accuracy is improved, but manufacturing cost increases and the system becomes vulnerable to occlusions and light interference
Solution Approach 1:
The patent removes the infrared LEDs from the controller, extracting the problematic hardware component that causes cost increases and occlusion vulnerabilities. The tracking system then uses alternative methods (hand feature detection combined with IMU data from the controller) to achieve tracking without requiring special LEDs on the controller.
Solution Approach 2:
The patent introduces an intermediary approach by using hand feature detection as a mediator between the user's hand and the controller tracking. Instead of directly tracking the controller with LEDs, the system tracks hand features and infers controller position, using IMU data as an additional mediator to refine the tracking accuracy.
2Device complexity
If hand tracking is used instead of controller tracking with LEDs, then manufacturing cost is reduced, but tracking accuracy deteriorates
Solution Approach 1:
The patent merges multiple tracking approaches: hand feature detection from camera images is combined with IMU data from the controller to create a hybrid tracking system. This combination allows the system to benefit from both the low cost of hand tracking and the accuracy provided by IMU data, achieving both cost reduction and accuracy maintenance.
Solution Approach 2:
The patent implements feedback by continuously refining the hand pose estimation using IMU data from the controller. The system uses the IMU information to correct and update the hand tracking results, creating a feedback loop that improves accuracy over time while maintaining the cost advantages of hand-based tracking.
3Productivity
If feature tracking from captured images is used to estimate hand pose, then tracking speed is improved, but accuracy may be affected by occlusions and lighting conditions
Solution Approach 1:
The patent uses IMU data as an intermediary to bridge the gap between fast camera-based feature tracking and accurate pose estimation. The IMU data serves as a mediator that provides complementary information about controller motion, helping to maintain accuracy even when visual tracking is affected by occlusions or lighting changes.
Solution Approach 2:
The patent implements feedback by using IMU data to continuously refine and correct the hand pose estimates derived from image features. This feedback mechanism allows the system to maintain high tracking speed from camera-based methods while using IMU information to compensate for inaccuracies caused by occlusions or poor lighting conditions.
Data Source
AI summary
In one embodiment, a method includes capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment, identifying one or more features from the first image to estimate a pose of the hand of the user, estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller, receiving IMU data of the controller, and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. The method utilizes multiple data sources to track the controller under various conditions of the environment to provide an accurate controller tracking consistently.