Multi-modal Hand Tracking for Avatar Animation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional virtual reality and augmented reality systems struggle to accurately animate a user's avatar hand movements, often resulting in errors due to IMU drift and limited vision algorithms that fail to identify the active hand or its orientation, leading to unrealistic and inaccurate representations.
Innovation Solution
A wearable system that combines multiple sensor modalities, including 6DOF external tracking, internal motion sensors, and external passive tracking, to determine the active hand and its orientation with enhanced confidence levels, using ergonomic and motion data to improve hand tracking accuracy beyond the field of view.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If IMU and vision algorithms are used to track hand movements, then the system can provide avatar animation, but the accuracy deteriorates due to IMU drift and limited vision algorithm performance
Solution Approach 1:
The patent combines multiple sensor modalities (IMU, external tracking sensors, and vision algorithms) into a unified hand tracking system. The IMU provides continuous motion data, external sensors provide 6DOF tracking information, and vision algorithms provide contextual understanding, merging these complementary data sources to overcome the limitations of any single modality and improve overall tracking accuracy and reliability
Solution Approach 2:
The system introduces an intermediary processing layer that fuses data from multiple sensors and algorithms. This intermediary layer reconciles the drift-prone IMU data with the more stable external tracking and vision data, acting as a mediator that produces a more accurate final hand position and orientation estimate than any single source could provide
2Measurement precision
If vision algorithms are used to identify active hand, then the system can determine hand orientation, but the reliability decreases near field of view boundaries
Solution Approach 1:
External tracking sensors serve as an intermediary system that provides reliable hand position and orientation data independent of field of view boundaries. This intermediary tracking system compensates for the vision algorithm's reduced reliability at FOV edges, ensuring continuous accurate tracking throughout the entire viewing area
Solution Approach 2:
The system changes the operational parameters of hand tracking by switching between or combining different sensing modalities depending on the situation. When vision algorithms approach FOV boundaries where reliability decreases, the system relies more heavily on external tracking sensors which maintain consistent performance across the entire field of view
3Reliability
If multiple sensor modalities are combined, then the confidence level in hand tracking increases, but the device complexity increases
Solution Approach 1:
The patent segments the hand tracking system into distinct functional modules: IMU processing module, external sensor tracking module, vision algorithm module, and data fusion module. Each module handles a specific sensing modality independently, processing its data through dedicated algorithms before combining results, which reduces the complexity of integrating multiple sensors compared to a monolithic approach
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Examples of systems and methods for improved hand tracking of a user in a mixed reality environment are disclosed. The systems and methods may be configured to estimate the hand pose and shape of a user's hands for applications such as animating a hand on a user's avatar. Data from multiple sources, such as a totem internal measurement unit ("IMU"), external totem location tracking, vision cameras, and depth sensors, may be manipulated using a set of rules that are based on historical data, ergonomics data, and motion data.