Articulated Model Fitting for Depth Sensor Hand Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current imaging systems face challenges in accurately capturing and interpreting the movements of articulated body portions, such as hands, due to high frame-to-frame coherency issues, self-occlusions, and ambiguity caused by similar-looking body parts, which limits their ability to provide high-fidelity imaging and control inputs for applications like gaming and human-computer interaction.
Innovation Solution
A processor-implemented method using a probabilistic and constraint-based approach that involves matching a model of articulated body portions to depth sensor data through a combination of rigid and non-rigid transformations, with iterative refinements and pixel-based adjustments to improve the accuracy of hand gesture recognition and control inputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a simple imaging system is used, then device complexity is reduced, but measurement precision of articulated body portions deteriorates
Solution Approach 1:
The articulated body is divided into multiple body portions with identified joints connecting them. The system segments the complex tracking problem into individual body portion tracking, joint location determination, and hierarchical relationship establishment, allowing precise measurement of each segment while managing overall system complexity through modular processing
Solution Approach 2:
The system transitions from 2D image plane coordinates to 3D spatial coordinates by detecting joints in multiple frames and establishing hierarchical relationships between body portions. This dimensional transformation enables accurate 3D pose estimation and articulated body tracking despite limitations of individual 2D frames
2Stability of the object's composition
If frame-to-frame coherence is increased, then stability of tracking is improved, but difficulty of detecting and measuring articulated portions worsens due to self-occlusions and similar-looking parts
Solution Approach 1:
The system performs preliminary identification of body portions and joints in each frame before establishing hierarchical relationships. By pre-identifying candidate body portions and their characteristics, the system reduces ambiguity in subsequent tracking steps and maintains frame-to-frame coherence through consistent identification criteria
Solution Approach 2:
The system uses detected joints as feedback to refine body portion identification in subsequent frames. The hierarchical relationships and joint locations from previous frames inform the search and identification process in current frames, creating a feedback loop that maintains tracking coherence while resolving ambiguities from self-occlusions and similar-looking parts
Data Source
AI summary
A depth sensor obtains images of articulated portions of a user's body such as the hand. A predefined model of the articulated body portions is provided. Representative attract points of the model are matched to centroids of the depth sensor data, and a rigid transform of the model is performed, in an initial, relatively coarse matching process. This matching process is then refined in a non-rigid transform of the model, using attract point-to-centroid matching. In a further refinement, an iterative process rasterizes the model to provide depth pixels of the model, and compares the depth pixels of the model to the depth pixels of the depth sensor. The refinement is guided by whether the depth pixels of the model are overlapping or non-overlapping with the depth pixels of the depth sensor. Collision, distance and angle constraints are also imposed on the model.