3D Hand Modeling from Heterogeneous Keypoints and Mesh Fitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hand modeling frameworks struggle to generate anatomically accurate and personalized 3D hand models from heterogeneous keypoint data collected by multiple tracking systems due to varying formats, coordinate systems, and accuracy issues, making data fusion difficult.
Innovation Solution
A method and system that aligns heterogeneous hand keypoints into a unified anatomical reference frame through coarse optimization, followed by a fine optimization process to fit a deformable hand mesh model, using pose, shape, and wrist orientation parameters, and derives anatomical joint positions with a trained neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If hand pose estimation models trained on single-source datasets are used, then the model performs well under controlled conditions, but it cannot handle heterogeneous keypoint data from multiple tracking systems
Solution Approach 1:
The patent transforms heterogeneous keypoint data from multiple tracking systems by applying coordinate system transformations and normalization parameters. The system converts keypoints from different coordinate systems (camera coordinates, device coordinates) into a unified anatomical reference frame using transformation matrices and scaling factors, enabling the model to process diverse inputs while maintaining accuracy
Solution Approach 2:
The patent introduces an intermediary processing layer that includes coordinate system transformation modules and keypoint alignment algorithms. This intermediary layer acts as a mediator between the diverse tracking system outputs and the hand pose estimation model, converting various keypoint formats into a standardized representation that the model can process reliably
2Ease of manufacture
If parametric hand meshes are fit directly to keypoints from specific tracking systems, then the fitting process is simple, but the resulting models lack anatomical accuracy when applied to heterogeneous data
Solution Approach 1:
The patent performs preliminary actions by pre-defining anatomical reference frames, pre-computing transformation matrices for different coordinate systems, and pre-aligning keypoint definitions before the actual mesh fitting process. This preparation ensures that when heterogeneous keypoints are transformed and fitted to the parametric hand mesh, the anatomical accuracy is maintained while the fitting process remains computationally efficient
3Quantity of substance
If multiple tracking systems are used to collect hand keypoints, then data coverage is improved, but the varying formats and coordinate systems make data fusion difficult
Solution Approach 1:
The patent creates a universal processing framework that can handle multiple types of tracking system outputs through a single unified interface. The system implements universal coordinate transformation functions and standardized keypoint alignment procedures that work across different tracking systems, reducing the need for system-specific processing code and simplifying the overall data fusion architecture
Data Source
AI summary
A method and a system for generating a 3D hand model are provided. The method includes: receiving heterogeneous hand keypoints collected from a plurality of tracking systems; performing a coarse optimization process to align the heterogeneous hand keypoints into an anatomical reference frame to produce unified hand keypoints; performing a fine optimization process to fit a hand mesh model to the unified hand keypoints; generating a 3D hand mesh using the hand mesh model fit to the unified hand keypoints; obtaining anatomical joint positions from the 3D hand mesh using a trained model; and outputting the 3D hand model including the 3D hand mesh and the anatomical joint positions.