3D Hand Mesh Pose Estimation via Graph CNN
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional VR and AR systems inaccurately represent user hands due to reliance on generic 3D hand models and sparse joint location data, lacking detailed hand shape and surface features, which hampers user interaction and realism.
Innovation Solution
Employing a graph Convolutional Neural Network (CNN) to generate a 3D hand mesh directly from a single RGB image, incorporating hierarchical graph convolutions and nonlinear activations to capture detailed hand shapes and poses without requiring depth maps, enabling accurate and realistic hand representation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If generic 3D hand models and sparse joint location data are used, then device complexity is reduced, but measurement precision and manufacturing precision of hand representation deteriorate
Solution Approach 1:
The patent replaces traditional mechanical/geometric hand modeling approaches with a machine learning-based graph CNN system. Instead of using generic 3D models fitted to joint locations, the system uses a trained neural network that directly predicts detailed hand mesh vertices from a single RGB image, achieving high precision without complex depth sensors or multiple cameras.
Solution Approach 2:
The patent transforms the hand representation from sparse joint coordinates to a dense 3D mesh with hundreds of vertices. The graph CNN processes the input image and outputs precise 3D coordinates for each mesh vertex, capturing detailed hand shape, pose, and surface features that were previously impossible to obtain from a single RGB image.
2Measurement precision
If depth sensors or multiple cameras are used to capture hand data, then measurement precision improves, but device complexity and cost increase
Solution Approach 1:
The patent substitutes physical depth sensors and multiple camera systems with a computational approach using a graph CNN. The neural network, trained on synthetic and real hand data, infers detailed 3D hand geometry from a single RGB image, eliminating the need for additional hardware sensors while achieving comparable or superior precision.
Solution Approach 2:
The patent creates a detailed 3D mesh copy of the hand from a 2D RGB image. The graph CNN generates virtual 3D coordinates for hand mesh vertices that accurately represent the real hand's shape and pose, effectively creating a photorealistic 3D copy without requiring depth information from specialized sensors.
3Manufacturing precision
If detailed hand mesh with surface features is generated, then manufacturing precision improves, but processing time and computational resources increase
Solution Approach 1:
The patent performs preliminary training of the graph CNN on extensive synthetic and real hand data before deployment. This pre-training allows the network to quickly infer detailed hand meshes from new images in real-time or near-real-time, as the complex computational heavy lifting has already been done during the offline training phase.
Solution Approach 2:
The patent replaces traditional iterative optimization methods for hand pose estimation with a direct feedforward neural network inference. The graph CNN processes the input image through multiple convolutional layers and graph convolution operations to directly output 3D mesh coordinates, significantly faster than traditional methods that require iterative refinement.
Data Source
AI summary
Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing a program and a method for receiving a monocular image that includes a depiction of a hand and extracting features of the monocular image using a plurality of machine learning techniques. The program and method further include modeling, based on the extracted features, a pose of the hand depicted in the monocular image by adjusting skeletal joint positions of a three-dimensional (3D) hand mesh using a trained graph convolutional neural network (CNN); modeling, based on the extracted features, a shape of the hand in the monocular image by adjusting blend shape values of the 3D hand mesh representing surface features of the hand depicted in the monocular image using the trained graph CNN; and generating, for display, the 3D hand mesh adjusted to model the pose and shape of the hand depicted in the monocular image.