3D Human Model Recovery Using Synthetic Data and Iterative Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning based techniques for human model recovery require annotated 3D training data, which are difficult to obtain, and lack accuracy when parts of the human body are covered or invisible.
Innovation Solution
The system uses multiple neural networks to determine body keypoints and pose/shape parameters from 2D and depth images, generating a 3D human model without relying on 3D annotations, and optimizes the model through iterative adjustments based on differences between inferred and original keypoints and normal maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If annotated 3D training data is used for deep learning based human model recovery, then the model can be trained to recover 3D human models, but obtaining such annotated 3D training data is very difficult and time-consuming
Solution Approach 1:
The patent uses synthetic 3D human models generated from SMPL parameters as virtual copies of real human bodies. These synthetic models are rendered to create training images with automatically generated 3D annotations (keypoints, pose parameters, shape parameters) without requiring manual annotation of real 3D data. This copying approach eliminates the time-consuming process of obtaining and annotating real 3D training data while providing sufficient training data for accurate model recovery
Solution Approach 2:
The system uses the SMPL parametric human model itself to generate its own training data. By sampling SMPL parameters and rendering the resulting 3D models, the system automatically creates training datasets with ground truth annotations without external intervention. The model essentially trains itself by generating its own training examples from the parametric model space
2Ease of operation
If existing deep learning techniques are used for human model recovery, then the system can process images, but accuracy deteriorates when parts of the human body are covered or invisible
Solution Approach 1:
The patent implements an iterative optimization process where the predicted 3D human model is rendered back to 2D, compared with the input image, and the loss is used to update the model parameters. This feedback loop allows the system to progressively refine the 3D model even when parts of the body are occluded, as the rendering-comparison-optimization cycle continuously improves accuracy by minimizing the difference between rendered and observed images
Solution Approach 2:
The system transitions from 2D image analysis to 3D model prediction and back to 2D rendering for comparison. By working in the 3D parameter space (SMPL parameters) and only comparing 2D projections, the system can infer 3D structure and pose even when 2D visual information is partially occluded, as the 3D model provides structural constraints that help recover hidden body parts
3Adaptability or versatility
If multiple neural networks are used to determine keypoints and parameters from 2D and depth images, then the system can generate 3D human models without 3D annotations, but the device complexity increases
Solution Approach 1:
The patent divides the complex task of 3D human model recovery into separate specialized neural networks: one for keypoint detection, one for pose parameter estimation, one for shape parameter estimation, and one for occlusion handling. Each network focuses on a specific aspect of the problem, making the overall system more manageable and trainably complex while achieving accurate 3D model recovery from 2D images without requiring 3D annotations
Data Source
AI summary
Human model recovery may be realized utilizing pre-trained artificially neural networks. A first neural network may be trained to determine body keypoints of a person based on image(s) of a person. A second neural network may be trained to predict pose parameters associated with the person based on the body keypoints. A third neural network may be trained to predict shape parameters associated with the person based on depth image(s) of the person. A 3D human model may then be generated based on the pose and shape parameters respectively predicted by the second and third neural networks. The training of the second neural network may be conducted using synthetically generated body keypoints and the training of the third neural network may be conducted using normal maps. The pose and shape parameters predicted by the second and third neural networks may be further optimized through an iterative optimization process.