3D Human Mesh Prediction via Synthetic Feedback and Loss Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D body mesh prediction algorithms and person detection models perform poorly when only a partial body, especially the upper body, is visible, due to a lack of relevant training data and insufficient constraints on invisible joints and limb orientations, leading to inaccurate pose and shape predictions.
Innovation Solution
The introduction of data augmentation techniques to generate partial body training samples and the implementation of two new loss terms, invisibility loss and limb orientation loss, to improve the performance of 3D human mesh prediction models by enforcing visibility constraints and correct limb orientations, even in difficult partial body view conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If model-based approaches fit 3D parametric models to estimate 3D shape and pose by minimizing 2D re-projection error, then 3D pose prediction can be achieved, but the solution is highly under-constrained and can be minimized via non-natural joint angles
Solution Approach 1:
The patent applies feedback by using the predicted 3D mesh to generate synthetic 2D detections and comparing them with actual 2D detections. This feedback loop refines the 3D pose estimation by minimizing the difference between synthetic and actual detections, ensuring natural joint angles while maintaining prediction accuracy. The system iteratively adjusts the 3D pose based on the error signal from detection comparison.
Solution Approach 2:
The patent introduces an intermediary object detection model as a mediator between the 3D pose prediction and the final 3D mesh generation. This intermediary translates 3D mesh predictions into 2D detection formats, enabling comparison with actual detections and providing a pathway to refine 3D pose estimates while constraining joint angles to natural values.
2Measurement precision
If additional 3D supervision including paired 2D-3D correspondences, unpaired 3D data, and synthetic data is used, then 2D to 3D prediction accuracy improves, but the complexity and data requirements increase
Solution Approach 1:
The patent uses copying by generating synthetic 2D detections from predicted 3D meshes as virtual training data. This synthetic data serves as a copy of what actual detections would look like, enabling the system to learn from unlimited synthetic examples without requiring additional real-world annotated data, thereby improving accuracy while avoiding the complexity of collecting and processing diverse real 3D supervision data.
Solution Approach 2:
The system performs self-service by generating its own training data through synthetic 2D detection generation from predicted 3D meshes. Instead of relying on external paired 2D-3D correspondences or unpaired 3D data, the system creates its own supervision signals from its predictions, enabling continuous self-improvement without external data dependencies.
3Ease of manufacture
If standard training data with full body views is used, then model training is straightforward, but performance deteriorates when only partial body, especially upper body, is visible
Solution Approach 1:
The patent applies dynamics by making the training data composition adaptive rather than static. The system dynamically adjusts the mix of full-body and upper-body-only samples in training based on performance needs. During training, the system can emphasize upper-body samples when improving partial view performance is the goal, while maintaining simplicity by using the same base dataset with different cropping configurations.
Solution Approach 2:
The patent implements preliminary action by pre-processing training images to create upper-body-only cropped versions before training begins. This preliminary data preparation ensures that when the model needs to handle partial body views, the training data is already optimized for this purpose, eliminating the need for complex runtime adjustments while maintaining training simplicity.
Data Source
AI summary
Devices and techniques are generally described for three dimensional mesh generation. In various examples, first two-dimensional (2D) image data representing a human may be received. In various further examples, bounding box data identifying a location of the human in the first 2D image data and joint data identifying locations of joints of the human may be received. Second 2D image data representing a cropped portion of the human may be generated using the bounding box data and the joint data. A three-dimensional (3D) mesh prediction model may be used to determine a pose, a shape, and a projection matrix for the human. The 3D mesh prediction model may be used to determine a transformed projection matrix for the portion of the human represented in the second 2D image data.