Multi-view 3D Mesh Generation via Siamese Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current 3D body mesh prediction algorithms struggle to accurately predict human poses that are not well-represented in training data, leading to subpar performance for underrepresented poses and unnatural fitting when viewed from different angles.
Innovation Solution
A neural network architecture that duplicates 3D mesh prediction models, enforcing consistency between pose and shape parameters across different viewpoints by using a Siamese Neural Network architecture, ensuring similar output for the same human from different perspectives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If model-based approaches fit 3D parametric models to minimize 2D re-projection error, then 3D pose prediction can be achieved, but the solution becomes highly under-constrained and can be minimized via non-natural joint angles
Solution Approach 1:
The patent applies feedback by using predicted 3D pose parameters to re-project back to 2D space and compare with original 2D keypoints. This creates a closed-loop constraint that prevents non-natural joint angles by continuously validating against the observed 2D data, resolving the under-constrained optimization problem
Solution Approach 2:
The patent changes the parameter space by introducing multiple view angles (different camera perspectives) as additional constraints. By optimizing 3D pose parameters across multiple viewpoints simultaneously, the solution space is sufficiently constrained to ensure natural joint angles while maintaining prediction accuracy
2Measurement precision
If 3D mesh prediction models are trained on standard datasets, then common poses are predicted well, but unusual poses underrepresented in training data show subpar performance
Solution Approach 1:
The patent makes the 3D mesh prediction model universal by training it on multi-view data from different camera angles. This multi-perspective training enables the model to generalize better to unseen poses and viewpoints, improving adaptability while maintaining accuracy on common poses through the robust multi-view constraint framework
3Device complexity
If single-view 3D mesh prediction is used, then computational complexity is lower, but the predicted mesh looks unnatural when viewed from different angles
Solution Approach 1:
The patent transitions from single-view (2D input) to multi-view (3D spatial information from multiple cameras) prediction. By incorporating multiple viewing angles as additional dimensional constraints, the model produces viewpoint-consistent 3D meshes that maintain natural appearance when rotated, resolving the viewpoint consistency issue
Data Source
AI summary
Devices and techniques are generally described for three dimensional mesh generation. In various examples, first two-dimensional (2D) image data representing a human body may be received from a first image sensor. Second 2D image data representing the human body may be received from a second image sensor. A first pose parameter and a first shape parameter may be determined using a first three-dimensional (3D) mesh prediction model and the first 2D image data. A second pose parameter and a second shape parameter may be determined using a second 3D mesh prediction model and the second 2D image data. In various examples, an updated 3D mesh prediction model may be generated from the first 3D mesh prediction model based at least in part on a first difference between the first pose parameter and the second pose parameter and a second difference between the first shape parameter and the second shape parameter.