3D Human Mesh Reconstruction With Perspective Pose Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for reconstructing 3D human meshes from single images face challenges due to non-linear mappings and assumptions of weak perspective cameras, leading to misaligned meshes in 3D space or image space, and lack of incorporating environmental constraints.
Innovation Solution
A method using a Pseudo-Linear-Inverse-Kinematic-Solver (PLIKS) approach that incorporates a perspective camera, reformulates the SMPL model as a linear system of equations, and utilizes a differentiable end-to-end training process to estimate 3D human shape and pose, incorporating depth and environmental constraints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a weak perspective camera model is used with a large focal length constant, then the camera model simplification is achieved, but the mesh alignment accuracy deteriorates
Solution Approach 1:
The patent changes the camera model parameters from a simplified weak perspective model with fixed large focal length to a full perspective camera model with accurate intrinsic parameters. This parameter change enables proper 3D to 2D projection geometry while maintaining computational tractability through analytical solutions.
Solution Approach 2:
The patent introduces an intermediary analytical solver that bridges the gap between the full perspective camera model and the neural network regression. This intermediary component computes accurate 3D to 2D projections using the perspective model, allowing the network to learn from geometrically correct projections without directly optimizing the complex camera model.
2Productivity
If neural networks directly regress shape and pose parameters from input images, then the approach is computationally efficient, but the mesh alignment in 3D space deteriorates
Solution Approach 1:
The patent introduces an intermediary analytical solver that computes accurate 3D to 2D projections using the perspective camera model. This intermediary component allows the neural network to regress parameters efficiently while the solver ensures geometric accuracy in the projection process, resolving the contradiction between computational efficiency and alignment precision.
Solution Approach 2:
The patent performs preliminary computation of accurate 3D to 2D projections using the analytical perspective camera model before the neural network regression. By pre-computing the geometric transformations, the network can focus on learning appearance and pose variations without struggling with the complex 3D to 2D projection geometry during optimization.
3Measurement precision
If non-parametric approaches directly regress mesh vertex coordinates with 2D projections, then the image alignment is improved, but the depth estimation realism deteriorates
Solution Approach 1:
The patent changes the regression target from direct mesh vertex coordinates to parametric shape and pose parameters. This parameter change allows the use of a realistic parametric human body model that enforces anatomical constraints, ensuring realistic depth and pose estimation while maintaining accurate image alignment through the analytical perspective camera model.
Solution Approach 2:
The patent introduces an intermediary analytical solver that computes accurate 3D to 2D projections using the perspective camera model. This intermediary ensures that the regressed parameters produce meshes that align accurately with the input image while maintaining realistic 3D geometry and depth relationships.
Data Source
AI summary
A 3D mesh of a person shall be reconstructed based on one single 2D image. For this purpose, 2D vertex projections and 3D vertex projections of the person are predicted based on the single 2D image. An approximated pose is estimated from the 3D vertex projections. The shape and/or pose, i.e. the 3D mesh of the person, is computed from the predicted 2D vertex projections and from the approximated pose by using a pregiven camera model and an articulated 3D mesh model of a human body.