Multiview Neural Human Prediction Using Implicit Differentiable Renderer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems in the entertainment industry struggle to predict 3D human models, including skeleton, body shape, and clothes appearance from multiview images, as they either fail to handle RGB inputs, predict clothes, or result in low-resolution human geometry and coarse geometry without mesh vertex correspondences.
Innovation Solution
A neural network architecture comprising a multiview stereo 3D convolutional neural network (MVS-3DCNN), a human mesh recovery multilayer perceptron (HMR MLP), and a neural radiance field multilayer perceptron (NeRF MLP) that processes multiview images to predict a layered 3D human model, including skeleton, body shape, and clothes displacement and appearance, generating a clothed body mesh with vertex correspondence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If implicit part network is used to predict both body and clothes from scanned point cloud, then clothes surface prediction is achieved, but 3D scan input is required and RGB images cannot be processed
Solution Approach 1:
The system changes the input parameter format from requiring 3D scan data to accepting standard RGB images. The neural network is designed to process 2D image inputs and automatically infer 3D human model parameters including body shape, pose, and clothes appearance, eliminating the need for specialized 3D scanning equipment.
Solution Approach 2:
The patent replaces the mechanical 3D scanning system with a computational approach using neural networks. Instead of physically scanning the subject with specialized equipment, the system uses multiple RGB cameras and neural rendering to reconstruct the 3D human model, substituting physical measurement with computational inference.
2Manufacturing precision
If neural radiance field is used to predict clothes and body, then appearance prediction is achieved, but dense latent code volume creation is required which limits resolution
Solution Approach 1:
The system segments the human model prediction into distinct components: body shape prediction, pose estimation, and clothes appearance prediction. Each component is processed separately through dedicated neural network modules, allowing high-resolution geometry prediction without requiring a single dense latent code volume, thus improving resolution while managing complexity.
Solution Approach 2:
The patent transitions from creating a dense 3D latent code volume to using a multi-view 2D image-based approach. By processing images from multiple camera views and synthesizing 3D information computationally, the system achieves high-resolution geometry prediction without the memory and computational constraints of dense volumetric representations.
3Measurement precision
If explicit numerical optimization is used for model fitting, then accurate human parameters are obtained, but processing time increases
Solution Approach 1:
The patent replaces explicit numerical optimization with a neural network-based direct regression approach. The neural network is trained to directly predict human parameters (body shape, pose, facial expression) from multiview images, substituting iterative optimization algorithms with a single-pass neural inference that achieves comparable accuracy while being significantly faster.
Solution Approach 2:
The system performs preliminary training offline where the neural network learns the complex mapping from images to human parameters. During actual processing, the pre-trained network directly outputs predictions without requiring runtime optimization, having already performed the computationally intensive learning phase in advance.
4Reliability
If single image input is used for prediction, then processing speed is maintained, but prediction quality and robustness decrease
Solution Approach 1:
The patent merges information from multiple camera views into a unified 3D human model prediction. The neural network processes images from multiple simultaneous camera positions and combines the observations to produce a more robust and accurate prediction, leveraging the complementary information from different viewing angles to improve reliability.
Solution Approach 2:
The system creates a universal prediction framework that can handle both single-image and multiview inputs through the same neural network architecture. The network is designed to accept variable numbers of input images and adaptively process them, providing improved robustness when multiple views are available while maintaining functionality with single image input.
Data Source
AI summary
A neural human performance capture framework (MVS-PERF) captures the skeleton, body shape and clothes displacement, and appearance of a person from a set of calibrated multiview images. It addresses the ambiguity of predicting the absolute position in monocular human mesh recovery, and bridges the volumetric representation from NeRF to animation-friendly performance capture. MVS-PERF includes three modules to extract feature maps from multiview images and fuse them to a feature volume, regress the feature volume to a naked human parameters vector, generating an SMPL-X skin-tight body mesh with a skeletal pose, body shape, and expression, and leverage a neural radiance field and a deformation field to infer the clothes as the displacement on the naked body using differentiable rendering. Clothed body mesh is obtained by adding the interpolated displacement vectors to the SMPL-X skin-tight body mesh vertices. The obtained radiance field is used for free-view volumetric rendering of the input subject.