3D Human Rendering from Monocular Image via Feature Disentanglement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in capturing the full appearance and three-dimensional shape of a human from a single monocular image, limiting applications in augmented reality, virtual reality, and other fields that require accurate human rendering.
Innovation Solution
A human rendering system utilizing a neural network that disentangles 3D geometry and texture features, includes a feature module for generating feature maps, a geometry module for determining density values, and a texture module for generating RGB colors, employing ResNet-18 and multilayer perceptron networks to generate accurate 3D renderings from a single monocular image without additional views.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple images from multiple views are used to capture human appearance and 3D shape, then the completeness and accuracy of human rendering is improved, but the device complexity and data processing requirements increase
Solution Approach 1:
The system segments the human rendering task into distinct modules: a feature module that extracts appearance features, a geometry module that determines 3D shape and density, and a texture module that generates surface properties. This segmentation allows each module to specialize in one aspect (appearance or geometry) while working together to achieve complete human rendering from a single image, resolving the contradiction between rendering accuracy and system complexity.
2Loss of information
If multiple images from multiple views are used to capture human appearance and 3D shape, then the completeness of human rendering is improved, but the loss of time for data collection and processing increases
Solution Approach 1:
The feature module performs preliminary action by extracting and encoding appearance features from the single input image before the geometry and texture modules process the data. This preliminary feature extraction captures essential appearance information early in the pipeline, enabling the subsequent modules to generate complete 3D human representations without requiring additional input images, thus eliminating data collection time while preserving information completeness.
3Device complexity
If a simple rendering system is used, then the device complexity is reduced, but the manufacturing precision of 3D human rendering deteriorates
Solution Approach 1:
The system transitions from 2D image processing to 3D space by generating a volumetric density map that represents the human body in three dimensions. The geometry module creates 3D spatial representations and the texture module maps 2D appearance features onto 3D surfaces, enabling precise 3D human rendering while maintaining relatively simple system architecture through this dimensional transformation approach.
Data Source
AI summary
A system includes: a feature module configured to generate a feature map based on a single image taken from a point of view (POV) including a human based on features of the human visible in the image and non-visible features of the human; a pixel features module configured to generate pixel features based on the feature map and a target POV; a feature mesh module configured to generate a feature mesh for the human based on the feature map; a geometry module configured to: generate voxel features based on the feature mesh; and generate a density value based on the voxel and pixel features; a texture module configured to generate RGB colors for pixels based on the density value and the pixel features; and a rendering module configured to generate a three dimensional rendering of the human from the target POV based on the RGB colors and the density value.