3D Model Reconstruction from Single Images Using Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for 3D face reconstruction, such as the Morphable Model framework, are limited in representing fine details and geometry, and struggle with aligning texture and shape for neural network training, especially when reconstructing full human busts or diverse objects, leading to poor results.
Innovation Solution
A neural network system that maps 3D models to pairs of images representing shape and texture, using a reference surface of a cylinder and half-sphere for projection, and employs deformable or dilated convolutions to align features, enabling the generation of 3D models from single images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If Morphable Model framework is used for 3D face reconstruction, then the method can handle diverse appearances and geometry, but the representational power is limited and fine details in geometry and texture cannot be described accurately
Solution Approach 1:
The patent changes the parameterization approach by using a neural network to directly regress 3D model parameters from 2D images, replacing the traditional Morphable Model framework. This allows the system to capture fine geometric and texture details through learned parameters while maintaining versatility across diverse appearances through the neural network's ability to generalize from training data.
2Measurement precision
If traditional parameterization methods are used to represent 3D models as images, then texture can be represented in 2D domain with minimal distortions, but shape is represented as 3D mesh which is poorly suited for neural networks
Solution Approach 1:
The patent merges the representation of shape and texture into a unified 2D image domain by projecting the 3D model onto a reference surface. This creates a single 2D representation that contains both shape and texture information, making it suitable for neural network processing while maintaining the accuracy of both components through the projection mapping.
Solution Approach 2:
The patent transforms the 3D model representation into a 2D image domain by projecting the 3D geometry and texture onto a 2D reference surface. This dimensional change enables the use of convolutional neural networks which are optimized for 2D data, while the projection preserves the essential geometric and textural information needed for accurate reconstruction.
3Productivity
If standard neural network architectures are used for 3D model reconstruction, then the system can process images efficiently, but the alignment between texture and shape images is poor leading to inaccurate results
Solution Approach 1:
The patent introduces a reference surface as an intermediary that serves as a common coordinate system for both shape and texture representations. The projection mapping from the 3D model to this reference surface ensures that corresponding points on the shape and texture images are properly aligned, enabling standard neural networks to process the data accurately without requiring complex alignment procedures.
Data Source
AI summary
The present invention provides a method to produce a 3D model of a person or an object from just one or several image. The method uses a neural network that is trained on pairs of 3D models of human heads and their frontal images, and then, given an image, infers a 3D model.