3D Mesh Generation from Single 2D Image via Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating three-dimensional meshes in computer animation require manual generation by designers or multiple scans from various angles, making the process time-consuming and inefficient, especially for creating life-like avatars.
Innovation Solution
An apparatus and method that uses a neural network trained with synthetic data to generate a three-dimensional mesh from a single two-dimensional image, inferring the back of the object not visible in the input image, by processing raw data through a pre-processing engine and a neural network engine to create detailed joint heatmaps and surface maps, ultimately forming a complete three-dimensional mesh.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual generation of three-dimensional meshes by designers is used, then the quality and accuracy of the mesh can be ensured, but the time consumption and productivity are significantly reduced
Solution Approach 1:
The patent uses a neural network trained on synthetic three-dimensional data to automatically generate three-dimensional meshes from two-dimensional images, copying the capabilities of manual designer work into an automated system. This eliminates the need for manual mesh generation while maintaining quality through the neural network's learning from extensive training data.
Solution Approach 2:
The patent replaces the mechanical process of manual mesh generation by designers with an automated neural network system. The neural network processes two-dimensional images and generates three-dimensional meshes through computational algorithms, substituting human manual work with an automated computational mechanism.
2Manufacturing precision
If multiple scans from various angles are used to generate three-dimensional meshes, then the completeness and accuracy of the mesh can be improved, but the complexity of the process and time required increase
Solution Approach 1:
The patent transitions from requiring multiple two-dimensional scans from different angles to generating a complete three-dimensional mesh from a single two-dimensional image. The neural network infers three-dimensional structure, including occluded regions, by learning from synthetic training data that provides multi-view information during training.
Solution Approach 2:
The patent performs preliminary training of the neural network using synthetic three-dimensional data and multi-view images before actual mesh generation. This pre-training phase allows the network to learn relationships between different views and three-dimensional structure, enabling it to generate complete meshes from single images without requiring multiple scans during operation.
3Manufacturing precision
If multiple scans from various angles are used to generate three-dimensional meshes, then the accuracy of the mesh can be improved, but the time required for processing increases
Solution Approach 1:
The patent copies the information from multiple views that would normally require multiple scans into the neural network's training data. The network learns to infer three-dimensional structure from single images by having been trained on paired multi-view images and corresponding three-dimensional meshes, effectively copying multi-view information into the model's knowledge.
Solution Approach 2:
The patent replaces the mechanical process of acquiring multiple scans with an automated neural network inference process. Instead of physically scanning objects from multiple angles, the system uses a trained neural network to predict the three-dimensional mesh from a single image, dramatically reducing processing time while maintaining accuracy.
Data Source
AI summary
An apparatus is provided. The apparatus includes a communications interface to receive raw data from an external source. The raw data includes a representation of an object. Furthermore, the apparatus includes a memory storage unit to store the raw data. The apparatus also includes a pre-processing engine to generate a coarse segmentation map and a joint heatmap from the raw data. The coarse segmentation map is to outline the object and the joint heatmap is to represent a point on the object. The apparatus further includes a neural network engine to receive the raw data, the coarse segmentation map, and the joint heatmap. The neural network engine is to generate a plurality of two-dimensional maps. Also, the apparatus includes a mesh creator engine to generate a three-dimensional mesh based on the plurality of two-dimensional maps.