Textured 3D Model Generation Using Voxel Latent Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Producing 3D models with texture is difficult due to high computational requirements and the challenge of training generative AI systems using irregular mesh representations, and generating 3D models from 2D images and textual descriptions is complex.
Innovation Solution
Using a regular voxel grid representation with signed distance fields for geometry and voxel colors for texture, encoding into a lower-dimensional latent space, and decoding back into a higher-dimensional representation with smaller neural networks, trained using a diffusion process with reference tuples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If irregular mesh representation is used for 3D models, then the model can represent complex geometries, but the computational requirements for training and generation become prohibitively high
Solution Approach 1:
The patent transforms the representation parameters of 3D models from irregular mesh structures to regular voxel grid structures. This parameter change simplifies the computational operations during training and generation while maintaining the ability to represent complex geometries through the voxel grid's spatial arrangement and signed distance field values.
Solution Approach 2:
The patent uses a regular voxel grid as a simplified copy or approximation of the irregular mesh representation. Instead of directly processing complex mesh structures, the system creates a regularized voxel-based copy that captures the essential geometric and textural information, making subsequent processing more computationally efficient.
2Manufacturing precision
If high-dimensional 3D model representations are used, then the models capture detailed information, but larger neural networks are required increasing training difficulty
Solution Approach 1:
The patent introduces a latent space dimension that compresses the high-dimensional voxel grid representation into a lower-dimensional embedding space. This dimensionality transformation allows the model to capture essential features in a compressed form, reducing the network size required while preserving the ability to generate detailed 3D models when needed.
Solution Approach 2:
The patent segments the 3D model representation into distinct components: geometric structure (voxel grid with signed distance fields) and textural information (voxel colors). This segmentation allows each component to be processed efficiently with specialized neural network pathways, reducing the overall network complexity compared to processing the complete high-dimensional representation as a single unit.
3Productivity
If regular voxel grid representation is used, then computational requirements are reduced, but the ability to represent irregular 3D geometries may be limited
Solution Approach 1:
The patent employs dynamic neural rendering techniques that can adapt the static voxel grid representation to produce diverse 3D geometries. The signed distance field values in the voxel grid are processed by neural networks that can dynamically generate different surface configurations, allowing the regular grid to represent irregular geometries through learned transformations rather than fixed structural constraints.
Data Source
AI summary
An artificial intelligence (AI) network or neural network is trained to generate three-dimensional (3D) models or shapes with color from two-dimensional (2D) input images and input text describing the 3D model with color. Example methods include converting a first three-dimensional (3D) model from a first representation to a second representation, the second representation including color information for the 3D model and inputting the second representation into an encoder to generate a third representation having a lower dimension than the second representation. The method further includes inputting the third representation into a decoder to generate a fourth representation having a same dimension as the second representation and generating a second 3D model from the fourth representation. The method further includes determining losses between the first 3D model and the second 3D model and updating weights of the encoder and the decoder based on the losses.