Text-to-3D Generation Using Pseudo Multi-View Model Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative machine-learned models, particularly Text-to-Image models, suffer from overfitting of viewpoint, leading to poor performance when generating viewpoints of subjects not present in the input images, limiting their accuracy and practical applications.
Innovation Solution
A fractional training process is applied to a machine-learned generative image model, followed by a fractional optimization of a 3D implicit representation model, using pseudo multi-view subject images to optimize the model for generating accurate 3D assets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional T2I models are used for 3D generation, then fast and efficient asset creation is achieved, but viewpoint overfitting occurs leading to poor performance on novel viewpoints
Solution Approach 1:
The patent segments the generation process into two independent stages: (1) a T2I model generates 2D images from text prompts, and (2) a 3D implicit representation model converts these 2D images into 3D assets. This segmentation allows each model to specialize in its specific task, with the T2I model handling text-to-image conversion and the 3D model handling image-to-3D conversion, thereby avoiding viewpoint overfitting while maintaining efficiency
Solution Approach 2:
The patent introduces 2D images as an intermediary representation between the textual prompt and the final 3D asset. The T2I model first generates intermediate 2D images from the text prompt, which then serve as input to the 3D implicit representation model. This intermediary 2D representation acts as a bridge that decouples the text understanding from the 3D generation, allowing the system to leverage pre-trained T2I models while achieving accurate 3D output
2Quantity of substance
If models are trained on limited input images, then training data requirements are reduced, but generalization to novel viewpoints deteriorates
Solution Approach 1:
The patent performs preliminary action by using a pre-trained T2I model to generate multiple 2D images from a single text prompt before feeding them into the 3D implicit representation model. This preliminary generation of multiple views from text allows the 3D model to learn diverse viewpoints without requiring actual multi-view training images, thereby improving viewpoint generalization while reducing training data requirements
Solution Approach 2:
The patent uses the T2I model to create synthetic 2D image copies of the 3D scene from different viewpoints. These synthetic 2D copies serve as training data for the 3D implicit representation model, allowing the system to learn from generated rather than real images. This copying approach enables the model to infer 3D structure from 2D projections without needing actual multi-view images of the target object
Data Source
AI summary
A fractional training process can be performed training images to an instance of a machine-learned generative image model to obtain a partially trained instance of the model. A fractional optimization process can be performed with the partially trained instance to an instance of a machine-learned three-dimensional (3D) implicit representation model obtain a partially optimized instance of the model. Based on the plurality of training images, pseudo multi-view subject images can be generated with the partially optimized instance of the 3D implicit representation model and a fully trained instance of the generative image model; The partially trained instance of the model can be trained with a set of training data. The partially optimized instance of the machine-learned 3D implicit representation model can be trained with the machine-learned multi-view image model.