3D Scene Image Generation with Gaussian Splatting Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-3D generation models struggle to generate complex scenes with multiple objects and lack object-level control during scene generation, leading to inefficiencies in training and testing datasets for machine learning systems.
Innovation Solution
A method and device that utilize text prompts to generate synthetic digital images of three-dimensional scenes by determining bounding boxes, assembling scenes with three-dimensional models, and applying Gaussian Splatting and stable diffusion techniques to render images with object-level control, allowing for training and testing of machine learning systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing text-to-3D generation models are used, then scene generation can be performed, but the models struggle to generate complex scenes with multiple objects and lack object-level control
Solution Approach 1:
The patent segments the scene generation process into distinct stages: text prompt interpretation, 2D bounding box generation, 3D bounding box conversion, object assembly, Gaussian Splatting representation, and stable diffusion rendering. This segmentation enables independent control at each stage, particularly allowing object-level manipulation during the assembly phase while maintaining the ability to generate complex multi-object scenes.
Solution Approach 2:
The patent transitions from 2D bounding boxes (generated from text prompts) to 3D bounding boxes by introducing depth information and spatial relationships. This dimensional transformation enables the system to interpret 2D spatial descriptions and convert them into accurate 3D object placements, thereby enhancing the capability to generate complex three-dimensional scenes with proper object positioning and orientations.
2Quantity of substance
If complex scenes with multiple objects are generated, then scene complexity increases, but control over individual objects during generation becomes difficult
Solution Approach 1:
The patent performs preliminary actions by first generating 2D bounding boxes from text prompts, then converting these to 3D bounding boxes before assembling objects. This preliminary structuring of spatial information allows for precise control over object placement and orientation even when multiple objects are present, as each object's position and orientation are determined before final assembly.
Solution Approach 2:
The patent introduces 3D bounding boxes as an intermediary representation between text prompts and final object assembly. These bounding boxes serve as mediators that carry spatial information (position, orientation, dimensions) from the textual description to the 3D object placement, enabling precise control over multiple objects during the assembly process.
3Manufacturing precision
If object-level control is implemented during scene generation, then precise object placement is achieved, but the generation process becomes more complex
Solution Approach 1:
The patent merges multiple functions into the assembly process: 3D bounding box interpretation, object retrieval from database, object placement, and orientation alignment are all performed during a single assembly operation. This merging reduces the overall process complexity while maintaining precise object-level control, as these functions work together in an integrated manner rather than as separate sequential steps.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
A device and a computer implemented method for generating a synthetic digital image (800) of a three-dimensional scene, in particular for a dataset for training and/or testing of a machine learning system, wherein the method comprises providing at least one text prompt, wherein the at least one text prompt comprises a description of a three-dimensional layout of the scene, wherein the at least one text prompt comprises a description of a style of the scene, generating the layout depending on the description of the layout, assembling the scene depending on the layout, determining a three-dimensional Gaussian Splatting representation of the assembled scene depending on the assembled scene, rendering a digital image from the three-dimensional Gaussian Splatting representation, and determining the synthetic digital image (800) with a stable diffusion depending on the digital image and the description of the style.