Controllable 3D Object Generation With Text and Image Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Designing three-dimensional models for animation or computer games is a time-consuming process that requires manual editing of properties, making it complicated to create plausible layouts and animations.
Innovation Solution
A machine-learning model generates three-dimensional models from conditional inputs such as images and text descriptions, using differentiable rendering and embedding models to update parameters based on semantic and style losses, allowing for automated editing and generation of 3D objects and animations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If manual editing of 3D model properties is used, then design control and precision are improved, but time consumption and productivity deteriorate
Solution Approach 1:
The system enables self-service by allowing the 3D model to be automatically generated and edited based on text descriptions and reference images. The machine learning model performs the design work autonomously, interpreting natural language instructions to modify model properties, place assets, and adjust parameters without requiring manual intervention for each design decision.
Solution Approach 2:
The patent replaces the mechanical manual editing process with an automated machine learning system. Instead of manually adjusting vertices, textures, and model properties, the system uses trained ML models that process text inputs and reference images to automatically generate and edit 3D models, substituting human manual operations with computational processes.
2Productivity
If automated machine learning generation is used, then productivity and time efficiency are improved, but design control and customization capability deteriorate
Solution Approach 1:
The system incorporates feedback mechanisms where the generated 3D models are evaluated against the original text descriptions and reference images. The machine learning model iteratively refines its output by comparing generated results with target properties, allowing users to provide feedback through text modifications or reference image adjustments to achieve desired design outcomes.
Solution Approach 2:
The machine learning model serves multiple functions: it can generate new 3D models from scratch, edit existing models based on text instructions, extract features from reference images, and adapt to different types of design tasks. This multi-functionality maintains design control while automating various aspects of the 3D modeling process through a single versatile system.
3Manufacturing precision
If complex manual rules for asset placement are created, then layout accuracy and plausibility are improved, but device complexity and ease of operation deteriorate
Solution Approach 1:
The patent replaces complex manual rule-based systems with machine learning models that have been trained on examples of proper asset placement and layout. Instead of programming explicit rules for furniture arrangement, lighting placement, and spatial relationships, the system uses trained neural networks that learn placement patterns from training data, automatically generating plausible layouts without requiring complex rule sets.
Solution Approach 2:
The system changes the approach from discrete rule-based decisions to continuous parameter optimization. The machine learning model adjusts multiple parameters simultaneously (asset positions, orientations, scales, lighting conditions) based on learned patterns, transforming the complex rule-based system into a parameter-driven approach that achieves layout accuracy through learned relationships rather than explicit rules.
Data Source
AI summary
Implementations disclosed herein are directed to using a machine learning model to generate three-dimensional models of objects. In some implementations, a computer implemented method can include the steps of: generating, by a machine-learning model, a three-dimensional model of an object from a three-dimensional model of an initial object and conditional input(s) specifying target properties of the generated object; generating two-dimensional image(s) of the generated object from the three-dimensional model of the generated object; generating a respective image embedding for each of the two-dimensional image(s) of the generated object using an image embedding model; generating a respective conditional embedding for each of the conditional input(s); and updating parameters of the machine-learning model based on a comparison of the respective conditional embedding for each of the conditional input(s) and the respective image embeddings for each of the two-dimensional image(s) of the generated object.