3D Model Style Transfer via Iterative Attribute Map Modification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image style transfer techniques are limited to two-dimensional models, failing to effectively apply the look and feel of reference images to three-dimensional models, which restricts their applications.
Innovation Solution
A computer-implemented method and system that uses machine-learned models, such as convolutional neural networks, to iteratively modify attribute rendering maps of three-dimensional models based on reference images, calculating style and content losses to generate consistent attribute rendering maps that mimic the style of reference images while preserving content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If image style transfer is applied to two-dimensional images using existing systems, then the look and feel of reference images can be successfully transferred, but the technique cannot be applied to three-dimensional models
Solution Approach 1:
The patent extends image style transfer from two-dimensional images to three-dimensional models by introducing a volumetric dimension. The system renders multiple two-dimensional views of the 3D model from different angles and applies style transfer to each view, then synthesizes the styled views back into a coherent three-dimensional model. This dimensional extension allows the technique to work with 3D objects while maintaining the proven 2D style transfer quality.
Solution Approach 2:
The patent segments the three-dimensional model into multiple two-dimensional projections or views that can be independently processed. Each view is treated as a separate 2D image for style transfer application, allowing the use of existing 2D style transfer algorithms. The segmented views are then reassembled to form the complete styled 3D model, ensuring consistent style application across all surfaces.
2Ease of manufacture
If style transfer is applied to three-dimensional models using existing two-dimensional techniques, then the process can be simplified, but the results exhibit seams and inconsistencies
Solution Approach 1:
The patent implements a feedback mechanism where the styled three-dimensional model is continuously evaluated for consistency across different views and lighting conditions. The system detects seams and inconsistencies in the rendered output and uses this feedback to iteratively adjust the style transfer parameters and rendering settings. This closed-loop approach maintains texture consistency while preserving the relative simplicity of the style transfer process.
Solution Approach 2:
The patent employs dynamic adjustment of rendering parameters and style transfer settings based on the specific characteristics of the three-dimensional model and reference images. The system adaptively modifies view angles, lighting configurations, and style transfer intensity to prevent seam formation and ensure consistent texture application across the 3D model's surfaces, balancing process simplicity with high precision results.
Data Source
AI summary
Example aspects of the present disclosure are directed to systems and methods that perform image style transfer for three-dimensional models. In some implementations, the systems and methods can use machine-learned models such as, for example, convolutional neural networks to generate image style and content information used to perform style transfer. The systems and methods of the present disclosure can operate in a rendered image space. In particular, a computing system can iteratively modify an attribute rendering map (e.g., texture map, bump map, etc.) based on information collected from a different rendering of the model at each of a plurality of iterations, with the end result being that the attribute rendering map mimics the style of one or more reference images in content-preserving way. In some implementations, a computation of style loss at each iteration can be performed using multi-viewpoint averaged scene statistics, instead of treating each viewpoint independently.