3D Object Shading With Self-Occlusion for Consistent Image Lighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to accurately harmonize foreground objects with backgrounds in digital images, particularly when the object surfaces have complex shapes, due to the lack of consideration of geometries and inconsistent lighting, leading to visual artifacts and inefficiencies in image synthesis.
Innovation Solution
The digital object shading system utilizes a light encoder neural network to generate a light representation embedding and a self-occlusion map, combined with a generator neural network to determine a shading map, ensuring consistent lighting for inserted three-dimensional objects by accounting for self-occlusions and background lighting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing systems utilize regression loss in image color space and perceptual feature space to improve synthesized results, then the quality of synthesized images is improved, but the systems are limited to operating under paired image settings and lack flexibility
Solution Approach 1:
The patent changes the parameter space from requiring paired images to working with unpaired images by introducing a different loss function (L2 loss in image space) that does not require corresponding target images. This allows the system to operate in both paired and unpaired settings, improving flexibility while maintaining synthesis quality.
2Adaptability or versatility
If existing systems add contrastive loss to guide encoder encoding between domains, then the systems attempt to overcome unpaired image limitations, but visual artifacts are frequently added into the generated results
Solution Approach 1:
The patent extracts and removes the contrastive loss component that causes visual artifacts, instead using a simpler L2 loss function that operates directly on image pixel values. This extraction eliminates the artifact problem while maintaining the ability to handle unpaired images.
3Ease of operation
If existing systems harmonize images by obtaining global information of the background from a neural network, then the systems attempt to improve image harmonization, but the systems fail to utilize information about geometries of foreground objects leading to inaccurate harmonization when object surfaces have complex shapes
Solution Approach 1:
The patent applies local quality by processing different parts of the image with different levels of detail. It uses a generator network that takes both global background information and local foreground object geometry information (through edge maps and depth maps) to produce harmonized results that accurately represent complex object surfaces.
Solution Approach 2:
The patent adds another dimension to the harmonization process by incorporating geometric information (edge maps, depth maps) as additional input channels to the generator network. This multi-dimensional approach allows the system to capture both appearance and geometric properties of foreground objects, improving accuracy for complex shapes.
Data Source
AI summary
This disclosure describes methods, non-transitory computer readable storage media, and systems that generate realistic shading for three-dimensional objects inserted into digital images. The disclosed system utilizes a light encoder neural network to generate a representation embedding of lighting in a digital image. Additionally, the disclosed system determines points of the three-dimensional object visible within a camera view. The disclosed system generates a self-occlusion map for the digital three-dimensional object by determining whether fixed sets of rays uniformly sampled from the points intersects with the digital three-dimensional object. The disclosed system utilizes a generator neural network to determine a shading map for the digital three-dimensional object based on the representation embedding of lighting in the digital image and the self-occlusion map. Additionally, the disclosed system generates a modified digital image with the three-dimensional object inserted into the digital image with consistent lighting of the three-dimensional object and the digital image.