Diffusion Image Restyling With Depth Maps and Human-Preserving Masks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI models struggle to accurately represent intricate human features and preserve depth information during image modification, often resulting in unrealistic or undesirable alterations, particularly when human subjects are involved.
Innovation Solution
A diffusion model is trained with a textual request, depth map, and user-selected mask to generate output images that modify specific objects while preserving depth and avoiding human subjects, using classifier-free guidance to enhance image quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If generative AI models are used to modify images with human subjects, then image generation capability is improved, but accuracy of human feature representation deteriorates
Solution Approach 1:
The patent segments the image into different regions using masks (user-selected mask, preserving mask, segmentation mask) to distinguish human subjects from non-human objects. This allows the diffusion model to apply different processing rules to different segments - modifying non-human objects while preserving human features, thereby resolving the contradiction between image generation capability and human feature representation accuracy.
Solution Approach 2:
The patent applies local quality by using depth maps and masks to differentiate between human and non-human regions, applying selective modification only to non-human objects while maintaining original quality for human subjects. This localized approach enables the model to improve overall image generation capability without compromising human feature accuracy.
2Adaptability or versatility
If diffusion model generates modified images, then creative flexibility is improved, but depth information preservation deteriorates
Solution Approach 1:
The patent performs preliminary action by generating a depth map from the initial image before applying modifications. This pre-computed depth information is then used as a constraint during the diffusion process to ensure that generated objects maintain appropriate depth relationships with existing objects, thus preserving depth structure while enabling creative flexibility.
Solution Approach 2:
The depth map acts as an intermediary between the creative modification process and the final image output. It mediates the conflict between creative flexibility and depth preservation by providing depth constraints that guide the diffusion model to generate modifications that respect the original image's depth structure.
3Adaptability or versatility
If diffusion model modifies selected objects, then object transformation capability is improved, but generation of hallucinations and artifacts increases
Solution Approach 1:
The patent implements feedback mechanisms through multiple masks (user-selected mask, preserving mask, segmentation mask) that continuously guide the diffusion process. These masks provide feedback about which regions should be modified and which should be preserved, enabling the model to achieve accurate object transformation while minimizing hallucinations and artifacts through iterative refinement.
Solution Approach 2:
The patent applies partial action by using masks to limit modifications only to specific non-human objects rather than the entire image. This selective approach reduces the risk of hallucinations and artifacts by confines the diffusion process to relevant regions only, improving reliability while maintaining object transformation capability.
4Manufacturing precision
If depth map is preserved during modification, then structural accuracy is improved, but modification freedom is reduced
Solution Approach 1:
The patent segments the modification process into two independent parts: depth preservation (through depth maps and masks) and object transformation (through diffusion). This segmentation allows the model to maintain structural accuracy by preserving depth information while simultaneously enabling modification freedom in the diffusion process for non-human objects.
Solution Approach 2:
The patent uses parameter changes by conditioning the diffusion model on depth maps and masks as additional inputs. These parameters guide the diffusion process to generate modifications that respect the original depth structure, thereby achieving both structural accuracy and modification freedom through parameter-based constraints.
Data Source
AI summary
A media application receives an initial image, user input that selects one or more objects in the initial image, and a textual request to generate an output image that modifies the one or more selected objects in the initial image. The media application generates a user-selected mask that includes object pixels corresponding to the one or more selected objects. A diffusion model receives the textual request to generate the output image, a depth map, and the user-selected mask, where the diffusion model is trained to generate output pixels that are not associated with a human subject. The diffusion model outputs the output image that satisfies the textual request.