Text-to-Image Changer Preserving Subject Identity via Latent Diffusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text-to-image models struggle to maintain the identity of a subject in a reference image while generating a realistic output that incorporates textual prompts, and they also fail to ensure privacy of the subject in the modified image.
Innovation Solution
The proposed architecture uses a trained machine learning model to determine features of a reference image and then employs latent diffusion models to modify the image based on these features and textual inputs, allowing for changes to the background, specific areas, or style of the image, while preserving the identity of subjects and ensuring privacy compliance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If text-to-image models generate images from textual descriptions, then visual content can be created, but the identity of subjects in reference images cannot be maintained
Solution Approach 1:
The patent segments the image into different regions (subject area vs. background area) and applies different processing strategies to each. The subject area is preserved using face recognition and masking techniques, while the background is generated or modified based on text prompts. This segmentation allows the model to maintain subject identity while still generating new visual content from textual descriptions.
Solution Approach 2:
The patent introduces an intermediary masking mechanism that separates the subject from the background. By creating a mask that identifies the subject region, the model can then apply text-based generation only to the background area while keeping the subject unchanged. This intermediary mask acts as a mediator between the text prompt and the final image generation, ensuring subject identity is preserved.
2Adaptability or versatility
If text-to-image models modify reference images, then textual prompts can be incorporated, but privacy of subjects cannot be ensured
Solution Approach 1:
The patent extracts the subject from the reference image by identifying and isolating the subject region using face recognition and masking. By taking out the subject area and applying text-based modifications only to the background or non-subject regions, the model incorporates text prompts while protecting subject privacy. The extracted subject mask ensures that no text generation occurs on identifiable person features.
3Manufacturing precision
If latent diffusion models are used to modify images, then visual quality can be improved, but processing time and computational resources increase
Solution Approach 1:
The patent applies local quality by using latent diffusion models only in specific regions where modification is needed (background areas) rather than processing the entire image. The subject area is excluded from diffusion processing, reducing computational time while maintaining high visual quality in the modified regions. This localized application of diffusion modeling balances quality and processing time.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The application describes method of modifying an image. The method may include a step of receiving, via an user interface of a service, a reference image and an input including text associated with the reference image. The method may also include a step of determining, via a trained machine learning (ML) model, one or more features of the reference image. The method may further include a step of modifying, via one or more trained latent diffusion models (LDMs), the reference image based upon the determined features and the received input. Any one or more of a background of the reference image, an area of the reference image or a style of the reference image may be modified. The method may even further include a step of causing to display, via the user interface of the service, the modified image.