Stable Diffusion Face Preservation in Image-to-Image Stylization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing image processing systems face challenges in maintaining face attributes while stylizing input images, particularly in augmented reality applications, due to computational intensity and the need for efficient processing of images under varying conditions.

Innovation Solution

The use of a stable diffusion generative model with neural networks to modify intermediate features by combining weighted values from segmentation networks, allowing for face attribute preservation and efficient image processing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If image stylization is performed using traditional image processing systems, then image style transformation is achieved, but face attributes are lost and computational load increases

Engineering Contradiction:

Improveimage style transformationVSAvoidface attribute preservation

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The system segments the input image into different regions using a segmentation network, specifically identifying face regions and non-face regions. This segmentation allows the model to apply different processing strategies to different parts of the image, preserving face attributes while transforming the overall style. The segmentation mask is used to guide the feature blending process, ensuring that face-related features are protected during stylization.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system applies local quality control by treating face regions differently from non-face regions. Face-related features are extracted and preserved through a dedicated face preservation module, while non-face regions undergo full stylization. This local differentiation ensures that critical face attributes maintain their original quality while the rest of the image achieves the desired style transformation.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If traditional image processing methods are used for stylization, then style transformation is achieved, but computational intensity increases

Engineering Contradiction:

Improveimage stylization capabilityVSAvoidcomputational load

Core Design Contradiction:

Adaptability or versatilityVSUse of energy by moving object

Solution Approach 1:

The system performs preliminary actions by pre-processing the input image through segmentation and initial feature extraction before the main stylization process. The segmentation network and feature extraction networks prepare the image data in advance, organizing it into structured representations that facilitate more efficient processing during the actual stylization phase. This preliminary organization reduces the computational complexity of subsequent operations.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system introduces intermediary components including a segmentation network, feature extraction networks, and a face preservation module that act as mediators between the input image and the final stylized output. These intermediary layers process and transform the image data in staged steps, allowing complex stylization to be achieved through multiple simpler transformations rather than a single computationally intensive operation.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If face attributes are preserved during stylization, then identity maintenance is improved, but processing complexity increases

Engineering Contradiction:

Improvefacial identity preservationVSAvoidmodel architecture complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The system merges multiple specialized networks (segmentation network, feature extraction networks, face preservation module) into a unified stable diffusion model framework. This integration allows the complex face preservation functionality to be combined with the stylization process in a cohesive architecture, where the various components work together synergistically rather than as separate sequential steps, thereby managing overall system complexity.

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The stable diffusion model serves multiple functions simultaneously: it performs image stylization, maintains face attribute preservation, and integrates segmentation guidance. This multi-functionality is achieved within a single unified model framework, reducing the need for multiple separate processing systems and thereby managing complexity through consolidation of diverse functionalities into one versatile architecture.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20250384526A1Face identity preservation for image-to-image models using stable diffusion generative model

Publication Date: 2025.12.18 SNAP INC

AI summary

The subject technology receives an input image and a segmentation mask of the input image. The subject technology obtains reconstructed noise of the input image using the input image and the segmentation mask. The subject technology determines a first set of features by performing a first portion of a forward pass of the reconstructed noise through a decoder. The subject technology determines a second set of features by processing the input image for stable diffusion using an image to image (IMG2IMG) model. The subject technology generates a third set of features based on combining, using the segmentation mask, the first set of features and the second set of features with the reconstructed noise. The subject technology generates an output image by performing a remaining portion of the forward pass of the third set of features through the decoder.