Neural Image Stylization With Pose and Likeness Preservation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image generation neural networks struggle to adapt the style of a given image while preserving the likeness and appearance of the agent depicted, often modifying the agent's likeness when changing the style or failing to consistently apply the target style.
Innovation Solution
Conditioning an image generation neural network with both a likeness embedding and a pose input to maintain the appearance of the agent, using a denoising neural network to iteratively update the image representation based on a style prompt, pose, and likeness embedding, while avoiding discriminative information that could identify the person.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing image generation neural networks are used to adapt the style of a given image, then the style can be changed, but the likeness of the agent is modified along with the style
Solution Approach 1:
The patent segments the image representation into two distinct components: a style embedding that captures stylistic attributes and a likeness embedding that preserves agent identity. By separating these representations, the system can independently control style transformation while maintaining likeness consistency, resolving the contradiction between style adaptability and likeness preservation.
Solution Approach 2:
The patent introduces pose inputs as an intermediary element that mediates between the style embedding and the final stylized image generation. The pose inputs provide structural guidance that helps maintain the agent's appearance and pose consistency during style adaptation, acting as a bridge that preserves likeness while allowing style changes.
2Adaptability or versatility
If existing image generation neural networks are used to adapt the style of a given image, then the style can be changed, but the appearance of the agent is modified
Solution Approach 1:
The patent segments the image representation into two distinct components: a style embedding that captures stylistic attributes and a likeness embedding that preserves agent identity. By separating these representations, the system can independently control style transformation while maintaining likeness consistency, resolving the contradiction between style adaptability and likeness preservation.
Solution Approach 2:
The patent changes the parameter representation by using embeddings instead of direct pixel manipulation. The style embedding and likeness embedding allow the system to adjust stylistic parameters independently from identity parameters, enabling reliable appearance consistency while achieving style adaptation through parameter space transformation.
3Manufacturing precision
If discriminative information is generated to preserve appearance, then likeness can be maintained, but privacy and security are compromised
Solution Approach 1:
The patent extracts only the necessary stylistic and pose information from the input image, creating a style embedding and pose inputs that exclude discriminative identity information. This extraction process removes harmful discriminative features while retaining the essential elements needed for style adaptation and appearance preservation, resolving the contradiction between likeness preservation and privacy protection.
Solution Approach 2:
The patent creates a simplified copy of the agent's appearance through pose inputs and likeness embeddings that capture structural and stylistic information without containing unique identifying features. This copy preserves the visual appearance needed for consistency while eliminating privacy risks associated with discriminative information.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating inputs using denoising neural networks. One of the methods includes receiving an input comprising an original image of a first agent; obtaining a style prompt representing a target style of a stylized image generated from the original image; generating, from the original image, a pose input that represents a pose of the first agent in the original image; generating, from the original image, a likeness embedding that represents a likeness of the first agent in the original image; and processing the style prompt, the pose input, and the likeness embedding using an image generation neural network to generate the stylized image that depicts the first agent in the target style.