Gradient Domain CNN for Natural Image Style Transform
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image style transform methods, such as those using neural networks, often result in face edge deformation and inconsistent skin color when applied to self-portrait images, leading to unnatural stylized outputs that do not resemble the original.
Innovation Solution
The implementation of a Convolutional Neural Network (CNN) structure based on an image gradient domain, which incorporates pixel-wise loss and perceptual loss to maintain structural and color information, ensuring more natural and accurate style transformations by focusing on stroke rather than color.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing neural network methods are used for image style transform, then the image can be stylized, but face edge deformation occurs and skin color becomes inconsistent
Solution Approach 1:
The patent changes the parameter space from pixel domain to gradient domain for image representation. By transforming the image into gradient domain and performing style transfer in this transformed space, the method preserves structural information (edges) more effectively while achieving style transformation, thus resolving the contradiction between style adaptability and structural precision
Solution Approach 2:
The patent introduces gradient domain as an intermediary representation between the original image and the stylized output. This intermediate gradient domain serves as a mediator that decouples the structural preservation from style transformation, allowing the neural network to operate in a space where both objectives can be simultaneously achieved
2Measurement precision
If pixel-wise loss is used for training, then color accuracy is improved, but structural information and edges are lost
Solution Approach 1:
The patent transitions from pixel-wise color matching to gradient-domain structural matching by changing the dimension of operation from intensity values to gradient magnitudes and directions. This dimensional change allows the loss function to operate on gradient information rather than pixel values, thereby preserving edges and structures while maintaining color accuracy through the gradient magnitude information
3Shape
If perceptual loss is used for training, then structural information is preserved, but color fidelity deteriorates
Solution Approach 1:
The patent creates a composite loss function that combines gradient domain loss and perceptual loss components. This composite approach integrates the strengths of both methods: gradient domain loss preserves structural information through gradient magnitude and direction, while perceptual loss components maintain color fidelity through feature map comparisons in the gradient domain, achieving both structural and color preservation simultaneously
Data Source
AI summary
An image style transform method includes: acquiring an initial image to be subjected to style transform; inputting a gradient of the initial image to an image style transform model, and obtaining a feature map of the initial image in a gradient domain from the image style transform model, where the image style transform model is obtained by being trained in the gradient domain based on a pixel-wise loss and a perceptual loss; and performing image reconstruction according to the feature map of the initial image in the gradient domain to obtain a style image.