Image Style Transfer Algorithm Using Covariance and Non-linear Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image color and tone transfer techniques often result in unrealistic artifacts due to aggressive color and tone modifications, failing to account for the features of the edited image, leading to poor visual results.
Innovation Solution
An image style transfer algorithm that applies a chrominance transfer function and a luminance transfer function to match the color and tone styles of a style image to an input image, while controlling modifications to prevent over-amplification and using non-linear optimization to estimate tone mapping curves, ensuring the modified image maintains the input image's characteristics without artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If complete color and tone transfer is applied to match the style image, then the color and tone statistics are fully transferred, but unrealistic artifacts are produced and visual quality deteriorates
Solution Approach 1:
The patent applies partial color and tone transfer by selectively transferring only certain statistical properties (mean and standard deviation) while preserving other characteristics of the original image. The chrominance transfer function applies covariance control to limit the extent of color modification, and the luminance transfer function uses histogram matching with clipping to prevent excessive tone adjustments, thereby avoiding artifacts while maintaining style transfer effectiveness.
Solution Approach 2:
The patent transforms the color and tone transfer process by changing key parameters: using covariance of input image color to control chrominance modification, employing non-linear optimization to estimate tone mapping curve parameters, and applying histogram matching with controlled clipping. These parameter changes enable precise control over the transfer process to prevent artifact generation.
2Adaptability or versatility
If aggressive color and tone modifications are applied to achieve style transfer, then the style matching is improved, but the input image features are distorted and visual realism is lost
Solution Approach 1:
The patent incorporates feedback mechanisms by using the covariance of the input image color as a control parameter in the chrominance transfer function. This feedback allows the system to adapt the degree of color modification based on the original image's color characteristics, preventing excessive distortion while achieving style transfer. The non-linear optimization process also uses feedback to estimate tone mapping parameters that preserve realism.
3Manufacturing precision
If full histogram matching is applied to match luminance distributions, then the tone style is fully transferred, but the natural luminance relationships are distorted
Solution Approach 1:
The patent applies partial histogram matching by transferring the luminance distribution style while clipping extreme values to prevent excessive modification. The luminance transfer function matches histograms within controlled bounds, transferring the essential tone characteristics without completely replacing the original luminance relationships, thereby maintaining stability while achieving style transfer.
Data Source
AI summary
In embodiments of image color and tone style transfer, a computing device implements an image style transfer algorithm to generate a modified image from an input image based on a color style and a tone style of a style image. A user can select the input image that includes color features, as well as select the style image that includes an example of the color style and the tone style to transfer to the input image. A chrominance transfer function can then be applied to transfer the color style to the input image, utilizing a covariance of an input image color of the input image to control modification of the input image color. A luminance transfer function can also be applied to transfer the tone style to the input image, utilizing a tone mapping curve based on a non-linear optimization to estimate luminance parameters of the tone mapping curve.