Style Transfer via Whitening Transform and Iterative Patch Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image editing applications that transfer the style of one image to another often result in blurry images, consume excessive processing resources, and fail to generalize to arbitrary styles, introducing undesirable distortions and being limited to 'seen' style classes, making them unsuitable for real-time applications with limited resources.
Innovation Solution
An image editing system utilizing a generative network with a whitening and color transform for one-pass encoding and decoding, featuring an encoder to extract image features, a feature transfer module for iterative patch matching, and a decoder to produce output images that accurately transfer style without significant delay or resource consumption, enabling generalization to 'unseen' style classes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If multiple passes of encoding and decoding are applied to transfer image style, then style transfer accuracy is improved, but processing time and memory consumption increase significantly
Solution Approach 1:
The patent segments the style transfer process into distinct functional modules: an encoder to extract content and style features, a whitening and color transform module to generate coarse features, and a feature transfer module to swap features iteratively. This segmentation allows each module to perform its specific function efficiently, achieving accurate style transfer in a single pass without requiring multiple repeated encoding-decoding cycles.
Solution Approach 2:
The patent introduces a whitening and color transform as an intermediary step between feature extraction and style transfer. This intermediary module generates coarse features by applying whitening and color transformation to the extracted features, serving as a bridge that enables accurate style transfer without requiring multiple passes of the entire encoding-decoding process.
2Adaptability or versatility
If transformations are applied to data between layers of a neural network for style transfer, then style transfer capability is improved, but processing resources and memory consumption increase
Solution Approach 1:
The patent applies transformations partially rather than exhaustively. Instead of applying transformations to all layers of the neural network, it selectively applies whitening and color transformation only to the coarse features generated by the encoder, reducing the computational burden while maintaining effective style transfer capability.
Solution Approach 2:
The patent changes the parameters of the feature representation through whitening and color transformation. By transforming the mean and covariance parameters of the extracted features, the system achieves style transfer without needing to process every layer of the neural network, thus reducing processing resource consumption.
3Productivity
If style transfer applications are trained on limited style classes, then training efficiency is improved, but generalization to arbitrary styles deteriorates
Solution Approach 1:
The patent creates a universal style transfer system through the feature transfer module that can handle both seen and unseen style classes. The module iteratively swaps features between content and style images, enabling the system to generalize to arbitrary styles not present in the training data, thus achieving multi-functionality across different style domains.
Solution Approach 2:
The feature transfer module employs an iterative process with feedback mechanisms to improve generalization. By repeatedly swapping features and comparing results, the system adapts to new style classes dynamically, allowing it to handle unseen styles effectively without requiring retraining for each new style category.
4Speed
If direct patch transfer from style image to output image is performed, then processing speed is improved, but image quality deteriorates due to unnatural features
Solution Approach 1:
The patent applies local quality by selectively transferring only the relevant style features to the content image. The feature transfer module identifies and swaps only the coarse features that correspond to style information, while preserving the content structure, thus achieving both speed and quality improvement compared to blind patch transfer.
Solution Approach 2:
The whitening and color transform serves as an intermediary that processes features before they are transferred. This intermediary step ensures that only appropriately transformed and normalized features are swapped, preventing the introduction of unnatural artifacts while maintaining processing efficiency.
Data Source
AI summary
In implementations of transferring image style to content of a digital image, an image editing system includes an encoder that extracts features from a content image and features from a style image. A whitening and color transform generates coarse features from the content and style features extracted by the encoder for one pass of encoding and decoding. Hence, the processing delay and memory requirements are low. A feature transfer module iteratively transfers style features to the coarse feature map and generates a fine feature map. The image editing system fuses the fine features with the coarse features, and a decoder generates an output image with content of the content image in a style of the style image from the fused features. Accordingly, the image editing system efficiently transfers an image style to image content in real-time, without undesirable artifacts in the output image.