Video Style Transfer Using Optical Flow for Frame Stability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video style transfer algorithms result in significant image instability between frames, leading to undesirable visual artefacts such as 'sizzling' and 'popping', and require deep understanding and modification of the algorithm, which is beyond many users.
Innovation Solution
A method that processes a content video by generating noise-added frames and preserving noise in low-frequency regions using optical flow data, applying a stylization algorithm without modifying the existing style transfer algorithm.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing video style transfer algorithms are applied directly, then style transfer can be achieved, but significant image instability between frames occurs causing visual artifacts
Solution Approach 1:
The patent applies preliminary actions by adding noise to low-frequency regions before style transfer processing, and by pre-calculating optical flow between frames. This preparatory noise addition creates a foundation that prevents visual artifacts during subsequent style transfer operations, while the pre-computed optical flow enables stable frame-to-frame transitions.
Solution Approach 2:
The patent introduces noise as an intermediary element in low-frequency regions and uses optical flow as a mediator between consecutive frames. The noise acts as a stabilizing intermediary that prevents direct artifacts, while optical flow serves as an intermediary transformation that ensures smooth transitions between stylized frames without causing instability.
2Reliability
If style transfer algorithms are modified to reduce artifacts, then visual quality improves, but algorithm complexity increases requiring deep understanding
Solution Approach 1:
The patent segments the image processing into distinct frequency components by adding noise specifically to low-frequency regions while preserving high-frequency content. This segmentation allows the style transfer algorithm to operate on stabilized low-frequency areas without requiring modifications to the core algorithm, maintaining visual quality while avoiding increased complexity.
Solution Approach 2:
The patent uses optical flow as an intermediary that bridges consecutive frames without requiring algorithm modification. By computing optical flow separately and applying it as a post-processing step, the patent achieves stable frame transitions while keeping the style transfer algorithm itself unchanged, thus improving visual quality without increasing algorithmic complexity.
3Stability of the object's composition
If noise is added to stabilize frames, then frame stability improves, but processing complexity increases
Solution Approach 1:
The patent applies noise addition locally only to low-frequency regions of the image spectrum rather than uniformly across all frequencies. This localized approach stabilizes frames where it is most needed (in smooth, low-frequency areas) while avoiding unnecessary processing in high-frequency regions, thereby improving frame stability without proportionally increasing processing complexity.
Solution Approach 2:
The patent performs noise addition as a preliminary step before the main style transfer processing. By preparing the noise-added frames in advance, the patent enables subsequent style transfer operations to proceed more efficiently with reduced iterations or adjustments needed, thus improving frame stability while the initial processing overhead is offset by reduced downstream computational requirements.
Data Source
AI summary
Described herein is a computer implemented method including: accessing an input video; generating a first output frame corresponding to a first input frame by: generating a noise-added frame by processing the first input frame to add noise to any low-frequency regions; and processing the noise-added frame in accordance with a stylization algorithm to generate the first output frame; generating a second output frame corresponding to a second input frame, where the second input frame is subsequent to the first input frame and is generated by: calculating first optical flow data describing an optical flow between the first and second input frame; generating a first noise-preserved frame by using the first optical flow data to deform the noise-added frame; and processing the first noise-preserved frame in accordance with the stylization algorithm to generate the second output frame; and encoding the first and second output frame into output video data.