Iterative Video Frame Interpolation for High-Resolution Slow Motion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video frame interpolation methods, such as CAIN, struggle with high computational cost and are limited to a fixed frame rate multiplier, resulting in poor image quality for high-resolution images, especially in broadcast applications where flexible slow-motion replays are desired without the use of expensive super slow-motion cameras.
Innovation Solution
A method and system that iteratively interpolates high-resolution intermediate images using a pyramidal approach with upscaled intermediate images, employing a modified CAIN network architecture that includes multiple iterations to enhance image quality, allowing flexible frame rate multipliers without explicit motion estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If optical flow estimation is used for video frame interpolation, then intermediate frames can be generated, but substantial computational cost in terms of time and memory is incurred
Solution Approach 1:
The patent extracts and removes the optical flow estimation step from the video frame interpolation process. Instead of estimating optical flow between frames, the invention directly processes image patches and their gradients to generate intermediate frames, thereby eliminating the substantial computational time and memory requirements associated with optical flow calculation while maintaining interpolation capability
Solution Approach 2:
The patent replaces the mechanical/optical approach of optical flow estimation with a direct image processing approach using gradient information. Instead of tracking pixel movements through optical flow algorithms, the invention substitutes this with a method that directly computes intermediate frames using gradient descent optimization on image patches, reducing computational complexity
2Device complexity
If CAIN approach is used for video frame interpolation, then intermediate frames are generated without explicit motion estimation, but the method is inadequate for high-resolution images and lacks flexibility in handling arbitrary frame rate multipliers
Solution Approach 1:
The patent segments the image into multiple patches and processes each patch independently using gradient information. This segmentation approach allows the method to handle high-resolution images effectively by breaking them down into manageable units, improving both computational efficiency and image quality preservation compared to the CAIN approach
Solution Approach 2:
The patent introduces a dynamic gradient descent optimization process that adapts to different frame rate multipliers and image resolutions. The method dynamically adjusts the number of intermediate frames and optimization parameters based on the desired frame rate multiplier, providing flexibility that the static CAIN approach lacks while maintaining high image quality
3Loss of information
If optical flow estimation is used for video frame interpolation, then motion information is captured, but the optical flow estimator limits the performance of video frame interpolation
Solution Approach 1:
The patent copies and utilizes gradient information directly from the input frames to construct intermediate frames, bypassing the need for optical flow estimation. By copying gradient information and using it to guide the interpolation process through gradient descent optimization, the method preserves motion information more reliably than optical flow estimators, which are inherently imperfect and limit interpolation performance
Data Source
Figure 1~3
Figure 2
Figure 4A~4B
AI summary
A method for interpolating high resolution intermediate images is suggested. The method relies on an iterative approach exploiting in a current interpolation step additional information contained in up sampled intermediate images from a previous interpolation step. In this way the proposed method overcomes shortcomings of known interpolation methods regarding the interpolation of high-resolution images. A broadcast production system leveraging the proposed interpolation method allows for replay of scenes selected by a production director in slow-motion without requiring dedicated slow-motion cameras.