Frame-Recurrent Video Super-Resolution for Temporal Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video super-resolution methods are computationally expensive and produce temporally inconsistent results due to independent frame processing, leading to flickering artifacts.
Innovation Solution
A machine-learned recurrent super-resolution model using a frame-recurrent video super-resolution framework that leverages a previous HR estimate to super-resolve subsequent LR frames, reducing computational cost by warping only one image per step and ensuring temporal consistency through recurrent processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a batch of LR frames is combined to estimate a single HR frame using sliding window approach, then multi-frame information is utilized for super-resolution, but computational cost increases significantly and temporal consistency deteriorates
Solution Approach 1:
The video super-resolution task is segmented into independent frame-level estimations rather than processing large batches together. Each frame is estimated independently using a recurrent model that leverages temporal information from previous frames, avoiding the computational burden of processing entire batches while maintaining super-resolution quality.
Solution Approach 2:
The recurrent model maintains continuous temporal information across frames by incorporating previous HR estimates into current frame estimation. This continuous processing approach ensures temporal consistency while avoiding the discrete batch processing that causes computational inefficiency and flickering artifacts.
2Device complexity
If each output frame is estimated independently conditioned on input frames, then frame processing is simplified, but temporal consistency deteriorates causing flickering artifacts
Solution Approach 1:
The model incorporates feedback from previously estimated HR frames into the current frame estimation process. By using the previous HR estimate as a condition for the current frame prediction, the model maintains temporal consistency and eliminates flickering artifacts while keeping the processing framework relatively simple.
Solution Approach 2:
The model performs preliminary estimation of HR frames in a sequential manner, where each frame estimation builds upon the previous one. This preliminary action of maintaining temporal context from prior frames ensures consistency throughout the video sequence without requiring complex post-processing.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
The present disclosure provides systems and methods to increase resolution of imagery. In one example embodiment, a computer-implemented method includes obtaining a current low-resolution image frame. The method includes obtaining a previous estimated high-resolution image frame, the previous estimated high-resolution frame being a high-resolution estimate of a previous low-resolution image frame. The method includes warping the previous estimated high-resolution image frame based on the current low-resolution image frame. The method includes inputting the warped previous estimated high-resolution image frame and the current low-resolution image frame into a machine-learned frame estimation model. The method includes receiving a current estimated high-resolution image frame as an output of the machine-learned frame estimation model, the current estimated high-resolution image frame being a high-resolution estimate of the current low-resolution image frame.