Compression-Aware Video Super-Resolution for Artifact Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video super-resolution methods perform poorly on compressed video data due to the lack of consideration for compression artifacts, leading to significant degradation and artifacts when applied to heavily compressed videos.
Innovation Solution
A bi-directional training approach for a machine-learned video super-resolution model that includes a bi-directional recurrent module to reduce warping errors, a detail-aware flow estimation module to recover high-resolution flow, and a Laplacian enhancement module to add high-frequency information, effectively addressing compression artifacts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing VSR methods are applied to compressed video data, then processing speed is maintained, but image quality deteriorates due to compression artifacts and warping errors
Solution Approach 1:
The patent trains the VSR model specifically on compressed video data rather than uncompressed data. By using compressed data as training input, the model learns to recognize and compensate for compression artifacts, effectively converting the harmful compression effects into training signals that improve performance on compressed video while maintaining general VSR capability
Solution Approach 2:
The patent modifies the training data parameters by using compressed video data instead of uncompressed data. This parameter change in the training process enables the model to adapt to compression artifacts and achieve better performance on compressed video inputs
2Object-affected harmful factors
If denoising model is applied before VSR on compressed video, then compression artifacts are reduced, but performance deteriorates due to changed degradation kernel
Solution Approach 1:
The patent performs preliminary training action by training the VSR model on compressed video data before actual inference. This preliminary training equips the model with the ability to handle compression artifacts directly, eliminating the need for separate denoising preprocessing steps and avoiding the degradation kernel mismatch problem
3Device complexity
If VSR model is trained on uncompressed video data, then model simplicity is maintained, but performance on compressed video deteriorates
Solution Approach 1:
The patent changes the training data parameter from uncompressed video to compressed video. This single parameter change in the training process enables the model to achieve superior performance on compressed video without requiring complex architectural modifications, maintaining model simplicity while improving specialized performance
Data Source
AI summary
Example aspects of the present disclosure are directed to systems and methods which feature a machine-learned video super-resolution (VSR) model which has been trained using a bi-directional training approach. In particular, the present disclosure provides a compression-informed (e.g., compression-aware) super-resolution model that can perform well on real-world videos with different levels of compression. Specifically, example models described herein can include three modules to robustly restore the missing information caused by video compression. First, a bi-directional recurrent module can be used to reduce the accumulated warping error from the random locations of the intra-frame from compressed video frames. Second, a detail-aware flow estimation module can be added to enable recovery of high resolution (HR) flow from compressed low resolution (LR) frames. Finally, a Laplacian enhancement module can add high-frequency information to the warped HR frames washed out by video encoding.