Codec-Assisted Video Enhancement for Compute-Limited Super-Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based video super-resolution requires significant computational resources and memory bandwidth, limiting its deployment on hardware platforms with limited resources, and existing complexity reduction techniques compromise video quality.
Innovation Solution
A framework that uses metadata from video decode to selectively apply deep learning networks based on quantization parameters and coding modes, bypassing deep learning for certain blocks to reduce computational complexity while maintaining quality, using hardware upsampling and pixel transfer for other blocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep learning networks are applied for video super-resolution, then video quality is improved, but computational complexity and memory bandwidth requirements increase significantly
Solution Approach 1:
The video is divided into multiple blocks, and the deep learning network is selectively applied only to certain blocks based on motion characteristics. Blocks with low motion or temporal correlation are processed using simpler methods, while blocks with high motion or significant changes are processed using the deep learning network, thus reducing overall computational complexity while maintaining quality where needed.
Solution Approach 2:
Different processing strategies are applied to different regions of the video based on local characteristics. The patent identifies regions with high motion or significant temporal changes and applies deep learning super-resolution only to those regions, while using simpler interpolation methods for regions with low motion or high temporal correlation, optimizing the trade-off between quality and computational cost.
2Manufacturing precision
If deep learning networks are applied for video super-resolution, then video quality is improved, but memory bandwidth requirements increase
Solution Approach 1:
The patent extracts and utilizes motion information from the video codec metadata to identify blocks that can be processed using simpler methods. By taking out the motion information early in the decoding process, the system can avoid applying the computationally intensive deep learning network to blocks where it would provide minimal benefit, thus reducing memory bandwidth requirements.
Solution Approach 2:
Motion estimation and block identification are performed in advance using the video codec metadata before the super-resolution processing. This preliminary action allows the system to pre-determine which blocks require deep learning processing and which can use simpler methods, reducing the amount of data that needs to be transferred to and from the deep learning network during processing.
3Productivity
If complexity reduction techniques are applied to deep learning networks, then computational speed is improved, but video super-resolution quality deteriorates
Solution Approach 1:
Instead of applying a simplified deep learning network to all blocks, the patent applies the full-capability deep learning network only to the specific blocks where it is most needed (those with high motion or significant temporal changes). For other blocks, simpler and faster processing methods are used, achieving a balance between overall processing speed and quality where the deep learning network provides the most value.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques related to accelerated video enhancement using deep learning selectively applied based on video codec information are discussed. Such techniques include applying a deep learning video enhancement network selectively to decoded non-skip blocks that are in low quantization parameter frames, bypassing the deep learning network for decoded skip blocks in low quantization parameter frames, and applying non-deep learning video enhancement to high quantization parameter frames.