Codec-Guided Video Enhancement for Low-Complexity Super-Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning-based video super-resolution techniques require significant computational resources and memory bandwidth, limiting their deployment on hardware platforms with limited resources, and existing complexity reduction methods compromise video quality.
Innovation Solution
A framework that uses metadata from video decode to selectively apply deep learning networks based on quantization parameters and coding modes, bypassing deep learning for certain blocks and using interpolation for others, thereby reducing computational complexity without sacrificing quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep learning networks are applied to all video blocks for super-resolution, then video quality is improved, but computational complexity and processing time increase significantly
Solution Approach 1:
The video frame is divided into multiple blocks, and deep learning super-resolution is selectively applied only to certain blocks based on their coding modes and quantization parameters, rather than processing the entire frame uniformly. This segmentation allows quality improvement where needed while reducing overall computational complexity.
Solution Approach 2:
Different processing strategies are applied to different regions of the video based on local characteristics. Blocks with inter coding modes and zero residual coefficients receive simple pixel value transfer, while other blocks receive deep learning-based super-resolution or interpolation, optimizing the balance between quality and complexity locally.
2Productivity
If deep learning networks are simplified to reduce computational complexity, then processing speed is improved, but video quality deteriorates
Solution Approach 1:
Instead of applying a simplified deep learning network to all blocks, the patent applies the full deep learning network only to specific blocks that require it (those with inter coding modes and zero residuals), while using simpler methods for other blocks. This partial application maintains quality where critical while achieving speed improvements overall.
3Device complexity
If temporal correlations are used to reduce complexity, then computational cost is reduced, but the motion vector computation network adds memory overhead
Solution Approach 1:
The patent leverages motion vectors and coding mode information that are already generated by the video decoding process itself, without requiring an additional neural network to compute them. This self-service approach uses existing metadata to guide selective super-resolution, avoiding the memory overhead of a separate motion estimation network.
Data Source
AI summary
Techniques related to accelerated video enhancement using deep learning selectively applied based on video codec information are discussed. Such techniques include applying a deep learning video enhancement network selectively to decoded non-skip blocks that are in low quantization parameter frames, bypassing the deep learning network for decoded skip blocks in low quantization parameter frames, and applying non-deep learning video enhancement to high quantization parameter frames.