Video Super-Resolution Using Pixel Unshuffle for Low-Latency Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning-based video super-resolution methods introduce complex motion estimation and compensation modules, increasing computational complexity and memory consumption, which impedes real-time and low latency performance, and can result in reconstruction errors.
Innovation Solution
A method and system that utilize pixel unshuffle, shallow feature extraction, and inverted residual blocks to transform low-resolution images into high-resolution images, reducing computational complexity and avoiding incorrect motion estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep learning based VSR methods with motion estimation and compensation modules are used, then video super-resolution quality is improved, but computational complexity and memory consumption increase, impeding real-time performance
Solution Approach 1:
The patent extracts and removes the complex motion estimation and compensation modules from the deep learning-based VSR system. By taking out these computationally intensive components, the system achieves real-time performance while maintaining acceptable super-resolution quality through alternative mechanisms such as frame interpolation and neural network-based enhancement without motion compensation.
Solution Approach 2:
The patent employs lighter, more computationally efficient neural network architectures that can be rapidly processed and discarded for each frame. Instead of using heavy, complex models requiring extensive computation, the system uses simplified networks that provide adequate quality improvement with much lower computational overhead, enabling real-time processing.
2Manufacturing precision
If deep learning based VSR methods with motion estimation and compensation modules are used, then video super-resolution quality is improved, but latency increases, impeding low latency performance
Solution Approach 1:
The patent removes motion estimation and compensation modules that introduce significant processing delays. By eliminating these time-consuming operations, the system reduces latency and achieves low-latency real-time video super-resolution processing while maintaining quality through streamlined neural network approaches.
Solution Approach 2:
The patent performs preliminary processing steps such as frame alignment and basic interpolation before applying neural network enhancement. This preliminary action prepares the data in a format that requires minimal additional processing, reducing overall latency while maintaining super-resolution quality.
3Loss of information
If motion estimation modules are used in VSR methods, then additional information from multiple frames is utilized, but reconstruction errors occur at receiver side
Solution Approach 1:
The patent extracts and removes motion estimation modules that are prone to errors. By taking out these unreliable components, the system avoids propagation of motion estimation errors to the reconstruction stage, thereby improving reliability and reducing reconstruction errors at the receiver side.
Solution Approach 2:
The patent uses direct copying and blending of corresponding regions from multiple frames through interpolation, rather than relying on motion estimation to warp and align frames. This copying approach avoids the complex transformations that lead to reconstruction errors, maintaining simpler and more reliable image synthesis.
Data Source
AI summary
A system and method for enhancing video resolution is disclosed. A Low-Resolution (LR) video including a plurality of LR images is received. The LR video is converted to a High-Resolution (HR) video by transforming at least one LR image to a HR image. Pixel unshuffle is performed to rearrange image portions in the LR image to generate a set of downsampled sub-images. Further, shallow features are extracted from the set of downsampled sub-image. The shallow features extracted corresponding to the set of downsampled sub-images configure a feature space. The shallow features are processed using a series of inverted residual blocks to generate enhanced features corresponding to the set of downsampled sub-images. The enhanced features are mapped to a high resolution feature space using pixel shuffle to generate the HR image corresponding to the LR image.