Super-Resolution Video Processing Using Compressed Stream Motion Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video super-resolution (VSR) techniques face challenges in achieving high-quality results while maintaining low computational intensity, making them unsuitable for real-time applications on devices with limited resources, such as mobile devices.
Innovation Solution
The method leverages motion information and residual information from compressed video streams to skip the time-consuming motion-estimation step, using a novel framework that accelerates VSR approaches by processing key frames with existing multi-frame techniques and generating super-resolution versions of inter frames using encoded motion and residual data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If multi-frame VSR approaches are used to generate super-resolution video, then image quality is improved, but computational intensity increases making it difficult to deploy on mobile devices
Solution Approach 1:
The patent segments the video processing into two distinct parts: (1) processing only key frames using computationally intensive multi-frame VSR techniques to achieve high image quality, and (2) processing inter-frames using lightweight motion-compensated frame difference methods. This segmentation allows the system to achieve high overall quality while reducing total computational intensity, enabling deployment on mobile devices with limited resources.
2Manufacturing precision
If motion estimation is performed for each frame in multi-frame VSR, then super-resolution quality is improved, but processing time increases
Solution Approach 1:
The patent performs motion estimation only on key frames in advance, before generating super-resolution versions. For subsequent inter-frames, the pre-computed motion information from key frames is reused through motion compensation techniques. This preliminary action eliminates the need for repeated motion estimation on every frame, dramatically reducing processing time while maintaining super-resolution quality through the reuse of motion data.
3Productivity
If conventional SISR approaches are used for video super-resolution, then processing speed is improved, but image quality deteriorates with blurry output and artifacts
Solution Approach 1:
The patent merges two different approaches: it combines the speed advantage of conventional SISR methods with the quality advantage of multi-frame VSR techniques. Specifically, it applies multi-frame VSR only to key frames to establish high-quality reference points, then uses fast SISR-like motion-compensated processing for inter-frames. This merging achieves both high processing speed and high image quality without the drawbacks of either approach alone.
Data Source
AI summary
A method, device and computer-readable medium for generating a super-resolution version of a compressed video stream. By leveraging the motion information and residual information in compressed video streams, described examples are able to skip the time-consuming motion-estimation step for most frames and make the most use of the SR results of key frames. A key frame SR module generates SR versions of I-frames and other key frames of a compressed video stream using techniques similar to existing multi-frame approaches to VSR. A non-key frame SR module generates SR version of the non-key inter frames between these key frames by making use of motion information and residual information used to encode the inter frames in the compressed video stream.