Video Super Resolution Using Optical Flow for Real-Time 4K Frames
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video super resolution technologies are unable to apply to real-time video due to resource-intensive processing, cause delays with frame dependencies, and struggle with extremely long video streams and poor noise and compression handling.
Innovation Solution
A video super resolution system utilizing a motion estimation device, warping device, and neural network super resolution device that processes only previous frames, employing optical flow, warping, and deep learning to generate high-quality frames with low power consumption, enabling real-time processing and handling long video streams.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video super resolution uses information from multiple future frames and past frames, then the super resolution quality is improved, but video delay occurs and real-time processing is compromised
Solution Approach 1:
The patent performs motion estimation and optical flow calculation in advance to predict the position and motion of objects in the current frame based only on the previous frame. This preliminary action allows the system to prepare super resolution processing without waiting for future frames, thereby eliminating video delay while maintaining processing quality
Solution Approach 2:
Instead of using multiple future and past frames to infer the current frame (conventional approach), the patent inverts the approach by using only the previous frame and motion estimation to predict and process the current frame. This inversion eliminates the need for future frame information, thus removing video delay while preserving super resolution capability
2Measurement precision
If video super resolution processes multiple frames with high resolution, then the super resolution quality is improved, but power consumption increases and real-time hardware processing becomes infeasible
Solution Approach 1:
The patent segments the video processing into independent frame-by-frame operations using recursive processing. Each frame is processed independently using only the previous frame as reference, avoiding the need to load and process multiple future frames simultaneously. This segmentation reduces memory bandwidth requirements and power consumption while maintaining super resolution quality
Solution Approach 2:
The patent changes the processing parameters by using lightweight architecture and quantization techniques. The neural network is optimized with reduced precision calculations (quantization) that significantly lower computational complexity and power consumption, enabling real-time hardware processing while maintaining acceptable super resolution quality
3Use of energy by moving object
If video super resolution uses a lightweight architecture with quantization, then power consumption is reduced, but processing capability must be optimized for real-time performance
Solution Approach 1:
The patent applies quantization to change the numerical precision parameters of the neural network computations. By using lower precision (e.g., 8-bit or 16-bit integers instead of 32-bit floats), the system reduces power consumption significantly while maintaining sufficient processing capability for real-time video super resolution through optimized computational kernels
Data Source
AI summary
A video super resolution system includes a motion estimation device, a warping device, and a neural network super resolution (NNSR) device. The motion estimation device calculates an optical flow according to a current frame and a previous frame. The warping device executes a warping process to the previous frame and a previous output to generate a warping frame and a warping output. The NNSR device executes a feature extraction to the current frame, the warping frame, the warping output, and a count value to generate at least one feature, executes a deep learning process to the at least one feature and a previous hidden state to generate a current hidden state and a deep learning result, and executes the feature extraction to the deep learning result to generate a current output. The NNSR device stores the current frame, the current hidden state, and the current output to a memory.