Video Super-Resolution Using Residual Frame Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing super resolution technologies fail to effectively restore high-frequency information in low-resolution images, resulting in images with low resolution and poor viewing experience.
Innovation Solution
An image processing method that utilizes motion vector information and residual frames to enhance high-frequency information by aligning and accumulating residual frames, which are then input into a neural network for super resolution processing, enhancing edge details without increasing hardware costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional super resolution algorithms are used to restore low-resolution images, then processing can be performed with existing hardware, but high-frequency information is lost and image resolution remains low
Solution Approach 1:
The patent applies preliminary action by performing motion compensation and frame alignment before super resolution processing. By pre-aligning multiple frames using motion vectors and accumulating residual frames, the method prepares enhanced high-frequency information in advance, which is then fed to the neural network for superior resolution restoration without losing edge details.
Solution Approach 2:
The patent transitions from single-frame processing to multi-frame temporal processing by accumulating residual frames across time. This temporal dimension accumulation reconstructs lost high-frequency information by combining data from multiple frames, enabling the neural network to recover edge details that would be impossible to obtain from a single low-resolution frame.
2Measurement precision
If high-resolution images are transmitted to improve picture quality, then viewing experience improves, but data transmission amount and storage costs increase
Solution Approach 1:
The patent replaces the mechanical approach of transmitting high-resolution data with an information processing approach. Instead of sending more data, the system transmits compressed low-resolution video streams and uses neural network-based super resolution with temporal accumulation to reconstruct high-frequency details, substituting data transmission with intelligent processing.
Solution Approach 2:
The patent changes the parameter of image resolution from the transmission stage to the processing stage. By transmitting low-resolution data and performing super resolution processing at the receiving end using accumulated residual frames, the system achieves high picture quality output without increasing the transmission data amount or storage requirements.
3Measurement precision
If multiple frames are processed to restore high-frequency information, then image resolution improves, but processing complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the super resolution process into distinct stages: motion compensation, frame alignment, residual accumulation, and neural network processing. This segmentation allows each component to be optimized independently, managing processing complexity through modular architecture while achieving high-resolution restoration.
Solution Approach 2:
The patent introduces residual frames as an intermediary element that bridges motion-compensated frames and the final super resolution output. By accumulating residuals and using them as additional input channels to the neural network, the system manages the complexity of multi-frame processing through this intermediate representation that captures high-frequency differences.
Data Source
Figure 1~2A
Figure 2B~3A
Figure 3B
AI summary
An image processing method and apparatus are disclosed. The method includes: obtaining a video stream, where the video stream includes a first frame of image, a second frame of image, and a third frame of image that are adjacent in time sequence; decoding the video stream to obtain a first alignment frame, a second alignment frame, and at least one residual between the first frame of image, the second frame of image, and the third frame of image; generating at least one residual frame based on the at least one residual; performing super resolution processing on the second frame of image based on the at least one residual frame, the first alignment frame, and the second alignment frame to obtain a second frame of image obtained after super resolution. According to the method, an alignment frame and a residual frame are used as inputs of a neural network. High-frequency information is output after the residual frame passes through the neural network, so that an edge detail can be enhanced by attaching the high-frequency information back to a luminance channel, thereby improving video picture quality without increasing hardware costs.