Multi-Frame Video Quality Enhancement via Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for enhancing the quality of lossily compressed video do not effectively utilize information from neighboring frames, limiting their performance in improving the quality of current frames.
Innovation Solution
A multi-frame quality enhancement method that identifies peak and non-peak quality frames using a support vector machine and employs a multi-frame convolutional neural network structure with a motion compensation subnet and a quality enhancement subnet to combine information from adjacent frames, enhancing the quality of non-peak-quality frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If existing quality enhancement approaches are used that process only the current frame, then the processing complexity is low, but the quality enhancement performance is limited
Solution Approach 1:
The patent combines multiple adjacent frames (current frame and neighboring frames) into a unified processing input for the convolutional neural network. By merging temporal information from multiple frames, the system achieves superior quality enhancement performance compared to processing single frames independently, while the neural network architecture efficiently handles the combined data without proportionally increasing complexity.
Solution Approach 2:
The patent transitions from processing a single frame in isolation to processing a sequence of frames in the temporal dimension. By adding the time dimension to the processing pipeline and utilizing temporal correlations between adjacent frames, the system extracts additional useful information that enhances quality reconstruction without requiring proportional increases in processing complexity.
2Loss of energy
If video compression is applied to save coding bit-rate, then the bandwidth usage is reduced, but compression artifacts are introduced that degrade quality
Solution Approach 1:
The patent acknowledges that compression artifacts are inevitable when reducing bandwidth usage, but converts this harmful effect into a beneficial one by using the temporal correlations between adjacent frames. The neural network learns to exploit the redundancy and correlations introduced by compression across multiple frames, reconstructing high-quality video from compressed representations without requiring increased bandwidth.
3Manufacturing precision
If information from neighboring frames is utilized for quality enhancement, then the playback quality is improved, but the processing complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-processing the sequence of adjacent frames before they enter the main quality enhancement network. The system prepares the multi-frame input data by aligning and organizing temporal information in advance, which simplifies the subsequent processing in the convolutional neural network and reduces the overall processing complexity while maintaining improved playback quality.
Data Source
AI summary
The present application provides a multi-frame quality enhancement method and device for a lossily compressed video. The method comprises: performing quality enhancement on the i-th frame of a decompressed video stream using m frames correlated with the i-th frame to play the i-th frame with enhanced quality. The m frames are frames in the video stream. The number of the same pixels or corresponding pixels of each of the m frames and the i-th frame is greater than a preset threshold. m is a natural number greater than 1. In an implementation, a non-peak-quality frame positioned between two peak quality frames may be enhanced using the peak quality frames. The method may mitigate the quality fluctuation across a plurality of frames for providing playback of the video stream and may improve the quality of all frames in the video after the lossy compression.