Video Super-Resolution Using Depth and Optical Flow Priors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for converting low-resolution images into high-resolution images do not effectively utilize prior information, leading to low accuracy and efficiency in super-sampling processes.
Innovation Solution
An image processing method that involves acquiring a first image and its corresponding depth image, along with N preceding frames and their respective depth and optical flow images, to determine a super-sampled image, utilizing a super-sampled network for improved accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a pre-trained super-sampled model is used to convert low-resolution images to high-resolution images, then the super-sampling process can be performed, but the accuracy of the image super-sampling is relatively low because the model does not acquire prior information of the low-resolution image
Solution Approach 1:
The patent applies preliminary action by acquiring N preceding frames (second images) and their corresponding depth images and optical flow images before performing super-sampling on the current low-resolution image. This allows the super-sampled model to utilize temporal prior information from previous frames, improving the accuracy of the super-sampling process without requiring additional processing of the current frame alone
Solution Approach 2:
The patent introduces depth images as an additional dimension of information beyond the standard 2D image data. By incorporating depth information from the current frame and previous frames, the system creates a multi-dimensional input space that provides richer contextual information for super-sampling, enabling more accurate reconstruction of high-resolution details
2Measurement precision
If super-sampled processing is performed on low-resolution images to obtain high-resolution images, then image resolution is improved, but the processing efficiency is reduced due to the large number of pixels involved in convolution
Solution Approach 1:
The patent segments the super-sampling process by performing it on individual frames (current frame and N preceding frames) independently rather than processing the entire video sequence as a single operation. This allows for parallel processing of multiple frames and reduces the computational burden on any single convolution operation, improving overall processing efficiency while maintaining high output resolution
3Measurement precision
If multiple frames and their corresponding depth and optical flow images are acquired for super-sampling, then the accuracy of image super-sampling is improved, but the device complexity and processing time increase
Solution Approach 1:
The patent makes the super-sampled model multi-functional by enabling it to process multiple types of input data (standard image data, depth image data, and optical flow data) simultaneously. This universal processing capability allows the same model architecture to handle diverse information types without requiring separate specialized models, thereby improving accuracy while controlling device complexity through model reuse
Data Source
Figure 1~2
Figure 3~4
Figure 5~6
AI summary
An image processing method and apparatus, and an electronic device. The method comprises: acquiring a first image of a video and a first depth image corresponding to the first image; acquiring N second image frames before the first image, and a second depth image and an optical flow image corresponding to each second image frame, wherein N is an integer greater than 0; and, on the basis of the first image, the first depth image, the N previous second image frames, the N second depth images and the N optical flow images, determining a super-sampling image corresponding to the first image. The super-sampling accuracy can be improved.