Super-Resolution Video Using Bidirectional Neural Network
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reference-based super-resolution methods face challenges in efficiently improving the resolution of ultra-wide-angle videos, particularly when using videos with different fields of view, as they require extensive calculations and are not practical for implementation, especially in asymmetric multi-camera systems.
Innovation Solution
A method utilizing a bidirectional neural network that accumulates confidence maps from past and future time points to enhance the resolution of ultra-wide-angle videos by incorporating wide-angle and telephoto camera inputs, trained through a two-step supervised learning scheme, allowing for efficient alignment and upsampling of video frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If reference-based super-resolution methods are applied to ultra-wide-angle videos using multi-camera inputs, then the resolution of the output video is improved, but the computational complexity increases significantly
Solution Approach 1:
The patent applies preliminary action by pre-calculating and accumulating confidence maps from past and future time points before processing the current frame. The bidirectional neural network accumulates temporal information in advance, storing confidence maps and intermediate features that are then reused during the super-resolution process, avoiding redundant calculations and reducing overall computational complexity
Solution Approach 2:
The patent segments the video processing into distinct temporal components (past time points, current time point, future time points) handled by separate forward and backward neural network cells. This segmentation allows independent processing and accumulation of temporal features, making the complex computation more manageable and efficient
2Ease of manufacture
If conventional frame-by-frame RefSR method is used, then the implementation is simple, but the computational cost becomes prohibitively high
Solution Approach 1:
The patent implements continuity of useful action by maintaining temporal coherence across frames through the bidirectional neural network. Confidence maps and intermediate features are accumulated and propagated continuously across time points, ensuring that useful computational results from past and future frames are reused, thereby improving computational efficiency while maintaining implementation feasibility
Solution Approach 2:
The bidirectional neural network performs preliminary accumulation of confidence maps and temporal features from multiple time points before the actual super-resolution inference. This pre-computation reduces the computational burden during real-time processing, achieving better efficiency without significantly complicating the implementation
3Manufacturing precision
If videos with different fields of view are used as reference, then the resolution improvement is enhanced, but the alignment difficulty increases
Solution Approach 1:
The patent introduces an intermediary mechanism - the confidence map - that mediates between videos with different fields of view. The confidence map encodes alignment information and spatial relationships, allowing the bidirectional neural network to effectively align and fuse features from ultra-wide-angle, wide-angle, and telephoto videos despite their different FoVs, thereby reducing alignment difficulty while maintaining resolution improvement
Data Source
AI summary
A method for generating a super-resolution video by using a multi-camera video may comprise: generating a resolution-improved ultra-wide-angle video frame at an arbitrary time step by inputting an ultra-wide-angle video frame of a first resolution at the arbitrary time step, ultra-wide-angle video frames right before and right after the arbitrary time step, and a wide-angle video frame for reference at the arbitrary time step, to a bidirectional neural network, wherein the generating of the resolution-improved ultra-wide-angle video frame is performed using accumulated information at a past time step based on the arbitrary time step, and accumulated information at a future time step based on the arbitrary time step, and wherein a second resolution, which is a resolution of the generated ultra-wide-angle video frame, is greater than the first resolution.