Super Resolution Image Generation Using Optical Flow and Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video capture systems often face limitations in resolution, especially when capturing objects at a distance, leading to low-resolution images that hinder advanced computer vision tasks due to factors like blur and low-light conditions, and existing upscaling techniques do not adequately meet the requirements for advanced analysis.
Innovation Solution
A method and system that generate a super-resolution image of an object by combining information from multiple video frames using a trained neural network, which calculates optical flow to accurately account for the object's 3D structure, comprising a correspondence estimation neural network and a reconstruction neural network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video capture resolution is increased, then the overall image quality improves, but the field of view decreases causing individual objects to remain low resolution
Solution Approach 1:
The patent segments the video processing task by extracting individual frames containing the object of interest and processing only those frames through the neural network, rather than processing the entire video stream. This allows high-resolution processing of specific objects while maintaining the ability to capture multiple objects across different frames.
Solution Approach 2:
The patent transitions from 2D image processing to 3D spatial reasoning by using optical flow to track object movement across frames and reconstructing the object's 3D structure. This enables the system to achieve high resolution for distant objects by leveraging temporal information and spatial relationships across multiple frames.
2Measurement precision
If upscaling techniques are applied to entire video, then resolution improves, but computational complexity increases significantly
Solution Approach 1:
The patent extracts only the necessary information for processing by identifying and isolating specific frames containing objects of interest, then applying the neural network upscaling technique only to these extracted frames. This selective processing approach dramatically reduces computational complexity compared to processing entire video streams.
Solution Approach 2:
The patent applies partial action by processing only the subset of frames that contain objects requiring high-resolution analysis, rather than uniformly processing all frames. This selective application of the computationally intensive neural network achieves the required resolution improvement while managing computational resources efficiently.
3Measurement precision
If standard upscaling techniques are used, then some resolution improvement is achieved, but the resolution is insufficient for advanced computer vision tasks
Solution Approach 1:
The patent replaces traditional mechanical upscaling methods (such as simple interpolation algorithms) with a neural network-based approach that uses deep learning to reconstruct high-resolution images. This substitution enables the system to achieve the high resolution required for advanced computer vision tasks like license plate recognition and text identification.
Solution Approach 2:
The patent introduces optical flow as an intermediary mechanism that bridges the gap between low-resolution video frames and high-resolution reconstruction. The optical flow field serves as a mediator that captures motion information and helps the neural network accurately reconstruct the 3D structure of objects, thereby improving the reliability of the upscaling process for advanced analysis tasks.
Data Source
AI summary
There is provided methods and systems for generating super resolution images of objects. In particular there is provided a method of generating a super resolution image of an object, the method comprising: receiving a plurality of frames of a video of the object; extracting from the plurality of frames a plurality of images of the object; selecting an image of the plurality of images as a target image; applying a trained model to the plurality of images to generate a super resolution image of the object, wherein the trained model comprises: (a) a correspondence estimation neural network configured to compute a respective optical flow between the target image and each other image of the plurality, and (b) a reconstruction neural network configured to generate a super resolution version of the target image using the plurality of images and the respective optical flows between the target image and each other image of the plurality.There is also provided a corresponding system, along with systems and methods for training said model.