Real-time Video Superresolution via Motion Registration

Solution Overview

Problem

Current video superresolution and transmission systems lack an integrated solution for real-time high resolution video production from low resolution encoded videos, especially in constrained environments like surveillance and defense applications, where high resolution images are desired but not feasible due to size, weight, power, and cost limitations.

Innovation Solution

An integrated system that combines motion registration, non-uniform image interpolation, image regularization, and image post-processing techniques, utilizing global motion vectors and compressive sensing to enhance low resolution videos to high resolution in real-time, directly on the decoder side, without requiring trained dictionaries on the decoder.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If high resolution sensors are used to capture high resolution images, then image quality and resolution are improved, but system size, weight, power consumption, and cost increase

Engineering Contradiction:

Improveimage resolutionVSAvoidsystem weight

Core Design Contradiction:

Measurement precisionVSWeight of moving object

Solution Approach 1:

The patent creates a high resolution image by synthesizing multiple low resolution images through superresolution algorithms. Instead of using a heavy high resolution sensor, the system captures multiple low resolution frames and computationally reconstructs a high resolution image, effectively copying the desired high resolution output through software processing rather than hardware sensing.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system performs preliminary actions by capturing multiple low resolution frames with sub-pixel displacements before the final image reconstruction. These preliminary frames are processed through motion registration and superresolution algorithms to produce the final high resolution image, allowing the system to achieve high resolution without requiring a high resolution sensor.

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If high resolution sensors are used to capture high resolution images, then image quality and resolution are improved, but system size, weight, power consumption, and cost increase

Engineering Contradiction:

Improveimage resolutionVSAvoidpower consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent creates a high resolution image by synthesizing multiple low resolution images through superresolution algorithms. Instead of using a heavy high resolution sensor, the system captures multiple low resolution frames and computationally reconstructs a high resolution image, effectively copying the desired high resolution output through software processing rather than hardware sensing.

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The system performs preliminary actions by capturing multiple low resolution frames with sub-pixel displacements before the final image reconstruction. These preliminary frames are processed through motion registration and superresolution algorithms to produce the final high resolution image, allowing the system to achieve high resolution without requiring a high resolution sensor.

Inventive Principle:

Principle #10Preliminary action

3Measurement precision

If multiple low resolution images are processed to create high resolution images through superresolution, then high resolution output is achieved with low cost hardware, but processing time and computational complexity increase

Engineering Contradiction:

Improveimage resolutionVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The system performs preliminary motion registration and alignment of multiple low resolution frames before the superresolution reconstruction. By pre-processing the frames to correct for motion and sub-pixel displacements, the system reduces the computational complexity of the final reconstruction step, thereby reducing overall processing time.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The superresolution process is divided into distinct segments: motion registration, sub-pixel displacement correction, and final image reconstruction. This segmentation allows each step to be optimized independently and enables parallel processing of multiple frames, reducing the total processing time required to generate high resolution output.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS8520736B2Real-time superresolution and video transmission

Publication Date: 2013.08.27 FASTVDO LLC

AI summary

A method and system of performing real-time video superresolution. A decoder receives a data stream representing a low resolution video and including global motion vectors relating to image motion between frames of the low resolution video. The decoder uses the global motion vectors from the received data stream and multiframe processing algorithms to derive a high resolution video from the low resolution video. The sharpness of the high resolution video may be enhanced.