Stream Video Frame Correction FPGA Memory Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current frame-based image correction methods in FPGAs result in high latency, which is not suitable for real-time applications, particularly in Degraded Visual Environment conditions where low latency is critical for aviation safety.
Innovation Solution
A stream-based video frame correction method using an integrated circuit that optimizes on-chip memory allocation by storing only a predefined portion of the input frame and employing a lookup table to interpolate pixel values, reducing memory access latency and eliminating the need for off-chip memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If frame-based correction approach is used to store entire frame on FPGA, then high update rate (60 Hz) is achieved, but pipeline delay increases to approximately 50 ms
Solution Approach 1:
The patent segments the frame processing into two distinct phases: a setup phase where LUT mesh points are loaded into on-chip memory blocks, and a stream-based processing phase where individual pixels are corrected on-the-fly as they stream through the FPGA. This segmentation eliminates the need to store and process the entire frame before correction, reducing pipeline delay from 50 ms to near real-time while maintaining 60 Hz update rates.
Solution Approach 2:
The patent performs preliminary loading of LUT mesh points into on-chip memory blocks before the actual pixel correction begins. By pre-positioning the correction data in fast on-chip memory, the system eliminates memory access latency during the critical pixel correction phase, enabling near real-time processing without sacrificing update rate.
2Loss of time
If on-chip memory blocks are used to store LUT mesh points and input pixels, then memory access latency is reduced, but memory capacity is limited
Solution Approach 1:
The patent extracts only the essential correction data (LUT mesh points) and a minimal portion of input pixels needed for correction into on-chip memory blocks. By taking out only the critical data elements rather than storing entire frames, the system fits the correction functionality within limited on-chip memory capacity while maintaining fast access speeds.
Solution Approach 2:
The patent implements partial action by storing only a predefined portion of the input frame in on-chip memory blocks rather than the entire frame. This partial storage approach provides sufficient data for correction operations while staying within the capacity constraints of on-chip memory, achieving a balance between latency reduction and capacity limitations.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and integrated circuit of video frame correction by an integrated circuit, IC, wherein the integrated circuit performs the following steps: receiving lookup table, LUT, mesh points from a software application, each LUT mesh point defining a projection from an output pixel of an output frame to a corresponding input pixel of an input frame, wherein a total number of LUT mesh points is less than a total number of pixels of the output frame; storing the received LUT mesh points on a first set of memory blocks of the integrated circuit; sequentially receiving input pixels of the input frame as captured by an image sensor of a camera; sequentially storing the received input pixels on a second set of memory blocks of the integrated circuit; and in response to storing a predefined portion of the input frame, generating the output frame by further performing the following steps within one fixed clock cycle: selecting one of the output pixels of the output frame as target pixel; determining a pixel value of the target pixel by: retrieving, from the first set of memory blocks, two or more LUT mesh points associated with the target pixel; determining a source pixel of the predefined portion for the target pixel on the input frame by interpolating data values of the retrieved two or more LUT mesh points; retrieving two or more input pixels from the second set of memory blocks that are located closest to the determined source pixel; determining the pixel value of the target pixel by interpolating between the two or more input pixels retrieved; outputting the pixel value of the target pixel; if all output pixel values of all output pixels are determined, terminating, or otherwise further performing the following steps: receiving a further input pixel from the image sensor of the camera; and storing the received further input pixel in the second set of memory blocks; selecting another of the output pixels of the output frame as target pixel.