Pixel Registration Between Event and Frame Cameras for Depth Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for image registration between frame-based and event-based cameras are inefficient due to the different nature of their signals, spatial resolution disparities, and the use of rolling shutters, which can bias depth estimation.
Innovation Solution
A method for pixel-by-pixel registration that includes synchronization, rectification, and correlation scoring to align and deblur images from frame-based and event-based cameras, using correlation functions and disparity maps to establish accurate depth information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional stereo vision methods are used for registration between frame-based and event-based cameras, then depth information can be retrieved, but the methods are inefficient due to different signal nature and spatial resolution disparities
Solution Approach 1:
The patent segments the registration process into multiple stages: first registering event-based camera pixels to a common reference frame, then registering frame-based camera pixels to the same reference frame. This segmentation allows each camera type to be processed according to its specific characteristics, improving overall efficiency while maintaining accuracy.
Solution Approach 2:
The patent introduces a common reference frame as an intermediary between the event-based and frame-based cameras. Both camera types are registered to this intermediate reference frame, which then serves as the basis for mutual registration. This intermediary approach resolves the incompatibility between different signal nature and spatial resolutions.
2Duration of action of moving object
If rolling shutter imaging is used in frame-based cameras, then capture time can be extended, but depth estimation becomes biased
Solution Approach 1:
The patent applies local quality by processing different regions of the image with different timing information. For each pixel or region, the method uses the specific exposure time of that region in the rolling shutter sequence, allowing accurate depth estimation for each local area while maintaining the benefits of extended capture time.
Solution Approach 2:
The patent dynamically adjusts the registration process to account for the rolling shutter effect by using time-varying exposure information for different pixel rows or regions. This dynamic approach allows the system to compensate for the biased depth estimation caused by sequential capture while maintaining extended capture duration.
3Measurement precision
If pixel-by-pixel registration is performed, then alignment precision is improved, but computational complexity increases
Solution Approach 1:
The patent segments the computational workload by first processing event-based camera pixels to establish a reference registration, then using this pre-computed registration to guide the frame-based camera pixel registration. This segmentation reduces the computational complexity of pixel-by-pixel registration while maintaining high alignment precision.
Solution Approach 2:
The patent performs preliminary registration of event-based camera pixels to establish a reference frame before registering frame-based camera pixels. This preliminary action provides a foundation that simplifies the subsequent registration process, reducing overall computational complexity while achieving precise pixel alignment.
Data Source
AI summary
A method for registering pixels provided in a pixel event stream comprising: acquiring image frames from a frame-based camera, each image frame being generated using an exposure period; generating a first point matrix from one or more of the image frames, the first point matrix being associated with an acquisition period of the image frames; acquiring a pixel event stream generated during the acquisition period; generating a second point matrix from pixel events of the pixel event stream, occurring during the acquisition period of the first point matrix; computing a correlation scoring function applied to at least a part of the points of the first and second point matrices, and estimating respective positions of points of the second point matrix in the first point matrix, due to depths of the points of the first point matrix related to the second point matrix, by maximizing the correlation scoring function.