Event Camera Image Reconstruction via Visible Fusion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Event cameras produce intensity images with noise due to environmental illumination changes, affecting the quality of reconstructed images.

Innovation Solution

A method that divides event streams into spatio-temporal windows, selects reference images from visible image sequences, and uses an enhancement module based on an improved Transformer model to perform texture feature migration and enhance the reconstructed images.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If event streams are used for image reconstruction, then high temporal resolution and high dynamic range are achieved, but image noise increases due to environmental illumination changes

Engineering Contradiction:

Improvetemporal resolutionVSAvoidimage noise

Core Design Contradiction:

SpeedVSObject-affected harmful factors

Solution Approach 1:

The patent divides the event stream into multiple spatio-temporal windows, processing events in segmented time intervals. This segmentation allows selective integration of events from different time windows, reducing the impact of illumination changes while preserving temporal resolution benefits

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent combines event stream data with visible image data to create enhanced reconstructed images. By merging the high temporal resolution of event cameras with the noise-resistant properties of visible images, the system achieves both high temporal resolution and reduced image noise

Inventive Principle:

Principle #5Merging (Combining)

2Manufacturing precision

If visible images are used for reference, then image quality is improved, but device complexity increases due to fusion processing requirements

Engineering Contradiction:

Improveimage qualityVSAvoidprocessing complexity

Core Design Contradiction:

Manufacturing precisionVSDevice complexity

Solution Approach 1:

The patent performs preliminary processing of visible images by extracting feature maps before fusion with event stream data. This preliminary action simplifies the subsequent fusion process by preparing standardized feature representations, reducing overall processing complexity while maintaining image quality improvement

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent introduces feature maps as an intermediary representation between visible images and event stream data. This intermediary form facilitates efficient fusion processing by providing a common feature space, reducing the complexity of directly fusing heterogeneous data types

Inventive Principle:

Principle #24Intermediary (Mediator)

Data Source

PatentUS12211179B2Method, medium and device for enhancing event camera image reconstruction by fusing visible images

Publication Date: 2025.01.28 ZHEJIANG GONGSHANG UNIVERSITY

AI summary

The disclosure discloses a method, a medium, and a device for enhancing event camera image reconstruction by fusing visible images, and belongs to the field of image processing. In order to solve the problem that the reconstructed image of the event camera is likely affected by noise and lacks of photorealistic, the method provided by the disclosure adopts an enhancement module based on the improved Transformer model to fuse the intensity image reconstructed from the event stream and the reference image selected from the visible image sequence captured by an optical sensor, so that makes the reconstructed image has an appearance close to the visible image while maintaining the characteristics of high dynamic range of the event camera.