Neuromorphic Compressive Sensing for Low-Light Image Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional sensors lose light information in low-light conditions due to saturation and sensitivity issues, leading to reduced image quality and untapped sparsity in neuromorphic vision (NMV) sensor outputs, which current compressive sensing methods are unable to effectively exploit.
Innovation Solution
A neuromorphic vision system with an array of NMV sensors that accumulate light until a threshold is reached, outputting binary signals, combined with a compressive sensing and reconstruction engine that processes spatiotemporal spike patterns to reconstruct images and videos, utilizing data-driven matrices and optimization techniques to recover hidden sparsity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If conventional sensors increase sensitivity to operate in low-light conditions, then light information capture is improved, but sensor saturation occurs more frequently causing loss of light information and noisy signals
Solution Approach 1:
The patent employs dynamic range compression by mapping the wide dynamic range of NMV sensor outputs to a compressed range suitable for compressive sensing reconstruction. This dynamic mapping allows the system to handle varying light intensities without saturation while maintaining signal quality in low-light conditions.
Solution Approach 2:
The system changes the parameter representation from conventional continuous intensity values to binary spike events with temporal coding. This parameter transformation enables the exploitation of sparsity in the temporal domain, allowing reliable reconstruction from fewer measurements and improving signal quality in low-light environments.
2Measurement precision
If NMV sensors output binary signals to reduce noise and energy consumption, then measurement precision is improved, but image reconstruction quality deteriorates due to untapped sparsity in spatial and temporal domains
Solution Approach 1:
The patent transitions from spatial-only reconstruction to spatiotemporal reconstruction by adding the temporal dimension. Video sequences provide temporal redundancy that, when combined with spatial sparsity, enables high-quality image reconstruction from compressed binary spike data that would be insufficient when considered spatially alone.
Solution Approach 2:
The system transforms the binary spike data into a sparse temporal representation suitable for compressive sensing. By encoding temporal information in the spike train patterns and exploiting the sparsity of natural video in appropriate bases, the system recovers full image quality despite the binary nature of the sensor outputs.
3Measurement precision
If polarization filters are used with NMV sensors to enhance image reconstruction, then image quality is improved, but light intensity is reduced making the environment more challenging
Solution Approach 1:
The patent maintains continuous accumulation of light energy in the NMV sensors despite the reduced intensity from polarization filters. The event-based integration mechanism continuously sums photons over time until a threshold is reached, allowing the system to compensate for reduced light intensity through temporal integration without sacrificing image reconstruction quality.
Solution Approach 2:
The system dynamically adjusts the effective integration time based on the reduced light intensity. By exploiting temporal sparsity and the dynamic range of NMV sensors, the system adapts to the lower photon flux from polarized light while maintaining reconstruction quality through compressive sensing techniques that are robust to varying signal rates.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables high-quality image and video reconstruction in low-light environments by capturing all available light information, reducing noise, and exploiting untapped sparsity, while eliminating the need for expensive and fragile high-sensitivity sensors.
Implementation Method 1
Passive vision in low-light environment is required for many military applications... Conventional sensors (e.g., complementary metal oxide semiconductor (CMOS) image sensors) used in vision devices lose light information when operating in low-light conditions... In comparison to conventional sensors, event-based (neuromorphic vision (NMV)) sensors accumulate light energy. The accumulated light energy is integrated until the integration reaches some fixed threshold δ. Upon reaching the threshold, a fixed binary signal is output
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is provided for reconstructing images or video from output of neuromorphic vision (NMV) sensors in a low-light environment. The method includes passively sensing light by an array of NMV sensors in the low-light environment, integrating the sensed light by each of the NMV sensors, outputting a time-stamped event signal per sensor of the array of NMV sensors upon a value of the integration exceeding a threshold value, resetting each NMV sensor after outputting an event signal for new integration of sensed light, combining the event signals, and reconstructing an image and/or video based on the combined event signals.