Single-Photon and CMOS HDR Fusion for Saturated Scene Reconstruction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image sensors have a limited dynamic range, leading to challenges in capturing high dynamic range images, especially in scenes with extreme brightness variations, where existing techniques suffer from artifacts like ghosting, light flicker, and spatially non-uniform signal-to-noise ratio dips.
Innovation Solution
A system utilizing a combination of CMOS image sensors and single-photon image sensors, with a machine learning model that processes data from both to generate high dynamic range images, leveraging the high sensitivity and resolution of single-photon sensors to reconstruct extremely bright and saturated regions that conventional techniques struggle with.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If conventional image sensors are used to capture images, then the device complexity is low, but the dynamic range is limited
Solution Approach 1:
The patent combines conventional image sensors and single-photon image sensors into a hybrid imaging system. The conventional sensor captures high-resolution spatial information while the single-photon sensor captures high dynamic range information, and their data are fused through machine learning to achieve both high dynamic range and high resolution simultaneously.
Solution Approach 2:
The patent introduces a machine learning model as an intermediary to process and fuse data from both sensor types. The model learns to combine the strengths of conventional and single-photon sensor data, transforming them into a unified high dynamic range image that neither sensor could produce alone.
2Adaptability or versatility
If single-photon image sensors are used to capture high dynamic range images, then the dynamic range is improved, but the resolution is reduced
Solution Approach 1:
The patent merges the high dynamic range capability of single-photon sensors with the high resolution capability of conventional sensors. By fusing their respective strengths through machine learning, the system achieves both high dynamic range and high resolution in the final image.
Solution Approach 2:
The patent applies different processing strategies to different regions of the image based on their characteristics. In regions where the single-photon sensor provides sufficient signal, it is used for high dynamic range capture, while in regions where resolution is critical, the conventional sensor data is weighted more heavily.
3Adaptability or versatility
If conventional high dynamic range techniques are used, then the device complexity is low, but artifacts like ghosting and light flicker occur
Solution Approach 1:
The machine learning model acts as an intermediary that intelligently fuses data from both sensors, avoiding the artifacts associated with traditional HDR techniques. The model learns to weight and combine the sensor data in a way that eliminates ghosting and light flicker while preserving true scene details.
Solution Approach 2:
The patent employs a machine learning model that is trained to minimize image quality metrics related to artifacts. The model receives feedback during training about the presence of ghosting and light flicker, learning to adjust its fusion strategy to eliminate these artifacts while maintaining high dynamic range performance.
4Reliability
If conventional image sensors are used, then the signal-to-noise ratio is acceptable in normal conditions, but it deteriorates in extreme brightness variations
Solution Approach 1:
The patent combines the linear response and good signal-to-noise ratio of conventional sensors with the high dynamic range capability of single-photon sensors. The machine learning fusion process leverages the strengths of each sensor type across different brightness levels, maintaining high signal-to-noise ratio throughout the entire brightness range.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system effectively produces high dynamic range images with improved signal-to-noise ratios and reduced artifacts, capable of capturing scenes with extreme brightness variations, outperforming traditional methods by maintaining high and uniform signal-to-noise ratios across all brightness levels.
Implementation Method 1
a first plurality of detectors, each configured to detect a level of photons arriving at the detector that is proportional to an incident photon flux at the detector
Implementation Method 2
a second plurality of detectors, each configured to detect arrival of individual photons
Data Source
AI summary
In accordance with some embodiments, systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data are provided. In some embodiments, the system comprises: first detectors configured to detect a level of photons proportional to incident photon flux; second detectors configured to detect arrival of individual photons; a processor programmed to: receive, from the first detectors, first values indicative of photon flux from a scene with a first resolution; receive, from the second detectors, second values indicative of photon flux from the scene with a lower resolution; provide a first encoder of a trained machine learning model first flux values based on the first values, provide the second encoder of the model second flux values; receive, as output, values indicative of photon flux from the scene; and generate a high dynamic range image based on the third plurality of values.