Low Bit Depth Image Sensor Reconstruction Using Temporal Neural Networks

Solution Overview

Problem

Existing image sensors that detect individual photons, such as quanta image sensors, produce high shot noise due to their short duration, leading to high noise and low bit depth frames that are unsuitable for generating high-quality digital images.

Innovation Solution

A system comprising a trained machine learning model with 3D convolutional layers, 2D convolutional LSTM layers, and concatenation layers is used to process low bit depth frames from image sensors like SPADs, converting them into high bit depth images by leveraging the high frame rate and temporal information from multiple frames.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If quanta image sensors are used to detect individual photons, then read noise is reduced and dynamic range is improved, but shot noise increases due to short duration frames

Engineering Contradiction:

Improveread noiseVSAvoidshot noise

Core Design Contradiction:

ReliabilityVSObject-generated harmful factors

Solution Approach 1:

The system performs preliminary accumulation of multiple low bit depth frames before processing. By collecting N frames (where N≥2) and summing their pixel values, the system prepares accumulated data that has reduced noise characteristics, which is then processed by the neural network to generate the final high bit depth image.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system merges multiple low bit depth frames by summing their pixel values to create an accumulated frame. This combining operation increases the signal-to-noise ratio and effective bit depth, transforming multiple noisy 1-bit frames into a higher quality intermediate representation that can be further processed.

Inventive Principle:

Principle #5Merging (Combining)

2Object-generated harmful factors

If multiple frames are accumulated to reduce noise, then shot noise is reduced, but processing complexity increases

Engineering Contradiction:

Improveshot noiseVSAvoidprocessing complexity

Core Design Contradiction:

Object-generated harmful factorsVSDevice complexity

Solution Approach 1:

The system replaces traditional mechanical or algorithmic noise reduction methods with a trained neural network model. The convolutional LSTM network automatically learns and applies complex noise reduction and bit depth enhancement operations, substituting manual processing complexity with pre-trained intelligent processing.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system changes the processing approach by using a trained neural network model with specific architecture (convolutional LSTM layers) rather than traditional processing methods. The model parameters are optimized during training to handle the specific characteristics of quanta image sensor data, enabling efficient noise reduction and bit depth conversion.

Inventive Principle:

Principle #35Parameter changes

3Productivity

If low bit depth frames are processed directly, then processing speed is maintained, but image quality deteriorates

Engineering Contradiction:

Improveprocessing speedVSAvoidimage quality

Core Design Contradiction:

ProductivityVSManufacturing precision

Solution Approach 1:

The system performs preliminary accumulation of multiple low bit depth frames before processing. By collecting N frames (where N≥2) and summing their pixel values, the system prepares accumulated data that has reduced noise characteristics, which is then processed by the neural network to generate the final high bit depth image.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system merges multiple low bit depth frames by summing their pixel values to create an accumulated frame. This combining operation increases the signal-to-noise ratio and effective bit depth, transforming multiple noisy 1-bit frames into a higher quality intermediate representation that can be further processed.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS20250259270A1Systems, methods, and media for generating digital images using low bit depth image sensor data

Publication Date: 2025.08.14 WISCONSIN ALUMNI RES FOUND

AI summary

In accordance with some embodiments, systems, methods, and media for generating digital images using low bit depth image sensor data are provided. In some embodiments, the system comprises: an image sensor; a processor programmed to: receive, from the image sensor, a series of low bit depth frames; provide low bit depth image information to a trained machine learning model comprising: a 3D convolutional layer; a 2D convolutional LSTM layer; a concatenation layer configured to generate a tensor that includes an output of the 2D convolutional LSTM layer and the low bit depth image information; and a 2D convolutional layer configured to generate an output based on the tensor; and generate a high bit depth image of a scene based on an output of the two-dimensional convolutional layer.