Interleaved CMOS Sensor HDR Pipeline

Solution Overview

Problem

CMOS image sensors in mobile devices have a limited dynamic range, leading to challenges in capturing images with high dynamic range (HDR) as they either saturate in bright areas or lose details in darker areas, and existing HDR techniques often sacrifice spatial resolution.

Innovation Solution

An interleaved image sensor captures images with two different exposure times simultaneously, and an image processing pipeline identifies subsets of pixels to generate HDR data by filtering and scaling values from each subset, maintaining spatial resolution while expanding the dynamic range.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Illumination intensity

If a single exposure time is used to capture an image, then the image sensor can capture details in either bright or dark areas, but cannot capture both simultaneously due to limited dynamic range

Engineering Contradiction:

Improvedynamic rangeVSAvoidloss of detail in bright or dark areas

Core Design Contradiction:

Illumination intensityVSLoss of information

Solution Approach 1:

The image sensor is divided into multiple pixel groups, where some pixels are exposed for a first exposure time optimized for bright areas while other pixels are exposed for a second exposure time optimized for dark areas. This segmentation allows simultaneous capture of both bright and dark region details, resolving the dynamic range limitation of single-exposure sensors.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a temporal dimension by using multiple exposure times simultaneously. Instead of capturing a single image with one exposure time, the system captures multiple images with different exposure times and combines them computationally to achieve extended dynamic range while maintaining spatial resolution.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Illumination intensity

If multiple images with different exposure times are captured and combined, then high dynamic range is achieved, but spatial resolution is reduced

Engineering Contradiction:

Improvedynamic rangeVSAvoidspatial resolution

Core Design Contradiction:

Illumination intensityVSManufacturing precision

Solution Approach 1:

The image sensor is divided into multiple pixel groups, where some pixels are exposed for a first exposure time optimized for bright areas while other pixels are exposed for a second exposure time optimized for dark areas. This segmentation allows simultaneous capture of both bright and dark region details, resolving the dynamic range limitation of single-exposure sensors.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent merges multiple images with different exposure times by combining data from different pixel groups that captured the same scene at different exposure times. This merging process preserves spatial resolution by maintaining the full pixel array structure while achieving extended dynamic range through computational combination.

Inventive Principle:

Principle #5Merging (Combining)

3Measurement precision

If exposure time is extended to capture details in dark areas, then dark area details become visible, but bright areas become overexposed

Engineering Contradiction:

Improvedetail capture in dark areasVSAvoidoverexposure in bright areas

Core Design Contradiction:

Measurement precisionVSIllumination intensity

Solution Approach 1:

The image sensor is divided into multiple pixel groups, where some pixels are exposed for a first exposure time optimized for bright areas while other pixels are exposed for a second exposure time optimized for dark areas. This segmentation allows simultaneous capture of both bright and dark region details, resolving the dynamic range limitation of single-exposure sensors.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different regions of the image sensor are assigned different exposure times based on their specific requirements. Some pixel groups use longer exposure times for dark area capture while other pixel groups use shorter exposure times for bright area capture, allowing each region to be optimized for its specific lighting conditions.

Inventive Principle:

Principle #3Local quality

Applied Scientific Principles

This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.

Function Achieved in This Case

The solution effectively generates HDR images with maintained spatial resolution, balancing exposure times to capture detailed information in both bright and dark areas without sacrificing resolution.

Implementation Method 1

Each pixel site in the CMOS image sensor functions like a capacitor, capturing photons focused on the image sensor by a lens during an exposure and building up a charge

Methodology Applied

Scientific EffectPhotoelectric Effect: Photoelectric Effect

Data Source

PatentUS9071765B2System, method, and computer program product implementing an image processing pipeline for high-dynamic range images

Publication Date: 2015.06.30 NVIDIA CORP

AI summary

A system, method, and computer program product for generating high-dynamic range image data is disclosed. The method includes the steps of receiving image sensor data from an interleaved image sensor. The interleaved the image sensor includes a first portion of pixels exposed for a first exposure time and a second portion of pixels exposed for a second exposure time that is shorter than the first exposure time. The method further includes the steps of identifying a first subset of pixels in the second portion having an intensity value above a first threshold value, identifying a second subset of pixels in the first portion having an intensity value below a second threshold value, and generating high-dynamic range (HDR) data based on the first subset and the second subset.