RGB-IR ISP Pipeline Using IR Control for Low-Light Image Quality
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional image signal processors (ISPs) that process RGB-IR image data neglect the valuable information in the infrared (IR) channel, leading to degraded signal-to-noise ratio (SNR) and inaccurate image rendering due to the omission of IR data, which is crucial for scene structure and detail representation.
Innovation Solution
An ISP pipeline that incorporates an IR-based control channel to propagate IR data through image color data channel processing stages, utilizing spatial and signal correlation to enhance visible wavelength color data processing, including the use of a pseudo-radiometric (PR) channel and image modulators derived from IR data to adjust visible wavelength channels.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If IR data is excluded from color channels to avoid contamination and noise, then color image processing purity is improved, but signal-to-noise ratio and image quality deteriorate
Solution Approach 1:
The patent separates IR data processing into a dedicated control channel that is distinct from the visible color data channels. The IR data is processed separately to generate image modulators, which then control the processing of visible wavelength channels. This segmentation allows IR information to be utilized without contaminating the color channels, resolving the contradiction between purity and signal-to-noise ratio.
Solution Approach 2:
The patent introduces image modulators as an intermediary element that translates IR data into control signals for visible channel processing. These modulators act as a mediator between the IR control channel and visible color channels, allowing IR information to influence visible image processing without direct mixing of data streams, thereby maintaining color purity while leveraging IR's high SNR.
2Reliability
If IR data is propagated through color data channel processing stages, then image quality and machine perception functions are improved, but device complexity increases
Solution Approach 1:
The ISP pipeline is segmented into distinct processing paths: an IR control channel for processing infrared data to generate image modulators, and separate visible wavelength color data channels for color image processing. This segmentation allows the system to handle complex multi-channel processing in an organized manner, improving image quality while managing device complexity through structured architecture.
Solution Approach 2:
The image modulators generated from IR data serve multiple functions within the processing pipeline, controlling various stages of visible channel processing including demosaicing, noise reduction, and tone mapping. This multi-functionality allows a single IR processing path to benefit multiple processing stages, improving overall image quality without proportionally increasing complexity.
3Device complexity
If conventional ISP processing is used without IR integration, then device complexity is reduced, but image rendering accuracy and machine perception functions deteriorate
Solution Approach 1:
Image modulators serve as intermediaries that carry processed IR information through the visible channel processing stages. These modulators enable accurate image rendering by providing IR-derived structural and luminance information to control various processing operations, thereby improving measurement precision without requiring complete redesign of the processing pipeline.
Solution Approach 2:
The system implements feedback mechanisms where IR data is processed to generate image modulators that continuously control and adjust visible channel processing. This feedback loop ensures that processing decisions are informed by IR information, improving image rendering accuracy and machine perception functions while maintaining a manageable processing architecture.
Data Source
AI summary
In various examples, image signal processing (ISP) using infrared (IR) image-based control for visible wavelength color data channel processing is presented. Image color data channel processing stages of an ISP pipeline are configured to exploit IR information represented in an IR-based control channel so that IR information may be used to enhance the image processing for human visualization and machine perception functions. In some embodiments, a pseudo-radiometric (PR) channel may be formed based on the color data and IR data channels and propagated through the ISP pipeline. An IR-based control channel may carry control data computed based on luma data, IR channel data, PR channel data, image statistics, or combinations thereof. Image modulators may be derived from the control channel, and the image modulator used by one or more of the image color data channel processing stages of the ISP pipeline to adjust the visible wavelength color data channels.