RGB-IR Sensor Processing With Day-Night IR Separation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
RGB-Infrared (RGB-IR) sensors present challenges for existing image signal processors due to their unique 4x4 color filter array pattern, leading to issues like clipping and banding artifacts, especially in night mode IR illuminated images, which do not map well to current processing systems.
Innovation Solution
A method and system for processing RGB-IR sensor data that determines whether to process in day mode or night mode, generating an infrared-subtracted raw Bayer image or an IR image, using a system-on-a-chip with an image signal processor (ISP) and a color filter array interpolation component, which includes filters for interpolation and IR subtraction, and employs thresholding and gain adjustments to mitigate artifacts and improve image quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RGB-IR sensor data is processed using conventional image signal processors designed for standard Bayer CFA, then the processing pipeline can operate with existing hardware, but image quality deteriorates due to clipping and banding artifacts especially in night mode
Solution Approach 1:
The patent segments the processing pipeline into distinct day mode and night mode pathways. In day mode, IR subtraction is performed to remove infrared contamination from visible color channels. In night mode, the pipeline selectively processes only IR pixels to generate high-quality infrared images, separating the handling of different pixel types to avoid artifacts.
Solution Approach 2:
The patent applies different processing strategies to different regions and pixel types within the sensor array. IR pixels are processed differently from visible color pixels, with dedicated interpolation and processing pipelines for infrared data, allowing optimal quality for each pixel type without compromising the other.
2Measurement precision
If the raw RGB-IR image is processed in day mode with IR subtraction, then color fidelity is improved, but processing complexity increases compared to standard Bayer images
Solution Approach 1:
The patent extracts the infrared component from the raw sensor data and removes it from the visible color channels through IR subtraction. This separation allows the visible color processing to proceed independently without infrared contamination, improving color fidelity while managing complexity through systematic data separation.
Solution Approach 2:
The patent performs preliminary IR subtraction or IR extraction before the main image processing pipeline. By removing or separating the infrared component early in the processing flow, subsequent processing steps can focus on visible color data without the added complexity of handling mixed RGB-IR data throughout the entire pipeline.
3Productivity
If all pixels including IR pixels are used for visible image generation, then sensor data utilization is maximized, but image quality deteriorates due to infrared contamination in visible channels
Solution Approach 1:
The patent extracts the infrared component from pixels that would otherwise be used for visible image generation. By separating the IR signal from visible color pixels, the full sensor array can be utilized for visible imaging while removing infrared contamination that would otherwise degrade color accuracy.
Solution Approach 2:
The patent converts the potentially harmful infrared contamination into a benefit by using the same sensor array to simultaneously capture both visible and infrared information. The IR pixels and IR content in visible pixels are processed to generate dedicated infrared images, while IR subtraction cleans the visible channels, making the infrared sensitivity a dual-purpose feature rather than a detriment.
4Ease of operation
If conventional CFA interpolation is applied to RGB-IR data, then the processing follows standard pipelines, but the additional IR pixel plane causes mapping issues and processing errors
Solution Approach 1:
The patent segments the CFA interpolation process into separate handling for visible color pixels and infrared pixels. By treating IR pixels as a distinct data plane rather than attempting to force them into standard Bayer interpolation routines, the system maintains compatibility with existing pipelines for visible data while developing specialized processing for the infrared component.
Data Source
AI summary
A method for processing RGB-Infrared (RGB-IR) sensor data is provided that includes receiving a raw RGB-IR image, determining whether to process the raw RGB-IR image in day mode or night mode, generating, when day mode is determined, an infrared (IR) subtracted raw Bayer image from the raw RGB-IR image and processing the IR subtracted raw Bayer image in an image signal processor (ISP), and generating, when night mode is determined, an IR image from the raw RGB-IR image.