Method and device for image processing, in particular for debayering
The method and device for image processing separate chroma and luminance processing, using artifact detection to selectively apply low-pass filtering, addressing Bayer sensor artifacts and maintaining high resolution and sharpness.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- PROTON CAMERA INNOVATIONS GMBH
- Filing Date
- 2025-12-10
- Publication Date
- 2026-06-18
AI Technical Summary
Existing image processing methods using Bayer sensors suffer from artifacts due to non-uniform patterns and geometric arrangements, leading to disruptive patterns and loss of detail when trying to remove these artifacts, often requiring compromises between resolution and sharpness.
A method and device for image processing that separates chroma and luminance processing, using a controllable low-pass filter to selectively apply filtering based on artifact detection, ensuring high resolution with minimal artifacts by independently optimizing brightness and color values.
Achieves high-resolution images with reduced artifacts by selectively applying low-pass filtering only where needed, maintaining sharpness in non-critical areas while smoothing edges to eliminate artifacts.
Smart Images
Figure IB2025062651_18062026_PF_FP_ABST
Abstract
Description
[0001]
[0002] Proton Camera Innovations GmbH 4763-1 PCT
[0003] Method and apparatus for image processing, in particular for debayering
[0004] The invention relates to a method and a device for image processing, in particular for debayering image signals from a Bayer sensor.
[0005] Bayer sensors are often used for photosensors to capture color images; these consist of a sensor chip with a matrix arrangement of sensor pixels and color filters placed in front of the individual sensor pixels.
[0006] Thus, an R-filter, G-filter, or B-filter of the RGB color values is placed in front of each sensor pixel, with its characteristic filter properties or absorption spectrum. For example, in the matrix, rows with GR arrangement and rows with BG arrangement alternate, so that the G color value is present in 50% of the pixels, and the R and B color values are each present in 25% of the pixels. The vertical columns are arranged in a correspondingly alternating pattern.
[0007] The Bayer sensor outputs a raw data signal, specifically a stream of successive pixel values, which initially represent intensity signals or values corresponding to their color filter. The undetected color values of each pixel are generally subsequently determined by Bayer interpolation from surrounding sensor pixels of other colors. Thus, for example, the green value G and the blue value B for the red color pixel R are determined by interpolation from the surrounding pixels.
[0008] 4763-1 PCT Bre / kl 10.12.2025 Since the Bayer pattern is not uniform, with varying numbers of color values and a geometric arrangement resembling a checkerboard, interpolation can result in disruptive patterns. This occurs, for example, at certain angles or color transitions. These effects, also known as artifacts, are particularly noticeable at sharp edges where the scene reaches the sensor's resolution.
[0009] This effect, or these artifacts, can be removed by blurring the image using a low-pass filter (OLPF), as shown, for example, in US2001 / 0052938A1. Alternatively, these artifacts can also be removed using a digital low-pass filter. To completely remove these artifacts, the image must be reduced to a quarter of its original resolution; however, this results in a loss of detail. The blurring can then be improved by sharpening. Therefore, it is always a matter of finding the best compromise.
[0010] Such Bayer interpolation and debayering with a digital low-pass filter are known from US2006 / 104537A1.
[0011] US 2011 / 52053 A1 describes an image processing device that converts a digital image from initial values into YCrCb color space values, wherein the image has an initial luminance Y layer and two initial Cr,Cb chrominance layers. The image processing device includes a first block that processes and modifies the initial luminance Y layer of the digital image to produce and output a modified luminance layer, and a further color artifact corrector block that operates in parallel with the first block A and modifies the initial values of the luminance in the Y and the CR,Cb chrominance layers through pixel-by-pixel processing.
[0012] 4763-1 PCT Bre / kl 10.12.2025 WO 2018 / 234616 describes a method comprising the following steps: receiving image data representing pixels of a two-dimensional image from an array of photosensors, each photosensor having a color filter to limit the photosensor's sensitivity to a single color range out of three color ranges; furthermore, estimating the luminance of the image data for a first set of pixels limited to a first color range using a first filter; separately estimating the luminance of the image data for a second and third set of pixels limited to the remaining color ranges; applying a second luminance evaluation filter to the first set of pixels; and estimating the luminance for the second and third sets of pixels to estimate an image luminance.
[0013] The invention is therefore based on the objective of creating a method and a device for image processing that enable image processing with high resolution and low artifacts with relatively little effort.
[0014] This problem is solved by a method according to claim 1 and a device according to claim 14. The dependent claims describe preferred embodiments.
[0015] The device according to the invention is particularly intended for carrying out the method according to the invention; the method according to the invention can in particular be carried out with the device according to the invention. The method according to the invention can in particular also be carried out wholly or partially with remote data transmission on a further device.
[0016] In particular, a device for image processing of raw data signals from a Bayer sensor is provided, especially for debayering,
[0017] 4763-1 PCT Bre / kl 10.12.2025, wherein the device comprises: a chroma interpolator configured to receive the raw data and output a chroma signal; a luminance interpolator configured to receive the raw data and output a first luma signal; an artifact detector configured to receive the raw data and generate a control signal; a controllable low-pass filter configured to receive the first luminance signal, wherein the controllable low-pass filter is configured to perform low-pass filtering of the pixel values of the luminance signal depending on the control signal, adjusting the low-pass filtering of the individual luminance values depending on the control signal, wherein the controllable low-pass filter is configured to determine a second luminance signal; and a conversion matrix configured to receive the chroma signal and the second luminance signal and to generate image values from them.
[0018] Thus, in particular, the raw data from the image sensor is processed separately in a chroma interpolator to determine the color values, especially as CbCr values, and separately in a luma interpolator to determine the luma values or brightness values. This alone achieves the advantage that both color values and brightness values can be optimized independently and filtered as needed or according to the inventive method.
[0019] Furthermore, after an initial luma interpolation, a controlled low-pass filtering of the luma values is performed, which is carried out by control signals from an artifact detector.
[0020] The luma values and the control signal with the control signal values are preferably each represented by a corresponding matrix with indexing i, j
[0021] 4763-1 PCT Bre / kl 10.12.2025 represents pixels i, j, i.e., pixel-related correction values are determined and used for low-pass filtering.
[0022] This offers the advantage that the luma values can be processed independently to address the artifacts, without involving the processing of the chroma values.
[0023] The process is characterized in particular by the following properties:
[0024] - The method preferably processes brightness information (luma or y) and color information (chroma) in separate processing steps. After processing, conversion back to RGB is possible.
[0025] - Separate processing allows each part to be optimized independently.
[0026] - The artifact detector identifies the areas in the image that contain disturbances.
[0027] A dynamic artifact filter, controlled by, for example, a difference calculator or as part of the artifact detector, removes artifacts only in the areas where problems exist. Other areas of the image retain maximum sharpness.
[0028] Other preferred qualifications and benefits include:
[0029] Brightness is processed as follows
[0030] The brightness signal or luma signal is interpolated by an adjustable low-pass filter; that is, with lower low-pass filtering, many details remain, while with high low-pass filtering, few details remain in the image.
[0031] The control signal is an image mask that preferably marks critical image areas.
[0032] At critical points, the image becomes less sharp, but this smooths out edges so that no image artifacts are visible.
[0033] 4763-1 PCT Bre / kl 10.12.2025 The image has maximum sharpness in the non-critical areas, without image artifacts.
[0034] According to a particularly preferred embodiment, the control signal is generated by a filtered image as follows: i. magnitude of RB; ii. normalized to Y by 1 / Y; iii. The masking area is optimized by a filter, wherein the filter in particular performs edge detection and optionally creates a blur.
[0035] The invention also includes a program and / or software for carrying out the method, in particular in a suitable device, e.g. a computer and in particular in the apparatus, as well as a data carrier with the stored program and / or software as a program product.
[0036] The invention is explained in more detail below with reference to the accompanying drawings. These show:
[0037] Fig. 1 shows a Bayer sensor,
[0038] Fig. 2 is a block diagram of a method and system for image processing according to the invention;
[0039] Fig. 3 shows another representation of the block diagram with a representation of the respective signals;
[0040] Fig. 4 shows a scheme of an image to be reproduced, as a matrix with image areas of different colors;
[0041] Figs. 5 and 6 show larger versions of the color charts CC1 to CC9 from Fig. 3.
[0042] Figure 1 shows an example of a Bayer sensor 1 as a photosensor, which for illustrative purposes is shown as a 9x9 matrix of individual pixels Pij with a
[0043] 4763-1 PCT Bre / kl 10.12.2025 characteristic color filter or Bayer filter 3 with a matrix of color values; real image sensors are generally larger. Thus, a color value, R, G, or B, with the characteristic filter property or absorption spectrum is provided in front of each pixel 2. In Figure 1, the positions of the matrix are indexed in the usual way, indicating the respective color value. Thus, in this matrix, rows with GR arrangement and rows with BG arrangement alternate, so that the color value G is present in 50% of the pixels, and the color values R and B are each present in 25% of the pixels. A correspondingly alternating arrangement of the vertical columns results.
[0044] The Bayer sensor 1 outputs a raw data signal S1, which is specifically configured as a stream of successive pixel values PWij, thus initially representing intensity signals, and is subsequently acquired and processed in an image processing device 5 according to the invention, so that an image signal S2 with the color values F-Rij, F-Gij, F-Bij is subsequently output. The image processing device 5 specifically performs Bayer interpolation with debayering.
[0045] Figure 2 shows a first embodiment of the device 5 according to the invention, which is described in more detail in Figure 3.
[0046] The pixel values PWij initially represent intensity values corresponding to their color filter. The undetected color values of each pixel Pij are subsequently determined by Bayer interpolation from surrounding sensor pixels of the other color values. Thus, for example, the green value G43 and the blue value B43 for the red color pixel R43 are determined by interpolation from the surrounding pixels.
[0047] The image processing device 5 comprises a chroma interpolator 8 and a luminance interpolator 9, each of which receives the raw data signal S1, i.e., the pixel values PWij. The chroma interpolator 8 performs a
[0048] 4763-1 PCT Bre / kl 10.12.2025 performs color interpolation or chroma interpolation and subsequently outputs a chroma image CbCr, in particular as CbCr data (complementary blue, complementary red), which allows the bandwidth or data volume to be kept low in the usual way.
[0049] The luminance interpolator 9 outputs luma values Y for the individual pixel values in the usual manner, i.e., Y11 to Y99, with this luma signal Y being output to a controllable low-pass filter 12. An artifact detector 10 is provided, which, as shown in more detail in Fig. 3, preferably comprises a difference calculator 15 and an artifact filter 11 and determines an artifact signal AS from the chroma values CbCr, or also from other color values that are determined differently from the raw data, as described in more detail below. This AS is subsequently filtered by the artifact filter 11, which then outputs a control signal CTR1 to a controllable low-pass filter 12 in order to define a low-pass filter for the individual luminance values Yj of the luminance signal Y.
[0050] The control signal CTR1 serves in particular to define the width of the low-pass filter in the luma signal Y for the individual luma values Yij, i.e., the averaging over neighboring luma values Yij. Without low-pass filtering, or "zero low-pass filtering," the luma value Yij is taken directly. For higher low-pass filtering, the directly adjacent luma values Yi-1, j, Yi+1, j, Yi, j+1, Yi, j-1, as well as Yi+1, j+1, ... can be used. For stronger low-pass filtering, further limiting luma values Yij can also be used.
[0051] The controllable low-pass filter 12 subsequently outputs a low-pass filtered luminance signal Y2ij to a conversion matrix 14.
[0052] The artifact detector 10 thus detects artifacts and outputs an artifact signal AS to an artifact filter 11, through edge detection and
[0053] 4763-1 PCT Bre / kl 10.12.2025 Smoothing creates a blurry artifact signal, which is then input as control signal CTR1 to the controllable low-pass filter 12.
[0054] The conversion matrix 14 thus receives the chroma signal C1, or CbCr, and the second luminance signal Y2ij and performs a conversion known as conversion matrix YCbCr in order to generate RGB values of the matrix, i.e., R11 to R99, G11 to G99, B11 to B99, of the 9x9 matrix shown in Figure 1.
[0055] Thus, the luma values or brightness information Y and the chroma values or color information CbCr are interpolated and determined in separate processing steps, with the controllable low-pass filter 12 being used additionally in the processing of the luma values.
[0056] The artifact detector 10 detects the areas in the image that contain disturbances.
[0057] The control signal CTR1 determined according to Figure 2 is formed in particular as a fuzzy artifact signal.
[0058] For illustration, Fig. 4 shows an image as a panel with twenty-four different colors and brightness levels, which is subsequently processed in the following figures. Here, artifacts and blurring can occur, particularly at the edges or transitions between the monochrome square tiles.
[0059] Figure 4 shows the chroma signal CbCr as an image of the recorded panels with image areas.
[0060] The control signal CTR1 thus represents an image mask that marks critical image areas. At the critical points, which are brighter in Figure 6,
[0061] 4763-1 PCT Bre / kl 10.12.2025: The image becomes less sharp due to stronger low-pass filtering, which smooths edges and thus reduces or completely prevents image artifacts. The dark areas in Figure 6 are correspondingly non-critical areas where the image shows maximum sharpness without image artifacts and are therefore subjected to little or no low-pass filtering.
[0062] According to a preferred embodiment, the control signal
[0063] CTR1 in the artifact detector 10, i.e., the difference calculator 15 and the artifact filter 11, is formed by the following calculation: The difference Rij - Bij is determined in the difference calculator 15, followed by normalization to the Y-value Y, i.e., division by Y:
[0064] (R - B) / Y. (Equation 1)
[0065] In the Arefakt filter 11, these values are converted into a masking area through edge detection and smoothing.
[0066] 4763-1 PCT Bre / kl 10.12.2025 Reference List
[0067] 1 Bayer sensor
[0068] 2 sensor pixels, Bayer pixels
[0069] 3 Bayer filters
[0070] 5 Image processing device
[0071] 8 Chroma Interpolator
[0072] 9 Luminescence Interpolator
[0073] 10 Artifact Detector
[0074] 12 controllable low-pass filters
[0075] 11 Artifact Filters
[0076] 14 Conversion matrix YCbCr to RGB
[0077] 15 Difference Calculator
[0078] CC1 Color Chart of S1 = (PWij)
[0079] CC2 Color Chart by Cb=Cbij
[0080] CC3 Color Chart by Cr=Crij
[0081] CC4 Color Chart by AS=ASij
[0082] CC5 Color Chart by CTR
[0083] CC6 Color Chart by Yij
[0084] CC7 Color Chart by F-Rij
[0085] CC8 Color Chart of F-Gij
[0086] CC9 Color Chart of F-Bij
[0087] 4763-1 PCT Bre / kl 10.12.2025
Claims
Patent claims 1. Methods for image processing, in particular for debayering, comprising at least the following steps: Providing raw data (S1) output by a Bayer sensor (1) with a matrix of pixels (Pij) and containing individual pixel values (PWij), Color interpolation from the raw data (S1) in a chroma interpolator (8) and determination of a chroma signal (CbCr-ij), Luminance interpolation from the raw data (S1 ) in a luminance interpolator (9) and determination of a first luma signal (Y1 ij) containing individual luminance values (Y1 ij), Detection of artifacts in an artifact detector (10) and determination of a control signal (CTR1 -ij), controllable low-pass filtering of the first luminance signal (Y1 ij) with the control signal (CTR1 ), adapting the low-pass filtering of the individual luminance values (Yij) depending on the control signal (CTR1 ), and determination of a second luminance signal (Y2ij), Receiving the chroma signal (CbCr-ij) and the second luminance signal (Y2ij) in a conversion matrix (14) and determining image values (F-Rij, F-Gij, F-Bij) to the pixel values (Pij).
2. Method according to one of the preceding claims, characterized in that the raw data (S1 ) are currently generated by the Bayer sensor (1 ) and processed directly.
3. Method according to one of the preceding claims, characterized in that the pixel values (PWij) of the raw data (S1) each contain intensity values and information about the upstream color filter (G11 , R21 , ... G99). 4763-1 PCT Bre / kl 10.12.2025 4. Method according to one of the preceding claims, characterized in that the detection of artifacts is carried out from: - the chroma signal (CbCr-ij), or - the chroma signal and the first luminance signal (Y1 ij), or - from a further chroma signal determined from the raw data (S1) with or without the luminance signal (Y1 ij).
5. Method according to one of the preceding claims, characterized in that in the step of detecting artifacts and determining a control signal (CTR1 -ij), relevant areas, preferably with high color differences, are first detected in the artifact detector (10), preferably in a difference generator (15) of the artifact detector (10), and the artifact detector (10), preferably the difference generator (15), determines an artifact signal (ASij) and a subsequent artifact filter (11), which is preferably part of the artifact detector (10), highlights and / or smooths the relevant areas, in particular edges, in the artifact signal (ASij), thereby forming the control signal (CTR1 -ij).
6. Method according to claim 5, characterized in that the artifact detector (10) comprises a difference calculator (15) and a subsequent artifact filter (11), wherein the difference calculator (15) detects the relevant areas, in particular high color differences, and determines an artifact signal (ASij), and the artifact signal (ASij) in the subsequent artifact filter (11) highlights and / or smooths the relevant areas, in particular edges, forming the control signal (CTR1 -ij). 4763-1 PCT Bre / kl 10.12.2025 7. Method according to one of the preceding claims, characterized in that the controllable low-pass filter (12) performs a low-pass filtering width, in particular the extent of the inclusion of neighboring pixels (Pij), depending on the control signal (CTR1 ).
8. Method according to one of the preceding claims, characterized in that in the second luminance signal (Y2ij) affected areas are unsharply masked by stronger low-pass filtering.
9. Method according to one of the preceding claims, characterized in that in the artifact detector (10), preferably the difference generator (15) and the artifact filter (11), the control signal (CTR1 -ij) for a pixel (Pij) is determined on the basis of: - the amount of the difference between the R-value and B-value of the pixel value (PWij), or the amount of the difference between the ground Cb-value of the pixel value (PWij), and - a normalization to the brightness value (Yij).
10. Method according to claim 9, characterized in that in the artifact detector (10), preferably in the artifact filter (11) of the artifact detector (10), the control signal (CTR1 ij) for a pixel Pij is determined from the quotient (Rij - Bij) / Yij, with Rij as the red value of pixel Pij and Bij as the blue value of pixel Pij, or the quotient (Cr-ij - Cb-ij) / Yij, with Cr-ij as the red-green difference signal of pixel Pij and Cb-ij as the blue-green difference signal of pixel Pij.
11. Method according to claim 9 or 10, characterized in that the control signal (CTR1 -ij) for a pixel value (Pij) without including 4763-1 PCT Bre / kl 10.12.2025 The G-value of the pixel value (Pij) is determined.
12. Method according to one of the preceding claims, characterized in that the chroma signal (CrCbij) is determined solely on the basis of the raw data (S1 ), without including the first or second luma signal, and / or the first luma signal (Y1 ij) is determined solely on the basis of the raw data (S1 ), without including the chroma signal (CrCbij).
13. Method according to one of the preceding claims, characterized in that the chroma signal (CbCr-ij) is determined as a CbCr signal in the chroma interpolator (8).
14. Device (5) for image processing of raw data signals (S1) of a Bayer sensor (1), in particular for debayering, wherein the device (5) comprises: a chroma interpolator (8) configured to receive the raw data (S1) and output a chroma signal (CbCr-ij), a luminance interpolator (9) configured to receive the raw data (S1) and output a first luma signal (Y1 ij), an artifact detector (10) configured to receive the raw data (S1) and generate a control signal (CTR1), a controllable low-pass filter (12) configured to receive the first luminance signal (Y1 ij) and wherein the controllable low-pass filter (12) is configured to perform low-pass filtering of the pixel values of the luminance signal (Y1) depending on the control signal (CTR), with adjustment of the low-pass filtering the individual luminance values (Yij) as a function of the control signal (CTR1) , wherein the controllable low-pass filter (12) is formed,to determine a second luminance signal (Y2ij), and, 4763-1 PCT Bre / kl 10.12.2025 a conversion matrix (14) configured to receive the chroma signal (CbCr) and the second luminance signal (Y2) and to generate image values (F-Rij, F-Gij, F-Bij) from them.
15. Device (5) according to claim 14, characterized in that the artifact detector (10) comprises: a difference processor (15) configured to generate the artifact signal (ASij), and an artifact filter (11) configured to receive the artifact signal (AS) and to generate the control signal (CTR1).
16. Device (5) according to claim 15, characterized in that the artifact filter (11 ) is configured to form the control signal (CTR1 ) by edge detection and smoothing. 4763-1 PCT Bre / kl 10.12.2025