Image sensor, electronic device, and image processing method

Abstract

The present application discloses an image sensor, an electronic device, and an image processing method. The image sensor provided in the present application comprises: a pixel array and a color filter array; the pixel array comprises at least one multispectral unit, and the multispectral unit comprises at least two first-type pixels for image imaging and at least one second-type pixel for multispectral imaging; the color filter array comprises a filter unit arranged opposite to the multispectral unit, the filter unit comprises at least two first-type filter elements and at least one second-type filter element, the first-type filter elements are arranged opposite to the first-type pixels; the second-type filter element is arranged opposite to the second-type pixel; and the first-type filter elements and the second-type filter element are transmissive to different wavelength ranges of light, and the at least one second-type filter element includes at least one filter element.

Description

Image sensors, electronic devices and image processing methods

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510385341.3, filed on March 28, 2025, entitled "Image Sensor, Electronic Device and Image Processing Method", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of camera device technology, specifically relating to an image sensor, electronic device and image processing method. Background Technology

[0004] With the rapid development of mobile phones and other electronic devices, the types of sensors integrated into these devices are also increasing. Electronic devices generally use image sensors for taking pictures and videos. In order to optimize the image quality of smartphones under different lighting conditions and improve color reproduction and performance, multispectral sensors that can identify the spectral characteristics of the environment have become an indispensable feature of smartphones.

[0005] Currently, in scenarios where multispectral sensors assist image sensors in image capture, the environmental spectral features identified by the multispectral sensor are used to adjust the images captured by the image sensor, improving color reproduction and performance. While image sensors use a single imaging camera for taking pictures, multispectral sensors use a separate camera to capture spectral information across multiple wavelength ranges, leading to increased hardware power consumption. Summary of the Invention

[0006] This application provides an image sensor, an electronic device, and an image processing method. The image sensor can perform both image imaging and multispectral imaging functions, eliminating the need for an additional multispectral sensor to identify multispectral information and reducing hardware power consumption.

[0007] In a first aspect, this application provides an image sensor, comprising: a pixel array and a color filter array; the pixel array includes at least one multispectral unit, the multispectral unit including at least two first-type pixels for image imaging and at least one second-type pixel for multispectral imaging; the color filter array includes filter units disposed opposite to the multispectral unit, the filter units including at least two first-type filter elements and at least one second-type filter element, the first-type filter elements being disposed opposite to the first-type pixels; the second-type filter elements being disposed opposite to the second-type pixels; the first-type filter elements and the second-type filter elements having different transmittance wavelength ranges, and at least one second-type filter element including at least one filter element.

[0008] In a second aspect, this application provides an electronic device including an image sensor as provided in the first aspect.

[0009] Thirdly, this application provides an image processing method executed by the electronic device provided in the second aspect. The image processing method includes: acquiring a preview image output by an image sensor, the preview image including a first type of pixel information and a second type of pixel information; and performing image processing on the preview image based on the first type of pixel information and the second type of pixel information; wherein the first type of pixel information is pixel data output by the first type of pixel; and the second type of pixel information is pixel data output by the second type of pixel.

[0010] Fourthly, this application provides an image processing method executed by the electronic device provided in the second aspect. The image processing method includes: acquiring second type of pixel information from a preview image output by an image sensor; the preview image includes first type of pixel information and second type of pixel information; wherein the first type of pixel information is pixel data output by the first type of pixels; the second type of pixel information is pixel data output by the second type of pixels; the second type of pixel information includes X types of pixel information, each type of pixel information corresponding to a color channel; X is an integer greater than 1; obtaining the phase difference of the X types of color channels based on the X types of pixel information; determining a focus offset based on the phase difference of the X types of color channels; and focusing on a target object in the preview image based on the focus offset.

[0011] In embodiments of this application, the image sensor includes a pixel array and a color filter array; the pixel array includes at least one multispectral unit, the multispectral unit includes at least two first-type pixels for image imaging and at least one second-type pixel for multispectral imaging; the color filter array includes filter units disposed opposite to the multispectral unit, the filter units include at least two first-type filter elements and at least one second-type filter element, the first-type filter elements are disposed opposite to the first-type pixels; the second-type filter elements are disposed opposite to the second-type pixels; the first-type filter elements and the second-type filter elements can transmit light wavelength ranges that are different, and at least one second-type filter element includes at least one filter element. Thus, in the image sensor, the multispectral unit includes a first type of pixels for image imaging and a second type of pixels for multispectral imaging. The filter unit includes a first type of filter element disposed opposite to the first type of pixels and a second type of filter element disposed opposite to the second type of pixels. The image sensor realizes image imaging through the color channel corresponding to the first type of filter element, and realizes multispectral imaging by recognizing multispectral information through the color channel corresponding to the second type of filter element. This allows the image sensor to perform both image imaging and multispectral imaging functions without the need for an additional multispectral sensor to recognize multispectral information, thereby reducing hardware power consumption. Attached Figure Description

[0012] Figure 1 is a schematic diagram of an image sensor provided in some embodiments of this application;

[0013] Figure 2 is a schematic diagram of an image sensor provided in some embodiments of this application;

[0014] Figure 3A is a schematic diagram of the smallest unit of an image sensor provided in some embodiments of this application;

[0015] Figure 3B is a schematic diagram of a filter unit provided in some embodiments of this application;

[0016] Figure 3C is a schematic diagram of a filter unit provided in some embodiments of this application;

[0017] Figure 3D is a schematic diagram of a filter unit provided in some embodiments of this application;

[0018] Figure 4A is a schematic diagram of a filter unit provided in some embodiments of this application;

[0019] Figure 4B is a schematic diagram of a filter unit provided in some embodiments of this application;

[0020] Figure 4C is a schematic diagram of a filter unit provided in some embodiments of this application;

[0021] Figure 4D is a schematic diagram of a filter unit provided in some embodiments of this application;

[0022] Figure 4E is a schematic diagram of a filter unit provided in some embodiments of this application;

[0023] Figure 5A is a schematic diagram of a color filter array provided in some embodiments of this application;

[0024] Figure 5B is a schematic diagram of a color filter array provided in some embodiments of this application;

[0025] Figure 6 is a schematic diagram of an electronic device provided in some embodiments of this application;

[0026] Figure 7A is a schematic flowchart of an image processing method provided in some embodiments of this application;

[0027] Figure 7B is a schematic flowchart of an image processing method provided in some embodiments of this application;

[0028] Figure 7C is a schematic diagram of a virtual multispectral channel diagram provided in some embodiments of this application;

[0029] Figure 8 is a schematic flowchart of an image processing method provided in some embodiments of this application.

[0030] Explanation of reference numerals in the attached figures:

[0031] 10 - Image sensor; 100 - Pixel array; 110 - Multispectral unit; 111 - First type pixel; 112 - Second type pixel; 200 - Color filter array; 210 - Filter unit; 211 - First type filter element; R - Red filter element; G - Green filter element; Gr - Greenish-red filter element; Gb - Greenish-blue filter element; B - Blue filter element; 212 - Second type filter element; C - Cyan filter element; Y - Yellow filter element; M - Magenta filter element; P - Violet filter element; O - Orange filter element; W - All-pass filter element; 220 - Smallest unit of image sensor; 300 - Microlens array; 301 - Microlens; 600 - Electronic device. Detailed Implementation

[0032] The embodiments of this application will now be described in detail. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0033] The terms "first" and "second" in the specification and claims of this application may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise stated, "multiple" means two or more. Furthermore, "and / or" in the specification and claims indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0034] In the description of this application, it should be understood that the terms "inner" and "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0035] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "connected" and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0036] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0037] The terminology used in the embodiments of this application will be explained below.

[0038] Image sensor: A device that converts light signals into electrical signals and generates images based on the electrical signals. An image sensor may include a pixel array, a color filter array, and a microlens array stacked sequentially. The pixel array may include a first type of pixel and a second type of pixel, and the color filter array may include a first type of filter element and a second type of filter element.

[0039] Pixels can be divided into first-class pixels and second-class pixels. A pixel is the smallest element in a pixel array.

[0040] First-class pixels: First-class pixels are used for image formation. For example, an image is synthesized using pixel data output from first-class pixels. In practical applications, during image synthesis, first-class pixels can be considered as valid pixels, while second-class pixels can be treated as dead pixels.

[0041] Type II pixels: Type II pixels are used for multispectral imaging, which refers to imaging using multiple spectral information. For example, imaging can be performed using the multiple spectral information output by Type II pixels. In practical applications, during multispectral imaging, the multiple spectral information output by Type II pixels can effectively improve the color reproduction of the image.

[0042] Type I filter element: The type I filter element is the filter element corresponding to the type I pixel. The type I pixel can sense the light transmitted through the type I filter element.

[0043] Type II filter elements: Type II filter elements are filter elements corresponding to Type II pixels. Type II pixels can sense light passing through Type II filter elements.

[0044] The image sensor, electronic device, and image processing method provided in the embodiments of this application are described in detail below with reference to the accompanying drawings.

[0045] Figure 1 is a schematic structural diagram of an image sensor provided in an embodiment of this application.

[0046] As shown in FIG1, an embodiment of this application provides an image sensor 10, which may include: a pixel array 100 and a color filter array 200; the pixel array 100 may include at least one multispectral unit 110, the multispectral unit 110 including at least two first-type pixels 111 for image imaging and at least one second-type pixel 112 for multispectral imaging; the color filter array 200 includes a filter unit 210 disposed opposite to the multispectral unit, the filter unit 210 including at least two first-type filter elements 211 and at least one second-type filter element 212, the first-type filter element 211 being disposed opposite to the first-type pixels 111; the second-type filter element 212 being disposed opposite to the second-type pixels 112; the first-type filter element 211 and the second-type filter element 212 have different transmittance wavelength ranges, and at least one second-type filter element 212 includes at least one filter element.

[0047] In the embodiments of this application, when the second type of filter element 212 includes multiple filter elements, each filter element corresponds to a range of transmittable light wavelengths, and the range of transmittable light wavelengths of the multiple filter elements are different.

[0048] In some embodiments of this application, as shown in FIG2, the pixel array 100 and the color filter array 200 can be stacked. The pixel array 100 may include multiple pixels, and the color filter array 200 may include multiple filter elements, with one pixel corresponding to one filter element.

[0049] For example, as shown in Figure 2, an image sensor may include a pixel array 100, a color filter array 200, and a microlens array 300 stacked sequentially. In the microlens array 300, each microlens 301 is used to better collect light; in the color filter array 200, filter elements such as first-type filter elements 211 and second-type filter elements 212 are used to filter light of different colors; in the pixel array 100, pixels such as first-type pixels 111 and second-type pixels 112 can sense the light transmitted through the filter elements, perform photoelectric conversion processing, and output an electrical signal.

[0050] In the pixel array 100, pixels such as the first type pixel 111 and the second type pixel 112 can adopt a typical four-transistor pinned photodiode (4T-PPD) pixel structure. The 4T-PPD pixel structure includes a sensing area of ​​one PPD and four transistors. Its working principle includes the following six steps: Step 1, Exposure. Electron-hole pairs generated by light irradiation separate due to the presence of the PPD's electric field; electrons move to the n-region, and holes move to the p-region. Step 2, Reset. At the end of exposure, the RST transistor is activated, resetting the readout area, i.e., the n+ region of the PPD, to a high level. Step 3, Reset Level Readout. After reset, the reset level is read out, including the op-amp's offset noise, 1 / f noise, and kTC noise introduced by the reset. The readout signal is stored in the first capacitor. Step 4, Charge Transfer. The TX transistor is activated, completely transferring charge from the photosensitive area to the n+ region for readout; this mechanism is similar to charge transfer in a CCD. Step 5, Signal Level Readout. The voltage signal in the n+ region is read out to the second capacitor. This signal includes: the signal generated by photoelectric conversion, the offset generated by the operational amplifier, 1 / f noise, and kTC noise introduced by the reset. Step six: Signal output. The signals stored in the two capacitors are subtracted, and the resulting signal is then amplified analogally and sampled by an analog-to-digital converter (ADC) for digital signal output.

[0051] Of course, in the pixel array 100, pixels such as the first type of pixel 111 and the second type of pixel 112 can also adopt other pixel circuit structures, and this application does not impose specific restrictions on them.

[0052] In this embodiment, the pixel array 100 may include at least one multispectral unit 110, and the color filter array 200 may include at least one filter unit 210. One multispectral unit 110 corresponds to one filter unit 210, and the number of multispectral units 110 is the same as the number of filter units 210. Furthermore, one pixel in one multispectral unit 110 corresponds to one filter element in one filter unit 210.

[0053] In the embodiments of this application, for a microlens array, one microlens can correspond to at least one filter element. For example, referring to FIG2, a filter element 111 can use a single microlens 301 to collect light. Alternatively, in other embodiments, multiple filter elements can share a single microlens to collect light, and this application does not impose specific limitations on this.

[0054] In the image sensor provided in this application embodiment, a multispectral unit 110 in the pixel array 100 consists of a first type of pixel 111 for image imaging and a second type of pixel 112 for multispectral imaging. Correspondingly, a filter unit 210 in the color filter array 200 consists of a first type of filter element 211 corresponding to the first type of pixel 111 and a second type of filter element 212 corresponding to the second type of pixel 112. Thus, the image sensor 10 achieves image imaging through the color channel corresponding to the first type of filter element 211, and recognizes multispectral information through the color channel corresponding to the second type of filter element 212 to achieve multispectral imaging. This allows the image sensor 10 to simultaneously perform image imaging and multispectral imaging functions through multispectral recognition, eliminating the need for an additional multispectral sensor to recognize multispectral information and reducing hardware power consumption.

[0055] An image sensor provided according to an embodiment of this application includes an image sensor comprising a pixel array and a color filter array; the pixel array includes at least one multispectral unit, the multispectral unit including at least two first-type pixels for image imaging and at least one second-type pixel for multispectral imaging; the color filter array includes filter units disposed opposite to the multispectral unit, the filter units including at least two first-type filter elements and at least one second-type filter element, the first-type filter elements being disposed opposite to the first-type pixels; the second-type filter elements being disposed opposite to the second-type pixels; the first-type filter elements and the second-type filter elements have different transmittance wavelength ranges, and at least one second-type filter element includes at least one filter element. Thus, in the image sensor, the multispectral unit includes a first type of pixels for image imaging and a second type of pixels for multispectral imaging. The filter unit includes a first type of filter element disposed opposite to the first type of pixels and a second type of filter element disposed opposite to the second type of pixels. The image sensor realizes image imaging through the color channel corresponding to the first type of filter element, and realizes multispectral imaging by recognizing multispectral information through the color channel corresponding to the second type of filter element. This allows the image sensor to perform both image imaging and multispectral imaging functions without the need for an additional multispectral sensor to recognize multispectral information, thereby reducing hardware power consumption.

[0056] In some embodiments of this application, in the color filter array 200, the first type of filter element 211 may include at least one color filter element, wherein the color filter element may include a red filter element R, a green filter element G, a blue filter element B, a cyan filter element C, a magenta filter element M, a yellow filter element Y, and an all-pass filter element W.

[0057] In one specific embodiment, taking the first type of filter element 211 as an example, which includes three types of filter elements, the three types of filter elements can transmit light from at least three color channels. The first type of filter element 211 can be arranged in the color filter array 200 according to RGGB, RYYB, RWWB or other methods to achieve the function of image imaging. This application does not impose specific limitations on the type and arrangement of the filter elements of the first type of filter element 211.

[0058] For example, taking the first type of filter element 211 as an example, which includes three types of filter elements, the first type of filter element 211 may include a red filter element R and a blue filter element B; the first type of filter element 211 may also include at least one of a green filter element G, a yellow filter element Y, and an all-pass filter element W. Among them, the wavelength range of light that can be transmitted by the red filter element R is the red light band, the wavelength range of light that can be transmitted by the blue filter element B is the blue light band, the wavelength range of light that can be transmitted by the green filter element G is the green light band, the wavelength range of light that can be transmitted by the yellow filter element Y is the yellow light band, and the wavelength range of light that can be transmitted by the all-pass filter element W is the global light band, which is white light.

[0059] For example, as shown in Figure 3A, when the first type of filter elements includes a red filter element R, a green filter element G, and a blue filter element B, the color filter array can include multiple minimum units. In one minimum unit 220 of the color filter array, the ratio of the number of red filter elements R, green filter elements G, and blue filter elements B is 1:2:1. The green filter element G adjacent to the red filter element R can be a green-to-red filter element Gr, and the green filter element G adjacent to the blue filter element B can be a green-to-blue filter element Gb. Thus, the first type of filter elements in the color filter array 200 can be arranged in an RGGB pattern, and a color filter array arranged in an RGGB pattern can be called a Bayer array.

[0060] For example, in other embodiments, when the first type of filter elements includes a red filter element R, a yellow filter element Y, and a blue filter element B, the ratio of the number of red filter elements R, yellow filter elements Y, and blue filter elements B in the color filter array is 1:2:1. Thus, compared to the embodiment shown in FIG. 3A, the green filter element G in the first type of filter elements can be replaced with the yellow filter element Y, and the first type of filter elements are arranged in the color filter array according to the RYYB pattern.

[0061] For example, in other embodiments, when the first type of filter elements includes a red filter element R, an all-pass filter element W, and a blue filter element B, the ratio of the number of red filter elements R, all-pass filter elements W, and blue filter elements B in the color filter array 200 is 1:2:1. Thus, compared to the embodiment shown in FIG. 3A, the green filter element G in the first type of filter elements can be replaced with the all-pass filter element W, and the first type of filter elements are arranged in the color filter array in an RWWB configuration. This application does not impose specific limitations on the arrangement of the first type of filter elements in the color filter array.

[0062] In some embodiments of this application, in order to achieve multispectral imaging, the color channel corresponding to the second type of filter element can identify spectral information in multiple band ranges, and there can be multiple types of filter elements corresponding to the second type of filter element 212.

[0063] In some embodiments of this application, the second type of filter element 212 may include visible light filter elements, infrared light filter elements, and ultraviolet light filter elements, etc.

[0064] For example, for visible light filtering elements, the second type of filtering element 212 may include filtering elements for various colors of light, such as cyan filtering element C, yellow filtering element Y, magenta filtering element M, violet filtering element P, orange filtering element O, and all-pass filtering element W.

[0065] The second type of filter element 212 may include filter elements with the same spectral curve but different transmittance. For example, the second type of filter element 212 may include a lighter-colored filter element with the same spectral shape and a transmittance of 80%, or a darker-colored filter element with the same spectral shape and a transmittance of 30%, and so on. For example, the second type of filter element 212 may include a yellow filter element Y, which may be further divided into a light yellow filter element with a transmittance of 80%, a dark yellow filter element with a transmittance of 30%, and so on. In this embodiment, the transmittance of the second type of filter element 212 under the same spectral shape is not specifically set.

[0066] The second type of filter element 212 may also include filter elements with different spectral forms, and different spectral forms correspond to different colors that the human eye can perceive within the visible light range. For example, the second type of filter element 212 may include a yellow-green filter element or a cyan-blue filter element in the visible light range, etc., and may include a near-infrared filter element that transmits 940nm to 945nm, a short-wave infrared filter element that transmits 1400nm to 1500nm, etc. in the non-visible light band. In this embodiment, the spectral transmittance curve of the second type of filter element 212 is not specifically set.

[0067] In some embodiments of this application, in the color filter array 200, the first type of filter element 211 may include at least one filter element, and the at least one filter element can transmit at least one optical wavelength range to obtain at least one color channel information for image imaging. The second type of filter element 212 may include at least one filter element, and the at least one filter element can transmit at least one optical wavelength range to satisfy the function of multispectral imaging.

[0068] In some embodiments of this application, the second type of filter element 212 may also include narrow-band filter elements and wide-band filter elements, etc.

[0069] The second type of filter element 212 can transmit light wavelengths of either a narrow or wide band, and this application does not impose any restrictions on this. The difference between narrow and wide band types lies in the following: when using spectral feature information for multispectral imaging, a single narrow band wavelength range can characterize one spectral feature, while at least two wide band wavelength ranges, after being processed as source data, can characterize another spectral feature. Furthermore, narrow band wavelength ranges are generally narrower and can directly characterize the spectral information of a specific color; while wide band wavelength ranges are generally wider, resulting in a greater amount of light entering the filter element and a larger amount of signal acquired.

[0070] For example, taking the second type of filter element 212 as an example where the wavelength range of light that can be transmitted is a narrow band type, as shown in Figures 4A to 4E, in a spectral unit 210, at least one second type of filter element may include at least one of the following: cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W.

[0071] For example, in some specific embodiments of this application, in a spectral unit 210, at least one second-type filter element may include one of a cyan filter element C, a yellow filter element Y, a magenta filter element M, a violet filter element P, an orange filter element O, and an all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include a cyan filter element C. For example, in a spectral unit 210, at least one second-type filter element may include a yellow filter element Y. For example, in a spectral unit 210, at least one second-type filter element may include a magenta filter element M. For example, in a spectral unit 210, at least one second-type filter element may include a violet filter element P. For example, in a spectral unit 210, at least one second-type filter element may include an orange filter element O. For example, in a spectral unit 210, at least one second-type filter element may include an all-pass filter element W.

[0072] For example, in some specific embodiments of this application, in a spectral unit 210, at least one second-type filter element may include two of the following: cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C and yellow filter element Y. For example, in a spectral unit 210, at least one second-type filter element may include magenta filter element M and violet filter element P. For example, in a spectral unit 210, at least one second-type filter element may include orange filter element O and all-pass filter element W.

[0073] For example, in some specific embodiments of this application, in a spectral unit 210, at least one second-type filter element may include three of the following: cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, and magenta filter element M. For example, in a spectral unit 210, at least one second-type filter element may include violet filter element P, orange filter element O, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, and all-pass filter element W.

[0074] For example, in some specific embodiments of this application, in a spectral unit 210, at least one second-type filter element may include four of the following: cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, magenta filter element M, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, magenta filter element M, and violet filter element P. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, magenta filter element M, and orange filter element O.

[0075] For example, in some specific embodiments of this application, in a spectral unit 210, at least one second-type filter element may include five of the following: cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, magenta filter element M, orange filter element O, and all-pass filter element W. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, magenta filter element M, orange filter element O, and violet filter element P. For example, in a spectral unit 210, at least one second-type filter element may include cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, and all-pass filter element W.

[0076] For example, in some specific embodiments of this application, in a spectral unit 210, at least one second type of filter element may include a cyan filter element C, a yellow filter element Y, a magenta filter element M, a violet filter element P, an orange filter element O, and an all-pass filter element W.

[0077] It should be noted that the type and number of the second type of filter element can be set according to actual needs in the embodiments of this application, and this application does not impose specific restrictions on this.

[0078] In multispectral imaging, when using spectral feature information obtained based on the second type of filter element 212, each of the cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W can characterize a spectral feature information.

[0079] Thus, by setting at least one of the following filter elements as a second type of filter element—cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W—in this embodiment of the application, the image sensor can achieve multispectral imaging function by recognizing at least one spectral information.

[0080] In a specific example, the more types of filters a second-class filter element has, the richer the multispectral information it can identify, and the better its multispectral imaging function. For example, as shown in Figure 4B, in a spectral unit 210, when the filter unit 210 includes at least six second-class filter elements, the at least six second-class filter elements may include a cyan filter element C, a yellow filter element Y, a magenta filter element M, a violet filter element P, an orange filter element O, and an all-pass filter element W.

[0081] Among them, the cyan filter element C transmits light in the wavelength range of 490 nm to 515 nm; the yellow filter element Y transmits light in the wavelength range of 580 nm to 595 nm; and the magenta filter element M transmits light in the wavelength range of 390 nm to 700 nm. In practical applications, the wavelength range transmitted by the magenta filter element M can be a mixture of red and blue light wavelengths obtained by mixing them in a preset ratio. The violet filter element P transmits light in the wavelength range of 380 nm to 410 nm; the orange filter element O transmits light in the wavelength range of 595 nm to 625 nm; and the all-pass filter element W transmits light in the entire wavelength range, which is white light.

[0082] Thus, by incorporating a cyan filter C, a yellow filter Y, a magenta filter M, a violet filter P, an orange filter O, and an all-pass filter W into the second type of filter element 212, this embodiment of the application enables the image sensor to achieve multispectral imaging by recognizing six types of spectral information. Furthermore, these six types of spectral information can be directly used as six spectral feature information for multispectral imaging, simplifying the multispectral imaging process.

[0083] In practical applications, image sensors typically use a traditional Bayer array arrangement, where each filter element has its own independent microlens to collect light. However, the image sensor provided in this application employs p×q on-chip lens (OCL) technology, allowing adjacent p×q filters of the same color to share a single optical unit for light collection. This optical unit can be an OCL. This approach achieves greater light integration while maintaining light transmission efficiency and provides higher phase-detection autofocus accuracy in low-light environments, thus significantly improving phase-detection autofocus performance.

[0084] For example, in a specific embodiment, the image sensor provided in this application embodiment may include r first-type filter elements 211, the r first-type filter elements 211 are distributed in a p×q array, the r first-type filter elements 211 share a single optical unit, and the wavelength range of light that can be transmitted by the r first-type filter elements 211 is the same, where p and q are both positive integers, and r = p×q.

[0085] In this way, when the first type of pixels with the first type of filter elements are set relative to each other as phase detection autofocus (PDAF) pixels, in the first type of filter elements, p×q filter elements share one optical unit. The optical unit can be a microlens, which can achieve more light integration while maintaining light transmission efficiency, and can provide higher phase focusing accuracy in low light environment, thereby significantly improving phase focusing performance.

[0086] For example, taking a 2×2 OCL Bayer array image sensor as an example, where p is 2 and q is 2, the Bayer array refers to a color filter array arranged in the RGGB pattern, and 2×2 OCL means that four adjacent color filter elements share a microlens. The green filter element G can specifically include: a green-to-red filter element Gr positioned adjacent to the red filter element R, and a green-to-blue filter element Gb positioned adjacent to the blue filter element B. The first type of filter element 211 can include the red filter element R, the green-to-blue filter element Gb, the green-to-red filter element Gr, and the blue filter element B.

[0087] As shown in Figure 3A, in the 2×2 OCL Bayer array image sensor, the smallest unit 220 consists of 16 filter elements. The smallest unit 220 may include 4 red filter elements R, 4 green-to-red filter elements Gr, 4 green-to-blue filter elements Gb, and 4 blue filter elements B. Specifically, in the smallest unit 220, the 4 red filter elements R are arranged in a 2×2 array, sharing one optical unit; the optical unit is represented by a dashed circle or ellipse. The 4 green-to-red filter elements Gr are arranged in a 2×2 array, sharing one optical unit; the 4 green-to-blue filter elements Gb are arranged in a 2×2 array, sharing one optical unit; and the 4 blue filter elements B are arranged in a 2×2 array, sharing one optical unit.

[0088] In this way, when the first type of pixels, which are arranged in a relative manner as red filter element R, green-blue filter element Gb, green-red filter element Gr, and blue filter element B, are used as PDAF pixels, 2×2 red filter elements R share one optical unit, 2×2 green-blue filter elements Gb share one optical unit, 2×2 green-red filter elements Gr share one optical unit, and 2×2 blue filter elements B share one optical unit. This can achieve more light integration while maintaining light transmission efficiency, and can provide higher phase focusing accuracy in low light environments, thereby significantly improving phase focusing performance.

[0089] Furthermore, since this embodiment also includes a second type of pixel 112 for multispectral imaging and a second type of filter element 212 disposed opposite to the second type of pixel 112 in the image sensor, the second type of pixel 112 can also be used as a PDAF pixel. When the second type of pixel 112 is a PDAF pixel, since the second type of filter element 212 disposed opposite to the second type of pixel 112 has richer color channels, incorporating the color channel information of the second type of filter element 212 into the PDAF algorithm can significantly improve phase focusing accuracy.

[0090] For example, in a specific embodiment, the filter unit 210 may include t second-type filter elements 212, the t second-type filter elements 212 are distributed in an array of m×n filter elements, the t second-type filter elements 212 share a single optical unit, and the t second-type filter elements 212 can transmit light of the same wavelength range, where m and n are both positive integers, and t = m×n.

[0091] Where m×n is greater than or equal to 1. When m×n equals 1, it corresponds to a 1×1 OCL image sensor design, where one type II filter element uses a single OCL to collect light. When m×n is greater than 1, multiple type II filter elements share one OCL to collect light. When multiple type II filter elements share one OCL, more light is integrated while maintaining light transmission efficiency, and higher phase-detection autofocus accuracy is provided in low-light environments, thus significantly improving phase-detection autofocus performance. Furthermore, since the type II filter element 212 corresponds to a richer color channel, incorporating the color channel information of the type II filter element 212 into the PDAF algorithm can further improve phase-detection autofocus accuracy.

[0092] For example, with m=1 and n=2, the multispectral unit 110 may include two second-type pixels arranged in a 1×2 array, and the filter unit 210 may include two second-type filter elements arranged in a 1×2 array. The two second-type pixels and the two second-type filter elements are set in a one-to-one correspondence.

[0093] In this configuration, 1×2 filter elements cover 1×2 second-type pixels, and the 1×2 filter elements and 1×2 second-type pixels can be aligned in the vertical direction.

[0094] As shown in Figure 3B, in the filter unit 210, the second type of filter element is a purple filter element P, and 1×2 purple filter elements P of the same color share one optical unit.

[0095] For example, with m=2 and n=1, the multispectral unit 110 may include two second-type pixels arranged in a 2×1 array, and the filter unit 210 may include two second-type filter elements arranged in a 2×1 array. The two second-type pixels and the two second-type filter elements are set in a one-to-one correspondence.

[0096] Among them, 2×1 filter elements cover 2×1 second-type pixels, and the 2×1 filter elements and 2×1 second-type pixels can be aligned in the vertical direction.

[0097] As shown in Figure 3C, in the filter unit 210, the second type of filter element is a cyan filter element C, and 2×1 cyan filter elements C of the same color share one optical unit.

[0098] In this case, in the second type of filter element, 1×2 filter elements share one optical unit, or 2×1 filter elements share one optical unit, and the amount of light entering a single color channel is increased to twice the original amount, which can achieve more light integration while maintaining light transmission efficiency.

[0099] For example, with m=2 and n=2, the multispectral unit 110 may include four second-type pixels arranged in a 2×2 array, and the filter unit 210 may include four second-type filter elements arranged in a 2×2 array. The four second-type pixels and the four second-type filter elements are set in a one-to-one correspondence.

[0100] Among them, 2×2 filter elements cover 2×2 second-type pixels, and the 2×2 filter elements and 2×2 second-type pixels can be aligned in the vertical direction.

[0101] As shown in Figure 3D, in the filter unit 210, the second type of filter element is a cyan filter element C and an orange filter element O. Two cyan filter elements C of the same color share one optical unit, and two orange filter elements O of the same color share one optical unit.

[0102] In this case, in the second type of filter element, 2×2 filter elements share one optical unit, and the amount of light entering a single color channel is increased to four times the original amount, which can achieve more light integration while maintaining light transmission efficiency.

[0103] This allows for greater light integration while maintaining light transmission efficiency, and provides higher phase-detection autofocus accuracy in low-light environments, thus significantly improving phase-detection autofocus performance. Furthermore, since the second-type filter element 212 corresponds to a richer color channel, incorporating its color channel information into the PDAF algorithm can further improve phase-detection autofocus accuracy.

[0104] In some embodiments of this application, when the second type of filter element includes at least one of cyan filter element C, yellow filter element Y, magenta filter element M, purple filter element P, orange filter element O and all-pass filter element W, the t second type of filter element 212 can be the same filter element selected from the above at least one filter element.

[0105] For example, taking the second type of filter element 212, which includes a cyan filter element C, a yellow filter element Y, a magenta filter element M, a violet filter element P, an orange filter element O, and an all-pass filter element W, as shown in Figure 4B, m is 2, n is 2, 2×2 cyan filter elements C share one optical unit, 2×2 yellow filter elements Y share one optical unit, 2×2 magenta filter elements M share one optical unit, 2×2 violet filter elements P share one optical unit, 2×2 orange filter elements O share one optical unit, and 2×2 all-pass filter elements W share one optical unit.

[0106] Thus, when the second type of pixels, which are relatively positioned relative to the second type of filter element 211, are used as PDAF pixels, in the second type of filter element 212, m×n filter elements share one optical unit. This allows for greater light integration while maintaining light transmission efficiency, and provides higher phase-detection autofocus accuracy in low-light environments, thereby significantly improving phase-detection autofocus performance. Furthermore, since the second type of filter element 212 corresponds to a richer color channel, incorporating the color channel information of the second type of filter element 212 into the PDAF algorithm can further improve phase-detection autofocus accuracy.

[0107] In practical applications, taking the Bayer array designed with 2×2OCL as an example, in the image sensor provided by this application, some of the original RGB filter elements in the Bayer array can be replaced with second-type filter elements to obtain filter unit 210 composed of first-type filter element 211 and second-type filter element 212, and at least one filter unit 210 constitutes a color filter array 200.

[0108] In some embodiments of this application, the second type of filter elements can be uniformly distributed in the filter unit 210 at a certain density. For example, in a filter unit 210 corresponding to a multispectral unit, the percentage of the number of second type filter elements ranges from 1% to 100%.

[0109] The proportion of the second type of filter element is the ratio of the number of the second type of filter element to the total number of filter elements in the filter unit 210.

[0110] Thus, in some embodiments of this application, a second type of filter element can be set in a filter unit 210 corresponding to a multispectral unit, with a proportion range of 1% to 100%, so that the second type of filter element is uniformly distributed in the filter unit 210 at a certain density.

[0111] It should be noted that the larger the proportion of the second type of filter elements, the greater the multispectral channel density of the color filter array, the greater the impact on image imaging function, and the better the effect of multispectral imaging function.

[0112] In the process of image formation using first-type pixels with first-type filter elements, second-type pixels with second-type filter elements can be treated as defective pixels. To avoid excessive defective pixels affecting the quality of the generated image, in a filter unit 210 corresponding to a multispectral unit, the proportion of first-type filter elements can be greater than the proportion of second-type filter elements. Specifically, the proportion of first-type filter elements is the ratio of the number of first-type filter elements to the total number of filter elements in filter unit 210, and the proportion of second-type filter elements is the ratio of the number of second-type filter elements to the total number of filter elements in filter unit 210.

[0113] For example, in some embodiments of this application, in a filter unit 210 corresponding to a multispectral unit, the proportion of the second type of filter element ranges from 1% to 50%.

[0114] In a filter unit, the proportion of the second type of filter element can be understood as the multispectral channel density range of the color filter array.

[0115] In this way, by setting the multispectral channel density range to 1% to 50%, it is possible to avoid the multispectral channel density range being too large and occupying too much of the color channel of the first type of pixel, thus affecting the image quality generated based on the first type of pixel.

[0116] In one specific embodiment, experiments have shown that the proportion of the second type of filter element in a filter unit 210 is 6%. Therefore, with a multispectral channel density of 6%, the image sensor can better balance image imaging function and multispectral imaging function.

[0117] Of course, in other embodiments, the proportion of the number of second-type filter elements can be set to 1%, 5.5%, 7%, 50%, 60%, 70%, 80%, 100%, or other values, etc., according to actual needs. This application does not impose specific limitations on this.

[0118] For example, taking the Bayer array with a 2×2 OCL design as an example, the image sensor provided in this application, as shown in Figure 3A, has a minimum unit 220 composed of 16 filter elements. The optical units are represented by dashed circles. In the minimum unit, 2×2 red filter elements R share one optical unit, 2×2 green-to-blue filter elements Gb share one optical unit, 2×2 green-to-red filter elements Gr share one optical unit, and 2×2 blue filter elements B share one optical unit.

[0119] In this embodiment, multiple minimum units 220 can be divided into a filter unit 210, and a filter unit 210 can include multiple minimum units. This application can select at least one minimum unit from the multiple minimum units and replace at least one first-type filter element in the at least one minimum unit with a second-type filter element, resulting in a filter unit 210 composed of a majority of first-type filter elements and a small portion of second-type filter elements.

[0120] In one filter unit 210, the number of minimum units can be set according to actual needs. For example, one filter unit 210 may include 5×5 minimum units, or 4×4 minimum units, etc. This application does not impose specific restrictions on this.

[0121] In some embodiments of this application, compared with the minimum unit 220 shown in Figure 3A, the red filter element R and / or the blue filter element B in the minimum unit can be replaced with a second type of filter element.

[0122] For example, as shown in Figure 3B, this application can replace the red filter element R in the smallest unit 220 with a second type of filter element, such as the purple filter element P.

[0123] For example, as shown in Figure 3C, this application can replace the blue filter element B in the smallest unit 220 with a second type of filter element, such as the cyan filter element C.

[0124] For example, as shown in Figure 3D, this application may also replace the red filter element R and the blue filter element B in the smallest unit 220 with a second type of filter element, such as the orange filter element O and the cyan filter element C.

[0125] Furthermore, in other embodiments of this application, the green-to-blue filter element Gb or the green-to-red filter element Gr can be replaced with a second type of filter element, and this application does not limit this.

[0126] As shown in Figures 4A to 4E, the filter unit 210 can be composed of 5×5 minimum units, each minimum unit including 4×4 filter elements, and one filter unit 210 can include 20×20=400 filter elements. The proportion of the second type of filter elements can be selected as 1%, 6%, 5.5%, 7% or 50%, etc., and this application does not limit the specific value of the proportion of the second type of filter elements.

[0127] In some embodiments of this application, as shown in FIG4A, the proportion of second-type filter elements in a filter unit 210 is 1%, 400 × 1% = 4. Therefore, this application can provide 4 second-type filter elements in a filter unit 210 composed of 400 filter elements. The second-type filter elements provided in the filter unit 210 may include four cyan filter elements C. Thus, 4 second-type filter elements are provided in the filter unit 210 composed of 400 filter elements, and the multispectral channel density is 1%.

[0128] In some embodiments of this application, as shown in FIG4B, the proportion of second-type filter elements in a filter unit 210 is 6%, 400 × 6% = 24. Therefore, this application can provide 24 second-type filter elements 211 in a filter unit 210 composed of 400 filter elements. The second-type filter elements provided in the filter unit 210 may include four cyan filter elements C, four yellow filter elements Y, four magenta filter elements M, four violet filter elements P, four orange filter elements O, and four all-pass filter elements W. Thus, 24 second-type filter elements are provided in the filter unit 210 composed of 400 filter elements, and the multispectral channel density is 6%.

[0129] In some embodiments of this application, as shown in FIG4C, the proportion of second-type filter elements in a filter unit 210 is 5.5%, 400 × 5.5% = 22. Therefore, 22 second-type filter elements can be set in a filter unit 210 composed of 400 filter elements, including four cyan filter elements C, four yellow filter elements Y, four magenta filter elements M, four violet filter elements P, four orange filter elements O, and two all-pass filter elements W. Compared with a 6% multispectral channel density, the image imaging function is better due to the higher proportion of first-type filter elements.

[0130] In some embodiments of this application, as shown in FIG4D, the proportion of second-type filter elements in a filter unit 210 is 7%, 400 × 7% = 28. Therefore, 28 second-type filter elements can be set in a filter unit 210 composed of 400 filter elements, including four cyan filter elements C, four yellow filter elements Y, four magenta filter elements M, four violet filter elements P, four orange filter elements O, and eight all-pass filter elements W. Compared with a 6% multispectral channel density, the multispectral imaging function is better due to the higher proportion of second-type filter elements.

[0131] In some embodiments of this application, as shown in FIG4E, the proportion of second-type filter elements in a filter unit 210 is 50%, 400 × 50% = 200. Therefore, this application can provide 200 second-type filter elements in a filter unit 210 composed of 400 filter elements. Thus, 200 second-type filter elements are provided in a filter unit 210 composed of 400 filter elements, and the multispectral channel density is 50%.

[0132] In other embodiments of this application, the proportion of second-type filter elements in a filter unit 210 can be 60%, i.e., 400 × 60% = 240. Therefore, this application can provide 240 second-type filter elements in a filter unit 210 composed of 400 filter elements. Thus, with 240 second-type filter elements provided in a filter unit 210 composed of 400 filter elements, the multispectral channel density is 60%.

[0133] In other embodiments of this application, the proportion of second-type filter elements in a filter unit 210 can be 70%, i.e., 400 × 70% = 280. Therefore, this application can provide 280 second-type filter elements in a filter unit 210 composed of 400 filter elements. Thus, with 280 second-type filter elements provided in a filter unit 210 composed of 400 filter elements, the multispectral channel density is 70%.

[0134] In other embodiments of this application, the proportion of second-type filter elements in a filter unit 210 can be 80%, 400 × 80% = 320. Therefore, this application can provide 320 second-type filter elements in a filter unit 210 composed of 400 filter elements. Thus, 320 second-type filter elements are provided in a filter unit 210 composed of 400 filter elements, and the multispectral channel density is 80%.

[0135] In other embodiments of this application, in a filter unit 210, the proportion of second-type filter elements can be 100%, 400 × 100% = 400. This application can provide 400 second-type filter elements in a filter unit 210 composed of 400 filter elements. Thus, 400 second-type filter elements are provided in the filter unit 210 composed of 400 filter elements, and the multispectral channel density is 100%.

[0136] It should be noted that in some embodiments of this application, the larger the proportion of the second type of filter elements in a filter unit 210, the greater the proportion of the second type of filter elements, the greater the multispectral channel density of the color filter array, the greater the impact on the image imaging function, and the better the effect of the multispectral imaging function.

[0137] In some embodiments of this application, to avoid the second type of filter elements being too densely distributed, resulting in excessive concentration of bad pixels during image imaging and affecting image quality, the second type of filter elements can be uniformly distributed in the color filter array 200. Specifically, the color filter array 200 may include multiple filter units 210, each filter unit 210 having the same number of second type of filter elements 211, thereby uniformly distributing the second type of filter elements 211 in each filter unit 210 and avoiding affecting image imaging quality.

[0138] In one specific embodiment, the number of filter elements in the color filter array 200 can be an integer multiple of the number of filter elements in the filter unit 210.

[0139] For example, as shown in Figure 5A, the color filter array (200) includes S filter elements, which are arranged in a K×N array; where K and N are both positive integers, and S = K×N;

[0140] The filter unit 210 includes c filter elements, which are arranged in an a×b array; where a and b are both positive integers, and c = a×b.

[0141] The color filter array (200) consists of J filter units; where J = (K / a) × (N / b), K is an integer multiple of a, and N is an integer multiple of b.

[0142] Since the color filter array 200 may include K×N filter elements, where K is the number of rows and N is the number of columns; the filter unit 210 may include a×b filter elements, where M is an integer multiple of a and N is an integer multiple of b; correspondingly, the color filter array 200 is composed of (K / a)×(N / b) filter units 210.

[0143] K and N, a and b can be set according to actual needs, and this application does not impose specific restrictions.

[0144] For example, a color filter array 200 may include 320 × 240 = 76,800 filter elements, and a filter unit 210 may include 20 × 20 filter elements. The color filter array 200 is composed of 16 × 12 = 192 filter units 210. Each filter unit 210 may include 400 × 6% = 24 second-type filter elements. The color filter array 200 includes a total of 192 × 24 = 4,608 second-type filter elements, representing 6% of the 76,800 filter elements in the color filter array.

[0145] For example, in some embodiments of this application, the color filter array 200 may include 8600 × 5800 = 49,880,000 filter elements, and a filter unit 210 may include 20 × 20 filter elements. The color filter array 200 is composed of 430 × 290 = 124,700 filter units 210. Each filter unit 210 may include 400 × 6% = 24 second-type filter elements. The color filter array 200 includes a total of 124,700 × 24 = 2,992,800 second-type filter elements, representing 6% of the 49,880,000 filter elements in the color filter array.

[0146] In this way, the color filter array 200 can include an integer multiple of filter units 210, so that the second type of filter element 211 can be evenly distributed in each filter unit 210, thereby making the second type of filter element 211 evenly distributed in the color filter array 200 and avoiding affecting the image imaging quality.

[0147] In one specific embodiment of this application, the number of filter elements in the color filter array 200 may be a non-integer multiple of the number of filter elements in the filter unit 210. In this case, multiple filter units 210 can be set in the central region of the color filter array 200, so that the second type of filter elements 211 are evenly distributed in the color filter array 200.

[0148] Specifically, the color filter array 200 may include a plurality of filter units 210; the plurality of filter units 210 are located in the central region of the color filter array 200, and a plurality of first-type filter elements 211 are disposed in the edge region of the color filter array 200.

[0149] For example, as shown in Figure 5B, a filter unit 210 may include 20×20 filter elements, and a color filter array may include [(20x+16)×(20y+8)] filter elements. The total number of filter elements in the color filter array is not an integer multiple of the number of filter elements in the filter unit. The non-integer part is removed by the edge so that multiple filter units are arranged as centrally as possible.

[0150] As shown in Figure 5B, the number of horizontal rows of the color filter array 200 is 20x+16, where x is a positive integer, and the number of vertical columns is 20y+8, where y is a positive integer. The horizontal edge of the color filter array 200 retains 8 rows of first-type filter elements, of which 4 rows of filter elements are retained at the top and bottom edges. The vertical edge of the color filter array 200 retains 16 rows of first-type filter elements, of which 8 rows of filter elements are retained at the left and right edges. The middle area of ​​the color filter array 200 can be completely set with x×y filter units 210.

[0151] In this way, when the number of filter elements in the color filter array is not an integer multiple of the number of filter elements in the filter unit, by setting multiple filter elements 210 in the middle region of the color filter array 200 and setting the original RGB filter elements at the edge position of the color filter array 200, the multiple filter units 210 are located in the middle position of the color filter array 200, so that the second type of filter elements 211 are evenly distributed in the color filter array 200.

[0152] In some embodiments of this application, in the color filter array 200, the filter surface shape of filter elements such as the first type of filter element 211 and the second type of filter element 212 can be any shape, such as a square, a circle, a triangle or a hexagon.

[0153] For example, the filter surface shape of the first type of filter element 211 can be square, triangular, hexagonal, or other shapes. The filter surface shape of the second type of filter element 212 can also be square, triangular, hexagonal, or other shapes. This application embodiment does not impose specific limitations on the filter surface shape of the filter elements.

[0154] Among them, square filters have better regularity, making them easier to implement on actual image input / output devices. Compared to square filters, hexagonal filters have a more uniform distribution, higher angular resolution, and better symmetry, which helps reduce the amount of filtering computation.

[0155] Based on the same concept as the image sensor provided in any of the above embodiments, this application also provides an electronic device.

[0156] As shown in Figure 6, this application embodiment provides an electronic device 600, including an image sensor 10.

[0157] It should be noted that the electronic device provided in this application includes the image sensor provided in any of the above embodiments, and can realize all the functions of the image sensor. To avoid repetition, it will not be described again here.

[0158] In the embodiments of this application, the electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a smartwatch, mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), etc. The embodiments of this application do not specifically limit the scope.

[0159] In practical applications, the color channel information of the second type of pixels embedded in the image sensor is combined with the RGB color channel information of the first type of pixels in a certain proportion to form virtual multispectral channel image information, realizing multispectral extension applications of the image, such as improving the accuracy and precision of image gain parameters such as white balance gain parameters and scene color temperature.

[0160] Furthermore, the second type of pixels embedded in the image sensor can serve as PDAF pixels, enabling automatic phase detection (PD) calculations and providing autofocus (AF) feedback to the system, thereby improving PDAF accuracy in monochrome, low light, and special lighting conditions and reducing focusing time. An example is provided below.

[0161] As shown in Figure 7A, this application embodiment provides an image processing method, executed by the electronic device provided in any of the above embodiments. The image processing method may include:

[0162] Step 710: Acquire a preview image output by the image sensor. The preview image includes first-type pixel information and second-type pixel information.

[0163] Step 720: Perform image processing on the preview image based on the first type of pixel information and the second type of pixel information;

[0164] Wherein, the first type of pixel information is the pixel data output by the first type of pixels; the second type of pixel information is the pixel data output by the second type of pixels.

[0165] In step 710, a multispectral camera can be activated, i.e., the image sensor provided in any of the above embodiments can be activated, and a preview image output by the image sensor can be acquired. The preview image includes a preview data stream, such as a RAW image. First-type pixel information and second-type pixel information can be extracted from the RAW image. The first-type pixel information includes color channel information of the first-type pixels, and the second-type pixel information includes color channel information of the second-type pixels.

[0166] In step 720, based on the first type of pixel information and the second type of pixel information in the preview image, image processing is performed on the preview image to obtain the image processing result. The image processing result can be a parameter, such as a target gain parameter, or an image, such as a target image after the preview image has been corrected.

[0167] Taking the image processing based on the first and second types of pixel information in the preview image to obtain the target gain parameter as an example, in special scenes such as large-area monochrome or low light, the second type of pixels contains richer spectral information and can provide more accurate white point / color temperature information compared to the first type of pixels. Therefore, determining the target gain parameter based on the second type of pixel information can improve the precision and accuracy of the target gain parameter.

[0168] In step 720, this application can further perform image processing on the preview image based on the target gain parameter to obtain the target image, which can improve the color performance of the target image.

[0169] Taking the target gain parameter, including the white balance gain parameter, as an example, in related technologies, the traditional Bayer-style Automatic White Balance (AWB) algorithm generally achieves color balance by statistically analyzing the image color distribution, calculating channel gains, and dynamically adjusting them. Its core process can be divided into three stages: statistical analysis, calculation, and feedback. In the statistical stage, the algorithm extracts the image color distribution features and identifies potential white / neutral regions. Various algorithms exist for implementation and judgment, and this application does not impose specific limitations. In the gain calculation stage, the R and B channel gain coefficients are calculated based on the statistical results. In the feedback and adjustment stage, the gain parameters are dynamically optimized to avoid overexposure or undersaturation. Specifically, a typical AWB algorithm implementation process includes: acquiring a RAW image; eliminating invalid regions through block statistics; calculating the mean / histogram; calculating R / G / B values ​​and white balance gain parameters; applying the white balance gain parameters to the preview image based on the assumptions; checking the image saturation; if the saturation does not meet the specified requirements, recalculating / adjusting the gain until the saturation meets the specified requirements; outputting the balanced image; and entering subsequent image enhancement processing steps, such as subsequent de-mosaic / color correction image enhancement processing steps.

[0170] It should be noted that traditional color filter arrays arranged in ways such as RGGB, RYYB, RCCB, and RWWB generally extract color information of the scene from 3 to 4 broadband colors. For large-area monochrome scenes, as well as complex lighting scenes such as mixed light sources and low illumination, the information obtained is limited, which restricts the capabilities of the AWB algorithm.

[0171] In this embodiment, after acquiring the RAW image from the multispectral camera, the second type of pixels, which are embedded at a ratio of 6% in the image sensor, and the original 94% of the first type of pixels, such as RGB pixels, in the Bayer array can be extracted at a certain ratio to form a virtual channel multispectral image. The AWB algorithm and the Image Signal Processor (ISP) link algorithm can use this virtual channel multispectral image to perform AWB calculation. By expanding the spectral dimension, enhancing physical interpretability, and improving statistical robustness, the multispectral channel solves the three core problems of the traditional Bayer-mode AWB algorithm: insufficient statistical dimension, rigid assumption model, and poor scene adaptability, thus obtaining more accurate scene color temperature / color information and achieving better white balance effect.

[0172] For example, as shown in Figure 7B, in order to improve the precision and accuracy of the white balance gain parameter, step 720 above, based on the first type of pixel information and the second type of pixel information in the preview image, may include the following image processing:

[0173] Step 721: Extract the first type of pixel information and the second type of pixel information from the preview image;

[0174] Step 722: Adjust the white balance based on the first type of pixel information and the second type of pixel information.

[0175] In step 721, this embodiment of the application can extract first type pixel information and second type pixel information from the preview image according to a specified ratio, combine the first type pixel information and second type pixel information to obtain first image information; perform defective pixel correction processing on the second type pixels in the preview image to obtain second image information; input the first image information and second image information into the automatic white balance (AWB) algorithm to calculate the white balance gain parameter.

[0176] In step 721, for each filter unit in the color filter array, referring to the filter unit shown in FIG4B, in a filter unit containing 20×20 filter elements, pixel data of the first type of pixel output and pixel data of the second type of pixel are extracted and combined according to a specified ratio to obtain a virtual multispectral channel map corresponding to one filter unit. The virtual multispectral channel map can be referred to FIG7C. Then, the same operation is performed on (M / 20)×(N / 20) filter units in the color filter array to obtain (M / 20)×(N / 20) virtual multispectral channel maps, thereby obtaining the first image information.

[0177] The virtual multispectral channel map includes all types of second-class pixels embedded in the image sensor, as well as a certain proportion of first-class pixels. It should be noted that Figure 7C shows one extraction and combination method, where one of each of the six types of second-class pixels and three types of first-class pixels is arranged. If a practical application requires more first-class pixels, more first-class pixels can be extracted as needed.

[0178] Specifically, the second type of pixel information in the preview image undergoes defect correction processing to obtain the second image information. Taking RGB pixels as an example, the second image information includes RGB channel information.

[0179] The AWB algorithm can be a traditional algorithm or a deep learning algorithm, and this application does not restrict the specific algorithm.

[0180] It should be noted that in the traditional AWB algorithm, the AWB algorithm can only obtain a full-frame RAW image with limited spectral information such as RGGB and RYYB to determine the white balance gain parameter.

[0181] In this embodiment, in step 721, the AWB algorithm obtains two data streams: one is a full-frame RAW image containing limited spectral information such as Bayer images or RYYB images, i.e., the second image information; the other is a multispectral virtual channel RAW image containing rich spectral information, i.e., the first image information. The multispectral virtual channel information contained in the first image information can provide more accurate white point / color temperature information in special scenarios such as large-area monochrome or low light.

[0182] In step 721, the AWB algorithm uses two-way image information and white balance white point white balance gain parameters, adding a mode compared to the traditional method, which helps to improve the accuracy of the white balance algorithm.

[0183] In step 722, the white balance of the preview image is adjusted based on the white balance gain parameter to obtain the target image. Specifically, the white balance gain parameter obtained by the AWB algorithm can be applied to the preview image to obtain the target image after white balance correction. The target image can then undergo subsequent image enhancement processing such as depigmentation / color correction.

[0184] In addition to calculating the white balance gain parameters by extracting the first type of pixel information and the second type of pixel information to adjust the white balance of the preview image, in other embodiments, the present application embodiments can also directly input the preview image into a pre-trained deep learning model to obtain the white balance corrected target image.

[0185] In this way, by using a specially designed image sensor, after acquiring a RAW image, the first image information containing multispectral pixel information is extracted and fed into the AWB algorithm for processing. This optimizes the traditional AWB algorithm and achieves the best color effect in various scenes and complex color / light source scenes, further improving the user experience.

[0186] Furthermore, in practical applications, Phase Detection Auto Focus (PDAF) hardware design achieves fast focusing by integrating a dedicated pixel structure into the image sensor. Its core principle is to determine the focus shift direction by detecting the phase difference of light rays. Commonly used PDAF pixel arrangements include 2×2 OCL pixels, 1×1 OCL 2PD pixels, 2PD pixels, sparsely occluded PD pixels, and combinations of multiple schemes. However, in image sensors with full pixel autofocus, although all pixels can participate in focusing, power consumption is relatively high. In some special modes, such as night scenes, only a portion of the pixels are used for focusing. Taking 2×2 OCL as an example, although all pixels can participate in phase focusing, in certain modes, only a portion of pixel information is read for PDAF focusing. The advantage of this approach is reduced power consumption, increased frame rate, and improved image quality simultaneously.

[0187] In this embodiment, when some pixels participate in focusing, a second type of pixel used for multispectral imaging can be used as a PDAF pixel. This improves the color robustness of PDAF. Focusing performance for objects of various colors is enhanced. Specifically, in the color filter arrays of related technologies, one of the R / G / B / C channels may be selected as a PDAF channel. When using a single channel, if the proportion of green objects in the scene is too high, such as in a forest scene, it may lead to phase difference signal saturation or misjudgment. However, this embodiment uses multispectral pixels for PDAF focusing, which can use multiple colors and different color channels for focusing on different locations in the image, improving focusing accuracy.

[0188] Furthermore, this application embodiment employs multispectral pixels for PDAF focusing, improving PDAF's robustness under varying lighting conditions, particularly enhancing focusing performance in low-light scenarios. Specifically, when related technologies use a single channel as the PDAF channel, such as when the R / B channel has low sensitivity, in low-light environments or under special light sources like those with low red or blue light proportions, insufficient light intake leads to high noise, resulting in decreased focusing performance. However, when this application embodiment uses multispectral pixels as PDAF pixels, different weights can be assigned to different color channels for focusing based on the actual scene, improving focusing accuracy.

[0189] Furthermore, in the embodiments of this application, the second type of pixels embedded in the image sensor for multispectral imaging will not negatively impact image quality in PDAF implementations such as sparse occlusion PD, but will only improve performance. In other implementations, because the second type of pixels are sparsely distributed, theoretically, bad pixel compensation algorithms and RMSC algorithms will not have much impact on the final image quality.

[0190] For example, in a specific embodiment, in order to improve focusing accuracy, as shown in FIG8, this application embodiment provides an image processing method, executed by the electronic device provided in any of the above embodiments, the image processing method may include:

[0191] Step 810: Obtain the second type of pixel information from the preview image output by the image sensor;

[0192] The preview image includes a first type of pixel information and a second type of pixel information; the first type of pixel information is the pixel data output by the first type of pixels; the second type of pixel information is the pixel data output by the second type of pixels; the second type of pixel information includes X types of pixel information, one type of pixel information corresponds to one color channel; X is an integer greater than 1;

[0193] Step 820: Based on X types of pixel information, obtain the phase difference of X color channels;

[0194] Step 830: Determine the focus offset based on the phase difference of the X color channels;

[0195] Step 840: Focus on the target object in the preview image based on the focus offset.

[0196] In step 810, this application can extract the second type of pixel information in the RAM image as PDAF pixel information. For an image sensor with 6% multispectral pixels, the pixel information of 6% of the multispectral pixels can be extracted, or, depending on actual needs, the pixel information of 6% of the multispectral pixels plus the pixel information of n% of the RGB pixels can be extracted for PDAF calculation.

[0197] In step 810, for example, when the second type of pixels includes six types of pixels corresponding to cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O and all-pass filter element W, the second type of pixel information includes six types of pixel information, which correspond one-to-one with cyan channel N1, yellow channel N2, magenta channel N3, violet channel N4, orange channel N5 and white channel N6 respectively.

[0198] Correspondingly, in step 820, the phase differences ΔX1, ΔX2, ΔX3, ΔX4, ΔX5 and ΔX6 of the six color channels are calculated respectively.

[0199] It should be noted that the relevant technologies are usually based on PD pixels with one or two color channels. Therefore, the results are obtained based on a few color channels. In practical applications, in some specific environments such as a large area of ​​red but PD pixels are blue, the poor signal-to-noise ratio causes inaccurate PDAF calculation.

[0200] In this embodiment, multispectral pixels are used for PDAF focusing, and PDAF algorithms, such as correlation detection / deep learning, are used to calculate the X phase differences corresponding to the X color channels.

[0201] In step 830, a phase difference result is determined based on the phase difference of the X color channels, and the focus offset Δd is determined based on the phase difference result.

[0202] The phase difference result can be the average value or weighted average value obtained based on the phase difference of X color channels. This application does not limit the specific calculation method of the phase difference result.

[0203] In step 840, the target object in the preview image is focused based on the focus offset. The target object can be a person, building, object, or other subject to be photographed, and can be located in various positions within the image.

[0204] Thus, by employing a second type of pixel for multispectral imaging for PDAF focusing, this embodiment of the application can use multiple color channels for focusing and use different color channels for focusing on different locations in the image, thereby improving focusing accuracy.

[0205] Furthermore, in a specific embodiment, this application embodiment can assign different weights to the multispectral channels of different colors for focusing according to the actual scenario. In step 830 above, determining the focus offset based on the phase difference of X color channels may include:

[0206] Select multiple phase differences from X color channels that meet the target requirements;

[0207] According to the preset weights of X color channels, the phase differences that meet the target requirements are calculated to obtain the phase difference results;

[0208] Based on the phase difference results, the focus offset is determined.

[0209] In this embodiment, the color channel that meets the target requirements can be determined from X color channels for focusing. For example, it can be determined whether there is a color channel with too low a signal or too high noise, or whether there is a channel whose phase difference is significantly different from other channels.

[0210] For example, if the second type of pixels includes six types of pixels corresponding to cyan filter element C, yellow filter element Y, magenta filter element M, violet filter element P, orange filter element O, and all-pass filter element W, phase difference calculation is performed using six multispectral channels: cyan channel N1, yellow channel N2, magenta channel N3, violet channel N4, orange channel N5, and white channel N6. The resulting phase differences are ΔX1 to ΔX6. The average value of ΔX1 to ΔX6 is denoted as ΔXave. A threshold T is set. When ΔXn - ΔXave > T, the fluctuation of ΔXn in that color channel is too large, and it is determined that ΔXn in that color channel will not participate in the PDAF calculation of the focus offset Δd. After the algorithm determines this, a specific channel is used to calculate the phase difference and the focus offset Δd based on the decision.

[0211] For example, if the phase difference ΔX6 of the white channel N6 is determined not to participate in the PDAF calculation of the focus offset Δd, then the multiple phase differences ΔX1 to ΔX5 that meet the target requirements are weighted and averaged according to the preset weights of the X color channels to obtain the final phase difference result ΔX. Then, the phase difference result ΔX is input into the AF algorithm to calculate the determined focus offset Δd. The actual AF algorithm may use partial or full-area PDAF pixel information, and may use simple phase difference, machine learning models to predict the trajectory of moving objects, real-time execution of convolutional networks by the edge NPU (Neural Network Processing Unit), and various other logics combined with traditional contrast focusing to determine the lens shift amount, and calculate the lens displacement Δd based on the pre-calibration data. The pre-calibration method can use traditional PDAF calibration; this application does not impose specific limitations.

[0212] In step 840, this embodiment of the application may further drive the motor to adjust the lens to the calculated position according to the calculated focus offset Δd, so as to focus on the target object in the preview image, and then verify whether it is in focus. If yes, the focus is locked; if no, iterative adjustment is performed.

[0213] Thus, by using the second type of pixels in the image sensor used for multispectral imaging as PDAF pixels for PDAF focusing, the embodiments of this application can improve the accuracy of PDAF under monochrome / low light / special light sources, reduce focusing time, reduce hardware power consumption, and further optimize the user experience.

[0214] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0215] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. An image sensor, comprising: Pixel array and color filter array; The pixel array includes at least one multispectral unit, which includes at least two first-class pixels for image imaging and at least one second-class pixel for multispectral imaging. The color filter array includes filter units disposed opposite to the multispectral unit. Each filter unit includes at least two first-type filter elements and at least one second-type filter element. The first-type filter elements are disposed opposite to the first-type pixels, and the second-type filter elements are disposed opposite to the second-type pixels. The first type of filter element and the second type of filter element have different wavelength ranges of light that can be transmitted, and the at least one second type of filter element includes at least one filter element.

2. The image sensor according to claim 1, wherein, The at least one second-type filter element includes at least one of the following: cyan filter element, yellow filter element, magenta filter element, purple filter element, orange filter element, and all-pass filter element.

3. The image sensor according to claim 2, wherein, When the filter unit includes at least six second-type filter elements, the at least six second-type filter elements include cyan filter elements, yellow filter elements, magenta filter elements, purple filter elements, orange filter elements, and all-pass filter elements.

4. The image sensor according to claim 3, wherein, The cyan filter element can transmit light wavelengths ranging from 490 nanometers to 515 nanometers. The yellow filter element can transmit light in the wavelength range of 580 nm to 595 nm. The magenta filter element can transmit light in the wavelength range of 390 nm to 700 nm. The purple filter element can transmit light in the wavelength range of 380 nanometers to 410 nanometers. The orange filter element allows light wavelengths ranging from 595 nm to 625 nm to pass through.

5. The image sensor according to any one of claims 1-4, wherein, In a filter unit corresponding to a multispectral unit, the proportion of the second type of filter element ranges from 1% to 100%. The proportion of the second type of filter element is the ratio of the number of the second type of filter element to the total number of filter elements in the filter unit.

6. The image sensor according to any one of claims 1-4, wherein, In a filter unit corresponding to a multispectral unit, the proportion of the second type of filter element ranges from 1% to 50%. The proportion of the second type of filter element is the ratio of the number of the second type of filter element to the total number of filter elements in the filter unit.

7. The image sensor according to claim 5, wherein, The second type of filter element accounts for 6% of the total number.

8. The image sensor according to claim 1, wherein, The color filter array includes S filter elements, which are arranged in a K×N array; where K and N are both positive integers, and S = K×N; The filtering unit includes c filtering elements, which are arranged in an a×b array; where a and b are both positive integers, and c = a×b. The color filter array consists of J filter units; where J = (K / a) × (N / b), K is an integer multiple of a, and N is an integer multiple of b.

9. The image sensor according to claim 1, wherein, The color filter array includes multiple filter units; The plurality of filter units are located in the central region of the color filter array, and the edge region of the color filter array is provided with a plurality of the first type of filter elements.

10. An electronic device comprising the image sensor according to any one of claims 1-9.

11. An image processing method, performed by the electronic device of claim 10, the method comprising: Acquire a preview image output by an image sensor, the preview image including a first type of pixel information and a second type of pixel information; Based on the first type of pixel information and the second type of pixel information, image processing is performed on the preview image; Wherein, the first type of pixel information is the pixel data output by the first type of pixels; the second type of pixel information is the pixel data output by the second type of pixels.

12. The method according to claim 11, wherein, The image processing based on the first type of pixel information and the second type of pixel information in the preview image includes: Extract the first type of pixel information and the second type of pixel information from the preview image; White balance adjustment is performed based on the first type of pixel information and the second type of pixel information.

13. An image processing method, performed by the electronic device of claim 10, the method comprising: A second type of pixel information is obtained from the preview image output by the image sensor; the preview image includes a first type of pixel information and a second type of pixel information; wherein, the first type of pixel information is the pixel data output by the first type of pixel; the second type of pixel information is the pixel data output by the second type of pixel; the second type of pixel information includes X types of pixel information, one type of pixel information corresponds to one color channel; X is an integer greater than 1; Based on the X types of pixel information, the phase difference of the X color channels is obtained; The focus offset is determined based on the phase difference of X color channels; Based on the focus offset, focus is applied to the target object in the preview image.

Images