Filter array of multispectral chip and imaging chip comprising multispectral imaging unit

Abstract

A filter array of a multispectral chip comprises: at least five filter units having different transmission spectrum curves, wherein the at least five filter units constitute periodically arranged array units, each array unit is composed of n*m filter units, n and m respectively being greater than or equal to 2; and the at least five filter units comprise at least one main filter unit, the proportion of the number of the at least one main filter unit to the number of filter units in the array units being greater than or equal to 50%. In this way, by using the filter arrays comprising at least five filter units to acquire both color image information and spectral information, a multispectral chip having a high spatial resolution and a good color restoration can be implemented.

Description

Filter arrays for multispectral chips and imaging chips containing multispectral imaging units Technical Field

[0001] This application relates to the field of spectral technology, and more specifically, to a filter array for a multispectral chip and an imaging chip containing a multispectral imaging unit. Background Technology

[0002] Spectral imaging technology is an emerging technology that organically combines traditional two-dimensional imaging and spectral techniques, based on image data across numerous spectral bands. For example, hyperspectral or multispectral imaging technologies acquire both spatial image information and spectral information of the object being measured. Spectral imaging features numerous spectral bands, high spectral resolution, narrow band width, wide spectral range, and unified image and spectrum representation, resulting in a wealth of acquired image information. Because hyperspectral or multispectral imaging technologies can more accurately characterize the spectral information of the physical world and achieve more accurate spectral detection, they have wide applications in fields such as accurate color and material detection.

[0003] Meanwhile, digital imaging has consistently had a profound impact on the quality and usability of camera technology. At the same time, camera consumers have increasingly higher expectations, especially regarding the image quality of imaging modules embedded in modern smartphones. Spectroscopic devices that function by detecting and / or acquiring incident light related to multiple wavelength ranges can be used to provide spectral information to assist camera imaging.

[0004] However, in existing technologies, a separate spectral sensor is often required to obtain spectral information, and digital imaging also requires a specific camera sensor. For consumer electronics, this means that both a camera sensor and a spectral sensor need to be configured, which will undoubtedly increase the power consumption and cost of consumer electronics.

[0005] Therefore, it is desirable to provide an improved filter array for a multispectral chip that can obtain more color and spectral information. Summary of the Invention

[0006] This application provides a filter array for a multispectral chip, which can achieve a multispectral chip with high spatial resolution and good color reproduction by simultaneously acquiring color image information and spectral information with a filter array including at least five filter units.

[0007] According to one aspect of this application, a filter array for a multispectral chip is provided, comprising: at least five filter units with different transmission spectral curves, the at least five filter units forming a periodically arranged array unit, each array unit consisting of n×m filter units, wherein n and m are both greater than or equal to 2, and the at least five filter units include at least one main filter unit, the ratio of the number of the at least one main filter unit to the number of filter units in the array unit being greater than or equal to 50%.

[0008] In the filter array of the aforementioned multispectral chip, the main filter units are distributed in adjacent positions along a diagonal direction that forms an angle of 45 degrees with the row and column directions of the array units.

[0009] In the filter array of the aforementioned multispectral chip, the main filter unit is a G filter unit, a Y filter unit, or a combination of a G filter unit and a Y filter unit.

[0010] In the filter array of the aforementioned multispectral chip, the filter array is rectangular, and the array units are symmetrical at least along one main diagonal of the rectangle.

[0011] In the filter array of the above-mentioned multispectral chip, the at least five filter units include auxiliary filter units, and the types of auxiliary filter units in the auxiliary filter units are all different.

[0012] In the filter array of the above-mentioned multispectral chip, the at least five filter units include auxiliary filter units, and at least one of the auxiliary filter units is the same as at least one of the main filter units.

[0013] In the filter array of the aforementioned multispectral chip, the at least five filter units include auxiliary filter units. The filter units adjacent to each other vertically and horizontally as auxiliary filter units are primary filter units. Two adjacent auxiliary filter units along a diagonal direction (45 degrees from the row and column directions of the filter array) are of the same type. In the filter array of the aforementioned multispectral chip, the height-to-width ratio of the filter units is less than 1.5.

[0014] In the filter array of the aforementioned multispectral chip, the ratio of the height to the width of the filter unit is less than 1.

[0015] In the filter array of the aforementioned multispectral chip, the thickness of the filter unit is from 0.3 μm to 2 μm.

[0016] In the filter array of the aforementioned multispectral chip, the thickness of the filter unit is 0.5 μm to 1 μm.

[0017] In the filter array of the aforementioned multispectral chip, when the filter array does not include an infrared filter unit, the thickness of the filter unit is 0.5 μm to 0.8 μm; when the filter array includes an infrared filter unit, the thickness of the infrared filter unit is 0.5 μm to 1 μm, and the thickness of the other filter units is 0.5 μm to 0.8 μm.

[0018] In the filter array of the aforementioned multispectral chip, the thickness of the same type of filter unit in different array units is the same.

[0019] In the filter array of the aforementioned multispectral chip, the optical response curves of the at least five filter units are not zero in the infrared band.

[0020] In the filter array of the aforementioned multispectral chip, the average transmittance of the light response curves of the at least five filter units in the infrared band is greater than 5%.

[0021] In the filter array of the above-mentioned multispectral chip, the total fitting error of the light response curves of the at least five filter units to the tristimulus value curves and IR channel curves is less than or equal to 0.25.

[0022] The filter array of the multispectral chip provided in this application embodiment can simultaneously acquire color image information and spectral information by using a filter array including at least five filter units, thereby achieving a multispectral chip with high spatial resolution and good color reproduction. Attached Figure Description

[0023] Various other advantages and benefits of this application will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiments below. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0024] Figure 1 illustrates a schematic diagram of the tristimulus value curves and IR channel curves of a multispectral chip according to an embodiment of this application.

[0025] Figure 2 illustrates the results of fitting the photoresponse curves of the five filter units in the filter array of the multispectral chip according to an embodiment of the present application to the tristimulus value curves and IR channel curves shown in Figure 1.

[0026] Figure 3 illustrates a schematic diagram of how the thickness of the filter unit in the filter array of a multispectral chip according to an embodiment of this application affects crosstalk.

[0027] Figures 4A and 4B illustrate schematic diagrams of a first example of a filter array for a multispectral chip according to an embodiment of this application.

[0028] Figures 5A to 5D illustrate schematic diagrams of a second example of a filter array for a multispectral chip according to an embodiment of this application.

[0029] Figures 6A and 6B illustrate schematic diagrams of a third example of a filter array for a multispectral chip according to an embodiment of the present application.

[0030] Figures 7A and 7B illustrate schematic diagrams of a fourth example of a filter array for a multispectral chip according to an embodiment of the present application.

[0031] Figures 8A and 8B illustrate a fifth example of a filter array for a multispectral chip according to an embodiment of this application.

[0032] Figures 9A and 9B illustrate a sixth example of a filter array for a multispectral chip according to an embodiment of this application.

[0033] Figures 10A and 10B illustrate a seventh example of a filter array for a multispectral chip according to an embodiment of the present application.

[0034] Figure 11 illustrates a schematic diagram of a Bayer array in an existing imaging chip.

[0035] Figures 12A to 12C illustrate schematic diagrams of variations of the Bayer array in existing imaging chips.

[0036] Figure 13 illustrates a schematic diagram of a first example of an array arrangement of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0037] Figure 14 illustrates a schematic diagram of the imaging principle of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0038] Figure 15 illustrates a schematic diagram of a first example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0039] Figure 16 illustrates a schematic diagram of a second example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0040] Figure 17 illustrates a schematic diagram of a third example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0041] Figure 18 illustrates a schematic diagram of a fourth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0042] Figure 19 illustrates a schematic diagram of a fifth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0043] Figure 20 illustrates a sixth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0044] Figure 21 illustrates a second example of an array arrangement of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0045] Figure 22 illustrates a schematic diagram of a seventh example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0046] Figure 23 illustrates a schematic diagram of an eighth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0047] Figure 24 illustrates a ninth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0048] Figure 25 illustrates a tenth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0049] Figure 26 illustrates a schematic diagram of an eleventh example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0050] Figure 27 illustrates a schematic diagram of a twelfth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application.

[0051] Figure 28 illustrates a schematic diagram of a first example of an imaging pixel unit comprising at least two types of multispectral imaging units according to an embodiment of this application.

[0052] Figure 29 illustrates a schematic diagram of a second example of an imaging pixel unit comprising at least two types of multispectral imaging units according to an embodiment of this application.

[0053] Figure 30 illustrates a cross-sectional schematic diagram of a first structural example of a filter for a multispectral chip according to an embodiment of this application.

[0054] Figure 31 illustrates a cross-sectional schematic diagram of a second structural example of a filter for a multispectral chip according to an embodiment of this application.

[0055] Figure 32 illustrates a cross-sectional schematic diagram of a third structure example of a filter for a multispectral chip according to an embodiment of this application.

[0056] Figure 33 illustrates a cross-sectional schematic diagram of a fourth structural example of a filter for a multispectral chip according to an embodiment of this application.

[0057] Figure 34 illustrates a cross-sectional schematic diagram of a fifth structural example of a filter for a multispectral chip according to an embodiment of this application.

[0058] Figure 35 illustrates a cross-sectional schematic diagram of a sixth structural example of a filter for a multispectral chip according to an embodiment of this application.

[0059] Figure 36 illustrates a cross-sectional schematic diagram of a seventh structure example of a filter for a multispectral chip according to an embodiment of this application.

[0060] Figure 37A illustrates a cross-sectional schematic diagram of an eighth structural example of a filter for a multispectral chip according to an embodiment of this application.

[0061] Figure 37B shows an enlarged schematic diagram of an example filter structure as shown in Figure 37A.

[0062] Figure 38A illustrates a cross-sectional schematic diagram of a ninth structure example of a filter for a multispectral chip according to an embodiment of this application.

[0063] Figure 38B shows an enlarged schematic diagram of an example filter structure as shown in Figure 38A. Detailed Implementation

[0064] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0065] Typically, the Bayer pattern is a common method used when acquiring digital images by employing charge-coupled devices (CCDs) or complementary metal-oxide semiconductor image sensors (CMOS image sensors) as optical sensors.

[0066] Image sensors convert light into electrical current; the brighter the light, the greater the current, and the dimmer the light, the smaller the current. However, image sensors have a serious drawback: they can only sense the intensity of light, not its wavelength. Since the color of light is determined by its wavelength, image sensors cannot record color. To obtain color images, one approach is to use a single image sensor with a color filter array (CFA) in front of it. The Bayer array does not place three color filters on each pixel; instead, it places a single-color filter (which can be a dye or pigment) at intervals on each pixel. In other words, each pixel of the image sensor is an independent photoelectric conversion unit, capable of converting the collected light intensity into a voltage signal before outputting it. That is, the pixel itself is a monochrome image sensor; it can record color because, before photoelectric conversion, light of a specific color is projected onto the pixel to collect its intensity, thus establishing a correspondence between the specific "monochromatic light" and the light intensity obtained by the photoelectric conversion unit. In this way, each channel can obtain an image with partially missing values, and these missing values ​​can be filled in using various interpolation methods.

[0067] The Bayer array is explained in detail below. The Bayer array is implemented using filters of different colors. Light is filtered into "monochromatic light" and then transmitted to individual pixels to record intensity, establishing a correspondence between the "monochromatic light" formed by the Bayer array and the voltage obtained by the photoelectric conversion unit. Typically, a Bayer array uses filters made of only red, green, and blue materials. Each filter corresponds one-to-one with a pixel in the image sensor, and each filter can only pass one of the three colors: red (R), green (G), or blue (B). The arrangement of these filters, which allow different colors of light, is systematic: green filters are located in the upper right and lower left of each filtering area, blue filters in the lower right, and red filters in the upper left. The number of green filters is twice the number of red or blue filters because research has shown that the human eye is most sensitive to green.

[0068] However, traditional Bayer arrays only produce images with three colors, which is clearly insufficient to represent the colors of the real world. Therefore, this application proposes an arrangement scheme using a filter array composed of at least five filter units, thereby providing higher-dimensional spectral sensing and color reproduction capabilities. Since each pixel receives only a single color of light, it can only record one color component at that location, lacking information from other color channels. To generate a complete RGB or hyperspectral image, it is necessary to recover the missing color channel components of each pixel. This process is called "demosaicing." Demosaicing algorithms analyze the color distribution of the pixel's neighborhood and use mathematical models and interpolation methods to estimate the missing color channel components of each pixel. Commonly used demosaicing algorithms include bilinear interpolation, directional interpolation, and neural network algorithms. The choice of algorithm affects the quality and detail restoration of the final image. For example, a simple interpolation method can be used to generate a complete RGB image on a Bayer array. For instance, the red value of a green pixel can be calculated by interpolating two adjacent red pixels. Similarly, the blue value can be calculated by interpolating two adjacent blue pixels, and so on, to obtain the final output RGB image.

[0069] Furthermore, to better understand the emerging technology of computational spectroscopy, the principles of computational spectroscopy will be introduced below.

[0070] The intensity signals of the incident light at different wavelengths λ are denoted as x(λ), and the transmission spectrum curve of the filter array is denoted as T(λ). The filter array has filter units of n kinds of filter materials (which can be dyes or pigments). The transmission spectrum of the filter unit corresponding to each filter material is different. Overall, the filter array can be denoted as Ti(λ) (i=1,2,3,…,n). The corresponding physical pixel below each filter material detects the light intensity bi modulated by the filter material.

[0071] The relationship between the spectral distribution of the incident light and the measurement value of the image sensor can be expressed by the following formula: bi=∫x(λ)*Ti(λ)*R(λ)dλ

[0072] After further discretization, we get: bi=Σ(x(λ)*Ti(λ)*R(λ))

[0073] Where R(λ) is the response of the image sensor, denoted as: Ai(λ)=Ti(λ)*R(λ)

[0074] The above equation can then be extended into matrix form:

[0075] Where bi (i = 1, 2, 3, ..., n) is the response of the image sensor after the light under test passes through the filter array, corresponding to the light intensity measurement values ​​of the photodetector layer corresponding to each of the n filter units. When one physical pixel corresponds to one filter unit, it can be understood as the light intensity measurement values ​​corresponding to n physical pixels, which is a vector of length n. A is the system's response to light of different wavelengths, determined by the transmittance of the filter structure and the quantum efficiency of the image sensor. A is a matrix, where each row vector corresponds to the response of the filter unit to incident light of different wavelengths. Here, the incident light is sampled discretely and uniformly, with a total of m sampling points. The number of columns in A is the same as the number of sampling points of the incident light. Here, x(λ) is the light intensity of the incident light at different wavelengths λ, which is the incident light spectrum to be measured.

[0076] Based on the above implementation method, arraying the spectral pixels can realize a snapshot-type spectral imaging device.

[0077] It should be understood that although computational spectroscopy has been described, the spectrum mentioned in this application embodiment is not limited to computational spectroscopy. It can also be a narrowband spectral technique, such as the spectrum obtained by filtering using dispersive elements, resonant cavities, or metal plasmas. Furthermore, in this application embodiment, the filter array can also be implemented as a metasurface, photonic crystal, nanopillar, multilayer film, dye, pigment, quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties.

[0078] It should also be noted that multiple physical pixels can correspond to one structural unit, for example, 2*2 physical units can correspond to one filter unit.

[0079] However, images acquired using only the RGB channels lack sufficient information to accurately depict the spectral information of the physical world. Therefore, it is necessary to acquire spectral information to aid imaging and enrich the image information. Often, obtaining spectral information requires a separate spectral sensor, which inevitably increases cost, power consumption of electronic devices, and space constraints.

[0080] To address the aforementioned issues, this application proposes a filter array composed of at least five filter units, specifically a filter array consisting of filter units with five different transmission spectrum curves. When the filter units are made of filter materials, the filter units with different transmission spectrum curves can be made of different types of filter materials, or they can be made of the same filter material but with different size parameters. The filter array includes periodically arranged array units, meaning the array unit is the smallest repeating unit that periodically lays flat to form the array. For example, it can be composed of n×m filter units, where n and m are both greater than or equal to 2, and preferably n and m are equal, or preferably n and m are not both 2. Taking a 4×4 filter unit configuration as an example, this 4×4 filter unit can have a similar arrangement to a common Bayer array, such as RGGB, thus allowing the reuse of the RGGB ISP chip algorithm based on certain types of arrangements. Furthermore, filter units different from RGGB can be added, providing a higher-dimensional spectral sensing capability. Specifically, among the five types of filter units, there is at least one main filter unit, whose number accounts for 50% or more of the minimum repeating units. Preferably, it is distributed in adjacent positions along the u-direction and / or v-direction. Here, the u-direction and v-direction refer to directions that form an angle of 45 degrees with the row and column directions of the array unit, such as the upper left to lower right direction or the upper right to lower left direction. That is, the u-direction and v-direction refer to directions that coincide with or are parallel to the main diagonal of the square array, but are not limited to the main diagonal; therefore, in this embodiment, they can be referred to as diagonal directions. It should be noted that the filter array is composed of a×b array units, where a and b are greater than or equal to 2, and a and b can be equal.

[0081] From a usage perspective, the most accurate image for that channel can be obtained by "de-mosaicing" at the main filter unit. Then, a guided filtering algorithm can correct the "de-mosaicing" channel results of other filter units. Preferably, the main filter unit is generally G and / or Y; that is, it can be implemented as G, Y, or a combination of G and Y, because G and Y are the two channels with the highest light throughput, thereby improving the overall signal-to-noise ratio. Furthermore, the human eye perceives G and Y most strongly. Moreover, if this arrangement is adopted, the filter array of the multispectral chip according to the embodiments of this application can have at least 50% overlap with mainstream RGGB or RYYB arrays. This allows for the reuse of currently common ISP chip algorithms to obtain an RGB image by simply re-mosaicing the filter units that do not overlap with the RGGB or RYYB arrays.

[0082] That is, according to the embodiments of this application, a filter array for a multispectral chip is provided. The filter array includes at least five filter units with different transmission spectrum curves. The at least five filter units form a periodically arranged array unit. Each array unit consists of n×m filter units, where n and m are both greater than or equal to 2. The at least five filter units include at least one main filter unit. The ratio of the number of the at least one main filter unit to the number of filter units in the array unit is greater than or equal to 50%.

[0083] Furthermore, in the filter array of the aforementioned multispectral chip, the main filter units are distributed in adjacent positions along a diagonal direction that forms an angle of 45 degrees with the row and column directions of the array units.

[0084] Furthermore, in the filter array of the aforementioned multispectral chip, the main filter unit is a G filter unit, a Y filter unit, or a combination of a G filter unit and a Y filter unit.

[0085] In this embodiment, the array unit of the multispectral chip may be rectangular. In this case, it needs to be equivalent to a square. Therefore, when m and n are equal, the array unit is a square (the shape may be rectangular, which is also equivalent to a square); if m and n are not equal, it can be equivalent to a square. Furthermore, in the square array unit, the filter unit satisfies the condition that it is symmetrical at least along one side of the main diagonal direction of the array unit, where the main diagonal direction is the u direction or the v direction, and the starting point is from the top left to the bottom right or from the top right to the bottom left of the array unit. In addition to the main filter unit, the five filter units also include four auxiliary filter units. These four auxiliary filter units can be any four different filter units. Selecting four filter units that are different from the main filter unit and also different from each other can maximize the spectral resolution of the multispectral chip. Of course, some auxiliary filter units can also be identical to the main filter unit. To further improve the accuracy of the main filter unit image after "de-mosaicing," they can also be identical to each other, thereby improving the accuracy of a certain auxiliary filter unit image after "de-mosaicing" to adapt to different product requirements. Preferably, the units that are adjacent vertically, horizontally, and vertically to the main filter unit are the four auxiliary filter units. Any two adjacent auxiliary filter units along the u or v direction are the same type of filter unit. When the above constraints exist, the four filter units adjacent to each main filter unit on the top, bottom, left, and right sides can be the same four types of filter units without considering the order. Due to crosstalk in the process, the advantage of this arrangement is that the main filter units are subjected to the same degree of crosstalk. That is, when a completely uniform light shines on the filter array, the response values ​​of all the main filter units are completely consistent without considering external factors such as random errors, dead pixels, and process deviations.

[0086] That is, in the filter array of the multispectral chip according to the embodiments of this application, the filter units are symmetrical along at least one main diagonal direction of the array unit.

[0087] Furthermore, in the filter array of the aforementioned multispectral chip, the at least five filter units include auxiliary filter units, and the types of auxiliary filter units in the auxiliary filter units are all different.

[0088] Furthermore, in the filter array of the aforementioned multispectral chip, at least one of the auxiliary filter units is identical to at least one of the main filter units.

[0089] In addition, in the filter array of the above-mentioned multispectral chip, the filter units adjacent to the auxiliary filter unit on the top, bottom, left and right are the main filter units, and the two adjacent auxiliary filter units along the diagonal direction with an angle of 45 degrees to the row and column directions of the filter array are of the same type.

[0090] Therefore, in the filter array of the multispectral chip according to the embodiments of this application, taking five filter units as an example, the five filter units used must be able to simultaneously acquire color information and infrared information in the visible light band. That is, the light response curves of the five filter units are required to have a non-zero response in the infrared band. Preferably, the average transmittance in the infrared band is greater than 5% to make the acquisition of infrared information more accurate. Transmittance refers to the ratio of transmitted light to incident light. The average transmittance is preferably the arithmetic mean of the infrared band, or a weighted average as required. Here, color information refers to the integral value of the overlap between a certain spectrum and the tristimulus value curve, and infrared information refers to the integral value of the overlap between the spectrum and the infrared channel curve, expressed by the formula: X=∫S x (λ)x(λ)dλ,Y=∫S y (λ)x(λ)dλ,Z=∫S z (λ)x(λ)dλ,IR=∫S ir (λ)x(λ)dλ

[0091] Where XYZ and IR represent the tristimulus value and IR channel value, respectively, and S x (λ),S y (λ),S z (λ),S ir(λ) represents the x, y, z stimulus curves and IR channel curve of the standard colorimetric observer in the 1931 CIE-XYZ system, respectively, where x(λ) is the incident light spectrum to be measured, and ∫ represents the integral operation over wavelength λ. The tristimulus curves and IR channel curves are shown in Figure 1, where the horizontal axis represents wavelength and the vertical axis represents relative intensity. Figure 1 illustrates a schematic diagram of the tristimulus curves and IR channel curves of the multispectral chip according to an embodiment of this application. The tristimulus curves are a set of curves defined by the International Commission on Illumination (CIE) to describe the sensitivity of the human visual system to different wavelengths of light, while the IR channel curve is defined as a Gaussian curve with a center wavelength of 840 nm and a full width at half maximum (FWHM) of 150 nm.

[0092] To obtain the XYZ and IR values ​​more accurately, the total fitting error of the light response curve Ai(λ) of the five filter units to the tristimulus value curve and the IR channel curve is less than or equal to 0.25. Here, the fitting error is defined as: error = sqrt(mean(S...) xfit S yfit S zfit S irfit -S x S y S z S ir ) 2 )

[0093] Where error is the fitting error, s sqrt refers to the square root, and mean refers to the average value. xfit S yfit S zfit S irfit The tristimulus value curves and IR channel curves are obtained by fitting the light response curves of the five filter units. The fitting process is as follows:

[0094] A is the photoresponse curve matrix of the five filter units, and M is the transformation matrix from the photoresponse curves of the five filter units to the tristimulus value curves and IR channel curves, specifically a 4x5 matrix. The matrix multiplication term M is calculated as follows:

[0095] pinv(A) is the pseudo-inverse of matrix A.

[0096] Figure 2 illustrates the results of fitting the photoresponse curves of the five filter units in the filter array of the multispectral chip according to an embodiment of this application to the tristimulus value curves and IR channel curves shown in Figure 1. As shown in Figure 2, the corresponding total fitting error is 0.1946.

[0097] That is, in the filter array of the multispectral chip according to the embodiments of this application, the optical response curve of the filter unit is not zero in the infrared band.

[0098] Furthermore, in the filter array of the aforementioned multispectral chip, the average transmittance of the light response curve of the filter unit in the infrared band is greater than 5%.

[0099] Furthermore, in the filter array of the multispectral chip according to the embodiments of this application, the total fitting error of the light response curve of the filter unit to the tristimulus value curve and the IR channel curve is less than or equal to 0.25.

[0100] Furthermore, multispectral chips used for imaging have certain requirements for imaging resolution, i.e., a high pixel count is needed. If the pixel count is high and the pixel size is too large, the total area of ​​the multispectral chip will be too large, resulting in high production costs. It will also require a relatively expensive large-size optical system, whose volume (especially height) is too large to meet the size requirements of portable devices. Therefore, the size of a single pixel in a multispectral chip used for imaging is generally no greater than 5µm, and mostly between 0.5 and 2.5µm. Considering the aspect ratio, and given the relatively small size of the filter unit, if the aspect ratio exceeds 1.5, it is prone to collapse. Therefore, the aspect ratio should be <1.5, preferably <1. Here, the height (thickness) of the filter unit refers to the thickness of the filter material formed on the image sensor, and the width refers to the size of the filter material formed on the image sensor in a plane parallel to the image sensor surface. If the pixel area covered by the filter unit is not square, the aspect ratio should consider the shorter side. The thickness of the filter unit should generally not exceed 2µm; for small pixels less than 2µm, the filter unit thickness should be further reduced. Meanwhile, since light needs to reach a certain depth to complete absorption and photoelectric conversion after entering the photodiode in a pixel, when the filter unit is thicker, the light has to travel a longer optical path and undergo more scattering before entering the photodiode. Further scattering inevitably occurs as the light penetrates deeper into the photodiode, leading to crosstalk between adjacent pixels, as shown in Figure 3. Therefore, this further limits the thickness of the filter unit to no more than 2 μm. Here, Figure 3 illustrates a schematic diagram of how the thickness of the filter unit in the filter array of the multispectral chip according to an embodiment of this application affects crosstalk.

[0101] On the other hand, if the thickness of the filter unit is too low, its absorption capacity for cutoff wavelength light will decrease, failing to meet performance requirements such as specific wavelength cutoff and signal-to-noise ratio. From a production perspective, during spin coating or spray coating operations, due to the influence of liquid surface tension, coating defects are prone to occur when the filter unit thickness is too low; this thickness limit is generally 0.3 μm. Therefore, it is necessary to limit the thickness of the filter unit. Generally, the thickness range of the filter unit is 0.3–2 μm, preferably 0.5–1 μm.

[0102] That is, in the filter array of the multispectral chip according to the embodiments of this application, the ratio of the height to the width of the filter unit is less than 1.5.

[0103] Furthermore, in the filter array of the aforementioned multispectral chip, the ratio of the height to the width of the filter unit is less than 1.

[0104] Furthermore, in the filter array of the multispectral chip according to the embodiments of this application, the thickness of the filter unit is from 0.3 μm to 2 μm.

[0105] Furthermore, in the filter array of the aforementioned multispectral chip, the thickness of the filter unit is 0.5 μm to 1 μm.

[0106] Furthermore, if the filter array does not include IRP filter units (infrared filter units), the preferred thickness of all filters is 0.5 μm to 0.8 μm. Below 0.5 μm, the filtering ability of each material to the cutoff wavelength decreases, resulting in a decrease in the signal-to-noise ratio of the transmitted / cutoff light signal; above 0.8 μm, the transmittance of the transmitted wavelength is reduced, similarly leading to a decrease in signal strength, insufficient sensitivity of the image sensor, and it cannot be applied to smaller pixels. For filter arrays containing IRP filter units, the preferred thickness of all non-IRP filter units is 0.5 μm to 0.8 μm, for the same reason as before. The preferred thickness of IRP filter units is 0.5 μm to 1 μm, because below 0.5 μm, similar to the above, the filtering effect of the IRP material on the visible light portion decreases, resulting in a decrease in the IR to visible light ratio and a reduction in the signal-to-noise ratio; if the thickness is greater than 1 μm, the difference between the IR thickness and the thickness of other filter units is too large, and the thickness consistency of the filter units will be significantly deteriorated.

[0107] Therefore, in the filter array of the multispectral chip according to the embodiments of this application, when the filter array does not include an infrared filter unit, the thickness of the filter unit is 0.5 μm to 0.8 μm; when the filter array includes an infrared filter unit, the thickness of the infrared filter unit is 0.5 μm to 1 μm, and the thickness of the other filter units is 0.5 μm to 0.8 μm.

[0108] In this embodiment, ensuring process consistency is relatively easy when different types of filter units have the same thickness. However, when the heights differ, the cured filter material obstructs liquid flow and provides anisotropic surface tension, leading to poor morphology and consistency of the subsequently installed filter units. Since the thickness of the filter unit determines the transmittance curve, different thicknesses may be required for different types of filter units to achieve better detection performance. Therefore, a balance between filter unit thickness consistency and performance requirements needs to be considered when setting combinations of different filter unit thicknesses. Preferably, all different types of filter units have the same thickness, but different thicknesses can be used depending on performance requirements. Furthermore, to ensure consistency among the same type of filter units in different array units, the thickness of the same type of filter units in different array units is the same.

[0109] That is, in the filter array of the multispectral chip according to the embodiments of this application, the same type of filter units in different array units have the same thickness.

[0110] Below, specific examples of filter arrays for multispectral chips according to embodiments of this application will be described.

[0111] In the first example, as shown in Figures 4A and 4B, unit a is the main filter unit, accounting for 50% of the smallest repeating units, while units b, c, d, and e are four auxiliary filter units, also accounting for 50% of the smallest repeating units. The left figure shows a symmetrical arrangement along the main diagonal from the top left to the bottom right of the smallest repeating units, and the right figure shows a symmetrical arrangement along the main diagonal from the top right to the bottom left of the smallest repeating units. It can be seen that the units adjacent to the four auxiliary filter units vertically and horizontally are the main filter units, and the auxiliary filter units adjacent along the u or v direction are the same type of filter unit. Furthermore, the four filter units adjacent to each main filter unit vertically and horizontally are the same four types of filter units regardless of their order. Figures 4A and 4B illustrate schematic diagrams of a first example of a filter array for a multispectral chip according to an embodiment of this application.

[0112] In the second example, as shown in Figures 5A to 5D, the main filter units are generally G and / or Y, accounting for more than 50% of the minimum repeating units, and are distributed in adjacent positions along the u and v directions. This is primarily because G or Y are the two channels with the highest light throughput, thus improving the overall signal-to-noise ratio. Secondly, the human eye perceives G and Y most strongly. From a usage perspective, on the one hand, "de-mosaicing" at the main filter unit can obtain the most accurate image of that channel, and then guided filtering algorithms can correct the "de-mosaicing" channel results of other filter units. On the other hand, this arrangement has at least 50% overlap with mainstream RGGB or RYYB arrays, allowing for simple re-mosaicing of filter units with different filter materials than RGGB or RYYB (i.e., filter units other than R, G, B, or Y in the filter array need to be re-mosaiced to convert them into the required filter units when performing image restoration), thus reusing the currently common ISP chip algorithm to obtain an RGB image. Figures 5A to 5D illustrate schematic diagrams of a first example of a filter array for a multispectral chip according to an embodiment of this application.

[0113] In the third example, as shown in Figures 6A and 6B, two of the four auxiliary filter units are R and B, thus achieving a higher overlap rate with mainstream RGGB or RYYB arrays. Furthermore, R and B are located on any main diagonal, and can be either RBRB or BRBR, allowing for the reuse of currently common ISP chip algorithms to obtain an RGB image by simply re-mosaicing the filter units that do not overlap with RGGB or RYYB. To further reduce the difficulty of re-mosaicing, preferably, the main filter unit can be G or Y. Figures 6A and 6B illustrate schematic diagrams of a third example of a filter array for a multispectral chip according to embodiments of this application.

[0114] That is, in the filter array of the multispectral chip according to the embodiments of this application, the auxiliary filter unit includes R and B filter units, and the R and B filter units are located on any one of the main diagonals within the array unit.

[0115] In the fourth example, as shown in Figures 7A and 7B, the main filter unit is G, accounting for more than 50% of the smallest repeating units, and is distributed in adjacent positions along the u and v directions. Two of the four auxiliary filter units are R and B, thus achieving a higher overlap rate with the mainstream RGGB or RYYB arrays. Furthermore, R and B are located on any main diagonal, and can be either RBRB or BRBR, allowing for simple re-mosaicing of filter units that do not overlap with RGGB or RYYB, enabling the reuse of currently common ISP chip algorithms to obtain an RGB image. Moreover, the transmittance curves of the five filter units selected according to the above requirements have a fitting error of less than 0.25 to the tristimulus value curves. For example, C and Y can meet the above requirements, and C and Y are the schemes with the smallest fitting error under the current available dye processes. In this case, the units adjacent to the four auxiliary filter units R, B, C, and Y on all sides are the main filter unit G. In this embodiment, the G filter unit and the Y filter unit can be interchanged depending on the requirements. Figures 7A and 7B illustrate schematic diagrams of a fourth example of a filter array for a multispectral chip according to an embodiment of the present application.

[0116] That is, in the filter array of the multispectral chip according to the embodiments of this application, the fitting error of the transmittance curve of the at least five filter units to the tristimulus value curve is less than 0.25.

[0117] Furthermore, in the filter array of the aforementioned multispectral chip, the at least five filter units include a main filter unit G and auxiliary filter units R, B, C, and Y. The main filter units G are distributed at adjacent positions along the diagonal direction, and the auxiliary filter units R and B are located on any one of the main diagonals of the array unit.

[0118] In the fifth example, as shown in Figures 8A and 8B, this example differs from the fourth example in that the G filter unit is replaced by a Y filter unit, and the Y filter unit is replaced by a G filter unit. Figures 8A and 8B illustrate schematic diagrams of a fifth example of a filter array for a multispectral chip according to embodiments of this application.

[0119] That is, in the filter array of the multispectral chip according to the embodiments of this application, the at least five filter units include a main filter unit Y and auxiliary filter units R, B, C and G. The main filter units Y are distributed at adjacent positions along the diagonal direction, and the auxiliary filter units R and B are located on any one of the main diagonals of the array unit.

[0120] In the sixth example, as shown in Figures 9A and 9B, this example differs from the fourth or fifth example in that the Y filter unit is replaced by a G1 filter unit. The G1 filter unit is also a green filter unit, but its transmission spectrum curve differs from that of the G filter unit, and their center wavelengths are different. In other examples, the G filter unit and the G1 filter unit can also be interchanged. Figures 9A and 9B illustrate a schematic diagram of a sixth example of a filter array for a multispectral chip according to embodiments of this application.

[0121] That is, in the filter array of the multispectral chip according to the embodiments of this application, the at least five filter units include a main filter unit G and auxiliary filter units R, B, C and G1. The main filter units G are distributed at adjacent positions along the diagonal direction, and the auxiliary filter units R and B are located on any one of the main diagonals of the array unit; or, the at least five filter units include a main filter unit G1 and auxiliary filter units R, B, C and G. The main filter unit G1 is distributed at adjacent positions along the diagonal direction, and the auxiliary filter units R and B are located on any one of the main diagonals of the array unit.

[0122] In the seventh example, as shown in Figures 10A and 10B, this example differs from the fourth or fifth example in that the Y filter unit is replaced by an M filter unit, wherein the M filter unit is a magenta filter unit. Figures 10A and 10B illustrate a schematic diagram of a seventh example of a filter array for a multispectral chip according to an embodiment of this application.

[0123] That is, in the filter array of the multispectral chip according to the embodiments of this application, the at least five filter units include a main filter unit G and auxiliary filter units R, B, C and M, the main filter units G are distributed at adjacent positions along the diagonal direction, and the auxiliary filter units R and B are located on any one of the main diagonals of the array unit; or, the at least five filter units include a main filter unit M and auxiliary filter units R, B, C and G, the main filter units M are distributed at adjacent positions along the diagonal direction, and the auxiliary filter units R and B are located on any one of the main diagonals of the array unit.

[0124] Although the above embodiments only have five filter units, this application includes six or more filter units. For example, in the above embodiments, the auxiliary filter unit can be composed of five or six filter units such as R, B, C, Y(G), M, and G1.

[0125] Spectral imaging technology is an emerging technology that combines spectral analysis and imaging technology. It can obtain three-dimensional data cubes containing two-dimensional spatial information and spectral information, which surpasses the perception ability of the human eye. It has important application prospects in many fields such as disease diagnosis and treatment, health monitoring, and precision agriculture.

[0126] A Bayer array is a widely used color filter array used to convert light captured by an image sensor into a color image. Commonly, it consists of red, green, and blue filters, or red, yellow, and blue filters arranged in repeating four-pixel units, with the number of green filters being twice that of red and blue. A Bayer array can also contain four colors of filters. During shooting, the image sensor acquires the intensity information of different colors of light through the Bayer array. Then, a demosaicing algorithm is used to convert this information into a complete color image. Bayer array-based color imaging technology is central to modern mobile phone photography, driving the rapid development of mobile imaging technology.

[0127] On the one hand, with the development of the smart era, spectral imaging devices are moving towards miniaturization and micro-miniaturization, with their size continuously shrinking, enabling them to be integrated into consumer electronic devices such as smartphones and tablets. On the other hand, consumers have increasingly higher expectations for mobile phone image quality, especially for the accuracy of color reproduction. However, existing Bayer arrays can only acquire image information from three or four channels, making it difficult to further improve the accuracy of color reproduction.

[0128] In existing technologies, obtaining multi-channel spectral information requires adding an additional spectral sensor. However, this sensor can only acquire the average spectral information of the imaging field of view, missing image information. Furthermore, adding an extra spectral sensor affects the phone's appearance design, increasing internal space, power consumption, and cost. While existing technologies can achieve multispectral imaging, they still require an additional component and suffer from issues such as aligning the spectral image with the main camera image and synchronizing exposure time.

[0129] Therefore, there is a need to provide an imaging chip solution with improved spectral and color imaging performance.

[0130] This application provides an imaging chip that includes a multispectral imaging unit. By setting imaging pixels containing five or more types of pixels and multispectral pixels in the imaging pixel unit used for imaging, spectral imaging and color imaging are integrated in the same imaging chip, thereby achieving more accurate color reproduction.

[0131] According to one aspect of this application, an imaging chip including a multispectral imaging unit is provided, comprising: periodically arranged imaging pixel units, each of the imaging pixel units comprising imaging pixels for imaging based on a predetermined imaging array and multispectral pixels of a different type for multispectral imaging, wherein the multispectral pixels form at least one multispectral imaging unit by replacing one or more imaging pixels, and the sum of the number of pixel types of the imaging pixels and the multispectral pixels in the multispectral imaging unit is greater than or equal to five, and the multispectral imaging unit is formed as a rectangular array composed of imaging pixels and multispectral pixels.

[0132] In the imaging chip containing the above-mentioned multispectral imaging units, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

[0133] In the imaging chip containing the multispectral imaging unit, the multispectral pixel of one of the multispectral imaging units is located in a different imaging pixel unit.

[0134] In the aforementioned imaging chip containing multispectral imaging units, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

[0135] In the imaging chip containing multispectral imaging units, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

[0136] In the aforementioned imaging chip containing multispectral imaging units, the imaging process includes: acquiring an original image of size H×W, where H and W are the height and width of a predetermined imaging array of the imaging chip, respectively; removing the pixel values ​​of the multispectral pixels of the multispectral imaging units and then interpolating to obtain an original RGB image of size H×W×3; extracting the pixel values ​​of m types of filters included in the multispectral imaging units as multispectral pixels from the original image to obtain an original multichannel image of size N×m, where N is the total number of the multispectral imaging units; and multiplying the original multichannel image by the color correction matrix of size m×3 to obtain... An N×3 XYZ image is generated; pixel values ​​of RGB filters from m filters in the original N×m original multi-channel image are extracted to obtain an original N×3 RGB multi-channel image; the pseudo-inverse matrix of the original RGB multi-channel image is multiplied by the XYZ image to obtain a dynamic color correction matrix; the last dimension of the original RGB image is multiplied by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and the high-fidelity XYZ image is converted to the sRGB color gamut through a standard transformation to obtain a color-restored sRGB image of size H×W×3.

[0137] In the aforementioned imaging chip containing a multispectral imaging unit, extracting the pixel values ​​of m types of filters (which serve as multispectral pixels) contained within the multispectral imaging unit from the original image to obtain an original multichannel image of size N×m includes:

[0138] The color gradient of each multispectral imaging unit is calculated based on the following formula: grad=max(f(S1),f(S2),…,f(S m ))

[0139] S i ∈ Multispectral Imaging Unit

[0140] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i It is the pixel value corresponding to the i-th filter in the multispectral imaging unit;

[0141] The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and

[0142] Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

[0143] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

[0144] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

[0145] In the imaging chip containing the above-mentioned multispectral imaging units, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral imaging unit, and less than or equal to 50%.

[0146] In the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

[0147] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 0.1%.

[0148] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

[0149] In the imaging chip containing the above-mentioned multispectral imaging units, each multispectral imaging unit contains at least one type of multispectral pixel or the number of imaging pixels is greater than 1.

[0150] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 4×4 rectangular array containing five types of pixels. Each imaging pixel unit contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)×(2b+4)).

[0151] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

[0152] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 4×4 rectangular array containing seven types of pixels. Each imaging pixel unit contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)×(2b+4)).

[0153] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

[0154] In the imaging chip containing the multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains two multispectral imaging units, and the size of the imaging pixel unit is 8×8. The horizontal and vertical spacing between adjacent multispectral imaging units is 1 pixel, and the ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 3.125%.

[0155] In the imaging chip containing the multispectral imaging unit, the imaging pixel is an R, G, B filter, and the multispectral pixel is a filter of other types besides the R, G, B filter.

[0156] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

[0157] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

[0158] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains two multispectral imaging units, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

[0159] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels in the imaging chip is 25%.

[0160] In the aforementioned imaging chip containing a multispectral imaging unit, the multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

[0161] In the imaging chip containing the multispectral imaging unit, the imaging pixels are R, G, and B pixels, the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels in the imaging chip is 25%.

[0162] The imaging chip containing a multispectral imaging unit provided in this application embodiment can integrate spectral imaging and color imaging in the same imaging chip by setting imaging pixels containing five or more types of pixels and multispectral pixels in the imaging pixel unit used for imaging, thereby achieving more accurate color reproduction.

[0163] As mentioned above, Bayer arrays are widely used in current imaging chips. Invented by Bryce Bayer in 1976, the Bayer array is a color filter array (CFA) widely used in digital cameras and mobile phone cameras to convert the light captured by the image sensor into a color image. The basic unit of a Bayer array is a 2×2 filter matrix consisting of two green, one red, and one blue filter, typically arranged in an "RGGB" pattern, as shown in Figure 11. Here, Figure 11 illustrates a schematic diagram of a Bayer array in an existing imaging chip. In a Bayer array, the number of green filters is twice that of red and blue filters because the human eye is more sensitive to green; this configuration improves image brightness and detail resolution.

[0164] When light reaches the image sensor through the Bayer array, each pixel can only detect the intensity of one color of light (red, green, or blue). Therefore, the raw data acquired by the sensor is a black-and-white image, called a "Bayer pattern," which needs to be demosaiced to generate a complete color image. Demosaic algorithms calculate the RGB value of each pixel using interpolation methods. Common demosaic algorithms include bilinear interpolation, malvar interpolation, and more complex adaptive algorithms. The choice of different algorithms affects image sharpness, color accuracy, and noise levels.

[0165] Bayer arrays are widely used in digital cameras, mobile phone cameras, and surveillance equipment. With technological advancements, variations of Bayer arrays have proliferated, such as RYYB, RGBW, and RGBIR, which consist of three, four, or more filters (blank can also be considered a filter), as shown in Figures 12A to 12C, to improve imaging performance in specific application scenarios. Figures 12A to 12C illustrate schematic diagrams of variations of Bayer arrays in existing imaging chips. For example, the RYYB array replaces the green (G) filter with a yellow (Y) filter. The yellow filter allows more light to pass through, thus enabling the RYYB array to capture more light in low-light environments, improving image brightness and quality. However, the RYYB array also requires complex algorithms to accurately reproduce image colors, potentially leading to color distortion. The RGBW array replaces part of the green (G) filter with a white (W) filter. The white filter allows all light to pass through, improving image brightness and dynamic range, but the addition of the white filter also makes color reproduction more complex. RGBIR arrays add an infrared (IR) filter to traditional RGB filters, enabling the simultaneous acquisition of visible and infrared information. In low-light or nighttime environments, the infrared light provides additional information, improving image clarity. However, specialized algorithms are required to process and integrate the visible and infrared information. These Bayer array variants, through different filter combinations and structural designs, optimize image quality and functionality for specific application scenarios.

[0166] Based on this, this application proposes an imaging chip that embeds a multispectral imaging unit in a Bayer array to improve the color accuracy of imaging and enable spectral imaging to assist applications such as mobile phone white balance. Here, the Bayer array in the imaging chip of this application can be a traditional RGGB arrangement or a variation of the Bayer array, which will not be distinguished below.

[0167] Taking a Bayer array with an "RGGB" arrangement as an example, the arrangement after embedding multispectral imaging units (also known as multispectral units) is shown in Figure 3. Here, Figure 13 illustrates a schematic diagram of a first example of the array arrangement of an imaging chip containing multispectral imaging units according to an embodiment of this application. As shown in Figure 13, the Bayer array arrangement after embedding multispectral units is still a periodic array. The minimum repeating unit of the periodic array is shown as a solid rectangle in Figure 13. In each minimum repeating unit, n (n≥1) groups of multispectral imaging units are embedded by replacing the original pixels in the Bayer array with multispectral pixels, so that each minimum repeating unit of the periodic array includes n multispectral imaging units. In the embodiments of this application, the embedded multispectral pixels are obtained by replacing the R, G, or B filters at one or more positions in the "RGGB" arrangement with filters other than those arranged in the Bayer array, such as R, G, and B filters in the "RGGB" arrangement. That is, each multispectral pixel corresponds to a filter of one color. It should be noted that two or more filter materials forming a multispectral pixel can also be considered as a filter of one color (based on the final transmission spectrum).

[0168] The multispectral imaging unit is shown as the dashed rectangle in the figure. When one or more multispectral imaging units are included, they may or may not overlap. Each multispectral imaging unit is a rectangular array of size h0×w0, containing multiple filters including R, G, and B filters, and the number of filter types in each multispectral imaging unit is m (m≥5). Thus, based on m types of filters, spectral reconstruction can be performed, thereby achieving spectral imaging. A more accurate Color Correction Matrix (CCM) can also be calculated for more accurate color reproduction, which will be explained in detail below. It should be noted that, preferably, the multispectral pixels of the same multispectral imaging unit are located within the smallest repeating unit, but this is not a limitation; filters constituting the same multispectral unit may also be located in different smallest repeating units.

[0169] Therefore, the imaging chip including a multispectral imaging unit according to embodiments of this application includes: periodically arranged imaging pixel units, each imaging pixel unit comprising imaging pixels for imaging based on a predetermined imaging array and multispectral pixels of a different type for multispectral imaging, wherein the multispectral pixels form at least one multispectral imaging unit by replacing one or more imaging pixels, and the sum of the number of pixel types of the imaging pixels and the multispectral pixels in the multispectral imaging unit is greater than or equal to five, and the multispectral imaging unit is formed as a rectangular array composed of imaging pixels and multispectral pixels. Wherein, the imaging pixels located in the multispectral imaging unit are also used for spectral imaging.

[0170] In other words, the imaging pixel unit here refers to the smallest repeating unit mentioned above, and it includes imaging pixels based on a predetermined imaging array, such as a Bayer array, for example, R, G, and B pixels, and multispectral pixels for multispectral imaging, such as C and Y pixels. Those skilled in the art will understand that the predetermined imaging array can also be other pixel array arrangements for imaging besides a Bayer array. In this embodiment, at least one multispectral imaging unit is formed in the imaging pixel unit by replacing one or more imaging pixels, such as R, G, and B pixels, in the predetermined imaging array with multispectral pixels of a different type than the imaging pixels, such as C and Y pixels. Furthermore, in order to achieve multispectral imaging through the multispectral imaging unit, the sum of the number of imaging pixels and the number of pixel types of the multispectral pixels in each multispectral imaging unit is greater than or equal to five, that is, the number of filter types included in the multispectral imaging unit for realizing pixels is greater than or equal to five. Moreover, the multispectral imaging unit is formed as a rectangular array composed of imaging pixels, such as R, G, and B pixels, and multispectral pixels, such as C and Y pixels.

[0171] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of the present application, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

[0172] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, the multispectral pixels of the multispectral imaging unit are located in different imaging pixel units.

[0173] The principle of spectral reconstruction calculation in the multispectral imaging unit of the imaging chip including the multispectral imaging unit according to the embodiments of this application will be explained below. Here, the spectrum of the incident light, that is, the intensity signal of the incident light at different wavelengths λ, is denoted as x(λ), and the transmission spectrum curves of various filters in the multispectral imaging unit are denoted as T. i(λ) (i = 1, 2, ..., m), each filter has a different transmission spectrum. The incident light, after being modulated by each filter, is detected by the corresponding image sensor pixels below, and the detected light intensity is y. i Here, each filter corresponds to one or more physical pixels of the image sensor. The image sensor's response curve is R(λ), representing the sensor's sensitivity to light of different wavelengths. The relationship between the incident light spectrum and the detected light intensity of the image sensor can be expressed by the following formula: y i =∫x(λ)*T i (λ)*R(λ)dλ

[0174] Discretizing the wavelength domain allows the above equation to be transformed into:

[0175] The corresponding matrix form is:

[0176] This can be simplified as: y = Ax, where A ij =T i (λ j )*R(λ j )

[0177] Matrix A represents the system's response to incident light of different wavelengths, determined by the filter transmittance and the quantum efficiency of the image sensor, and can be obtained by calibrating the system. By solving the above equations, the incident light spectrum can be reconstructed. After calculating the spectrum at the location of each multispectral imaging unit, spectral imaging (either spectral reconstruction or spectral imaging) is achieved.

[0178] It should be noted that the process of acquiring spectral information based on the above-mentioned multispectral imaging unit is not limited to calculating spectral reconstruction. A multispectral imaging unit can also be implemented using a narrowband filter, for example, by using a dispersive element for spectral dispersion or by using a resonant cavity, metal plasma, etc., to achieve narrowband filtering. Thus, the spectral information of the incident light can be directly obtained without calculating reconstruction. In the embodiments of this application, the filter can be implemented as a metasurface, photonic crystal, nanopillar, multilayer film, filter material (dye, pigment, etc.), quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties.

[0179] Based on the above principle of spectral reconstruction, it is known that in order to accurately reconstruct the spectrum at the location of each multispectral imaging unit, the incident light spectrum received by each filter in the multispectral imaging unit must be the same. In order to meet this condition as much as possible, in the embodiments of this application, the size of a single multispectral imaging unit cannot be too large. Specifically, for example, h0≤10, w0≤10, preferably h0≤4, w0≤4, h0 and w0 can be different. Preferably, the multispectral imaging unit is rectangular (it is defined and divided into rectangles according to requirements. For example, when it is not a standard rectangle, it can be expanded into a rectangle, which is also covered by this invention).

[0180] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

[0181] Furthermore, in the imaging chip including multispectral imaging units according to embodiments of this application, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

[0182] The principle of using a multispectral imaging unit to calculate a more accurate color correction matrix is ​​described below, as shown in Figure 14. Figure 14 illustrates a schematic diagram of the imaging principle of an imaging chip containing a multispectral imaging unit according to an embodiment of this application. Specifically, based on the array arrangement of the imaging chip containing the multispectral imaging unit shown in Figure 13, after imaging a scene, a raw image of size H×W can be obtained. First, the multispectral pixels in the embedded multispectral imaging unit (which serve as filters for other types of multispectral pixels that replace the filters for imaging pixels in the Bayer array) are removed as bad pixels. Bad pixel removal is generally achieved through neighborhood interpolation, that is, interpolating using the normal pixel values ​​of the same channel around the bad pixel to replace the bad pixel value. Commonly used interpolation methods include the averaging method, bilinear interpolation, etc. After removing bad pixels, de-mosaic is performed to obtain a raw RGB image of size H×W×3, which is denoted as image A. Furthermore, the pixel values ​​of the m types of filters contained in each multispectral imaging unit are extracted from the original raw image. This extraction is done by averaging the pixel values ​​of the same type of filter within each multispectral unit, resulting in a raw multichannel image of size N×m, where N corresponds to the number of all multispectral imaging units. Further, multiplying the last dimension of the raw multichannel image by a CCM of size m×3 yields a high-fidelity XYZ image of size N×3, denoted as image B. It should be noted that the CCM can be pre-calibrated and calculated for the multispectral imaging units, where XYZ represents tristimulus values. Since m (m>3) channels are used to calculate the tristimulus values, this method is more accurate than calculating them using only the RGB channels, resulting in a high-fidelity XYZ image. Additionally, taking only the RGB components of the last dimension of the raw multichannel image yields a raw RGB multichannel image of size N×3, denoted as image C. By extracting the XYZ values ​​and corresponding raw RGB values ​​of all pixels in images B and C, we can obtain an N×3 XYZ matrix and an N×3 raw RGB matrix. Using the least squares method, a 3×3 CCM (called the dynamic CCM) can be calculated. This dynamic CCM maps the raw RGB matrix to the XYZ matrix. The calculation method of the dynamic CCM is as follows:

[0183] Among them, M RGB and M XYZ These represent the raw RGB matrix and the XYZ matrix, respectively. Pinv(M) represents matrix multiplication. RGB ) is M RGBThe pseudo-inverse is then calculated. Finally, multiplying the last dimension of image A by the dynamic CCM yields a high spatial resolution, high-fidelity XYZ image of size H×W×3. This image is then converted to the sRGB color gamut using a standard transform, resulting in a more accurate sRGB image of size H×W×3. As seen in the calculations above, the dynamic CCM is the optimal CCM calculated based on the color components of the target scene. This means that the calculated dynamic CCM changes with the shooting scene, achieving higher color reproduction accuracy compared to traditional CCM.

[0184] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the imaging process includes:

[0185] Obtain an original image of size H×W, where H and W are the height and width of the predetermined imaging array of the imaging chip, respectively;

[0186] After removing the pixel values ​​of the multispectral pixels of the multispectral imaging unit (i.e., removing multispectral pixels of other types besides the imaging pixel types in the Bayer array), interpolation is performed to obtain an original RGB image of size H×W×3.

[0187] The pixel values ​​of m types of filters contained in the multispectral imaging unit as multispectral pixels are extracted from the original image to obtain an original multichannel image of size N×m, where N is the total number in the multispectral imaging unit.

[0188] The original multi-channel image is multiplied by the color correction matrix of size m×3 to obtain an XYZ image of size N×3;

[0189] Extract the pixel values ​​of the RGB filters from the m filters in the original N×m multi-channel image to obtain an original RGB multi-channel image of size N×3;

[0190] The pseudo-inverse matrix of the original RGB multi-channel image is multiplied with the XYZ image to obtain the dynamic color correction matrix;

[0191] Multiply the last dimension of the original RGB image by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and

[0192] The high-fidelity XYZ image is converted to the sRGB color gamut using a standard transformation to obtain a color-reproduced sRGB image with a size of H×W×3.

[0193] Furthermore, based on the above, it is also possible to determine whether a multispectral imaging unit is located at the edge of an image by calculating the color gradient within the multispectral imaging unit. The color gradient represents the degree of color change within the multi-source imaging unit and can be calculated using the following formula: grad=max(f(S1),f(S2),…,f(S m ))

[0194] S i ∈ Multispectral Imaging Unit

[0195] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i This refers to the pixel value corresponding to the i-th filter within the multispectral imaging unit. When the color gradient is less than a certain threshold (for example, a threshold of 0.05 can be used), the multispectral imaging unit is considered not to be at the image edge. By calculating the color gradient of each multispectral imaging unit, multispectral imaging units not at the edge can be filtered out. Then, in images B and C, only those pixel values ​​not at the edge (where N ≤ the total number of multispectral imaging units) are extracted for dynamic CCM calculation. This avoids the impact of edge-induced false color on the accuracy of dynamic CCM calculation, achieving more accurate dynamic CCM calculation.

[0196] That is, in the imaging chip containing the multispectral imaging unit, extracting the pixel values ​​of m types of filters (which serve as multispectral pixels) contained in the multispectral imaging unit from the original image to obtain an original multichannel image of size N×m includes: calculating the color gradient of each multispectral imaging unit based on the following formula: grad=max(f(S1),f(S2),…,f(S... m ))

[0197] S i ∈ Multispectral Imaging Unit

[0198] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i It is the pixel value corresponding to the i-th filter in the multispectral imaging unit;

[0199] The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and

[0200] Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

[0201] In the above calculation process, on the one hand, in order to reduce the impact of bad pixel removal on image spatial resolution, the density of bad pixels should be as small as possible, that is, the embedding density of multispectral imaging units should be as small as possible. Here, it can be specified that the proportion of multispectral pixels in the embedded multispectral imaging units (excluding imaging pixels used for imaging, such as R, G, B pixels or R, Y, B pixels used for multispectral imaging) to the total number of pixels in the imaging chip should be less than or equal to 25%. That is, the multispectral pixels are other types of filters besides the filters used for imaging included in the predetermined imaging array. For example, when the predetermined imaging array is a Bayer array, and the imaging pixels it includes are RGGB pixels, the multispectral pixels can be variants of C, Y, M, G, W, IR, etc. If the imaging pixels of the Bayer array are RYYB, the multispectral pixels can be C, M, G and their variants, W, IR, etc.; preferably, its proportion should be less than or equal to 10%. Furthermore, preferably, within each multispectral imaging unit, the proportion of multispectral pixels should be greater than or equal to 2 / (h0×w0) and less than or equal to 50%, preferably less than or equal to 25%. On the other hand, to ensure the accuracy of dynamic CCM calculation, the number of multispectral imaging units cannot be too small, i.e., N cannot be too small. Here, it can be specified that the total number of multispectral pixels in all multispectral imaging units accounts for a proportion greater than 0.1% of the total number of pixels in the imaging chip, preferably greater than 1%. In addition, to facilitate the calculation of color gradients within multispectral imaging units, it is preferably required that the number of at least one type of filter in each multispectral imaging unit is greater than 1.

[0202] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

[0203] Furthermore, in the imaging chip containing the multispectral imaging unit described above, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

[0204] Furthermore, in the imaging chip containing the multispectral imaging units, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral pixel unit, and less than or equal to 50%.

[0205] Furthermore, in the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

[0206] Furthermore, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the ratio of the number of multispectral pixels of the multispectral imaging unit to the total number of pixels of the imaging chip is greater than 0.1%.

[0207] Furthermore, in the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

[0208] Furthermore, in the imaging chip containing multispectral imaging units according to embodiments of this application, each multispectral imaging unit contains at least one type of pixel, i.e., the number of multispectral pixels or imaging pixels is greater than 1.

[0209] Below, specific examples of an imaging chip including a multispectral imaging unit according to embodiments of this application will be described.

[0210] Figure 15 illustrates a schematic diagram of a first example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 15, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents the multispectral imaging unit, which is a 4×4 rectangle (h0=4, w0=4) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, the multispectral pixels can also be other types of filters, such as C and M filters; or Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a pixels and the vertical spacing is 2b pixels. In this arrangement, G accounts for 50%, and C and Y each account for 2 / ((2a+4)×(2b+4)). Furthermore, in this example, the multispectral imaging unit may also include a predetermined imaging array composed of other types of imaging pixels, such as imaging arrays other than Bayer arrays, including other types of imaging pixels, such as RYYB, RGBW, etc., as are the other examples described below.

[0211] Figure 16 illustrates a schematic diagram of a second example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 16, unlike the first example, one set of C and Y in the multispectral imaging unit is replaced with two other filters M and W, while the other set of C and Y remains unchanged. In this way, each multispectral unit contains 7 filters (m=7), enabling more accurate spectral reconstruction and color reproduction.

[0212] Figure 17 illustrates a schematic diagram of a third example of an imaging chip including multispectral imaging units according to an embodiment of this application. As shown in Figure 17, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least two multispectral imaging units (n=2). That is, although two multispectral imaging units are shown in Figure 17, this example may also include more than two multispectral imaging units. The size of the minimum repeating unit is 8×8, and the horizontal and vertical spacing between adjacent multispectral imaging units is 1 pixel. In this arrangement, C and Y each account for 3.125%.

[0213] Figure 18 illustrates a schematic diagram of a fourth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 18, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five types of filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, G accounts for 50%, while C and Y each account for 1 / ((2a+4)×(2b+4)).

[0214] Figure 19 illustrates a schematic diagram of a fifth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 19, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains two multispectral imaging units (n=2). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, R and B each account for 25%, while C and Y each account for 1 / ((2a+4)×(2b+4)).

[0215] Figure 20 illustrates a sixth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 20, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, R and B each account for 25%, while C and Y each account for 2 / ((2a+4)×(2b+4)).

[0216] In the examples corresponding to Figures 15-20 above, RGGB is used as an example for illustration, but it can cover other types, such as RYYB, where the Y and G filters can be considered as being replaced. Other types of Bayer arrays can also follow the above principle; therefore, although RGGB is used as an example, it covers other array types. The description of the proportions of R, G, and B filters in the embodiments can be understood as a specific description of the proportions of imaging pixels in the Bayer array, and does not limit the Bayer array to an RGGB array.

[0217] According to another aspect of the embodiments of this application, an imaging chip including a multispectral imaging unit is provided, comprising: at least one imaging pixel unit, the imaging pixel unit including imaging pixels for image imaging and multispectral imaging and multispectral pixels for multispectral imaging, the imaging pixels and the multispectral pixels being pixels of different types with different transmission spectrum curves, the imaging pixels and the multispectral pixels constituting at least one multispectral imaging unit, the sum of the number of types of the imaging pixels and the multispectral pixels in the multispectral imaging unit being greater than or equal to five, and the multispectral imaging unit being formed as a rectangular array composed of imaging pixels and multispectral pixels.

[0218] In the imaging chip containing the multispectral imaging unit, the different types of pixels include pixels with different filter materials having different transmission spectrum curves, and / or pixels with the same filter material having different transmission spectrum curves but different filter material parameters.

[0219] In the imaging chip containing the multispectral imaging unit described above, the different types of pixels include pixels using the following filter materials:

[0220] Red filter material: transmittance >20% for wavelengths above 580nm, and transmittance <20% for other wavelengths;

[0221] The first green filter material has a transmittance of more than 20% at wavelengths of 475–630 nm and above 690 nm, and less than 20% at other wavelengths.

[0222] The second green filter material has a transmittance of more than 20% at wavelengths of 475–610 nm and above 690 nm, and less than 20% at other wavelengths.

[0223] The third type of green filter material has a transmittance of more than 20% at wavelengths of 470–650 nm and above 660 nm, and less than 20% at other wavelengths.

[0224] Blue filter material: transmittance greater than 20% at wavelengths less than 520nm and greater than 785nm, and less than 20% at other wavelengths;

[0225] Cyan filter material: transmittance greater than 20% at wavelengths less than 570nm and greater than 730nm, and less than 20% at other wavelengths;

[0226] Yellow filter material: transmittance is greater than 20% at wavelengths greater than 470nm, and less than 20% at other wavelengths;

[0227] Magenta filter material: transmittance less than 20% at wavelengths of 520–580 nm, and greater than 20% at other wavelengths; and

[0228] Infrared filter material: transmittance greater than 20% for wavelengths above 790nm, and less than 20% for other wavelengths.

[0229] In the imaging chip containing the above-mentioned multispectral imaging units, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

[0230] In the aforementioned imaging chip containing a multispectral imaging unit, multispectral pixels belonging to the same multispectral imaging unit are located in different imaging pixel units.

[0231] In the aforementioned imaging chip containing multispectral imaging units, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

[0232] In the imaging chip containing multispectral imaging units, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

[0233] In the imaging chip containing the above-mentioned multispectral imaging unit, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units are different, wherein the different multispectral imaging units include at least one different type of multispectral pixel.

[0234] In the imaging chip containing multispectral imaging units, there is at least one multispectral imaging unit with a quantity greater than or equal to one.

[0235] In the imaging chip containing the above-mentioned multispectral imaging units, the types of multispectral pixels in the different multispectral imaging units are completely different or partially different.

[0236] In the imaging chip containing the above-mentioned multispectral imaging units, the number of each type of multispectral imaging unit is greater than one.

[0237] In the imaging chip containing the multispectral imaging unit described above, the imaging process includes:

[0238] Obtain an original image of size H×W, where H and W are the height and width of the predetermined imaging array of the imaging chip, respectively;

[0239] After removing the pixel values ​​of the multispectral pixels of the multispectral imaging unit, interpolation is performed to obtain an original RGB image of size H×W×3.

[0240] The pixel values ​​of m types of filters contained in the multispectral imaging unit as multispectral pixels are extracted from the original image to obtain an original multichannel image of size N×m, where N is the total number of the multispectral imaging units.

[0241] The original multi-channel image is multiplied by the color correction matrix of size m×3 to obtain an XYZ image of size N×3;

[0242] Extract the pixel values ​​of the RGB filters from the m filters in the original N×m multi-channel image to obtain an original RGB multi-channel image of size N×3;

[0243] The pseudo-inverse matrix of the original RGB multi-channel image is multiplied with the XYZ image to obtain the dynamic color correction matrix;

[0244] Multiply the last dimension of the original RGB image by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and

[0245] The high-fidelity XYZ image is converted to the sRGB color gamut using a standard transformation to obtain a color-reproduced sRGB image with a size of H×W×3.

[0246] In the aforementioned imaging chip containing a multispectral imaging unit, extracting the pixel values ​​of m types of filters (which serve as multispectral pixels) contained within the multispectral imaging unit from the original image to obtain an original multichannel image of size N×m includes:

[0247] The color gradient of each multispectral imaging unit is calculated based on the following formula: grad=max(f(S1),f(S2),…,f(S m ))

[0248] S i ∈ Multispectral Imaging Unit

[0249] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i It is the pixel value corresponding to the i-th filter in the multispectral imaging unit;

[0250] The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and

[0251] Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

[0252] In the aforementioned imaging chip containing multispectral imaging units, calculations are performed on a per-multispectral imaging unit computing unit basis, and each multispectral imaging unit computing unit includes different multispectral imaging units and / or the same multispectral imaging units.

[0253] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

[0254] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

[0255] In the imaging chip containing the above-mentioned multispectral imaging units, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral imaging unit, and less than or equal to 50%.

[0256] In the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

[0257] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 0.1%.

[0258] In the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

[0259] In the imaging chip containing the above-mentioned multispectral imaging units, each multispectral imaging unit contains at least one type of multispectral pixel or the number of imaging pixels is greater than 1.

[0260] In the imaging chip containing the above-mentioned multispectral imaging unit, the at least one multispectral imaging unit includes at least two types of multispectral imaging units, wherein the first multispectral imaging unit and the second multispectral imaging unit in the at least two types of multispectral imaging units are 2×4 rectangles and each contains five types of pixels.

[0261] In the imaging chip containing the multispectral imaging unit described above, the first multispectral imaging unit includes R, G, and B pixels as imaging pixels and C and Y pixels as multispectral pixels, and the second multispectral imaging unit includes R, G, and B pixels as imaging pixels and GR and M pixels as multispectral pixels.

[0262] In the imaging chip containing the above-mentioned multispectral imaging units, there are two of each of the first multispectral imaging units and the second multispectral imaging units.

[0263] In the imaging chip containing the multispectral imaging unit described above, the ratio of each type of multispectral pixel to the total number of pixels in the imaging chip is 0.78%.

[0264] In the imaging chip containing the above-mentioned multispectral imaging units, each of the first multispectral imaging unit and the second multispectral unit is used as a separate multispectral unit computing unit for calculation, or one or more first multispectral imaging units and one or more second multispectral units are combined into a single multispectral imaging unit computing unit for calculation.

[0265] In the imaging chip containing the above-mentioned multispectral imaging units, the number of the first multispectral imaging unit and the number of the second multispectral imaging units are both four.

[0266] In the imaging chip containing the multispectral imaging unit, the ratio of each multispectral pixel to the total number of pixels in the imaging chip is 1.56%.

[0267] In the aforementioned imaging chip containing multispectral imaging units, each of the first multispectral imaging unit and the second multispectral unit is used as a separate multispectral unit calculation unit for calculation, or a first number of first multispectral imaging units and a second number of second multispectral units are combined into a single multispectral imaging unit calculation unit for calculation, or a third number of first multispectral imaging units and a fourth number of second multispectral units are combined into a single multispectral imaging unit calculation unit for calculation, wherein the first number is equal to or not equal to the second number and not equal to the third number, and the second number is not equal to the fourth number.

[0268] The imaging chip containing a multispectral imaging unit provided in this application embodiment can integrate spectral imaging and color imaging in the same imaging chip by setting imaging pixels containing five or more types of pixels and multispectral pixels in the imaging pixel unit used for imaging, thereby achieving more accurate color reproduction.

[0269] Based on the Bayer arrays shown in Figures 11 and 12A to 12C, taking the “RGGB” arrangement of the Bayer array as an example, the arrangement after embedding multispectral imaging units (also known as multispectral units) is shown in Figure 21. Here, Figure 21 illustrates a schematic diagram of a second example of the array arrangement of an imaging chip containing multispectral imaging units according to an embodiment of this application. As shown in Figure 21, the Bayer array arrangement after embedding multispectral units is still a periodic array. The minimum repeating unit of the periodic array is shown as a solid-line rectangle in Figure 21. In each minimum repeating unit, n (n≥1) groups of multispectral imaging units are embedded by replacing the original pixels in the Bayer array with multispectral pixels, so that each minimum repeating unit of the periodic array includes n multispectral imaging units. In the embodiments of this application, the embedded multispectral pixels are obtained by replacing the R, G, or B filters at one or more positions in the "RGGB" arrangement with filters other than those arranged in the Bayer array, such as R, G, and B filters in the "RGGB" arrangement. That is, each multispectral pixel corresponds to a filter of one color. It should be noted that two or more filter materials forming a multispectral pixel can also be considered as a filter of one color (based on the final transmission spectrum), and blank pixels can also be considered as filters of one color.

[0270] The multispectral imaging unit is shown as the dashed rectangle in the figure. When one or more multispectral imaging units are included, they may or may not overlap. Each multispectral imaging unit is a rectangular array of size h0×w0, containing multiple filters including R, G, and B filters, and the number of filter types in each multispectral imaging unit is m (m≥5). Thus, based on m types of filters, spectral reconstruction can be performed, thereby achieving spectral imaging. A more accurate Color Correction Matrix (CCM) can also be calculated for more accurate color reproduction, which will be explained in detail below. It should be noted that, preferably, the multispectral pixels of the same multispectral imaging unit are located within the smallest repeating unit, but this is not a limitation; filters constituting the same multispectral imaging unit may also be located in different smallest repeating units.

[0271] Furthermore, in some embodiments, the imaging pixel unit is composed of filters other than R, G, and B filters. In this case, the R, G, and B filters can be embedded as multispectral pixels in an array of other filters. For example, if the filters constituting the imaging pixel unit are C, Y, M, W, and / or IR, then the R, G, and B filters can be implemented as multispectral pixels. For example, if the imaging pixel unit is composed of C, Y, and M filters, then other filters besides C, Y, and M filters, such as R, G, B, W, and / or IR filters, can be embedded. As another example, if the imaging pixel unit is composed of C, Y, M, and IR filters, then R, G, B, and / or W filters can be embedded.

[0272] In other words, in this embodiment, the imaging pixels and multispectral pixels are not limited to specific filters, but rather multispectral pixels are embedded in an array of imaging pixel units to obtain an imaging chip containing multispectral imaging units. Therefore, those skilled in the art will understand that in the imaging chip containing multispectral imaging units according to this embodiment, any pixel with a transmission spectrum curve different from that of the imaging pixels in the imaging pixel units can be defined as a multispectral pixel. This can be due to different filter materials, or the same filter material with different parameters, such as different dimensional parameters like thickness, resulting in different transmission spectrum curves for the same filter material. Therefore, these can also be considered multispectral pixels.

[0273] To better understand filter materials, for example, the transmittance curves of various filter materials in the visible and near-infrared bands can be defined with the following characteristics. It should be noted that filter materials tested or manufactured differently may vary. Therefore, in the embodiments of this application, the following characteristics are merely examples and are not limited to filter materials meeting the following requirements:

[0274] Filter material 1 (R, red filter material): transmittance >20% for wavelengths above approximately 580nm, and <20% for other wavelengths;

[0275] Filter material 2 (G1, green filter material): transmittance >20% at wavelengths of approximately 475–630 nm and above 690 nm, and <20% at other wavelengths;

[0276] Filter material 3 (G2, green filter material): transmittance >20% at wavelengths of approximately 475–610 nm and above 690 nm, and <20% at other wavelengths;

[0277] Filter material 4 (G3, green filter material): transmittance >20% at wavelengths of approximately 470–650 nm and above 660 nm, and <20% at other wavelengths;

[0278] Filter material 5 (B, blue filter material): transmittance >20% at wavelengths <520nm and >785nm, and <20% at other wavelengths;

[0279] Filter material 6 (C, cyan filter material): transmittance >20% at wavelengths <570nm and >730nm, and <20% at other wavelengths;

[0280] Filter material 7 (Y, yellow filter material): transmittance >20% at wavelengths >470nm, <20% at other wavelengths;

[0281] Filter material 8 (M, magenta filter material): transmittance <20% at wavelengths of approximately 520–580 nm, and >20% at other wavelengths;

[0282] Filter material 9 (IR, infrared filter material): transmittance >20% for wavelengths above approximately 790nm, and <20% for other wavelengths.

[0283] Therefore, the imaging chip including a multispectral imaging unit according to the embodiments of this application includes: at least one imaging pixel unit, the imaging pixel unit including imaging pixels for image imaging and multispectral imaging and multispectral pixels for multispectral imaging, the imaging pixels and the multispectral pixels being pixels of different types with different transmission spectrum curves, the imaging pixels and the multispectral pixels constituting at least one multispectral imaging unit, the sum of the number of types of the imaging pixels and the multispectral pixels in the multispectral imaging unit being greater than or equal to five, and the multispectral imaging unit being formed as a rectangular array composed of imaging pixels and multispectral pixels.

[0284] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, different types of pixels include pixels with different filter materials having different transmission spectrum curves, and / or pixels with the same filter material having different transmission spectrum curves but different parameters of the filter material.

[0285] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, the different types of pixels include pixels with the following filter materials:

[0286] Red filter material: transmittance >20% for wavelengths above 580nm, and transmittance <20% for other wavelengths;

[0287] The first green filter material has a transmittance of more than 20% at wavelengths of 475–630 nm and above 690 nm, and less than 20% at other wavelengths.

[0288] The second green filter material has a transmittance of more than 20% at wavelengths of 475–610 nm and above 690 nm, and less than 20% at other wavelengths.

[0289] The third type of green filter material has a transmittance of more than 20% at wavelengths of 470–650 nm and above 660 nm, and less than 20% at other wavelengths.

[0290] Blue filter material: transmittance greater than 20% at wavelengths less than 520nm and greater than 785nm, and less than 20% at other wavelengths;

[0291] Cyan filter material: transmittance greater than 20% at wavelengths less than 570nm and greater than 730nm, and less than 20% at other wavelengths;

[0292] Yellow filter material: transmittance is greater than 20% at wavelengths greater than 470nm, and less than 20% at other wavelengths;

[0293] Magenta filter material: transmittance less than 20% at wavelengths of 520–580 nm, and greater than 20% at other wavelengths; and

[0294] Infrared filter material: transmittance greater than 20% for wavelengths above 790nm, and less than 20% for other wavelengths.

[0295] In other words, the imaging pixel unit here refers to the smallest repeating unit mentioned above, and it includes imaging pixels for image imaging, such as R, G, B pixels, or C, Y pixels, and multispectral pixels for multispectral imaging, such as C, Y pixels other than R, G, B pixels, or R, G pixels other than C, Y, M pixels, etc. Furthermore, in the embodiments of this application, the imaging pixels for image imaging can also be used for multispectral imaging; that is, a multispectral image is obtained by combining the multispectral pixels for multispectral imaging and the imaging pixels for image imaging.

[0296] Specifically, at least one multispectral imaging unit can be formed in the imaging pixel unit by replacing one or more imaging pixels in the predetermined imaging array, such as R, G, and B pixels, with multispectral pixels of a different type than the imaging pixels, such as C, Y, and M pixels. Further explaining the replacement, as mentioned above, most imaging chips use imaging pixels arranged periodically, while in this embodiment, multispectral pixels are embedded in the imaging chip according to certain rules, so that pixels that should theoretically be imaging pixels become multispectral pixels.

[0297] Furthermore, in order to achieve multispectral imaging through the multispectral imaging unit, the sum of the number of imaging pixels and the number of pixel types of the multispectral pixels in each multispectral imaging unit is greater than or equal to five, that is, the number of filter types included in the multispectral imaging unit for realizing pixels is greater than or equal to five. Moreover, the multispectral imaging unit is formed as a rectangular array consisting of imaging pixels, such as R, G, B pixels or C, M, Y pixels, and multispectral pixels, such as C, Y pixels other than R, G, B pixels, or R, G pixels other than C, M, Y pixels.

[0298] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of the present application, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

[0299] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, multispectral pixels belonging to the same multispectral imaging unit are located in different imaging pixel units.

[0300] Furthermore, in the imaging chip containing a multispectral imaging unit according to the embodiments of this application, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units are not the same. That is, there may be at least two different multispectral imaging units in an imaging pixel unit, that is, at least one of the different multispectral imaging units has different multispectral pixels constituting the multispectral imaging unit.

[0301] Furthermore, within the same imaging pixel unit, the number of identical multispectral imaging units is greater than or equal to one, meaning that at least two multispectral imaging units can be identical. There are at least two types of multispectral imaging units within the imaging pixel unit. Taking two types of multispectral imaging units as an example, multispectral pixel 1 and multispectral pixel 2 are embedded in the first multispectral imaging unit, and multispectral pixel 3 and multispectral pixel 4 are embedded in the second multispectral imaging unit. Multispectral pixels 1 and 2, and multispectral pixels 3 and 4, can be composed of different filter materials, or they can be composed of the same filter material. Furthermore, different multispectral imaging units can contain the same multispectral pixels. In some embodiments, taking two types of multispectral imaging units as an example, multispectral pixel 1 and multispectral pixel 2 are embedded in the first multispectral imaging unit, and multispectral pixel 2 and multispectral pixel 3 are embedded in the second multispectral imaging unit. Furthermore, the number of the first multispectral imaging unit and / or the second multispectral imaging unit can each exceed one, for example, two, three, or four, thereby improving accuracy. Similarly, the imaging pixel unit can contain three, four, or more multispectral imaging units. For example, the first multispectral imaging unit is composed of R, G, B, C, and Y, and the corresponding second multispectral imaging unit is composed of R, G, B, C, and M. Likewise, in the embodiments of this application, different multispectral pixels can also be obtained by designing the same filter material with different parameters. Furthermore, pixels without a filter material can also be considered as a type of multispectral pixel or imaging pixel.

[0302] It should also be noted that in some cases, there may be some functional pixels or unusable pixels in the imaging chip. Such pixels may not participate in imaging or spectral restoration, but this does not conflict with the purpose of this application. They can still be equivalent to normal pixels (imaging pixels or multispectral pixels) or removed.

[0303] Therefore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, the at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units are different, wherein the different multispectral imaging units contain at least one different type of multispectral pixel. Furthermore, the number of different multispectral imaging units corresponding to the same imaging pixel unit can be the same or different. For example, it may include a first multispectral imaging units and b second multispectral imaging units, where a and b can be equal or unequal, and a and b are both greater than or equal to 1.

[0304] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of the present application, there is at least one multispectral imaging unit with a quantity greater than or equal to one.

[0305] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of this application, the types of multispectral pixels in the different multispectral imaging units are completely different or partially different.

[0306] Furthermore, in an imaging chip including a multispectral imaging unit according to a certain embodiment of the present application, the number of each type of multispectral imaging unit is greater than one.

[0307] The principle of spectral reconstruction calculation in the multispectral imaging unit of the imaging chip including the multispectral imaging unit according to the embodiments of this application will be explained below. Here, the spectrum of the incident light, that is, the intensity signal of the incident light at different wavelengths λ, is denoted as x(λ), and the transmission spectrum curves of various filters in the multispectral imaging unit are denoted as T. i (λ) (i = 1, 2, ..., m), each filter has a different transmission spectrum. The incident light, after being modulated by each filter, is detected by the corresponding image sensor pixels below, and the detected light intensity is y. i Here, each filter corresponds to one or more physical pixels of the image sensor. The image sensor's response curve is R(λ), representing the sensor's sensitivity to light of different wavelengths. The relationship between the incident light spectrum and the detected light intensity of the image sensor can be expressed by the following formula: y i =∫x(λ)*T i (λ)*R(λ)dλ

[0308] Discretizing the wavelength domain allows the above equation to be transformed into:

[0309] The corresponding matrix form is:

[0310] This can be simplified as: y = Ax, where A ij =T i (λ j )*R(λ j )

[0311] Matrix A represents the system's response to incident light of different wavelengths, determined by the filter transmittance and the quantum efficiency of the image sensor, and can be obtained by calibrating the system. By solving the above equations, the incident light spectrum can be reconstructed. After calculating the spectrum at the location of each multispectral imaging unit, spectral imaging (either spectral reconstruction or spectral imaging) is achieved.

[0312] It should be noted that the process of acquiring spectral information based on the above-mentioned multispectral imaging unit is not limited to calculating spectral reconstruction. A multispectral imaging unit can also be implemented using a narrowband filter, for example, by using a dispersive element for spectral dispersion or by using a resonant cavity, metal plasma, etc., to achieve narrowband filtering. Thus, the spectral information of the incident light can be directly obtained without calculating reconstruction. In the embodiments of this application, the filter can be implemented as a metasurface, photonic crystal, nanopillar, multilayer film, filter material (dye, pigment, etc.), quantum dot, MEMS (microelectromechanical systems), FP etalon, cavity layer, waveguide layer, diffraction element, or other structures or materials with filtering properties.

[0313] Based on the above principle of spectral reconstruction, it is known that in order to accurately reconstruct the spectrum at the location of each multispectral imaging unit, the incident light spectrum received by each filter in the multispectral imaging unit must be the same. In order to meet this condition as much as possible, in the embodiments of this application, the size of a single multispectral imaging unit cannot be too large. Specifically, for example, h0≤10, w0≤10, preferably h0≤4, w0≤4, h0 and w0 can be different. Preferably, the multispectral imaging unit is rectangular (it is defined and divided into rectangles according to requirements. For example, when it is not a standard rectangle, it can be expanded into a rectangle, which is also covered by this invention).

[0314] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

[0315] Furthermore, in the imaging chip including multispectral imaging units according to embodiments of this application, the length and width of the rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

[0316] The principle of using a multispectral imaging unit to calculate a more accurate color correction matrix is ​​described below, as shown in Figure 14. Figure 14 illustrates a schematic diagram of the imaging principle of an imaging chip containing a multispectral imaging unit according to an embodiment of this application. Specifically, based on the array arrangement of the imaging chip containing the multispectral imaging unit as shown in Figure 21, after imaging a scene, a raw image of size H×W can be obtained. First, the multispectral pixels in the embedded multispectral imaging unit (which serve as filters for other types of multispectral pixels that replace the filters for imaging pixels in the Bayer array) are removed as bad pixels. Bad pixel removal is generally achieved through neighborhood interpolation, that is, interpolating using the normal pixel values ​​of the same channel around the bad pixel to replace the bad pixel value. Commonly used interpolation methods include the averaging method, bilinear interpolation, etc. After removing bad pixels, de-mosaic is performed to obtain a raw RGB image of size H×W×3, which is denoted as image A. Furthermore, the pixel values ​​of the m types of filters contained in each multispectral imaging unit are extracted from the original raw image. This extraction is done by averaging the pixel values ​​of the same type of filter within each multispectral unit, resulting in a raw multichannel image of size N×m, where N corresponds to the number of all multispectral imaging units. Further, multiplying the last dimension of the raw multichannel image by a CCM of size m×3 yields a high-fidelity XYZ image of size N×3, denoted as image B. It should be noted that the CCM can be pre-calibrated and calculated for the multispectral imaging units, where XYZ represents tristimulus values. Since m (m>3) channels are used to calculate the tristimulus values, this method is more accurate than calculating them using only the RGB channels, resulting in a high-fidelity XYZ image. Additionally, taking only the RGB components of the last dimension of the raw multichannel image yields a raw RGB multichannel image of size N×3, denoted as image C. By extracting the XYZ values ​​and corresponding raw RGB values ​​of all pixels in images B and C, we can obtain an N×3 XYZ matrix and an N×3 raw RGB matrix. Using the least squares method, a 3×3 CCM (called the dynamic CCM) can be calculated. This dynamic CCM maps the raw RGB matrix to the XYZ matrix. The calculation method of the dynamic CCM is as follows:

[0317] Among them, M RGB and M XYZ These represent the raw RGB matrix and the XYZ matrix, respectively. Pinv(M) represents matrix multiplication. RGB ) is M RGBThe pseudo-inverse is then calculated. Finally, multiplying the last dimension of image A by the dynamic CCM yields a high spatial resolution, high-fidelity XYZ image of size H×W×3. This image is then converted to the sRGB color gamut using a standard transform, resulting in a more accurate sRGB image of size H×W×3. As seen in the calculations above, the dynamic CCM is the optimal CCM calculated based on the color components of the target scene. This means that the calculated dynamic CCM changes with the shooting scene, achieving higher color reproduction accuracy compared to traditional CCM.

[0318] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the imaging process includes:

[0319] Obtain an original image of size H×W, where H and W are the height and width of the predetermined imaging array of the imaging chip, respectively;

[0320] After removing the pixel values ​​of the multispectral pixels of the multispectral imaging unit (i.e., removing multispectral pixels of other types besides the imaging pixel types in the Bayer array), interpolation is performed to obtain an original RGB image of size H×W×3.

[0321] The pixel values ​​of m types of filters contained in the multispectral imaging unit as multispectral pixels are extracted from the original image to obtain an original multichannel image of size N×m, where N is the total number in the multispectral imaging unit.

[0322] The original multi-channel image is multiplied by the color correction matrix of size m×3 to obtain an XYZ image of size N×3;

[0323] Extract the pixel values ​​of the RGB filters from the m filters in the original N×m multi-channel image to obtain an original RGB multi-channel image of size N×3;

[0324] The pseudo-inverse matrix of the original RGB multi-channel image is multiplied with the XYZ image to obtain the dynamic color correction matrix;

[0325] Multiply the last dimension of the original RGB image by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and

[0326] The high-fidelity XYZ image is converted to the sRGB color gamut using a standard transformation to obtain a color-reproduced sRGB image with a size of H×W×3.

[0327] Furthermore, based on the above, it is also possible to determine whether a multispectral imaging unit is located at the edge of an image by calculating the color gradient within the multispectral imaging unit. The color gradient represents the degree of color change within the multi-source imaging unit and can be calculated using the following formula: grad=max(f(S1),f(S2),…,f(S m ))

[0328] S i ∈ Multispectral Imaging Unit

[0329] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i This refers to the pixel value corresponding to the i-th filter within the multispectral imaging unit. When the color gradient is less than a certain threshold (for example, a threshold of 0.05 can be used), the multispectral imaging unit is considered not to be at the image edge. By calculating the color gradient of each multispectral imaging unit, multispectral imaging units not at the edge can be filtered out. Then, in images B and C, only those pixel values ​​not at the edge (where N ≤ the total number of multispectral imaging units) are extracted for dynamic CCM calculation. This avoids the impact of edge-induced false color on the accuracy of dynamic CCM calculation, achieving more accurate dynamic CCM calculation.

[0330] That is, in the imaging chip containing the multispectral imaging unit, extracting the pixel values ​​of m types of filters (which serve as multispectral pixels) contained in the multispectral imaging unit from the original image to obtain an original multichannel image of size N×m includes: calculating the color gradient of each multispectral imaging unit based on the following formula: grad=max(f(S1),f(S2),…,f(S... m ))

[0331] S i ∈ Multispectral Imaging Unit

[0332] Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i It is the pixel value corresponding to the i-th filter in the multispectral imaging unit;

[0333] The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and

[0334] Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

[0335] It should be noted that the above calculation process is merely an example and does not constitute a limitation. In some embodiments, multiple multispectral imaging units can be used together for calculation. That is, the multispectral imaging units, in terms of structural arrangement, conform to the starting point of this application. In actual calculation, multiple multispectral imaging units can be combined to form a multispectral imaging unit calculation unit for calculation. In particular, when an imaging pixel unit includes two or more multispectral imaging units, and at least two types of multispectral imaging units exist among the two or more multispectral imaging units (at least one multispectral pixel is different), the different multispectral imaging units can be combined to form a multispectral imaging unit calculation unit for calculation. That is, a single multispectral imaging unit calculation unit includes n different multispectral imaging units; wherein, in some embodiments, at least two of these n different multispectral imaging units can be the same. Alternatively, multiple identical multispectral imaging units can also be combined to form a multispectral imaging unit calculation unit for calculation.

[0336] Therefore, in the imaging chip containing the multispectral imaging unit, the calculation is performed on a unit basis, and each multispectral imaging unit calculation unit includes different multispectral imaging units and / or the same multispectral imaging unit.

[0337] In the above calculation process, on the one hand, in order to reduce the impact of bad pixel removal on image spatial resolution, the density of bad pixels should be as small as possible, that is, the embedding density of multispectral imaging units should be as small as possible. Here, it can be stipulated that the proportion of multispectral pixels in the embedded multispectral imaging units (excluding imaging pixels used for imaging, such as C, M, Y pixels, and the total number of multispectral pixels used for multispectral imaging, such as R, G, B pixels) to the total number of pixels in the imaging chip should be less than or equal to 25%. That is, for example, if the imaging pixels used for image imaging are based on a Bayer array, then when the imaging pixels included are RGGB pixels, the multispectral pixels can be C, Y, M, GRAY (gray, hereinafter referred to as GR), variants of G, W, IR, etc., while if the imaging pixels of the Bayer array are RYYB, the multispectral pixels can be C, M, GR, G and its variants, W, IR, etc. Preferably, the ratio is less than or equal to 10%. Furthermore, preferably, within each multispectral imaging unit, the proportion of multispectral pixels should be greater than or equal to 2 / (h0×w0) and less than or equal to 50%, preferably less than or equal to 25%. On the other hand, to ensure the accuracy of dynamic CCM calculation, the number of multispectral imaging units cannot be too small, i.e., N cannot be too small. Here, it can be specified that the total number of multispectral pixels in all multispectral imaging units accounts for a proportion greater than 0.1% of the total number of pixels in the imaging chip, preferably greater than 1%. In addition, to facilitate the calculation of color gradients within multispectral imaging units, it is preferably required that the number of at least one type of filter in each multispectral imaging unit is greater than 1.

[0338] That is, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

[0339] Furthermore, in the imaging chip containing the multispectral imaging unit described above, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

[0340] Furthermore, in the imaging chip containing the multispectral imaging units, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral pixel unit, and less than or equal to 50%.

[0341] Furthermore, in the imaging chip containing the multispectral imaging units described above, the ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

[0342] Furthermore, in the imaging chip including a multispectral imaging unit according to the embodiments of this application, the ratio of the number of multispectral pixels of the multispectral imaging unit to the total number of pixels of the imaging chip is greater than 0.1%.

[0343] Furthermore, in the imaging chip containing the multispectral imaging unit, the ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

[0344] Furthermore, in the imaging chip containing multispectral imaging units according to embodiments of this application, each multispectral imaging unit contains at least one type of pixel, i.e., the number of multispectral pixels or imaging pixels is greater than 1.

[0345] Below, specific examples of an imaging chip including a multispectral imaging unit according to embodiments of this application will be described.

[0346] Figure 22 illustrates a schematic diagram of a seventh example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 22, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents the multispectral imaging unit, which is a 4×4 rectangle (h0=4, w0=4) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, the multispectral pixels can also be other types of filters, such as C and M filters; or Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a pixels and the vertical spacing is 2b pixels. In this arrangement, G accounts for 50%, and C and Y each account for 2 / ((2a+4)×(2b+4)). Furthermore, in this example, the multispectral imaging unit may also include a predetermined imaging array composed of other types of imaging pixels, such as imaging arrays other than Bayer arrays, including other types of imaging pixels, such as RYYB, RGBW, etc., as are the other examples described below.

[0347] Figure 23 illustrates a schematic diagram of an eighth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 23, unlike the seventh example, one set of C and Y in the multispectral imaging unit is replaced with two other filters M and W, while the other set of C and Y remains unchanged. Thus, each multispectral unit contains 7 filters (m=7), enabling more accurate spectral reconstruction and color reproduction. A modified embodiment exists, containing 6 filters (m=6), where one multispectral pixel replaces two imaging pixels.

[0348] Figure 24 illustrates a ninth example of an imaging chip including multispectral imaging units according to an embodiment of this application. As shown in Figure 24, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least two multispectral imaging units (n=2). That is, although two multispectral imaging units are shown in Figure 7, this example can also include more than two multispectral imaging units. The size of the minimum repeating unit is 8×8, and the horizontal and vertical spacing between adjacent multispectral imaging units is 1 pixel. In this arrangement, C and Y each account for 3.125%.

[0349] Figure 25 illustrates a schematic diagram of a tenth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 25, multispectral pixels C and Y are embedded in a Bayer array arranged in "RGGB". The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five types of filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, G accounts for 50%, while C and Y each account for 1 / ((2a+4)×(2b+4)).

[0350] Figure 26 illustrates a schematic diagram of an eleventh example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 26, multispectral pixels C and Y are embedded in a Bayer array arranged in “RGGB”. The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five types of filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can also be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains two multispectral imaging units (n=2). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, R and B each account for 25%, while C and Y each account for 1 / ((2a+4)×(2b+4)).

[0351] Figure 27 illustrates a schematic diagram of a twelfth example of an imaging chip including a multispectral imaging unit according to an embodiment of this application. As shown in Figure 27, multispectral pixels C and Y are embedded in a Bayer array arranged in “RGGB”. The dashed rectangle in the figure represents a multispectral imaging unit, which is a 3×3 rectangle (h0=3, w0=3) containing five types of filters (m=5). For example, it can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. The multispectral pixels can be other types of filters, such as C and M filters, Y and M filters, etc. The solid rectangle in the figure represents the imaging pixel unit as the minimum repeating unit. Each minimum repeating unit contains at least one multispectral imaging unit (n=1). The size of the minimum repeating unit is (2a+4)×(2b+4), that is, the horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. In this arrangement, R and B each account for 25%, while C and Y each account for 2 / ((2a+4)×(2b+4)).

[0352] Figure 28 illustrates a schematic diagram of a first example of an imaging pixel unit comprising at least two types of multispectral imaging units according to an embodiment of this application. As shown in Figure 28, the imaging pixel unit includes at least two types of multispectral imaging units, wherein the first multispectral imaging unit and the second multispectral imaging unit are each implemented as a 2×4 rectangle, each containing five types of filters (m=5). For example, the first multispectral imaging unit can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters. Of course, multispectral pixels can also be other types of filters; while the second multispectral imaging unit can be implemented as R, G, B, GR, M filters.

[0353] Furthermore, the imaging pixel unit may include at least one first multispectral imaging unit, such as two as shown in the figure; the imaging pixel unit may also include at least one second multispectral imaging unit, such as two as shown in the figure. In this arrangement, the proportion of each multispectral pixel is 0.78%, that is, the proportion of C, Y, GR, and M is 0.78%. During the calculation process, the first multispectral imaging unit and the second multispectral unit can be used as separate multispectral unit calculation units, or they can be combined into a single multispectral imaging unit calculation unit for calculation. For example, two first multispectral pixels and two second multispectral pixels can be combined into a 4×16 multispectral imaging unit calculation unit for calculation.

[0354] Figure 29 illustrates a schematic diagram of a second example of an imaging pixel unit comprising at least two types of multispectral imaging units according to an embodiment of this application. As shown in Figure 29, when the imaging pixel unit includes at least two types of multispectral imaging units, the first multispectral imaging unit and the second multispectral imaging unit are each implemented as a 2×4 rectangle, each containing five types of filters (m=5). For example, the first multispectral imaging unit can be implemented as R, G, B, C, Y filters, that is, multispectral pixels other than RGB are C and Y filters, and multispectral pixels can also be other types of filters; while the second multispectral imaging unit can be implemented as R, G, B, GR, M filters. Simultaneously, the imaging pixel unit includes at least one of the first multispectral imaging units, for example, four as shown in the figure; and the imaging pixel unit includes at least one of the second multispectral imaging units, for example, four as shown in the figure. In this arrangement, the proportion of each multispectral pixel is 1.56%, that is, the proportion of C, Y, W, and M is 1.56%. During the calculation process, the first multispectral imaging unit and the second multispectral unit can be used as separate multispectral unit calculation units for calculation; alternatively, the first multispectral imaging unit and the second multispectral unit can be combined into a single multispectral unit calculation unit for calculation; for example, two first multispectral pixels and two second multispectral pixels can be combined into a 4×16 multispectral unit calculation unit for calculation; alternatively, four first multispectral pixels and four second multispectral pixels can be combined into an 8×32 multispectral unit calculation unit for calculation.

[0355] Of course, in the examples shown in Figures 28 and 29 above, the multispectral pixels in the first and second multispectral imaging units can also be implemented as other types of filters. For example, the first multispectral pixel can be an R, G, B, C, Y filter, and the second multispectral pixel can be an R, G, B, C, M filter. That is, the same multispectral pixels can also exist. In this arrangement, the proportion of one type of multispectral pixel is twice or more integer multiples of the proportions of other multispectral pixels, that is, the proportion of C is twice the proportion of M or Y.

[0356] In the examples corresponding to Figures 22-29 above, RGGB is used as an example for illustration, but it can cover other types, such as RYYB, which can be considered as replacing the Y and G filters; or CYYM, RGBW, and other chips. Other types of Bayer arrays can also follow the above principle. Therefore, although RGGB is used as an example, it covers other array types. The description of the proportion of R, G, and B filters in the embodiments can be understood as a specific description of the proportion of imaging pixels in the Bayer array, and does not limit the Bayer array to an RGGB array.

[0357] Furthermore, in this application, the filter can adopt a photonic crystal merging hybrid structure. Figure 30 illustrates a cross-sectional schematic diagram of a first structural example of the filter of a multispectral chip according to an embodiment of this application. As shown in Figure 30, in the multispectral chip, for example, the imaging pixel channel uses red, green, and blue filters, while the multispectral pixels are composed of filter materials (dyes or pigments) with different transmission spectrum curves than the red, green, and blue filters. The height of the filters above all image sensor pixels is as similar as possible, i.e., the height difference does not exceed 0.4 μm, and there may be a microlens structure above the filter, which is not shown in the figure.

[0358] Figure 31 illustrates a cross-sectional schematic diagram of a second structural example of a filter for a multispectral chip according to an embodiment of this application. As shown in Figure 31, the multispectral pixel can be composed of a filter structure formed by at least one single-layer metal or dielectric photonic crystal, wherein the height (thickness) from the bottom of the filter structure to the top of the image sensor pixel is approximately 300–500 nm, and preferably filled with a dielectric material of 400 nm height. Further, the filter structure is covered with a dielectric material of 200–400 nm to protect the filter structure and also to make the upper surface flat. In this example, the filter structure can be a metal pillar or a dielectric pillar or a metal hole or a dielectric hole. Here, the pillar and hole are not limited to circles and squares, but can also be rectangular, strip-shaped or other irregular shapes. Preferably, the gaps between the pillars or the holes are filled with dielectric material. Further, a filter includes at least one filter structure (metal pillar or dielectric pillar, or metal hole or dielectric hole), wherein the height of the filter structure (a single pillar or hole, etc.) is 10–500 nm, and the corresponding lateral dimension is 70–600 nm.

[0359] Figure 32 illustrates a cross-sectional schematic diagram of a third structure example of a filter for a multispectral chip according to an embodiment of this application. As shown in Figure 32, the multispectral pixel is composed of at least one filter structure, wherein the filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. The height from the bottom of the filter structure to the top of the image sensor pixel is 100-300 nm, preferably 200 nm, and preferably filled with a dielectric material. Preferably, the filter structure is covered with a dielectric material of 200-400 nm to protect the filter structure and also to make the upper surface flat. In this example, the filter structure can be a metal pillar or a dielectric pillar or a metal hole or a dielectric hole. Here, the pillar and hole are not limited to circles and squares, but can also be rectangular, strip-shaped or other irregular shapes. Preferably, the gaps between the pillars or the holes are filled with dielectric material. Further, a filter includes at least one filter structure (metal pillar or dielectric pillar, or metal hole or dielectric hole), wherein the height of the filter structure is 10-500 nm, and the corresponding lateral dimension is 70-600 nm.

[0360] Figure 33 illustrates a cross-sectional schematic diagram of a fourth structure example of a filter for a multispectral chip according to an embodiment of this application. As shown in Figure 33, the multispectral pixel is composed of at least one filter structure and a filter material covering the filter structure. The filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. The height from the bottom of the filter structure to the top of the image sensor pixel is 100-300 nm, preferably 200 nm, and preferably, it is filled with dielectric material. Preferably, the filter structure is covered with 200-400 nm of dielectric material to protect the filter structure and also to make the upper surface flat. In this embodiment, the filter structure can be a metal pillar or a dielectric pillar, or a metal hole or a dielectric hole. Here, the pillar and hole are not limited to circles and squares, but can also be rectangular, strip-shaped or other irregular shapes. Preferably, the gaps between the pillars or the holes are filled with dielectric material. Further, a filter includes at least one filter structure (metal pillar or dielectric pillar, or metal hole or dielectric hole), wherein the height of the filter structure is 10-500 nm, and the corresponding lateral dimension is 70-600 nm. Furthermore, the height difference between the multispectral pixel and the imaging pixel is less than or equal to 0.4 μm.

[0361] Figure 34 illustrates a cross-sectional schematic diagram of a fifth structure example of a filter for a multispectral chip according to an embodiment of this application. As shown in Figure 34, the multispectral pixel is composed of at least two layers, each layer consisting of at least one filter structure. The filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. The height from the bottom of the filter structure to the top of the image sensor pixel is 100–300 nm, preferably 200 nm. The thickness of the dielectric material between each layer is 100–300 nm, preferably 200 nm, and preferably, it is filled with dielectric material. Taking two layers as an example, the distance between the first and second layers is 100–300 nm, preferably 200 nm. Preferably, the filter structure is covered with a 200–400 nm layer of dielectric material to protect the filter structure and also to make the upper surface flat. In this embodiment, the filter structure can be a metal pillar, a dielectric pillar, or a metal hole or a dielectric hole. The pillars and holes are not limited to circles and squares; they can also be rectangular, strip-shaped, or other irregular shapes. Preferably, the gaps between the pillars or the holes are filled with a dielectric material. Furthermore, a filter includes at least one filter structure (metal pillar or dielectric pillar, or metal hole or dielectric hole), wherein the height of the filter structure is 10–500 nm, and the corresponding lateral dimension is 70–600 nm. Furthermore, the two layers can be identical or different.

[0362] Figure 35 illustrates a cross-sectional schematic diagram of a sixth structural example of a filter for a multispectral chip according to an embodiment of this application. As shown in Figure 35, the multispectral pixel is composed of at least two layers, each layer consisting of at least one filter structure. The filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. The height from the bottom of the filter structure to the top of the image sensor pixel is 300–700 nm, preferably 400 nm. The thickness of the dielectric material between each layer is 100–300 nm, preferably 200 nm, and preferably, it is filled with dielectric material. Taking two layers as an example, the distance between the first and second layers is 100–300 nm, preferably 200 nm. Preferably, the filter structure is covered with a 200–400 nm layer of dielectric material to protect the filter structure and also to make the upper surface flat. In this embodiment, the filter structure can be a metal pillar, a dielectric pillar, or a metal hole or a dielectric hole. The pillars and holes are not limited to circles and squares; they can also be rectangular, strip-shaped, or other irregular shapes. Preferably, the gaps between the pillars or the holes are filled with a dielectric material. Further, a filter includes at least one filter structure (metal pillar or dielectric pillar, or metal hole or dielectric hole), wherein the height of the filter structure is 10–500 nm, and the corresponding lateral dimension is 70–600 nm. Further, the two structures can be identical or different. Further, the imaging chip includes a metal grid structure formed between the image sensor and the filter structure, thereby ensuring that the filter structure and the filter of the imaging chip have the same initial height during manufacturing and are free from MG interference, resulting in a simpler process and lower cost.

[0363] Figure 36 illustrates a cross-sectional schematic diagram of a seventh structure example of a filter for a multispectral chip according to an embodiment of this application. As shown in Figure 36, the multispectral pixel is composed of at least one filter structure and a filter material covering the filter structure. The filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. The height from the bottom of the filter structure to the top of the image sensor pixel is 300-700 nm, preferably 400 nm, and preferably, it is filled with dielectric material. Preferably, the filter structure is covered with 200-400 nm of dielectric material to protect the filter structure and also to make the upper surface flat. In this embodiment, the filter structure can be a metal pillar or a dielectric pillar, or a metal hole or a dielectric hole. Here, the pillar and hole are not limited to circles and squares, but can also be rectangular, strip-shaped or other irregular shapes. Preferably, the gaps between the pillars or the holes are filled with dielectric material. Further, a filter includes at least one filter structure (metal pillar or dielectric pillar, or metal hole or dielectric hole), wherein the height of the filter structure is 10-500 nm, and the corresponding lateral dimension is 70-600 nm. Furthermore, the height difference between the multispectral pixels and the imaging pixels is less than or equal to 0.4 μm. Furthermore, the imaging chip includes a metal grid structure formed between the image sensor and the filter structure, thereby ensuring that the filter structure and the filter of the imaging chip have the same initial height during manufacturing and are free from MG interference, resulting in a simpler process and lower cost.

[0364] Figure 37A illustrates a cross-sectional schematic diagram of an eighth structure example of a filter for a multispectral chip according to an embodiment of this application. Figure 37B illustrates an enlarged schematic diagram of the filter structure example shown in Figure 37A. As shown in Figures 37A and 37B, the multispectral pixel is composed of a multilayer film and at least one filter structure. Specifically, the multilayer film of the multispectral pixel is divided into upper and lower reflective layers, wherein the upper and lower reflective layers are formed by sequentially stacking high-refractive-index and low-refractive-index materials, respectively, and a cavity is formed between the upper and lower reflective layers. At least one filter structure is formed in the cavity, and the filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. The height difference between the multispectral pixel and the imaging pixel is less than or equal to 0.4 μm. In some embodiments, both the upper and lower reflective layers may have metal layers formed, thereby reducing the overall height. In some embodiments, the upper and lower reflective layers may be composed of a multilayer film and a metal layer, respectively.

[0365] Figure 38A illustrates a cross-sectional schematic diagram of a ninth structure example of a filter for a multispectral chip according to an embodiment of this application. Figure 38B illustrates an enlarged schematic diagram of the filter structure example shown in Figure 37A. As shown in Figures 38A and 38B, the multispectral pixel is composed of a multilayer film and at least one filter structure. Specifically, the multilayer film of the multispectral pixel is divided into upper and lower reflective layers, wherein the upper and lower reflective layers are formed by sequentially stacking high-refractive-index and low-refractive-index materials, respectively, forming a cavity between the upper and lower reflective layers. At least one filter structure is formed in the cavity, and the filter structure can be composed of a single-layer metal or dielectric photonic crystal structure. Further, the imaging chip includes a metal grid structure, which is formed between the image sensor and the filter structure, so that the filter structure and the filter of the imaging chip have the same initial height during manufacturing and there is no MG interference, resulting in a simpler process and lower cost. In some examples, both the upper and lower reflective layers can have metal layers formed, thereby reducing the overall height. In some examples, the upper and lower reflective layers may be composed of multilayer films and metal layers, respectively.

[0366] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0367] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0368] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0369] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0370] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A filter array for a multispectral chip, comprising: The array comprises at least five filter units with different transmission spectral curves, arranged in a periodic array. Each array consists of n×m filter units, where n and m are both greater than or equal to 2. The at least five filter units include at least one main filter unit, and the ratio of the number of the at least one main filter unit to the number of filter units in the array is greater than or equal to 50%.

2. The filter array of the multispectral chip as described in claim 1, wherein, The main filter units are distributed at adjacent positions along a diagonal direction that forms a 45-degree angle with the row and column directions of the array units.

3. The filter array of the multispectral chip as described in claim 1, wherein, The main filter unit is a G filter unit, a Y filter unit, or a combination of a G filter unit and a Y filter unit.

4. The filter array of the multispectral chip as described in claim 1, wherein, The filter unit is symmetrical along at least one main diagonal of the array unit.

5. The filter array of the multispectral chip as described in claim 1, wherein, The at least five filter units include auxiliary filter units, and the types of auxiliary filter units in the auxiliary filter units are all different.

6. The filter array of the multispectral chip as described in claim 1, wherein, The at least five filter units include an auxiliary filter unit, and at least one of the auxiliary filter units is the same as at least one of the main filter units.

7. The filter array of the multispectral chip as described in claim 1, wherein, The at least five types of filter units include auxiliary filter units. The filter units adjacent to the auxiliary filter units on the top, bottom, left, and right sides are the main filter units. The two adjacent auxiliary filter units of the auxiliary filter unit along the diagonal direction with an angle of 45 degrees to the row and column directions of the filter array are of the same type.

8. The filter array of the multispectral chip as described in claim 1, wherein, The ratio of the height to the width of the filter unit is less than 1.

5.

9. The filter array of the multispectral chip as described in claim 8, wherein, The ratio of the height to the width of the filter unit is less than 1.

10. The filter array of the multispectral chip as described in claim 1, wherein, The thickness of the filter unit is 0.3 μm to 2 μm.

11. The filter array of the multispectral chip as described in claim 10, wherein, The thickness of the filter unit is 0.5 μm to 1 μm.

12. The filter array of the multispectral chip as described in claim 1, wherein, When the filter array does not include an infrared filter unit, the thickness of the filter unit is 0.5 μm to 0.8 μm; when the filter array includes an infrared filter unit, the thickness of the infrared filter unit is 0.5 μm to 1 μm, and the thickness of the other filter units is 0.5 μm to 0.8 μm.

13. The filter array of the multispectral chip as described in claim 1, wherein, The thickness of the same type of filter unit in different array units is the same.

14. The filter array of the multispectral chip as described in claim 1, wherein, The optical response curves of the at least five filter units are not zero in the infrared band.

15. The filter array of the multispectral chip as described in claim 14, wherein, The light response curves of the at least five filter units have an average transmittance of greater than 5% in the infrared band.

16. The filter array of the multispectral chip as described in claim 1, wherein, The total fitting error of the light response curves of the at least five filter units to the tristimulus value curves and IR channel curves is less than or equal to 0.

25.

17. An imaging chip comprising a multispectral imaging unit, comprising: The imaging pixel units are arranged in a periodic manner. Each imaging pixel unit includes imaging pixels for imaging based on a predetermined imaging array and multispectral pixels of a different type for multispectral imaging. The multispectral pixels replace one or more imaging pixels to form at least one multispectral imaging unit. The sum of the number of pixel types of the imaging pixels and the multispectral pixels in the multispectral imaging unit is greater than or equal to five. The multispectral imaging unit is formed as a rectangular array composed of imaging pixels and multispectral pixels.

18. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

19. The imaging chip comprising a multispectral imaging unit as described in claim 18, wherein, The multispectral pixels of one of the multispectral imaging units are located in different imaging pixel units.

20. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The length and width of a rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

21. The imaging chip comprising a multispectral imaging unit as described in claim 20, wherein, The length and width of a rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

22. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The imaging process includes: Obtain an original image of size H×W, where H and W are the height and width of the predetermined imaging array of the imaging chip, respectively; After removing the pixel values ​​of the multispectral pixels of the multispectral imaging unit, interpolation is performed to obtain an original RGB image of size H×W×3. The pixel values ​​of m types of filters contained in the multispectral imaging unit as multispectral pixels are extracted from the original image to obtain an original multichannel image of size N×m, where N is the total number of the multispectral imaging units. The original multi-channel image is multiplied by the color correction matrix of size m×3 to obtain an XYZ image of size N×3; Extract the pixel values ​​of the RGB filters from the m filters in the original N×m multi-channel image to obtain an original RGB multi-channel image of size N×3; The pseudo-inverse matrix of the original RGB multi-channel image is multiplied with the XYZ image to obtain the dynamic color correction matrix; Multiply the last dimension of the original RGB image by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and The high-fidelity XYZ image is converted to the sRGB color gamut using a standard transformation to obtain a color-reproduced sRGB image with a size of H×W×3.

23. The imaging chip comprising a multispectral imaging unit as described in claim 22, wherein, Extracting pixel values ​​from the m filters (which serve as multispectral pixels) within the multispectral imaging unit from the original image to obtain an original multichannel image of size N×m includes: The color gradient of each multispectral imaging unit is calculated based on the following formula: grad = max(f(S1), f(S2),..., f(S m )) S i ∈ multispectral imaging unit wherein grad denotes the color gradient, max denotes the maximum value, min denotes the minimum value, S i is the pixel value corresponding to the i-th filter in the multispectral imaging unit; The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

24. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

25. The imaging chip comprising a multispectral imaging unit as described in claim 24, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

26. The imaging chip comprising a multispectral imaging unit as described in claim 25, wherein, The ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral imaging unit, and less than or equal to 50%.

27. The imaging chip comprising a multispectral imaging unit as described in claim 26, wherein, The ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

28. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 0.1%.

29. The imaging chip comprising a multispectral imaging unit as described in claim 28, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

30. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, Each multispectral imaging unit contains at least one type of multispectral pixel or the number of imaging pixels is greater than 1.

31. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The multispectral imaging unit is a 4×4 rectangular array containing five types of pixels. Each imaging pixel unit contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)×(2b+4)).

32. The imaging chip comprising a multispectral imaging unit as described in claim 31, wherein, The imaging pixels are R, G, and B pixels, and the multispectral pixels are other types of pixels besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

33. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The multispectral imaging unit is a 4×4 rectangular array containing seven types of pixels. Each imaging pixel unit contains one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a pixels, and the vertical spacing is 2b pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 2 / ((2a+4)×(2b+4)).

34. The imaging chip comprising a multispectral imaging unit as described in claim 33, wherein, The imaging pixels are R, G, and B pixels, and the multispectral pixels are other types of pixels besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

35. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains two multispectral imaging units, and the size of the imaging pixel unit is 8×8. The horizontal and vertical spacing between adjacent multispectral imaging units is 1 pixel. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 3.125%.

36. The imaging chip comprising a multispectral imaging unit as described in claim 35, wherein, The imaging pixels are R, G, and B filters, and the multispectral pixels are other types of filters besides R, G, and B filters.

37. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

38. The imaging chip comprising a multispectral imaging unit as described in claim 37, wherein, The imaging pixels are R, G, and B pixels, and the multispectral pixels are other types of pixels besides R, G, and B pixels, and the ratio of G pixels to the total number of pixels in the imaging chip is 50%.

39. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains two multispectral imaging units, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

40. The imaging chip comprising a multispectral imaging unit as described in claim 39, wherein, The imaging pixels are R, G, and B pixels, and the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels in the imaging chip is 25%.

41. The imaging chip comprising a multispectral imaging unit as described in claim 17, wherein, The multispectral imaging unit is a 3×3 rectangular array containing five types of pixels. Each imaging pixel unit contains at least one multispectral imaging unit, and the size of the imaging pixel unit is (2a+4)×(2b+4). The horizontal spacing between adjacent multispectral imaging units is 2a+1 pixels, and the vertical spacing is 2b+1 pixels. The ratio of the number of pixels of each type of multispectral pixel to the total number of pixels of the imaging chip is 1 / ((2a+4)×(2b+4)).

42. The imaging chip comprising a multispectral imaging unit as described in claim 41, wherein, The imaging pixels are R, G, and B pixels, and the multispectral pixels are pixels of other types besides R, G, and B pixels, and the ratio of the R pixels and the B pixels to the total number of pixels in the imaging chip is 25%.

43. An imaging chip comprising a multispectral imaging unit, comprising: At least one imaging pixel unit, the imaging pixel unit comprising imaging pixels for image imaging and multispectral imaging and multispectral pixels for multispectral imaging, the imaging pixels and the multispectral pixels being pixels of different types with different transmission spectrum curves, the imaging pixels and the multispectral pixels constituting at least one multispectral imaging unit, the sum of the number of types of the imaging pixels and the multispectral pixels in the multispectral imaging unit being greater than or equal to five, and the multispectral imaging unit being formed as a rectangular array composed of imaging pixels and multispectral pixels.

44. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The different types of pixels include pixels made of different filter materials with different transmission spectrum curves, and / or pixels made of the same filter material with different transmission spectrum curves but different parameters of the filter material.

45. The imaging chip comprising a multispectral imaging unit as described in claim 44, wherein, The different types of pixels include pixels using the following filter materials: Red filter material: transmittance >20% for wavelengths above 580nm, and transmittance <20% for other wavelengths; The first green filter material has a transmittance of more than 20% at wavelengths of 475–630 nm and above 690 nm, and less than 20% at other wavelengths. The second green filter material has a transmittance of more than 20% at wavelengths of 475–610 nm and above 690 nm, and less than 20% at other wavelengths. The third type of green filter material has a transmittance of more than 20% at wavelengths of 470–650 nm and above 660 nm, and less than 20% at other wavelengths. Blue filter material: transmittance greater than 20% at wavelengths less than 520nm and greater than 785nm, and less than 20% at other wavelengths; Cyan filter material: transmittance greater than 20% at wavelengths less than 570nm and greater than 730nm, and less than 20% at other wavelengths; Yellow filter material: transmittance is greater than 20% at wavelengths greater than 470nm, and less than 20% at other wavelengths; Magenta filter material: transmittance is less than 20% at wavelengths of 520–580 nm, and greater than 20% at other wavelengths; as well as Infrared filter material: transmittance greater than 20% for wavelengths above 790nm, and less than 20% for other wavelengths.

46. ​​The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units may overlap or not overlap in the imaging pixel unit.

47. The imaging chip comprising a multispectral imaging unit as described in claim 46, wherein, Multispectral pixels belonging to the same multispectral imaging unit are located in different imaging pixel units.

48. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The length and width of a rectangular array of a single multispectral imaging unit are less than or equal to ten multispectral pixels.

49. The imaging chip comprising a multispectral imaging unit as described in claim 48, wherein, The length and width of a rectangular array of a single multispectral imaging unit are less than or equal to four multispectral pixels.

50. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The at least one multispectral imaging unit is two or more multispectral imaging units, and the two or more multispectral imaging units are different, wherein the different multispectral imaging units include at least one different type of multispectral pixel.

51. The imaging chip comprising a multispectral imaging unit as described in claim 50, wherein, There exists at least one multispectral imaging unit with a quantity greater than or equal to one.

52. The imaging chip comprising a multispectral imaging unit as described in claim 50, wherein, The types of multispectral pixels in the different multispectral imaging units are completely different or partially different.

53. The imaging chip comprising a multispectral imaging unit as described in claim 50, wherein, The number of each type of multispectral imaging unit in the different multispectral imaging units is greater than one.

54. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The imaging process includes: Obtain an original image of size H×W, where H and W are the height and width of the predetermined imaging array of the imaging chip, respectively; After removing the pixel values ​​of the multispectral pixels of the multispectral imaging unit, interpolation is performed to obtain an original RGB image of size H×W×3. The pixel values ​​of m types of filters contained in the multispectral imaging unit as multispectral pixels are extracted from the original image to obtain an original multichannel image of size N×m, where N is the total number of the multispectral imaging units. The original multi-channel image is multiplied by the color correction matrix of size m×3 to obtain an XYZ image of size N×3; Extract the pixel values ​​of the RGB filters from the m filters in the original N×m multi-channel image to obtain an original RGB multi-channel image of size N×3; The pseudo-inverse matrix of the original RGB multi-channel image is multiplied with the XYZ image to obtain the dynamic color correction matrix; Multiply the last dimension of the original RGB image by the dynamic color correction matrix to obtain a high-fidelity XYZ image of size H×W×3; and The high-fidelity XYZ image is converted to the sRGB color gamut using a standard transformation to obtain a color-reproduced sRGB image with a size of H×W×3.

55. The imaging chip comprising a multispectral imaging unit as described in claim 54, wherein, Extracting pixel values ​​from the m filters (which serve as multispectral pixels) within the multispectral imaging unit from the original image to obtain an original multichannel image of size N×m includes: The color gradient of each multispectral imaging unit is calculated based on the following formula: grad=max(f(S1),f(S2),…,f(S m )) S i ∈ Multispectral Imaging Unit Where grad represents the color gradient, max represents the maximum value, min represents the minimum value, and S i It is the pixel value corresponding to the i-th filter in the multispectral imaging unit; The multispectral imaging unit is determined to have a color gradient less than or equal to a predetermined threshold; and Extract the pixel values ​​of m filters from the determined multispectral imaging units to obtain an original multichannel image of size N×m, where N is the total number of determined multispectral imaging units.

56. The imaging chip comprising a multispectral imaging unit as described in claim 54, wherein, The calculation is performed on a unit basis, and each multispectral imaging unit calculation unit includes different multispectral imaging units and / or the same multispectral imaging units.

57. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 25%.

58. The imaging chip comprising a multispectral imaging unit as described in claim 57, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is less than or equal to 10%.

59. The imaging chip comprising a multispectral imaging unit as described in claim 58, wherein, The ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is greater than or equal to 2 / the total number of pixels in the multispectral imaging unit, and less than or equal to 50%.

60. The imaging chip comprising a multispectral imaging unit as described in claim 59, wherein, The ratio between the number of multispectral pixels in each multispectral imaging unit and the total number of pixels in the multispectral imaging unit is less than or equal to 25%.

61. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 0.1%.

62. The imaging chip comprising a multispectral imaging unit as described in claim 61, wherein, The ratio of the number of multispectral pixels in the multispectral imaging unit to the total number of pixels in the imaging chip is greater than 1%.

63. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, Each multispectral imaging unit contains at least one type of multispectral pixel or the number of imaging pixels is greater than 1.

64. The imaging chip comprising a multispectral imaging unit as described in claim 43, wherein, The at least one multispectral imaging unit includes at least two types of multispectral imaging units, wherein the first multispectral imaging unit and the second multispectral imaging unit in the at least two types of multispectral imaging units are 2×4 rectangles and each contains five types of pixels.

65. The imaging chip comprising a multispectral imaging unit as described in claim 64, wherein, The first multispectral imaging unit includes R, G, and B pixels as imaging pixels and C and Y pixels as multispectral pixels, and the second multispectral imaging unit includes R, G, and B pixels as imaging pixels and GR and M pixels as multispectral pixels.

66. The imaging chip comprising a multispectral imaging unit as described in claim 64, wherein, The number of the first multispectral imaging unit and the number of the second multispectral imaging unit are both two.

67. The imaging chip comprising a multispectral imaging unit as described in claim 66, wherein, The ratio of each type of multispectral pixel to the total number of pixels in the imaging chip is 0.78%.

68. The imaging chip comprising a multispectral imaging unit as described in claim 64, wherein, Each of the first multispectral imaging unit and the second multispectral unit is used as a separate multispectral unit computing unit for calculation, or one or more first multispectral imaging units and one or more second multispectral units are combined into a single multispectral imaging unit computing unit for calculation.

69. The imaging chip comprising a multispectral imaging unit as described in claim 64, wherein, The number of the first multispectral imaging unit and the number of the second multispectral imaging unit are both four.

70. The imaging chip comprising a multispectral imaging unit as described in claim 69, wherein, The ratio of each type of multispectral pixel to the total number of pixels in the imaging chip is 1.56%.

71. The imaging chip comprising a multispectral imaging unit as described in claim 69, wherein, Each of the first multispectral imaging unit and the second multispectral unit is used as a separate multispectral unit calculation unit for calculation, or a first number of first multispectral imaging units and a second number of second multispectral units are combined into a single multispectral imaging unit calculation unit for calculation, or a third number of first multispectral imaging units and a fourth number of second multispectral units are combined into a single multispectral imaging unit calculation unit for calculation, wherein the first number is equal to or not equal to the second number and not equal to the third number, and the second number is not equal to the fourth number.

Images