Image processing method and apparatus

Abstract

An image processing method and apparatus, which belong to the field of image processing. The method is applied to an electronic device, which comprises a first camera and a second camera, wherein the first camera comprises a micro-lens layer, a filter layer and a photosensitive layer that are stacked, and the filter layer is used for multi-spectral filtering and polarization filtering; and the second camera is an imaging camera. The method comprises: an electronic device acquiring multi-spectral information and polarization information by means of a first camera; the electronic device acquiring image information by means of a second camera; and the electronic device processing the image information on the basis of the multi-spectral information and the polarization information. In the solution, polarization information and multi-spectral information are acquired by means of a camera other than an imaging camera (i.e., a second camera), and the polarization information is effectively acquired and used by means of a first camera without sacrificing the light sensitivity of the imaging camera and reducing the resolution of an image, thereby effectively improving image quality.

Description

Image processing methods and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202510354080.9, filed on March 24, 2025, entitled "Image Processing Method and Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more specifically, to an image processing method and apparatus. Background Technology

[0003] During the imaging process of a scene, processing the image using the scene's polarization information can improve image sharpness and contrast. For example, electronic devices can acquire polarization information through the polarization structure in an imaging camera and process the image based on this information, thereby improving image sharpness and contrast. However, the polarization structure reduces the overall amount of light entering the imaging camera and causes uneven light reception, resulting in a loss of the camera's sensitivity, reduced image resolution, and a decline in image quality.

[0004] Therefore, how to effectively improve image quality is an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and apparatus that can effectively improve image quality.

[0006] In a first aspect, an image processing method is provided. This method is applied to an electronic device, which includes a first camera and a second camera. The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer has at least four spectral channels. The second camera is an imaging camera. The method includes: the electronic device acquiring multispectral information and polarization information through the first camera; the electronic device acquiring image information through the second camera; and the electronic device processing the image information based on the multispectral information and the polarization information.

[0007] Based on the above scheme, the first camera can be equipped with a filter layer that has multispectral filtering and polarization filtering functions. In this way, the electronic device can acquire multispectral and polarization information through the first camera, and process the image information acquired by the second camera, thereby improving the contrast, clarity, and color accuracy of the image information. In the above scheme, the polarization and multispectral information are acquired by a camera other than the imaging camera (i.e., the second camera). Therefore, the above scheme effectively acquires and uses polarization information through the first camera without sacrificing the sensitivity of the imaging camera or reducing the image resolution, thus effectively improving image quality.

[0008] In some implementations, the filter layer includes a multispectral layer and a polarization filter layer.

[0009] Based on the above scheme, the filter layer can include a multispectral layer and a polarization filter layer. The multispectral layer provides multispectral information, and the polarization filter layer provides polarization information. Thus, by using these two layers, a filter layer with both multispectral and polarization filtering functions can be easily implemented.

[0010] In some implementations, the filter layer has a micro / nano structure.

[0011] Based on the above scheme, the filter layer can have a micro / nano structure, which can have either multispectral filtering or polarization filtering functions, or both. Therefore, the filter layer can be a single layer, saving space.

[0012] In some implementations, the first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

[0013] Based on the above scheme, multiple first pixels with the same spectral channel are arranged adjacently. This allows the electronic device to simultaneously read multiple first pixels, thereby quickly acquiring multispectral information. The multiple first pixels have different polarization angles. This allows the electronic device to read each pixel of the multiple first pixels one by one, thus simultaneously acquiring multispectral and polarization information. Therefore, the above scheme supports the electronic device in conveniently and flexibly acquiring only multispectral information, or simultaneously acquiring multispectral and polarization information, through two different reading methods.

[0014] In some implementations, the filter layer has at least two polarization channels.

[0015] In some implementations, the second camera is an RGB camera, a RYB camera, or an RGBW camera.

[0016] In some implementations, the electronic device acquires multispectral information and polarization information through the first camera, including: the electronic device performs visible light imaging through the first camera to acquire the multispectral information and polarization information.

[0017] In some implementations, before the electronic device processes the image information based on the multispectral information and the polarization information, the method further includes: the electronic device determining, based on the image information, that the image information corresponds to a preset scene.

[0018] Secondly, an image processing apparatus is provided. This apparatus is applied to an electronic device, which includes a first camera and a second camera. The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer has at least four spectral channels. The second camera is an imaging camera.

[0019] As an example, the image processing apparatus may include modules, units, or means corresponding one-to-one with the methods / operations / steps / actions described in the first aspect and any possible implementation of the first aspect, which may be hardware circuits, software, or a combination of hardware circuits and software implementation.

[0020] For example, the image processing apparatus includes an acquisition unit and a processing unit; the acquisition unit is used to acquire multispectral information and polarization information through the first camera; the acquisition unit is also used to acquire image information through the second camera; the processing unit is used to process the image information based on the multispectral information and the polarization information.

[0021] In some implementations, the filter layer includes a multispectral layer and a polarization filter layer.

[0022] In some implementations, the filter layer has a micro / nano structure.

[0023] In some implementations, the first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

[0024] In some implementations, the filter layer has at least two polarization channels.

[0025] In some implementations, the second camera is an RGB camera, a RYB camera, or an RGBW camera.

[0026] In some implementations, the processing unit is specifically used to perform visible light imaging through the first camera to acquire the multispectral information and the polarization information.

[0027] In some implementations, the processing unit is also used to determine, based on the image information, that the image information corresponds to a preset scene.

[0028] Thirdly, an electronic device is provided, comprising: a first camera, a second camera, and a processor (or processing circuitry). The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer, the filter layer being located between the microlens layer and the photosensitive layer, the filter layer being used to perform multispectral filtering and polarization filtering on light from the microlens layer, and the filter layer having at least four spectral channels; the second camera is an imaging camera; and the processor is used to execute the first aspect and any possible method thereof.

[0029] In some implementations, the electronic device also includes memory. In one possible implementation, the processor and memory are integrated together. In another possible implementation, the memory is located outside the electronic device. The processor may include one or more processors.

[0030] In some implementations, the processor is used to communicate with other devices via interface circuitry and to execute the first aspect and any possible method of the first aspect described above.

[0031] Fourthly, a computer-readable storage medium is provided that stores a computer program or instructions that, when executed on a computer, cause the first aspect and any possible method of the first aspect to be performed.

[0032] Fifthly, a computer program product is provided, comprising a computer program or instructions that, when executed on a computer, cause the first aspect and any possible method of the first aspect to be performed (or implemented).

[0033] In a sixth aspect, a chip is provided, including a processor for invoking a computer program or computer instructions in memory to cause the processor to execute any of the implementations in the first aspect described above.

[0034] In some implementations, the processor is coupled to the memory via an interface.

[0035] The description of the beneficial effects of any of the second to sixth aspects can be referred to the description of the beneficial effects of the first aspect. Attached Figure Description

[0036] Figure 1 is a schematic structural diagram of an electronic device according to an embodiment of this application.

[0037] Figure 2 is a schematic exploded view of a camera provided in an embodiment of this application.

[0038] Figure 3 is a schematic cross-sectional view of a camera provided in an embodiment of this application.

[0039] Figure 4 is a schematic cross-sectional view of the image sensor provided in an embodiment of this application.

[0040] Figure 5 is another schematic cross-sectional view of the image sensor provided in an embodiment of this application.

[0041] Figure 6 is a schematic diagram of the smallest repeating unit of the sensor pattern of the image sensor provided in the embodiment of this application.

[0042] Figure 7 is a schematic flowchart of an image processing method provided in an embodiment of this application.

[0043] Figure 8 is a schematic flowchart of another image processing method provided in an embodiment of this application.

[0044] Figure 9 is a schematic flowchart of another image processing method provided in an embodiment of this application.

[0045] Figure 10 is a schematic block diagram of another electronic device according to an embodiment of this application.

[0046] Figure 11 is a schematic block diagram of an image processing apparatus according to an embodiment of this application. Detailed Implementation

[0047] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0048] In the embodiments of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more, and "at least one" and "one or more" refer to one, two, or more than two. The singular expressions "a," "an," "the," "the," "this," and "this" are intended to also include expressions such as "one or more," unless the context explicitly indicates otherwise.

[0049] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0050] In the description of the embodiments of this application, the terms "upper," "lower," "inner," "outer," "vertical," and "horizontal," etc., indicate orientations or positional relationships relative to the indicated placement of components in the accompanying drawings. It should be understood that these directional terms are relative concepts, used for relative description and clarification, and not to indicate or imply a specific orientation that the device or component must have, or its construction and operation in a specific orientation. They can change accordingly depending on the orientation of the components in the accompanying drawings, and therefore should not be construed as limiting this application. Furthermore, "vertical" in this application is not strictly vertical, but within the allowable error range. "Parallel" is not strictly parallel, but within the allowable error range.

[0051] In the embodiments of this application, the same reference numerals are used to denote the same component or part. Furthermore, the parts in the drawings are not drawn to scale, and the dimensions and sizes of the parts shown are merely exemplary and should not be construed as limiting this application.

[0052] This application can relate to electronic devices. As an example, the aforementioned electronic device can be an electronic device with imaging capabilities, such as a mobile phone, personal digital assistant (PDA), tablet computer, laptop computer, camera, video recorder, smartwatch, smart wristband, point of sale (POS), in-vehicle system, television (e.g., smart screen), or wearable device. This application does not impose any special limitations on the specific form of the electronic device. For ease of explanation and understanding, the following description, with reference to Figure 1, uses a mobile phone as an example.

[0053] Figure 1 shows a schematic structural diagram of an electronic device according to an embodiment of this application.

[0054] For example, Figures 1(a) and (b) schematically show the front and back of the electronic device 100, respectively. As shown in Figure 1, the electronic device 100 may include a housing 101, a display panel (DP) 102, and a camera array 103.

[0055] As an example, housing 101 can form a receiving space for accommodating components of electronic device 100. Housing 101 also serves to protect electronic device 100 and support the entire device. Display screen 102 and camera array 103 can be disposed within the receiving space of housing 101 and connected to housing 101. In some examples, housing 101 may include a back cover opposite to display screen 102 and a mid-frame disposed between the back cover and display screen 102, with display screen 102 and camera array 103 fixed to the mid-frame. Housing 101 can be made of metal, plastic, ceramic, or glass, etc.

[0056] As an example, display screen 102 can be used to display images, such as images captured by camera array 103. Display screen 102 can be a liquid crystal display (LCD) screen, or an organic light emitting diode (OLED) screen, etc. Display screen 102 can be a regular screen, an irregularly shaped screen, or a foldable screen, etc. Display screen 102 can be located on the front and / or back of electronic device 100. Here, the front of electronic device 100 can be understood as the side facing the user when using electronic device 100, and the back of electronic device 100 can be understood as the side facing away from the user when using electronic device 100.

[0057] As an example, camera array 103 can be used to capture still images or videos. Camera array 103 includes multiple camera compact modules (CCMs) (or simply cameras). Exemplarily, as shown in FIG1, the camera array 103 may include at least one imaging camera 103a and at least one multispectral camera 103b.

[0058] For example, the imaging camera 103a can be a telephoto camera, a wide-angle camera, an ultra-wide-angle camera, or a depth-sensing camera.

[0059] For example, the multispectral camera 103b may include a lens assembly and an image sensor (or, a multispectral image sensor). Exemplarily, the multispectral sensor may include a filter array with multiple transmittances to obtain more comprehensive spectral information (or multispectral information, or color information) of the subject. As an example, this multispectral information can provide a basis for color reproduction of the subject in an image, and can also be used for spectral detection.

[0060] For example, multispectral image sensors can use multiple filter materials in their filter arrays, each with a different transmittance. Thus, the pixel (or the photosensitive unit or photodiode within that pixel) corresponding to each filter material can only acquire spectral information within the band corresponding to that transmittance. Compared to traditional image sensors, multispectral image sensors can obtain more spectral information (called multispectral information), thereby enabling color reproduction.

[0061] For example, spectral detection can be understood as follows: because different objects have different spectral characteristics (e.g., emission or reflection spectra), multispectral image sensors can determine the spectral information of objects by analyzing and calibrating the spectra, thereby detecting the type or characteristics of the objects. For instance, spectral detection may include freshness detection, or sunscreen detection, and so on.

[0062] Multispectral sensors can have multiple spectral channels (or color channels, or filter channels). For example, spectral channels can be defined based on peak wavelengths and transmittance. As an example, for practical applications, a range of peak wavelengths and transmittances can be defined, with each range corresponding to a spectral channel. This application does not limit the specific values ​​for defining spectral channels. In this way, different spectral channels have different transmittance curves, ensuring that a pixel corresponding to a particular spectral channel (or, the photosensitive unit or photodiode within that pixel) can only acquire (or, in other words, detect) spectral information within a fixed wavelength band.

[0063] In some examples, the number of spectral channels in a multispectral sensor can be greater than 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, or other values. As a specific example, the spectral channels of a multispectral sensor may include at least one of the following: green channel, yellow channel, blue channel, white channel, red channel, cyan channel, magenta channel, gray channel, or violet channel.

[0064] The following describes an exemplary meaning of multispectral information. Multispectral information can be spectral information obtained from a multispectral image sensor. Taking a spectral channel count of 4 as an example, the transmittance of the 4 spectral channels can vary significantly. Therefore, a multispectral image sensor can obtain the spectral characteristics of the subject in four ways. After inverse processing using an algorithm, the multispectral information of the subject can be obtained, thus enabling color restoration.

[0065] When the camera array 103 includes multiple multispectral cameras 103b, the multiple multispectral cameras 103b can be completely identical or different. For example, the multiple multispectral cameras 103b may have different focal lengths, different optical structures, or different optical parameters.

[0066] Similarly, when the camera array 103 includes multiple imaging cameras 103a, the multiple imaging cameras 103a can be completely identical or different. For example, the multiple imaging cameras 103a may have different focal lengths, different optical structures, or different optical parameters.

[0067] In some examples, each of the at least one multispectral camera 103b is capable of independent imaging, and the captured images can be directly presented to the user for viewing.

[0068] In some examples, each of the at least one imaging camera 103a is capable of independent imaging, and the captured images can be directly presented to the user for viewing. One or more of the at least one imaging camera 103a can serve as the main camera. Typically, the main camera is responsible for the primary shooting task, usually having the highest pixel count, providing higher resolution and a more powerful sensor, thereby meeting the user's photography needs in different scenarios.

[0069] In some examples, images acquired by multispectral camera 103b can be used to perform color calibration on images acquired by imaging camera 103a. For instance, an electronic device can use a multispectral image of a scene acquired by multispectral camera 103b (e.g., containing color distribution information of the scene) as a reference to perform color calibration on an image of the same scene acquired by imaging camera 103a. Exemplarily, imaging camera 103a can be a red-green-blue (RGB) camera, a red-yellow-blue (RYB) camera, or a red-green-blue-white (RGBW) camera, etc. Accordingly, images acquired by imaging camera 103a can be RGB images, RYB images, or RGBW images, etc.

[0070] One possible principle for color calibration of images acquired by imaging camera 103a using images acquired by multispectral camera 103b is that the spectral response curve of imaging camera 103a differs somewhat from that of the human eye, and it can perceive a smaller number of spectral bands, resulting in weaker color reproduction and a tendency for certain colors to be off-color in the acquired images. In contrast, the spectral response curve of multispectral camera 103b can be made less different from that of the human eye, and it has a stronger spectral perception capability (i.e., it can perceive more than 3 spectral bands, or even more). Therefore, it has a stronger color reproduction capability, and the colors in the acquired images are closer to the true colors after restoration. Thus, using images acquired by multispectral camera 103b as a benchmark for color calibration of imaging camera 103a can improve the color fidelity, or in other words, the color accuracy, of the output images from imaging camera 103a.

[0071] In some examples, the multispectral camera 103b or imaging camera 103a can be a vertical module or a foldable module (or periscope camera). A vertical camera can be understood as light entering the camera directly hitting the image sensor without bending the light path. A foldable camera can be understood as light entering the camera needing to pass through optical elements such as mirrors, lenses, and prisms before hitting the image sensor, resulting in a folded light path.

[0072] In some examples, the camera array 103 can be positioned on the front and / or back of the electronic device 100. When the camera array 103 is positioned on the front of the electronic device 100, it can also be referred to as a front-facing camera. When the camera array 103 is positioned on the back of the electronic device 100, it can also be referred to as a rear-facing camera. In some examples, when the display screen 102 can be folded, the camera array 103 can function as either a front-facing or rear-facing camera as the display screen 102 folds. It is understood that the placement of the camera array 103 can be determined according to actual needs, and the installation position shown in Figure 1 is merely illustrative.

[0073] In some examples, the electronic device 100 may also include a protective lens 104 for protecting the cameras in the camera array 103. The protective lens 104 is disposed on the housing 101 and covers the cameras. For example, when the protective lens 104 is used to protect the front-facing camera, the protective lens 104 may cover only the front-facing camera or cover the entire front of the electronic device 100. When the protective lens 104 covers the entire front of the electronic device 100, it can also be used to protect the display screen 102, in which case the protective lens 104 is the cover glass (CG). As another example, when the protective lens 104 is used to protect the rear-facing camera, the protective lens 104 may cover the entire back of the electronic device 100 or may be disposed only at the position corresponding to the rear-facing camera.

[0074] In some examples, the protective lens 104 may be made of glass, sapphire, ceramic, etc., and this application does not impose any special limitations on it. For example, the protective lens 104 may be transparent, allowing light from outside the electronic device 100 to enter the camera through the protective lens 104.

[0075] In some examples, the electronic device 100 may also include a circuit board and an image processor (not shown in FIG. 1). Exemplarily, the circuit board and image processor may be located within a receiving space formed by the housing 101, with the image processor fixed to and electrically connected to the circuit board. The image processor may be communicatively connected to a camera (e.g., a multispectral camera 103b or an imaging camera 103a) in the camera array 103. The image processor may be used to acquire and process image data from the camera. The communication connection between the camera and the image processor may include data transmission via electrical connections such as wiring, or data transmission via coupling or other means. It is understood that the camera and the image processor may also be communicatively connected through other means capable of data transmission.

[0076] In some examples, the electronic device 100 may also include an analog-to-digital converter (also known as an A / D converter, not shown in Figure 1). The analog-to-digital converter can be connected between the camera and the image processor. The analog-to-digital converter can be used to convert the signal generated by the camera into a digital image signal and transmit it to the image processor, whereby the image processor processes the digital image signal and finally displays the image or video on the display screen 102.

[0077] In some examples, the electronic device 100 may also include a memory (not shown in Figure 1), which can be communicatively connected to an image processor. After processing the digital image signal, the image processor can transfer the image to the memory so that the image can be retrieved from the memory and displayed on the display screen 102 at any time when it is needed to view the image later. In some examples, the image processor may also compress the processed digital image signal before storing it in the memory to save memory space.

[0078] For example, the image processor described above may also be called a processor or other names, which are not limited in this application.

[0079] It is understood that the structure shown in Figure 1 does not constitute a specific limitation on the electronic device 100. The electronic device 100 may include more or fewer components than shown in the figure. For example, the electronic device 100 may also include one or more of the following components: battery, flash, earpiece, buttons, sensors, etc. Alternatively, the electronic device 100 may not include the display screen 102, or the electronic device 100 may have a different component arrangement than shown in the figure.

[0080] Figures 2 and 3 show schematic diagrams of a camera provided in an embodiment of this application. Figure 2 is a schematic exploded view of the camera 200, and Figure 3 is a schematic cross-sectional view of the camera 200. The camera 200 in Figure 2 can be an exemplary structure of the multispectral camera or imaging camera in Figure 1. The structure of the camera 200 will be briefly described below with reference to Figures 2 and 3.

[0081] For ease of description, the optical axis direction of the camera 200 is defined as the Z direction, and the two directions perpendicular to the optical axis are the X direction and the Y direction, with the X direction perpendicular to the Y direction. In the Z direction, the side facing the object being photographed is the front side, and the side facing away from the object is the rear side. In the X and Y directions, the direction closer to the optical axis is the inner side, and the direction facing away from the optical axis is the outer side. In this embodiment, the optical axis direction is the direction in which the optical system transmits light.

[0082] Here, the definitions of X, Y, Z directions and front, back, inside, and outside also apply to the various figures described below. It should be noted that the above definitions of X, Y, Z directions and front, back, inside, and outside are merely for the convenience of describing the positional, connection, or motion relationships between the components in the embodiments of this application, and should not be construed as limiting the embodiments of this application.

[0083] As shown in Figures 2 and 3, the camera module 200 may include a housing 210, a lens assembly 220, a lens actuator 230, and a light sensing component 240.

[0084] The housing 210 has a receiving space for accommodating the lens assembly 220, the lens actuator 230, the light sensing assembly 240, etc. Additionally, the housing 210 also serves a protective and support function. It is understood that the structure of the housing 210 shown in Figures 2 and 3 is merely exemplary and should not be construed as limiting the scope of this application. Those skilled in the art can design the shape of the housing 210 according to actual needs.

[0085] The lens assembly 220 (or optical lens) mainly includes a lens group 221 and a lens barrel 222, with the lens group 221 housed within the receiving space formed by the lens barrel 222. The lens assembly 220 is used to image the scene on the object side onto the image plane on the image side. In some embodiments, the lens assembly 220 can also perform certain processing on the received imaging beam, such as aberration correction and chromatic aberration elimination. Here, the imaging beam refers to the beam formed by the light incident on the camera 200.

[0086] Lens group 221 may include at least one lens. The at least one lens may be different or at least partially the same. This application embodiment does not specifically limit the number of lenses included in lens group 221. Those skilled in the art can set the number of lenses according to actual needs, such as 1, 2, 3, 5, 8 or more.

[0087] The focal length of lens group 221 can be fixed, and correspondingly, lens assembly 220 is a prime lens. Alternatively, the focal length of lens group 221 can be adjusted, and correspondingly, lens assembly 220 is a zoom lens. For example, the focal length of lens group 221 can be adjusted by changing the relative positions of the lenses within it.

[0088] The lens barrel 222 has a receiving space, primarily for accommodating the lens assembly 221. In some embodiments, the lens barrel 222 can be a single unit, with the lens assembly 221 housed within this single unit. In other embodiments, the lens barrel 222 may also comprise multiple lens barrel sections, with the lenses of the lens assembly 221 grouped and disposed within these multiple lens barrel sections, wherein each lens barrel section and the lenses housed therein can be referred to as a lens group. Exemplarily, the relative positions between these multiple lens barrel sections can be adjusted, enabling optical zoom by adjusting the relative positions of the lenses.

[0089] It is understood that the structure of the lens barrel 222 and the connection method between the lens group 221 and the lens barrel 222 in Figures 2 and 3 are merely exemplary and do not impose any limitations on the embodiments of this application.

[0090] The lens actuator 230 is used to move the lens assembly 220 to achieve autofocus and / or optical image stabilization. In some embodiments, the lens actuator 230 may also be referred to as a lens motor, or simply a motor.

[0091] As shown in Figure 3, the lens assembly actuator 230 may include a motor (hereinafter referred to as the AF motor) 231 for moving the lens assembly 220 for autofocus (AF) and / or a motor (hereinafter referred to as the OIS motor) 232 for moving the lens assembly 220 for optical image stabilization (OIS). Specifically, the AF motor 231 can be used to move the lens assembly 220 for autofocus in the Z direction, and the OIS motor 232 can be used to move the lens assembly 220 for optical image stabilization in the X and / or Y directions. In some embodiments, the AF motor 231 and the OIS motor 232 may be two independent components, which independently drive the lens assembly 220 for AF and OIS respectively. Alternatively, the AF motor 231 and the OIS motor 232 may be integrated into one unit, with one motor driving the lens assembly 220 for AF and OIS. Figure 3 exemplarily illustrates that the lens assembly actuator 230 includes independent AF motor 231 and OIS motor 232, but it should be understood that the embodiments of this application are not limited thereto.

[0092] In some embodiments, the AF motor 231 or the OIS motor 232 can be used to move the entire lens assembly 220, or it can be used to move a part of the lens assembly 220. For example, if a part of the lens assembly 220 (such as the first lens group) is relatively fixed and another part (such as the second lens group) is movable, the AF motor 231 or the OIS motor 232 can drive the movable part to move, thereby changing the optical path to achieve the desired function.

[0093] In some embodiments, the AF motor 231 or the OIS motor 232 may be a voice coil motor (VCM), a shape memory alloy (SMA) motor, a stepping motor, a piezoelectric motor, etc. It should be understood that the specific structure of the AF motor 231 or the OIS motor 232 may be designed and selected according to the selected driving method, and the embodiments of this application do not limit this.

[0094] The light sensing component 240 is disposed on the rear side of the lens assembly 220 and is mainly used for imaging. Exemplarily, the light sensing component 240 may include a filter 241, an image sensor 242, and a circuit board 243.

[0095] A filter 241 can be disposed between the lens assembly 220 and the image sensor 242. The filter 241 can eliminate unwanted light projected onto the image sensor 242, preventing problems such as ghosting, stray light, and color cast during image formation. For example, the filter 241 can be an infrared cut-off filter (filtering out long-wavelength light other than visible light), a bandpass filter (allowing light of a specific wavelength to pass through while blocking light outside the passband), or a filter for filtering out other wavelengths of light. The filter 241 can have different effective ranges depending on the application scenario.

[0096] Image sensor 242 can be a semiconductor chip used to convert collected external light signals into electrical signals. Specifically, the surface of image sensor 242 contains hundreds of thousands to millions of photodiodes, which generate charges when illuminated, thereby converting the light signals collected by lens assembly 220 into electrical signals. For example, image sensor 242 can be a CCD or a CMOS sensor.

[0097] The image sensor 242 includes multiple photosensitive units, which can convert light signals into electrical charges to form an electronic image corresponding to the scene. Each photosensitive unit corresponds to one pixel; the more pixels, the clearer the image.

[0098] As an example, in the image sensor 242 used in the imaging camera 130a, each photosensitive unit includes three different spectral channels, such as red, green, and blue filter channels, or red, yellow, and blue filter channels, etc. For example, different spectral channels can be formed by placing three different colored filters on the photosensitive unit.

[0099] As an example, in the image sensor 242 (hereinafter referred to as multispectral image sensor for ease of description) used in the multispectral camera 130b, each photosensitive unit includes more spectral channels (the number of spectral channels is more than 3, 4, 5, 6, 7, 8, 9, 10 or other values). For example, different color filter arrays (CFAs) can be formed by chemical dyes, coating interference, or metasurface micro / nano structures. Therefore, the multispectral image sensor can extract spectral information from multiple channels to acquire images. Because the multispectral image sensor has more spectral channels and stronger spectral sensing capabilities, the color fidelity of the perceived multispectral image is higher. The multispectral image sensor can acquire multispectral images through multiple spectral channels and can transmit the acquired multispectral images to the color restoration module for analysis, realizing color imaging or color information extraction. The aforementioned multispectral image can also be referred to as multispectral information or other names.

[0100] For example, a multispectral image sensor may include a spectral modulation region and a photoelectric conversion region. The spectral modulation region forms multiple spectral channels through various materials or optical structures to split the incident light. The photoelectric conversion region converts the split light signal into an electrical signal, and then outputs a digital signal or encoding through analog-to-digital conversion. A color restoration module (e.g., an image signal processor) is electrically connected to the multispectral image sensor. The color restoration module can calculate spectral information or color information based on the light signal and pixel position information input from the multispectral image sensor. Based on the color transformation matrix from signals of multiple spectral channels to color space, the color restoration module can also convert the multispectral signals acquired by the multispectral image sensor into color space information of the image (e.g., RGB information, XYZ tristimulus values, or YUV information). Here, the color transformation matrix from signals of multiple spectral channels to color space is established based on the visual response curve of the human eye (e.g., the spectral tristimulus value curve, including the response curve of the human eye to red, green, and blue light) and the spectral response curve of the multispectral camera (including the spectral response curve of the multispectral camera to different wavelengths). In some embodiments, the color restoration module can also use the spectral information acquired by the multispectral camera to perform color calibration on other cameras (e.g., imaging cameras). For example, based on the transformation relationship between the spectral response curve of the multispectral camera and the spectral response curve of other cameras, the color space information corresponding to the multispectral camera is mapped onto other cameras to correct their colors, making the image colors of other cameras more accurate.

[0101] In some embodiments, camera modules with different focal lengths may have their own image sensors, or camera modules with different focal lengths may share the same image sensor.

[0102] In some embodiments, the light sensing component 240 may include an image signal processing (ISP) module. The ISP module is used to process the signals acquired by the image sensor 242, such as performing linear correction, noise reduction, automatic white balance, automatic exposure control, color correction, etc., converting the raw data acquired by the image sensor 242 into a format supported by the algorithm. Exemplarily, the image signal processing unit may be integrated with the image sensor 242.

[0103] Circuit board 243 is used to transmit electrical signals, and it can be a flexible printed circuit (FPC) or a printed circuit board (PCB). Image sensor 242 can be electrically connected to circuit board 243 via wires to extract signals.

[0104] In some embodiments, the light sensing component 240 may further include a microelectromechanical system (MEMS) actuator 244, which drives the image sensor 242 to move along the optical axis and / or perpendicular to the optical axis, thereby achieving autofocus and / or optical image stabilization. The MEMS actuator 244 can be driven by electrostatic force, magnetoelectric force, piezoelectric force, thermoelectric force, etc. It should be understood that the specific structure of the MEMS actuator 244 can be designed and selected according to the chosen driving method, and this application does not limit it in this regard.

[0105] It should be understood that the structures illustrated in Figures 2 and 3 do not constitute a specific limitation on camera 200. Camera 200 may include more or fewer components than illustrated. For example, camera 200 may also include connectors and peripheral electronic components, or camera 200 may not include lens actuator 230, which will not be described in detail here.

[0106] When electronic devices image a scene, processing the image using the scene's polarization information can improve image sharpness and contrast. For example, in a scene containing mirrors such as a lake or a mirror, the electronic device can use the scene's polarization information to eliminate the decrease in sharpness caused by reflected light from the mirrors, thus improving image sharpness. As another example, in a scene containing metallic objects in grass, the electronic device can use the scene's polarization information to further enhance the contrast between the metallic objects and the grass, thereby improving image contrast.

[0107] In some related solutions, the camera used for image capture (referred to as the imaging camera) can incorporate a polarization structure. For example, a pixelated polarization array can be incorporated into the complementary metal-oxide-semiconductor (CMOS) image sensor of an electronic device. This allows the electronic device to acquire and utilize polarization information simultaneously with image capture, thereby improving image sharpness and contrast. Alternatively, a polarizing mirror or similar component can be placed in front of the CMOS image sensor or charge-coupled device (CCD) image sensor of an electronic device. This allows the electronic device to utilize temporal or spatial domain techniques to obtain polarization information of the subject from various angles through the polarizing mirror or similar component, generating images with high contrast and sharpness.

[0108] In other words, in the aforementioned solutions, electronic devices can acquire polarization information through the polarization structure in the imaging camera. However, the polarization structure reduces the overall amount of light entering the imaging camera and causes uneven light reception. Therefore, these solutions reduce the camera's sensitivity and image resolution, resulting in a decrease in image quality.

[0109] Therefore, how to effectively improve image quality is an urgent problem to be solved.

[0110] In view of this, embodiments of this application provide an image processing method. This method can be applied to an electronic device. The electronic device may include a first camera and a second camera. The first camera is capable of providing multispectral information and polarization information. The second camera may be an imaging camera capable of providing image information. Thus, the electronic device can acquire multispectral and polarization information through the first camera and image information through the second camera. Furthermore, the electronic device can use the multispectral and polarization information to process the image information, thereby improving the image's contrast, sharpness, and color accuracy, that is, effectively improving image quality.

[0111] The following are examples of the first camera.

[0112] As an example, the first camera can be obtained by further designing the aforementioned multispectral camera 103b. For instance, a polarization structure can be added to the multispectral camera 103b, enabling it to provide polarization and multispectral information. The meaning of multispectral information can be found in the description of the multispectral camera 103b in Figure 1 above, and will not be repeated here.

[0113] As an example, polarization information may include the degree of polarization, a polarization image, the polarization angle, or other polarization-related information.

[0114] For example, the first camera includes a microlens layer, a filter layer, and a photosensitive layer stacked together. The filter layer may be located between the microlens layer and the photosensitive layer. The filter layer can be used to perform multispectral filtering and polarization filtering on the light from the microlens layer, and the filter layer may have at least four spectral channels. For ease of understanding, the structure of the first camera is described in detail below with reference to Figure 4.

[0115] Figure 4 is a schematic cross-sectional view of the image sensor 400 provided in an embodiment of this application.

[0116] For example, the first camera may include an image sensor 400, which may include a microlens layer 410, a filter layer 420, and a photosensitive layer 430 stacked together. The filter layer 420 may be located between the microlens layer 410 and the photosensitive layer 430.

[0117] As an example, the microlens layer 410 can be used to converge light.

[0118] As an example, the filter layer 420 can be used to perform multispectral filtering and polarization filtering on light from the microlens layer 410. Exemplarily, the filter layer 420 has at least four spectral channels, or at least five spectral channels, or at least six spectral channels, or at least seven spectral channels, or at least eight spectral channels, or at least nine spectral channels, or at least ten spectral channels, or at least eleven spectral channels, or at least twelve spectral channels, or at least thirteen spectral channels, or at least fourteen spectral channels, or at least fifteen spectral channels, or at least sixteen spectral channels.

[0119] Taking filter layer 420 with five spectral channels as an example, these five spectral channels can be different from each other; in other words, any two of the five spectral channels can be different. For example, the five spectral channels can include a blue channel, a red channel, a green channel, a yellow channel, and a white channel. Examples of other numbers and specific forms of spectral channels will not be elaborated further.

[0120] The aforementioned spectral channels may also be referred to as color channels, multispectral channels, or other names, and this application does not limit them to any particular name.

[0121] For example, the filter layer 420 may have at least two polarization channels, or at least three polarization channels, or at least four polarization channels. Different polarization channels may correspond to different polarization angles. Taking the filter layer 420 having two polarization channels as an example, the filter layer 420 can obtain polarization information at two different polarization angles based on the two polarization channels.

[0122] As an example, the photosensitive layer 430 can be used to perform photoelectric conversion on light from the filter layer 420. For instance, the photosensitive layer 430 can convert optical signals into electrical signals.

[0123] In some examples, the first camera may also include other components, such as some or all of the components in camera 200 in FIG. 2. Exemplarily, the image sensor 400 in the first camera may be located at the position of image sensor 242 in camera 200, or at other locations, which are not limited in this application.

[0124] The image sensor 400 may also be called a multispectral image sensor, a multispectral polarization image sensor, or other names. This application does not limit the specific name of the image sensor 400.

[0125] To facilitate understanding, examples of some structures of the filter layer 420 are described below with reference to Figure 5.

[0126] Figure 5 is another schematic cross-sectional view of the image sensor 400 provided in an embodiment of this application. For the sake of brevity, the reference numerals for the microlens layer 410 and the photosensitive layer 430 are shown in Figure 4, and will not be repeated in Figure 5.

[0127] In some examples, filter layer 420 may include two layers: a multispectral layer 422 and a polarization filter layer 424. For example, Figures 5(a), (b), and (c) illustrate some of the above examples.

[0128] This application does not limit the relative positions of the multispectral layer 422 and the polarization filter layer 424. For example, referring to Figures 5(a) and (b), the multispectral layer 422 can be located closer to the microlens layer 410, and the polarization filter layer 424 can be located farther from the microlens layer 410. As another example, referring to Figure 5(c), the multispectral layer 422 can be located farther from the microlens layer 410, and the polarization filter layer 424 can be located closer to the microlens layer 410. Alternatively, the positions of the multispectral layer 422 and the polarization filter layer 424 can be interchanged.

[0129] As an example, the multispectral layer 422 can be achieved through dyes, coatings, or micro / nano structures.

[0130] As an example, the polarization filter layer 424 can be implemented using gratings, micro / nano structures, or other polarization sensing structures.

[0131] Image sensor 400 may include one or more pixels. For example, a pixel may include a microlens in microlens layer 410, a multispectral component and / or polarization filter in filter layer 420, and a photodiode (or photosensitive unit) in photosensitive layer 430. For instance, Figures 5(a) through (d) show four pixels respectively.

[0132] For example, a multispectral component can be a component with color filtering function that is implemented in the filter layer 420 by dye, coating or micro / nano structure.

[0133] For example, the polarization filter component can be a component with polarization filtering function that is implemented by a grating or micro / nano structure in the filter layer 420.

[0134] As an example, a pixel in the image sensor 400 may have both a multispectral component and a polarization filter component. As another example, a pixel in the image sensor 400 may have either a multispectral component or a polarization filter component.

[0135] For example, referring to Figures 5(a) and (c), the four pixels in the image sensor 400 can simultaneously have a multispectral component and a polarization filter component. As another example, referring to Figure 5(b), pixels 401 and 404 can simultaneously have a multispectral component and a polarization filter component; pixel 402 can have a multispectral component but not a polarization filter component; and pixel 403 can have a polarization filter component but not a multispectral component. Thus, the electronic device can read pixels 401 and 404 when both multispectral and polarization information are needed; the electronic device can read pixel 402 when only multispectral information is needed; and the electronic device can read pixel 403 when only polarization information is needed.

[0136] As an example, when a pixel does not have a multispectral component, the pixel may include a planar material in the multispectral layer 422. When a pixel does not have a polarization filter component, the pixel may include a planar material in the polarization filter layer 424.

[0137] In some examples, the filter layer 420 can be a single layer. The filter layer 420 can have a micro / nano structure. The micro / nano structure can have one of the functions of multispectral filtering or polarization filtering, or it can have both multispectral filtering and polarization filtering functions simultaneously.

[0138] For example, as shown in Figure 5(d), the shape of the micro / nanostructure cross-section can be triangular or rectangular. This application does not limit this; the shape of the micro / nanostructure cross-section can also be other shapes, such as trapezoidal, elliptical, or other shapes. It is understood that by designing the size, period, shape, or fabrication conditions of the micro / nanostructure, it is possible to make the micro / nanostructure simultaneously possess polarization filtering and multispectral filtering functions.

[0139] As can be understood, as mentioned earlier, a pixel with color filtering functionality implemented through micro-nano structures in the filter layer 420 can be called a multispectral component; a pixel with polarization filtering functionality implemented through micro-nano structures in the filter layer 420 can be called a polarization filtering component. Referring to the scheme shown in Figure 5(d), if a pixel's micro-nano structure only has color filtering or polarization filtering functionality, the pixel can be understood as having either a multispectral component or a polarization filtering component; if a pixel's micro-nano structure has both color filtering and polarization filtering functionality, the pixel can be understood as having both a multispectral component and a polarization filtering component.

[0140] For example, the arrangement of the spectral channels and polarization angles of the individual pixels in the image sensor 400 of the first camera can be referred to as a "sensor pattern". In some examples, the sensor pattern can be obtained by repeating the smallest repeating unit multiple times on the plane where the filter layer 420 is located. In other examples, the sensor pattern itself can be regarded as the smallest repeating unit of the sensor pattern.

[0141] To facilitate understanding, some examples of the smallest repeating unit of the sensor pattern are introduced below with reference to Figure 6.

[0142] Figure 6 is a schematic diagram of the smallest repeating unit of the sensor pattern of the image sensor 400 provided in this application embodiment. In Figure 6, one square can correspond to one pixel. Or, one square can represent a polarization filter component and / or a multispectral component in one pixel. The shading in different directions in Figure 6 can represent polarization filter components with different polarization angles. Different polarization angles can correspond to different polarization channels. Figure 6 shows a scenario with four polarization channels, but those skilled in the art will understand that the image sensor 400 of this application can have more or fewer polarization channels. The numbers in Figure 6 can represent spectral channel numbers; the same number can represent the same spectral channel, and different numbers can represent different spectral channels. To clearly represent the numbers, in Figure 6, when a square contains both shading and a number (i.e., the number corresponding to the square simultaneously has both a polarization filter component and a multispectral component), the number is surrounded by white fill. Those skilled in the art will understand that the white fill is only for clarity and does not imply a change in the actual structure of the polarization filter component.

[0143] As an example, the smallest repeating unit of a sensor pattern may include two, three, four, or more pixels with polarization filters. For example, referring to Figures 6(a) through (c), the smallest repeating unit of a sensor pattern may include four pixels with polarization filters. As another example, referring to Figure 6(d), the smallest repeating unit of a sensor pattern may include nine pixels with polarization filters. Yet another example, referring to Figure 6(e), the smallest repeating unit of a sensor pattern may include 16 pixels with polarization filters. And yet another example, referring to Figure 6(f), the smallest repeating unit of a sensor pattern may include 64 pixels with polarization filters.

[0144] For example, a grating or micro / nano structure can be provided in the polarization filter layer 424 of the image sensor 400 as a polarization filter component, so that only light with a certain polarization angle can pass through the polarization filter component and be detected by the photosensitive layer 430. As an example, the difference in polarization angle between two polarization filter components with different polarization angles in the smallest repeating unit of the sensor pattern is ±45°, ±60°, ±90°, ±135° or other polarization angles. "±" indicates "positive or negative".

[0145] This application does not limit the number of spectral channels included in the smallest repeating unit of a sensor pattern. For example, referring to Figures 6(a), (b), and (d), the smallest repeating unit of a sensor pattern may include 5 spectral channels. As another example, referring to Figure 6(e), the smallest repeating unit of a sensor pattern may include 7 spectral channels. Yet another example, referring to Figure 6(f), the smallest repeating unit of a sensor pattern may include 8 spectral channels. And yet another example, referring to Figure 6(c), the smallest repeating unit of a sensor pattern may include 9 spectral channels.

[0146] Furthermore, this application does not limit the number of pixels with multispectral components in the smallest repeating unit of the sensor pattern. For example, referring to Figures 6(a), (b), and (d), the smallest repeating unit of the sensor pattern may include 5 pixels with multispectral components. As another example, referring to Figure 6(c), the smallest repeating unit of the sensor pattern may include 9 pixels with multispectral components. Yet another example, referring to Figure 6(e), the smallest repeating unit of the sensor pattern may include 16 pixels with multispectral components. And yet another example, referring to Figure 6(f), the smallest repeating unit of the sensor pattern may include 64 pixels with multispectral components.

[0147] A pixel in the smallest repeating unit of a sensor pattern can have one of the following components: a polarization filter or a multispectral component, or both.

[0148] Considering the application scenarios and manufacturing difficulties of multispectral and polarization information, the following section, with reference to Figure 6, introduces some specific examples of the minimum repeating unit of the sensor pattern provided in the embodiments of this application.

[0149] For example, assume that the smallest repeating unit of the sensor pattern is an N*N square matrix. Here, N is an integer greater than 1, and "*" represents a multiplication sign.

[0150] In some examples, 2, 3, or 4 pixels are selected in the smallest repeating unit of N*N for polarization detection (or, polarization assessment, or, providing polarization information). That is, 2, 3, or 4 pixels in the smallest repeating unit of N*N have polarization filters, and the remaining pixels can be used for multispectral color detection (or, providing multispectral information). That is, the pixels in the smallest repeating unit of N*N, excluding those with polarization filters, have multispectral components. For example, pixels with polarization filters can be adjacent, see Figure 6(a). Alternatively, pixels with polarization filters can be non-adjacent, see Figure 6(b).

[0151] Building upon the example above, each pixel in the smallest repeating unit of N*N is provided with a polarization filter or a multispectral component. Thus, some pixels can have both a polarization filter and a multispectral component, while others have only one of either. See, for example, Figure 6(c).

[0152] In other examples, each pixel of the N*N minimum repeating unit simultaneously possesses both polarization filtering and multispectral components. See, for example, Figure 6(e). In the case shown in Figure 6(e), the electronic device can simultaneously obtain multispectral and polarization information by reading the pixels on the image sensor obtained from the minimum repeating unit of the sensor pattern. Furthermore, the electronic device can process image information using both multispectral and polarization information, or it can extract one of the multispectral and polarization information using an algorithm and process the image information using that information (i.e., either the multispectral information or the polarization information).

[0153] For example, an N*N minimum repeating unit includes adjacent pixels with the same spectral channel and different polarization angles. Since an image sensor can be obtained by repeating an N*N minimum repeating unit once or multiple times, or it can be the N*N minimum repeating unit itself, the above example can also be described as: a first camera includes multiple first pixels arranged adjacently, the multiple first pixels having the same spectral channel and different polarization angles.

[0154] As an example, see Figure 6(f). The aforementioned multiple first pixels can be four pixels with the same spectral channel and adjacent to each other. For example, the four pixels corresponding to the four squares with spectral channel number 1 and different polarization angles in the upper left corner of Figure 6(f).

[0155] Based on the image sensor obtained from the smallest repeating unit shown in Figure 6(f), the electronic device can acquire either polarization information or multispectral information through different readout methods (or circuit readout methods). For example, by reading a single pixel at a time, the electronic device can simultaneously acquire polarization and multispectral information. Alternatively, by reading four pixels at a time (e.g., four adjacent pixels with the same spectral channel), the electronic device can acquire only multispectral information.

[0156] To facilitate understanding, the following is a flowchart of the image processing method with reference to Figure 7.

[0157] Figure 7 is a schematic flowchart of an image processing method 700 provided in an embodiment of this application. This method 700 can be applied to an electronic device. The electronic device may include a first camera and a second camera. The first camera can be referred to in the examples related to Figures 1 to 6 above, and will not be described again. The second camera may be an imaging camera, specifically referred to in the examples related to Figures 1 to 3, and will not be described again. Optional operations in method 700 are indicated by dashed lines in Figure 7. The various operations of method 700 will be described below with reference to Figure 7.

[0158] S710: The electronic device acquires multispectral and polarization information through a first camera.

[0159] In some examples, the electronic device can control a first camera to perform visible light imaging, acquiring multispectral and polarization information. That is, the multispectral information may include (or only include) the spectral information of visible light. In other examples, the multispectral information may include the spectral information of other light sources, such as infrared light, and / or ultraviolet light.

[0160] In other examples, S710 described above can be replaced by the electronic device acquiring either multispectral information or polarization information via the first camera. For example, the electronic device can acquire multispectral information via the first camera to perform spectral detection. Examples of spectral detection are given in Figure 1 above, which describes the multispectral camera 103b, and will not be repeated here. As another example, the electronic device can acquire polarization information via the first camera to perform polarization detection. As an example, the polarization detection described above can include scratch detection, roughness detection, etc.

[0161] In the S720, electronic devices acquire image information through a second camera.

[0162] For example, electronic devices can control a second camera to capture images and thus obtain image information.

[0163] This application does not limit the execution order of S710 and S720. For example, S710 may be executed before S720, or S720 may be executed before S721, or S710 and S720 may be executed simultaneously.

[0164] The S730 is an electronic device that processes image information based on multispectral and polarization information.

[0165] For example, electronic devices can improve color accuracy by processing image information based on multispectral information. In other words, the color accuracy of the image information processed in S730 can be higher than the color accuracy of the image information before processing in S720.

[0166] For example, electronic devices can improve the contrast and sharpness of image information by processing it based on polarization information. In other words, the contrast and sharpness of the image information processed in S730 can be higher than the contrast and sharpness of the image information before processing in S720.

[0167] In other examples, S730 described above can be replaced by: the electronic device processing image information based on either multispectral information or polarization information. For example, if S710 is replaced by the electronic device acquiring polarization information via a first camera, or if the electronic device extracts polarization information using an algorithm after obtaining multispectral and polarization information through S710, then S730 can be replaced by: the electronic device processing image information based on polarization information.

[0168] For example, if the aforementioned S710 is replaced by the case where the electronic device can acquire multispectral information through the first camera, or if the electronic device extracts multispectral information through an algorithm after obtaining multispectral and polarization information by executing S710, then the aforementioned S730 can be replaced by: the electronic device processes image information based on multispectral information.

[0169] In some possible implementations, method 700 also includes S725 prior to S730.

[0170] S720: The electronic device determines that the image information corresponds to a preset scene based on the image information.

[0171] As an example, the aforementioned preset scene can be a typical scenario where polarization imaging is advantageous. For instance, the preset scene could include a mirror, a water surface, or other scenes with specular reflection. After the electronic device detects the aforementioned scene, it can execute S730, that is, process the image information using polarization information to eliminate the imaging effect of reflected light on the object in front of the mirror (or on the water surface), thereby significantly improving the contrast and sharpness of the object.

[0172] It is understandable that many objects in nature exhibit differences in polarization due to variations in their materials. In other examples, when the processing load on electronic devices is relatively light or when there is a user demand, the electronic device can use a first camera to simultaneously acquire polarization and multispectral information, thereby improving image quality.

[0173] To facilitate understanding, a specific example of method 700 is presented below with reference to Figure 8.

[0174] Figure 8 is a schematic flowchart of another image processing method provided in an embodiment of this application.

[0175] Referring to Figure 8, the electronic device can simultaneously perceive the subject using a first camera and a second camera. The first camera can provide multispectral and polarization information. As an example, the first camera can acquire an image containing both multispectral and polarization information. Using an inverse kinematics algorithm, combined with calibration data, the spectral and polarization information of the subject can be extracted from this image. The second camera can provide image information. The electronic device fuses the aforementioned multispectral, polarization, and image information using an algorithm to obtain image information with improved contrast, sharpness, and color accuracy, thereby enhancing the imaging effect of previews, videos, or finished products.

[0176] Referring to Figure 8, the electronic device can access the first camera to acquire multispectral and polarization information of the object being photographed. Furthermore, the electronic device can perform spectral and polarization detection of the substance based on the multispectral and polarization information. Examples of spectral and polarization detection can be found in the relevant content of S710 above, and will not be repeated here.

[0177] To facilitate understanding, another specific example of method 700 is introduced below with reference to Figure 9.

[0178] Figure 9 is a schematic flowchart of another image processing method provided in an embodiment of this application.

[0179] In some examples, the first camera can operate in multiple modes; for instance, it can acquire only multispectral or polarization information. As an example, if polarization enhancement is required, both multispectral and polarization information can be acquired and then fused with the image information acquired by the second camera using an algorithm to obtain image information with improved contrast, sharpness, and color accuracy (i.e., image information with improved quality).

[0180] For example, the aforementioned situations requiring polarization enhancement may include: the image information corresponding to a preset scene, or other situations, such as user-defined settings. As an example, the electronic device can perform content detection on the image information acquired by the second camera, and determine that polarization enhancement is required if it identifies that the image information corresponds to a preset scene. Specific examples are described in the preceding S725 section and will not be repeated here.

[0181] As another example, if no polarization enhancement is required, multispectral information can be acquired and then fused with the image information acquired by the second camera based on an algorithm to obtain image information with improved color accuracy (i.e., image information with improved quality).

[0182] The following describes the apparatus embodiments corresponding to the method embodiments of this application. Only a brief description of the apparatus is provided below; for specific implementation steps and details, please refer to the preceding method embodiments.

[0183] To achieve the functions of the methods provided in this application, the communication device may include hardware structures and / or software modules, implementing the aforementioned functions in the form of hardware structures, software modules, or a combination of hardware structures and software modules. Whether a particular function is implemented in the form of hardware structures, software modules, or a combination of hardware structures and software modules depends on the specific application and design constraints of the technical solution.

[0184] Figure 10 is a schematic block diagram of another electronic device 1000 according to an embodiment of this application. The electronic device 1000 includes a processor 1010, a first camera 1020, and a second camera 1030. Optionally, the processor 1010, the first camera 1020, and the second camera 1030 can be interconnected via a bus. As an example, the electronic device 1000 in Figure 10 may also include other components described for the electronic device 100 in Figure 1. For example, the electronic device 1000 may also include a housing 101, a display screen 102, or other components not shown in Figure 1.

[0185] Optionally, the electronic device 1000 may further include a memory 1040. The memory 1040 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), cache, erasable programmable read-only memory (EPROM), synchronous dynamic random access memory (SDRAM), hard disk drive (HDD), solid-state drive (SSD), or compact disc read-only memory (CD-ROM). The memory 1040 is used to store related instructions and / or data. The memory 1040 may be integrated with the processor 1010 or disposed separately.

[0186] Processor 1010 may be one or more central processing units (CPUs). When processor 1010 is a CPU, it may be a single-core CPU or a multi-core CPU. However, this application is not limited in this respect; processor 1010 may also be one or more graphics processing units (GPUs). Processor 1010 may be an image processor, a chip, or other integrated circuit capable of implementing the methods of this application, or a portion of the circuitry within the aforementioned processor, chip, or integrated circuit for processing functions.

[0187] For example, the processor 1010 is configured to perform the following operations: acquire multispectral information and polarization information through the first camera; acquire image information through the second camera; and process the image information based on the multispectral information and the polarization information.

[0188] The above description is for illustrative purposes only. The electronic device 1000 may be responsible for executing the methods or steps related to Figures 7 to 9 in the foregoing method embodiments.

[0189] For details on other implementation methods, please refer to the detailed descriptions of the embodiments shown in Figures 7 to 9 above, which will not be repeated here. It should be understood that the specific processes by which each component performs the corresponding processes described above have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0190] Figure 11 is a schematic block diagram of an image processing apparatus 1100 according to an embodiment of this application. The image processing apparatus 1100 can be an electronic device 1000, or a chip or module of the electronic device 1000, used to implement the methods involved in the embodiments shown in Figures 7 to 9. Please refer to the relevant descriptions in the above method embodiments for details.

[0191] The image processing apparatus 1100 includes an acquisition unit 1110 and a processing unit 1120.

[0192] For example, the acquisition unit 1110 is used to acquire multispectral information and polarization information through the first camera; the acquisition unit 1110 is also used to acquire image information through the second camera; the processing unit 1120 is used to process the image information according to the multispectral information and the polarization information.

[0193] In some possible implementations, the processing unit 1120 is specifically used to perform visible light imaging through the first camera to acquire the multispectral information and the polarization information.

[0194] In some possible implementations, the processing unit 1120 is further configured to determine, based on the image information, that the image information corresponds to a preset scene.

[0195] The above description is for illustrative purposes only. The image processing device 1100 will be responsible for executing the relevant methods or steps in the foregoing method embodiments. It should be understood that the specific processes by which each component performs the corresponding processes have been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0196] This application also provides a chip, including a processor, for calling and executing instructions stored in a memory, causing an electronic device on which the chip is installed to perform the methods in the examples above.

[0197] This application also provides another chip, including: an input interface, an output interface, and a processor. The input interface, output interface, and processor are connected via an internal connection path. The processor is used to execute code in memory. When the code is executed, the processor is used to perform the methods in the examples described above. Optionally, the chip also includes a memory for storing computer programs or code.

[0198] This application also provides a processor for coupling with a memory for performing the methods and functions of an image processing apparatus or electronic device in any of the above embodiments.

[0199] In another embodiment of this application, a computer program product comprising a computer program or instructions is provided, wherein the method of the foregoing embodiments is implemented when the computer program product is run on a computer.

[0200] This application also provides a computer program product that, when run on a computer, enables the implementation of the methods described in the foregoing embodiments.

[0201] In another embodiment of this application, a computer-readable storage medium is provided, which stores a computer program that, when executed by a computer, implements the methods described in the foregoing embodiments.

[0202] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0204] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0205] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0206] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0207] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0208] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, which includes a first camera and a second camera; The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on the light from the microlens layer. The filter layer has at least four spectral channels. The second camera is an imaging camera; The method includes: The electronic device acquires multispectral and polarization information through the first camera; The electronic device acquires image information through the second camera; The electronic device processes the image information based on the multispectral information and the polarization information.

2. The method according to claim 1, characterized in that, The filter layer includes a multispectral layer and a polarization filter layer.

3. The method according to claim 1 or 2, characterized in that, The filter layer has a micro-nano structure.

4. The method according to any one of claims 1 to 3, characterized in that, The first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

5. The method according to any one of claims 1 to 4, characterized in that, The filter layer has at least two polarization channels.

6. The method according to any one of claims 1 to 5, characterized in that, The second camera is: a red-green-blue RGB camera, a red-yellow-blue RYB camera, or a red-green-blue-white RGBW camera.

7. The method according to any one of claims 1 to 6, characterized in that, The electronic device acquires multispectral and polarization information through the first camera, including: The electronic device performs visible light imaging through the first camera to acquire the multispectral information and the polarization information.

8. The method according to any one of claims 1 to 7, characterized in that, Before the electronic device processes the image information based on the multispectral information and the polarization information, the method further includes: The electronic device determines that the image information corresponds to a preset scene based on the image information.

9. An image processing apparatus characterized by comprising: The device is applied to an electronic device, which includes a first camera and a second camera; The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on the light from the microlens layer. The filter layer has at least four spectral channels. The second camera is an imaging camera; The device includes an acquisition unit and a processing unit; The acquisition unit is used to acquire multispectral information and polarization information through the first camera; The acquisition unit is further configured to acquire image information through the second camera; The processing unit is used to process the image information based on the multispectral information and the polarization information.

10. The apparatus of claim 9, wherein, The filter layer includes a multispectral layer and a polarization filter layer.

11. The apparatus of claim 9 or 10, wherein, The filter layer has a micro-nano structure.

12. The apparatus of any one of claims 9-11, wherein, The first camera includes a plurality of first pixels arranged adjacent to each other, the plurality of first pixels having the same spectral channel and different polarization angles.

13. The apparatus of any one of claims 9-12, wherein, The filter layer has at least two polarization channels.

14. The apparatus of any one of claims 9-13, wherein, The second camera is: a red-green-blue RGB camera, a red-yellow-blue RYB camera, or a red-green-blue-white RGBW camera.

15. The apparatus according to any one of claims 9 to 14, characterized in that, The processing unit is specifically used to perform visible light imaging through the first camera to obtain the multispectral information and the polarization information.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The processing unit is further configured to determine, based on the image information, that the image information corresponds to a preset scene.

17. An electronic device, comprising: include: First camera, second camera, and processor; The first camera includes a stacked microlens layer, a filter layer, and a photosensitive layer. The filter layer is located between the microlens layer and the photosensitive layer. The filter layer is used to perform multispectral filtering and polarization filtering on the light from the microlens layer. The filter layer has at least four spectral channels. The second camera is an imaging camera; The processor is configured to execute the method of any one of claims 1 to 8 by executing a computer program or instructions.

18. The electronic device of claim 17, wherein, The electronic device further includes a memory for storing the computer program or the instructions.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions, and when the computer program or instructions are run, the method of any one of claims 1 to 8 is performed.

20. A computer program product, characterised in that, It includes a computer program or instructions that, when run, implement the method as described in any one of claims 1 to 8.

Images