Multispectral imaging module, mobile terminal, and image processing method
By using a filter unit and a multispectral sensor in the multispectral imaging module, the transmittance of the light signal and the lens angle are controlled, solving the problem of inter-channel interference and improving the spectral restoration accuracy and imaging quality.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- GUANGDONG OPPO MOBILE TELECOMMUNICATIONS CORP LTD
- Filing Date
- 2025-10-28
- Publication Date
- 2026-06-25
AI Technical Summary
Interference between channels in a multispectral imaging module affects the accuracy of imaging results, especially the signal crosstalk problem between adjacent bands.
The optical signal is filtered by a filter unit, allowing at least two wavelengths of optical signal to pass through while limiting the intensity of some of the wavelengths. The optical signal is separated into multiple wavelengths by a multispectral sensor and converted into an electrical signal. Signal crosstalk is reduced by controlling the transmittance of the filter unit and the design of the lens's main ray angle.
It improves the accuracy of spectral reconstruction, reduces signal interference between adjacent bands, and enhances the spectral reconstruction effect of the shooting scene.
Smart Images
Figure CN2025130681_25062026_PF_FP_ABST
Abstract
Description
A multispectral imaging module, a mobile terminal, and an image processing method
[0001] Cross-reference to related applications
[0002] This application is based on and claims priority to a prior Chinese patent application with application number 202411879921.X, application date December 18, 2024, entitled "A Multispectral Imaging Module, Mobile Terminal and Image Processing Method", the entire contents of which are incorporated herein by reference. Technical Field
[0003] This application relates to the field of computer vision technology, and in particular to a multispectral imaging module, a mobile terminal, and an image processing method. Background Technology
[0004] A multispectral camera is an imaging system capable of capturing and utilizing the light radiation information of a target object across different spectral bands to perform complex target inspection and analysis. Compared to full-color imaging, multispectral imaging is not limited to the three basic color channels of red, green, and blue within the visible light range, but can cover a spectral range from ultraviolet to infrared and even wider. This means that multispectral cameras can capture the unique reflection, emission, or absorption characteristics of a target object across multiple discrete or continuous spectral bands, thus providing richer and more realistic visual data. With the continuous maturation of the technology and the expansion of its application areas, multispectral cameras will play an increasingly important role in the future development of machine vision technology.
[0005] The beam splitter is one of the core components of a multispectral imaging module. It is responsible for dispersing the focused light into different wavelengths of light. Beam splitters may employ different techniques such as prism beam splitting or filter beam splitting. Prism beam splitting uses the principle of refraction to disperse light, while filter beam splitting uses filters of specific wavelengths to allow only light of that corresponding wavelength to pass through.
[0006] However, the design and implementation of the spectrometer result in interference between channels, which affects the accuracy of multispectral imaging results. Summary of the Invention
[0007] This application aims to provide a multispectral imaging module, a mobile terminal, and an image processing method to reduce interference between channels and improve spectral reconstruction accuracy.
[0008] The technical solution of this application is implemented as follows:
[0009] In a first aspect, a multispectral imaging module is provided, characterized in that it includes:
[0010] A lens is used to receive and focus light signals;
[0011] A filtering unit is used to filter the light signal after it has been focused by the lens, so as to allow light signals of at least two wavelengths to pass through and limit the light signal intensity of at least a portion of the at least two wavelengths.
[0012] A multispectral sensor is used to separate the light signal filtered by the filter unit into light signals of multiple bands and convert the light signals into electrical signals to obtain spectral image data in multiple bands.
[0013] In a second aspect, a mobile terminal is provided, including the multispectral imaging module described in the first aspect above.
[0014] Thirdly, a mobile terminal is provided, comprising the multispectral imaging module and the first imaging module described in the first aspect.
[0015] The multispectral imaging module is used to sense light signals in multiple bands of the current scene and obtain the spectral information of the current scene.
[0016] The first imaging module is used to sense the visible light band light signal of the current scene based on the spectral information of the current scene, and obtain the image data of the current scene.
[0017] Fourthly, an image processing method is provided, including:
[0018] Acquire the spectral information of the current scene, wherein the spectral information is determined based on the multispectral imaging module described in the first aspect;
[0019] Based on the spectral information of the current scene, the shooting parameters of the first imaging module are determined;
[0020] Based on the shooting parameters of the first imaging module, the first imaging module is controlled to acquire image data of the current scene;
[0021] Based on the spectral information of the current scene, image processing is performed on the image data of the current scene to obtain processed image data.
[0022] Fifthly, an image processing apparatus is provided, comprising:
[0023] An acquisition unit is used to acquire spectral information of the current scene, wherein the spectral information is determined based on the multispectral imaging module described in the first aspect;
[0024] The processing unit is used to determine the shooting parameters of the first imaging module based on the spectral information of the current scene;
[0025] The control unit is used to acquire image data of the current scene based on the shooting parameters determined by the spectral information;
[0026] The processing unit is also used to perform image processing on the image data of the current scene based on the spectral information of the current scene, so as to obtain processed image data.
[0027] In a sixth aspect, an image processing apparatus is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor.
[0028] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.
[0029] In a seventh aspect, a computer-readable storage medium is provided having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.
[0030] Eighthly, a computer program product includes a computer program that, when executed by a processor, implements the steps of the aforementioned method.
[0031] This application provides a multispectral imaging module, a mobile terminal, and an image processing method. The multispectral imaging module specifically includes: a lens for receiving and focusing light signals; a filter unit for filtering the light signals focused by the lens to allow light signals of at least two wavelengths to pass through, and limiting the intensity of light signals in at least a portion of the at least two wavelengths; a multispectral sensor for separating the light signals filtered by the filter unit into light signals of multiple wavelengths, and converting the light signals into electrical signals to obtain spectral image data in multiple wavelengths; and an image processor for obtaining spectral information of the current scene based on the spectral image data in multiple wavelengths. Thus, by limiting the intensity of light signals in the allowed wavelengths through the filter unit, interference from these wavelengths to other wavelengths can be reduced, while retaining some information from these wavelengths, thereby improving the spectral reconstruction accuracy of the captured scene. Attached Figure Description
[0032] Figure 1 is a schematic diagram of the composition structure of the multispectral imaging module in an embodiment of this application;
[0033] Figure 2 is a schematic diagram of the composition structure of the multispectral sensor in the embodiment of this application;
[0034] Figure 3 is a schematic diagram of the spectral response curve of the multispectral sensor in the embodiment of this application;
[0035] Figure 4 is a schematic diagram of the transmittance curves of the multilayer membrane structure under different incident angles in the embodiments of this application;
[0036] Figure 5 is a schematic diagram of the optical design principle of a small-angle CRA lens in an embodiment of this application;
[0037] Figure 6 is a schematic diagram of the composition structure of the multispectral imaging module in an embodiment of this application;
[0038] Figure 7 is a schematic diagram of the composition structure of the mobile terminal in an embodiment of this application;
[0039] Figure 8 is a schematic diagram of the composition structure of the mobile terminal in an embodiment of this application;
[0040] Figure 9 is a flowchart illustrating the image processing method in an embodiment of this application;
[0041] Figure 10 is a schematic diagram of the composition structure of the image processing device in an embodiment of this application;
[0042] Figure 11 is a schematic diagram of the composition structure of the image processing device in an embodiment of this application. Detailed Implementation
[0043] This application provides a multispectral imaging module, including:
[0044] A lens is used to receive and focus light signals;
[0045] A filter unit is used to filter the light signal after it has been focused by the lens, so as to allow light signals of at least two wavelengths to pass through and limit the light signal intensity of at least a portion of the at least two wavelengths.
[0046] A multispectral sensor is used to separate the light signal filtered by the filter unit into light signals of multiple bands and convert the light signals into electrical signals to obtain spectral image data in multiple bands.
[0047] In some embodiments, the filtering unit allows optical signals of the first band and the second band to pass through, and limits the intensity of the second band optical signal. The first band and the second band are adjacent bands.
[0048] In some embodiments, the first band is the visible light band and the second band is the infrared band. The filtering unit allows light signals from both the visible light band and the infrared band to pass through, while limiting the intensity of the infrared band light signal. This reduces crosstalk between the infrared band and the visible light band when the multispectral sensor responds to light signals from multiple bands.
[0049] In some embodiments, the filtering unit limits the intensity of the second-band light signal by controlling the transmittance of the second-band light signal.
[0050] In some embodiments, one of the following is included:
[0051] The transmittance of the filter unit for the second band of optical signal is less than the first preset value;
[0052] The transmittance of the filter unit to the second band of optical signal is greater than or equal to the first preset value;
[0053] In the first light source scenario, the transmittance of the filter unit to the second band of light signal is less than the first preset value; in the second light source scenario, the transmittance of the filter unit to the second band of light signal is greater than or equal to the first preset value.
[0054] In some embodiments, the multispectral sensor includes at least a spectrometer array, a photosensitive unit array, and a signal processing circuit.
[0055] A beam splitter array is used to separate optical signals into multiple wavelength bands.
[0056] Photosensitive unit array is used to convert optical signals of multiple wavelengths into electrical signals;
[0057] The signal processing circuit is used to process electrical signals in multiple bands and output spectral image data in multiple bands.
[0058] In some embodiments, the spectral dispersive unit is formed using a resin dye coating technique to create a filter.
[0059] In some embodiments, the principal ray angle of the lens is less than a second preset value, and the principal ray angle of the lens and the principal ray angle of the multispectral sensor meet the matching condition.
[0060] In some embodiments, the image processor is used to compensate for spectral image data of multiple bands output by a multispectral sensor to obtain spectral image data after compensation of multiple bands; and to perform spectral analysis based on the spectral image data after compensation of multiple bands to obtain spectral information.
[0061] This application also provides a mobile terminal, including any of the multispectral imaging modules in this application.
[0062] This application also provides another mobile terminal, including the multispectral imaging module and the first imaging module as described in any of the embodiments of this application.
[0063] The multispectral imaging module is used to sense light signals in multiple bands of the current scene and obtain the spectral information of the current scene.
[0064] The first imaging module is used to acquire image data of the current scene based on the shooting parameters determined by spectral information;
[0065] The first imaging module is also used to perform image processing on the image data of the current scene based on the spectral information of the current scene, so as to obtain the processed image data.
[0066] This application also provides an image processing method, the method comprising:
[0067] The spectral information of the current scene is obtained, and the spectral information is determined based on the multispectral imaging module of any one of the embodiments of this application;
[0068] Based on the spectral information of the current scene, determine the shooting parameters of the first imaging module;
[0069] The first imaging module is controlled to acquire image data of the current scene based on the shooting parameters of the first imaging module.
[0070] Image processing is performed on the image data of the current scene based on the spectral information of the current scene to obtain processed image data.
[0071] This application also provides an image processing apparatus, including:
[0072] An acquisition unit is used to acquire spectral information of the current scene, wherein the spectral information is determined based on the multispectral imaging module described in the first aspect;
[0073] The processing unit is used to determine the shooting parameters of the first imaging module based on the spectral information of the current scene;
[0074] The control unit is used to acquire image data of the current scene based on the shooting parameters determined by the spectral information;
[0075] The processing unit is also used to perform image processing on the image data of the current scene based on the spectral information of the current scene, so as to obtain processed image data.
[0076] This application also provides an image processing apparatus, including: a processor and a memory configured to store a computer program capable of running on the processor.
[0077] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.
[0078] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the steps of the aforementioned method.
[0079] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the aforementioned method.
[0080] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.
[0081] Figure 1 is a schematic diagram of the composition structure of the multispectral imaging module in an embodiment of this application. As shown in Figure 1, the multispectral imaging module 100 may specifically include:
[0082] Lens 110 is used to receive and focus light signals;
[0083] The filter unit 120 is used to filter the light signal after it has been focused by the lens, so as to allow light signals of at least two wavelengths to pass through and limit the light signal intensity of at least a portion of the at least two wavelengths.
[0084] The multispectral sensor 130 is used to separate the light signal filtered by the filter unit into light signals of multiple bands and convert the light signals into electrical signals to obtain spectral image data of multiple bands.
[0085] Image processor 140 is used to obtain spectral information of the current scene based on spectral image data in multiple bands.
[0086] In some embodiments, the filtering unit allows optical signals of the first band and the second band to pass through, and limits the intensity of the second band optical signal. The first band and the second band are adjacent bands.
[0087] It should be noted that since the first and second bands are adjacent bands, when the multispectral sensor responds to optical signals from multiple bands, there is signal crosstalk between the two bands, which affects the accuracy of the response spectrum reconstruction. Therefore, this application reduces or eliminates this signal crosstalk by setting a filter unit to limit the optical signal intensity of the second band.
[0088] The first band includes bands containing useful spectral information. For example, the first band includes the visible light band, which is the portion of the electromagnetic spectrum perceptible to the human eye. Blue light has a wavelength range of approximately 380-500 nanometers (nm) and is perceived as blue by the retina. Green light has a wavelength range of approximately 500-570 nm, located in the middle of the visible spectrum, to which the human eye is most sensitive. Red light has a wavelength range of approximately 625-740 nm (or 700 nm), and is the longest wavelength in the visible spectrum, perceived as red by the retina. The first band can specifically be a portion or the entire visible light band.
[0089] The second band contains the bands containing useful spectral information and is adjacent to the first band. In multispectral imaging technology, multiple filters of different bands are typically used to acquire optical signals of different bands separately. However, due to potential errors in the design and manufacturing of the filters, or factors such as scattering and reflection of the optical signal during transmission, optical signals from adjacent bands may overlap or cross over, resulting in crosstalk. For example, the first band is the visible light band, which is adjacent to the ultraviolet band (below 390 nm) and the infrared band (above 760 nm). During imaging, if the system's spectral resolution is insufficient, or the filter design is inaccurate, optical signals from the ultraviolet or infrared bands may leak into the visible light band, causing crosstalk.
[0090] For example, the first band is the visible light band, and the second band is the infrared band. The filtering unit allows light signals from both the visible and infrared bands to pass through, while limiting the intensity of the infrared light signal. This reduces crosstalk between the infrared band and the visible light band when the multispectral sensor responds to light signals from multiple bands.
[0091] For example, the first band is the visible light band, and the second band is the ultraviolet band; the first band is the blue light band, and the second band is the green light band; the first band is the green light band, and the second band is the red light band.
[0092] In some embodiments, the filtering unit limits the intensity of the second-band light signal by controlling the transmittance of the second-band light signal.
[0093] For example, different transmittance can be set for different light source scenarios, thereby limiting the intensity of the second-band light signal to different degrees.
[0094] In some embodiments, the transmittance of the filter unit for the second band of light signal is less than a first preset value. It is understood that, based on the spectral response curve characteristics of the second band, the transmittance of the filter unit for the second band of light signal is fixed to a value less than the first preset value, thereby reducing the transmittance of the second band of light signal in all light source scenarios and reducing the intensity of the second band of light signal.
[0095] The first preset value is the upper limit of the transmittance of the second band. For example, the first preset value can be 90%, 85%, 80%, etc.
[0096] In some embodiments, the transmittance of the filter unit for the second band of light signal is greater than or equal to a first preset value. It is understood that, based on the spectral response curve characteristics of the second band, the transmittance of the filter unit for the second band of light signal is fixed to a value greater than or equal to the first preset value, thereby improving the transmittance of the second band of light signal in all light source scenarios and ensuring the acquisition of sufficient second band spectral information.
[0097] In some embodiments, under the first light source scenario, the transmittance of the filter unit to the light signal of the second band is less than a first preset value; under the second light source scenario, the transmittance of the filter unit to the light signal of the second band is greater than or equal to the first preset value.
[0098] The first light source scene can be a scene where the intensity of the second band light signal is relatively strong. The second band has a greater interference with other bands. In the first light source scene, reducing the transmittance of the second band light signal can reduce the interference of these bands to other bands, while retaining some information of these bands, thereby improving the spectral reproduction accuracy of the shooting scene.
[0099] The second light source scenario can be a scenario where the light signal intensity of the second band is relatively weak. The second band has less interference with other bands. In the second light source scenario, the transmittance of the light signal of the second band is increased, and sufficient information of these bands is retained, thereby improving the spectral reproduction accuracy of the shooting scene.
[0100] This application's embodiments include, but are not limited to, the two light source scenarios described above. Different transmittance levels are set for different light source scenarios based on the spectral response curve characteristics of the second band.
[0101] For example, a light source scene can be a single light source scene or a multi-light source scene. Light sources specifically include natural light sources, artificial light sources, and other special light sources.
[0102] In some embodiments, as shown in FIG2, the multispectral sensor 130 includes at least a beam splitting unit array 131, a photosensitive unit array 132, and a signal processing circuit 133; the beam splitting unit array 131 is used to separate optical signals into optical signals of multiple bands; the photosensitive unit array 132 is used to convert optical signals of multiple bands into electrical signals; and the signal processing circuit 133 is used to process the electrical signals of multiple bands and output spectral image data of multiple bands.
[0103] The beam-splitting unit array 131 is composed of multiple filters arranged in a two-dimensional matrix. The photosensitive unit array 132 can specifically be a pixel array, composed of multiple pixel units arranged in a two-dimensional matrix. Each pixel unit can receive and convert optical signals into electrical signals. Each filter covers one or more pixel units.
[0104] In some embodiments, the spectral dispersive unit is formed using a resin dye coating technique to create a filter.
[0105] Due to the material properties of resin dyes, warping issues typically exist in the infrared band. In scenarios with strong infrared light, such as sunlight, this can cause significant crosstalk to other spectral channels, leading to increased spectral reconstruction errors. Figure 3 is a schematic diagram of the spectral response curve of the multispectral sensor in this embodiment. The spectral response curve is a graphical representation of spectral response characteristics, showing the relationship between the photocurrent or photoelectric signal generated when light of different wavelengths illuminates a multispectral sensor and the wavelength. It can be seen that obvious infrared warping occurs in the nine channels of the infrared band.
[0106] The signal processing circuit is responsible for amplifying, filtering, and digitizing the electrical signal output by the photosensitive unit for subsequent data processing.
[0107] This application embodiment allows visible and infrared light to pass through by adding a dual-channel or multi-channel filter unit. Furthermore, the transmittance of the infrared band is controllable. By reducing the transmittance of red light, the infrared component is reduced, thereby minimizing interference from the infrared band to other bands, rather than eliminating it entirely. In other words, by limiting the light signal intensity of the infrared band, interference from the infrared band to other bands can be reduced while retaining some information from the infrared band, thus improving the spectral accuracy of the captured scene.
[0108] In some embodiments, the filter unit is a multilayer film filter unit. The working principle of the multilayer film filter unit is mainly based on the interference effect of light. When light passes through the multilayer film structure, light of different wavelengths undergoes multiple reflections and interferences between the film layers. This interference causes light waves of specific wavelengths to reinforce each other (constructive interference), while light waves of other wavelengths cancel each other out (destructive interference). Therefore, only wavelengths that meet specific conditions can pass through the filter, while light of other wavelengths is reflected or absorbed. Types of multilayer film structures include, but are not limited to: interference filters, absorption filters, and reflection filters.
[0109] For a multilayer film filter unit, when light is incident on the multilayer film structure at different incident angles, the reflection and refraction paths of the light between the film layers will change, resulting in changes in interference phenomena and optical properties (such as reflectivity, transmittance, color, etc.). Figure 4 is a schematic diagram of the transmittance curves of the multilayer film structure at different incident angles in the embodiment of this application. It shows the transmittance of different wavelengths when light at different incident angles is incident on the multispectral sensor. It can be seen that in some bands, such as 350nm-400nm, 450nm-550nm, and 900nm-1050nm, there are significant differences in transmittance between 0° incident angle and 30° incident angle.
[0110] It should be noted that, considering the impact of angular effects on imaging performance after adding a multi-layered filter unit, this embodiment employs a lens with a small chief ray angle (CRA) to reduce the influence of angular effects on the filter unit. CRA is an important parameter in optical imaging systems, defining the maximum incident angle of light rays as they pass through the lens and reach the sensor pixel.
[0111] In some embodiments, the chief ray angle (CRA) of the lens is less than a second preset value, and the chief ray angle of the lens and the chief ray angle of the spectral sensor meet the matching condition. The second preset value is the upper limit of the lens CRA, and can be 10°, 8°, 5°, etc. By controlling the size of the CRA at various field-of-view positions, the CRA of different fields of view is made as close to 0° as possible to reduce the impact of angular effects on the filter unit.
[0112] Matching conditions are used to limit the difference between the lens CRA and the spectral sensor CRA. For example, matching conditions include limiting the absolute difference between the lens CRA and the spectral sensor CRA to less than a third preset value. For example, the third preset value can be 3°, and a more stringent third preset value can be 2°.
[0113] In some embodiments, the size of the lens CRA is determined based on parameters such as the lens assembly, focal length, aperture size, and total track length (TTL). Figure 5 is a schematic diagram of the optical design principle of a small-angle CRA lens according to an embodiment of this application. As shown in Figure 5, the small-angle CRA lens is composed of seven different lenses. During the imaging process, the incident light is refracted to varying degrees and projected onto an imaging plane of finite size. The edge field of view CRA of the imaging plane is smaller than a second preset value.
[0114] To better illustrate the purpose of this application, based on the above embodiments, a multispectral imaging device is further illustrated. Figure 6 is a schematic diagram of the composition structure of the multispectral imaging module in an embodiment of this application. As shown in Figure 6, the multispectral imaging module 600 may specifically include:
[0115] Lens 610, used to receive and focus light signals, consists of multiple lenses.
[0116] The dual-channel infrared filter 620 is used to filter the light signal after it has been focused by the lens, so as to allow light signals in the visible light band and the infrared band to pass through, and to limit the light signal intensity of at least a portion of the infrared band.
[0117] The multispectral sensor 630 is used to separate the light signal filtered by the filter unit into light signals of multiple wavelengths and convert the light signals into electrical signals to obtain spectral image data in multiple wavelengths. The multispectral sensor 630 specifically includes: a lens array, a filter array, a pixel array, and a signal processing circuit. The filter array includes multiple filters formed based on resin dye coating technology.
[0118] The fixing unit 640 is used to support and fix components such as lens 610, dual-channel infrared filter 620, and multispectral sensor 630, ensuring the relative position stability between these components, thereby guaranteeing the overall performance and imaging quality of the multispectral imaging module.
[0119] The image processor is mounted on the printed circuit board 650 (PCB). The wires and connection points on the PCB enable the image processor to connect with other electronic components, thereby forming a complete circuit system for the multispectral imaging module.
[0120] The Board-to-Board (BTB) connector 660 is a type of connector used to connect two or more printed circuit boards (PCBs). In terminals, BTB connectors are typically used to connect different circuit boards such as main circuit boards and sub-circuit boards, sensor boards, etc., to achieve signal transmission and power connection between them.
[0121] The multispectral imaging module 600 may specifically include: a power management unit, hardware interfaces, and a glass cover (not shown in the figure). The power management unit is responsible for providing a stable operating voltage and current to the multispectral imaging device to ensure its normal operation. The hardware interfaces include a camera interface and a control interface for transmitting data and control information acquired by the sensor. A glass cover is typically placed in front of the lens to protect it from dust, dirt, and other contaminants.
[0122] By using the aforementioned multispectral imaging module, the intensity of the light signal in the infrared band is limited by a dual-channel infrared filter. This reduces the inherent infrared crosstalk problem of resin dyes while retaining the effective spectral information in the infrared band, thereby improving the spectral accuracy of the shooting scene.
[0123] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide a mobile terminal, as shown in FIG7. The mobile terminal 700 includes any of the multispectral imaging modules 710 provided in the embodiments of this application.
[0124] The multispectral imaging module 710 includes: a lens for receiving and focusing light signals;
[0125] A filter unit is used to filter the light signal after it has been focused by the lens, so as to allow light signals of at least two wavelengths to pass through and limit the light signal intensity of at least a portion of the at least two wavelengths.
[0126] A multispectral sensor is used to separate the light signal filtered by the filter unit into light signals of multiple bands and convert the light signals into electrical signals to obtain spectral image data of multiple bands.
[0127] An image processor is used to obtain spectral information of the current scene based on spectral image data in multiple bands.
[0128] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide a mobile terminal, as shown in FIG8. The mobile terminal 800 includes: any of the multispectral imaging modules 810 and the first imaging module 820 provided in the embodiments of this application.
[0129] The multispectral imaging module 810 is used to sense light signals in multiple bands of the current scene and obtain the spectral information of the current scene.
[0130] The first imaging module 820 is used to acquire image data of the current scene based on the shooting parameters determined by spectral information;
[0131] The first imaging module 820 is also used to perform image processing on the image data of the current scene based on the spectral information of the current scene, so as to obtain the processed image data.
[0132] The first imaging module is a conventional RGB imaging module. In RGB imaging modules, an RGB sensor is typically used to capture images. However, the RGB sensor only records information from three bands of the visible spectrum, while the true spectral information contains a wider range of more detailed spectral information. This means that RGB sensors have limitations in capturing and recording spectral information; different objects may have different reflection and transmission characteristics at different spectral frequency bands, which may lead to certain errors in color reproduction by the RGB camera. This application's embodiment utilizes a multispectral imaging module to provide richer and more detailed spectral information, adjusts the shooting parameters of the first imaging module, and performs image processing on the image data acquired by the first imaging module. The multispectral imaging module and the first imaging module work together to improve image quality and the accuracy of color reproduction.
[0133] For example, the shooting parameters of the first imaging module, such as aperture, shutter speed, ISO, exposure time, white balance, and gain, are adjusted according to spectral information to optimize the shooting effect.
[0134] For example, when using a deep learning model for image processing, inputting spectral information as auxiliary information into the deep learning model can optimize the image processing results.
[0135] In some embodiments, the filtering unit allows optical signals of the first band and the second band to pass through, and limits the intensity of the second band optical signal. The first band and the second band are adjacent bands.
[0136] In some embodiments, the first band is the visible light band and the second band is the infrared band. The filtering unit allows light signals from both the visible light band and the infrared band to pass through, while limiting the intensity of the infrared band light signal. This reduces crosstalk between the infrared band and the visible light band when the multispectral sensor responds to light signals from multiple bands.
[0137] In some embodiments, the filtering unit limits the intensity of the second-band light signal by controlling the transmittance of the second-band light signal.
[0138] In some embodiments, one of the following is included:
[0139] The transmittance of the filter unit for the second band of optical signal is less than the first preset value;
[0140] The transmittance of the filter unit to the second band of optical signal is greater than or equal to the first preset value;
[0141] In the first light source scenario, the transmittance of the filter unit to the second band of light signal is less than the first preset value; in the second light source scenario, the transmittance of the filter unit to the second band of light signal is greater than or equal to the first preset value.
[0142] In some embodiments, the multispectral sensor includes at least a spectrometer array, a photosensitive unit array, and a signal processing circuit.
[0143] A beam splitter array is used to separate optical signals into multiple wavelength bands.
[0144] Photosensitive unit array is used to convert optical signals of multiple wavelengths into electrical signals;
[0145] The signal processing circuit is used to process electrical signals in multiple bands and output spectral image data in multiple bands.
[0146] In some embodiments, the multispectral sensor further includes a microlens array: the microlens array is used to further focus the light and guide it onto the sensor. The microlens array can improve the spectral acquisition efficiency of the sensor and reduce aberrations.
[0147] In some embodiments, the spectral dispersive unit is formed using a resin dye coating technique to create a filter.
[0148] In some embodiments, the principal ray angle of the lens is less than a second preset value, and the principal ray angle of the lens and the principal ray angle of the multispectral sensor meet the matching condition.
[0149] In some embodiments, the image processor is used to compensate for spectral image data of multiple bands output by a multispectral sensor to obtain spectral image data after compensation of multiple bands; and to perform spectral analysis based on the spectral image data after compensation of multiple bands to obtain spectral information.
[0150] For example, based on computer vision (CV) algorithms, the raw data of 9 channels are compensated. Due to the influence of angle effects, the infrared band components of different fields of view (i.e., different image positions) are different. Therefore, different compensation coefficients are used to compensate the infrared components of different fields of view so that the infrared components of different field of view positions are the same. The spectral characteristics of the environment are obtained by performing spectral calculations based on the compensated multispectral channel data.
[0151] In some embodiments, the first imaging module 820 may include one or more imaging modules. For example, the first imaging module 820 includes a main camera module, a wide-angle module, and a telephoto module. The main camera module, the wide-angle module, and the telephoto module acquire light signals in the visible light band to obtain image data of the current scene.
[0152] The mobile terminals described in this application may include devices such as mobile phones, tablets, laptops, PDAs, wearable devices, cameras, and smart home appliances.
[0153] Based on the above embodiments of this application, an image processing method is also provided, as further illustrated in Figure 9. This method specifically includes:
[0154] Step 901: Obtain the spectral information of the current scene;
[0155] The spectral information is determined based on any of the multispectral imaging modules provided in the embodiments of this application.
[0156] Step 902: Determine the shooting parameters of the first imaging module based on the spectral information of the current scene;
[0157] The first imaging module is a conventional RGB imaging module, which typically uses an RGB sensor to capture images. The multispectral imaging module provides richer and more detailed spectral information, allowing for adjustments to the first imaging module's aperture, shutter speed, ISO, exposure time, white balance, gain, and other shooting parameters to optimize the shooting results.
[0158] Step 903: Control the first imaging module to acquire image data of the current scene based on the shooting parameters of the first imaging module;
[0159] Step 904: Perform image processing on the image data of the current scene based on the spectral information of the current scene to obtain the processed image data.
[0160] For example, image processing includes at least one of the following: image preprocessing, image enhancement, image compression, etc.
[0161] For example, image data and spectral information can be input into a deep learning model for image processing to obtain processed image data. Spectral information can be used as auxiliary information input into the deep learning model to optimize the image processing effect.
[0162] Because multispectral imaging modules can capture a wider spectral range than traditional imaging modules, including visible, infrared, and ultraviolet bands, they can acquire the reflection, absorption, and transmission characteristics of objects at different spectral levels during image capture. This provides richer information and plays a crucial role in both traditional imaging module shooting and image processing.
[0163] To implement the method of the embodiments of this application, based on the same inventive concept, the embodiments of this application also provide an image processing apparatus, as shown in FIG10. The image processing apparatus 1000 includes:
[0164] The acquisition unit 1010 is used to acquire the spectral information of the current scene, which is determined based on the multispectral imaging module of the first aspect.
[0165] The processing unit 1020 is used to determine the shooting parameters of the first imaging module based on the spectral information of the current scene.
[0166] The control unit 1030 is used to acquire image data of the current scene based on the shooting parameters determined by spectral information;
[0167] The processing unit 1020 is also used to perform image processing on the image data of the current scene based on the spectral information of the current scene, so as to obtain the processed image data.
[0168] In practical applications, the aforementioned device can be a mobile terminal or a chip used in mobile devices. In this application, the device can implement the functions of multiple units through software, hardware, or a combination of both, enabling the device to execute the image processing method provided in any of the above embodiments. Furthermore, the technical effects of each technical solution of the device can be referenced to the technical effects of the corresponding technical solutions in the image processing method; therefore, this application will not elaborate further on these effects.
[0169] Based on the hardware implementation of each unit in the above-mentioned image processing device, this application embodiment also provides an image processing device, as shown in FIG11. The image processing device 1100 includes: a processor 1110 and a memory 1120 configured to store a computer program capable of running on the processor.
[0170] When the processor 1110 is configured to run a computer program, it executes the method steps described in the foregoing embodiments.
[0171] Of course, in practical applications, as shown in Figure 11, the various components in the image processing device 1100 are coupled together via a bus system 1130. It can be understood that the bus system 1130 is used to achieve communication between these components. In addition to the data bus, the bus system 1130 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 1130 in the figure.
[0172] In practical applications, the aforementioned processor can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), controller, microcontroller, and microprocessor. It is understood that, for different devices, the electronic device used to implement the above processor functions can also be other types, and the embodiments of this application do not specifically limit this.
[0173] The aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.
[0174] Optionally, the image processing device 1100 can be a chip, which may further include an input interface. The processor can control the input interface to communicate with other devices or chips, specifically, it can acquire information or data sent by other devices or chips.
[0175] Optionally, the chip may also include an output interface. The processor can control this output interface to communicate with other devices or chips; specifically, it can output information or data to other devices or chips.
[0176] In an exemplary embodiment, this application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an image processing device to perform the steps of the aforementioned method.
[0177] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the methods in this application.
[0178] Optionally, the computer program product can be applied to the image processing device in the embodiments of this application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the image processing device in the various methods of the embodiments of this application. For the sake of brevity, they will not be described in detail here.
[0179] This application also provides a computer program.
[0180] Optionally, the computer program can be applied to the image processing device in the embodiments of this application. When the computer program is run on a computer, it causes the computer to execute the corresponding processes implemented by the image processing device in the various methods of the embodiments of this application. For the sake of brevity, it will not be described in detail here.
[0181] It should be understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0182] It should be understood that the terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items. The expressions “having,” “may have,” “comprising,” and “including,” or “may include” and “may contain” used herein may be used to indicate the presence of a corresponding feature (e.g., an element such as a number, function, operation, or component), but do not exclude the presence of additional features.
[0183] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of this invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information.
[0184] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.
[0185] In the several embodiments provided in this application, it should be understood that the disclosed methods, apparatus, and devices can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.
[0186] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0187] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0188] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Industrial applicability
[0189] This application provides a multispectral imaging module, a mobile terminal, and an image processing method. The multispectral imaging module specifically includes: a lens for receiving and focusing light signals; a filter unit for filtering the light signals focused by the lens to allow light signals of at least two wavelengths to pass through, and limiting the intensity of light signals in at least a portion of the at least two wavelengths; a multispectral sensor for separating the light signals filtered by the filter unit into light signals of multiple wavelengths, and converting the light signals into electrical signals to obtain spectral image data in multiple wavelengths; and an image processor for obtaining spectral information of the current scene based on the spectral image data in multiple wavelengths. Thus, by limiting the intensity of light signals in the allowed wavelengths through the filter unit, interference from these wavelengths to other wavelengths can be reduced, while retaining some information from these wavelengths, thereby improving the spectral reconstruction accuracy of the captured scene.
Claims
A multispectral imaging module, wherein, include: A lens is used to receive and focus light signals; A filtering unit is used to filter the light signal after it has been focused by the lens, so as to allow light signals of at least two wavelengths to pass through and limit the light signal intensity of at least a portion of the at least two wavelengths. A multispectral sensor is used to separate the light signal filtered by the filter unit into light signals of multiple bands and convert the light signals into electrical signals to obtain spectral image data of multiple bands. An image processor is used to obtain spectral information of the current scene based on spectral image data in the multiple bands. The multispectral imaging module of claim 1, wherein The filtering unit allows optical signals of the first band and the second band to pass through, and limits the intensity of the optical signal of the second band. The first band and the second band are adjacent bands. The multispectral imaging module of claim 2, wherein, The first band is the visible light band, and the second band is the infrared band. The filtering unit allows light signals from the visible light band and the infrared band to pass through, and limits the intensity of the infrared band light signal, so that when the multispectral sensor responds to the light signals of the multiple bands, it reduces the signal crosstalk of the infrared band to the visible light band. The multispectral imaging module of claim 2, wherein The filtering unit limits the light signal intensity of the second band by controlling the light signal transmittance of the second band. The multispectral imaging module of claim 4, wherein, Including one of the following: The transmittance of the filter unit for the second band of light signal is less than a first preset value; The transmittance of the filter unit to the optical signal in the second band is greater than or equal to the first preset value; In the first light source scenario, the transmittance of the filter unit to the light signal of the second band is less than a first preset value; in the second light source scenario, the transmittance of the filter unit to the light signal of the second band is greater than or equal to the first preset value. The multispectral imaging module of claim 1, wherein The multispectral sensor includes at least a spectrometer array, a photosensitive unit array, and a signal processing circuit. The optical splitting unit array is used to separate optical signals into multiple optical bands; The photosensitive unit array is used to convert optical signals of the multiple wavelength bands into electrical signals; The signal processing circuit is used to process the electrical signals of the multiple bands and output spectral image data of the multiple bands. The multispectral imaging module of claim 6, wherein The spectral splitting unit is formed by forming a filter based on resin dye coating technology. The multispectral imaging module according to any one of claims 1 to 7, wherein The principal ray angle of the lens is less than the second preset value, and the principal ray angle of the lens and the principal ray angle of the multispectral sensor meet the matching condition. The multispectral imaging module according to claim 8, wherein, The image processor is used to compensate the spectral image data of the multiple bands output by the multispectral sensor to obtain the compensated spectral image data of the multiple bands. Spectral analysis is performed on the spectral image data after compensation for the multiple bands to obtain spectral information. A mobile terminal, wherein, Includes the multispectral imaging module as described in any one of claims 1-9. A mobile terminal, wherein, Includes the multispectral imaging module and the first imaging module as described in any one of claims 1-9. A multispectral imaging module is used to sense light signals in multiple bands of the current scene and obtain the spectral information of the current scene. The first imaging module is used to acquire image data of the current scene based on the shooting parameters determined by the spectral information; The first imaging module is further configured to perform image processing on the image data of the current scene based on the spectral information of the current scene, to obtain processed image data. An image processing method, wherein, The method includes: Acquire spectral information of the current scene, wherein the spectral information is determined based on the multispectral imaging module as described in any one of claims 1 to 9; Based on the spectral information of the current scene, the shooting parameters of the first imaging module are determined; Based on the shooting parameters of the first imaging module, the first imaging module is controlled to acquire image data of the current scene; Based on the spectral information of the current scene, image processing is performed on the image data of the current scene to obtain processed image data.