Image processing method and apparatus, electronic device, and computer-readable medium

Abstract

The present application discloses an image processing method and apparatus, a device, and a medium. The method comprises: acquiring a visible light image and a non-visible light image of a target low-brightness scene; preprocessing the visible light image to generate a preprocessed visible light image; on the basis of the preprocessed visible light image and the non-visible light image, generating an enhanced non-visible light image; and on the basis of the enhanced non-visible light image, brightening the preprocessed visible light image to enhance the preprocessed visible light image so as to obtain an enhanced visible light image.

Description

Image processing methods, apparatuses, electronic devices and computer-readable media

[0001] This application claims priority to Chinese Patent Application No. CN202510099452.8, filed on January 22, 2025, entitled "Image Processing Method, Apparatus, Electronic Device and Computer-Readable Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] Embodiments of this application relate to the field of computer technology, and more specifically to image processing methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0003] In low-light scenarios, night vision equipment is typically used to acquire images of the target scene. Currently, night vision equipment typically acquires low-light scene images by using infrared imaging to provide a black and white image under nighttime or low-light conditions, or by performing RGB image enhancement on a single full-color RGB image to achieve normal lighting and color levels. However, when using these methods to acquire low-light scene images, the following technical problems often arise:

[0004] Black and white images generated by infrared imaging lack color information, resulting in the loss of necessary features in the captured image. When enhancing a single full-color RGB image with RGB imagery, the lack of prior exposure information amplifies the noise and color cast of the original image, resulting in unclear outlines in the enhanced image. Consequently, the enhanced image cannot clearly display information about low-light scenes.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that is not prior art known to those skilled in the art. Technical issues

[0006] This application provides an image processing method, apparatus, electronic device, and computer-readable medium that can effectively enhance visible light images, thereby generating clear visible light images, avoiding the loss of necessary features in captured images, and avoiding the situation where the enhanced image cannot clearly display low-light scene information. Technical solutions

[0007] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this application provide image processing methods, apparatuses, electronic devices, and computer-readable media to solve one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this application provide an image processing method, the method comprising: acquiring a visible light image of a target low-brightness scene and a non-visible light image of the same scene; preprocessing the visible light image to generate a preprocessed visible light image; performing an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image; and brightening the preprocessed visible light image based on the enhanced non-visible light image to enhance the preprocessed visible light image, thereby obtaining an enhanced visible light image.

[0010] Secondly, some embodiments of this application provide an image processing apparatus, comprising: an acquisition unit configured to acquire a visible light image of a target low-brightness scene and a non-visible light image of the same scene; a preprocessing unit configured to preprocess the visible light image to generate a preprocessed visible light image; an execution unit configured to perform an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image; and a brightening unit configured to brighten the preprocessed visible light image based on the enhanced non-visible light image to enhance the preprocessed visible light image and obtain an enhanced visible light image.

[0011] Thirdly, some embodiments of this application provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this application provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above. Beneficial effects

[0013] The above embodiments of this application have the following beneficial effects: the image processing methods of some embodiments of this application avoid the loss of necessary features in the captured image and the inability of the enhanced image to clearly display low-light scene information. Specifically, the reasons for the loss of necessary features in the captured image and the inability of the enhanced image to clearly display low-light scene information are: the black and white image generated by infrared light imaging lacks color information, resulting in the loss of necessary features in the captured image; and when enhancing a single full-color RGB image using RGB image enhancement, the lack of prior exposure information amplifies the noise and color cast of the original image, resulting in unclear outlines in the enhanced image, which in turn leads to the inability of the enhanced image to clearly display low-light scene information. Based on this, the image processing methods of some embodiments of this application first acquire a visible light image of the target low-brightness scene and a non-visible light image of the same scene. Thus, RGB visible light and non-visible light images of the low-brightness scene can be acquired. Second, the visible light image is preprocessed to generate a preprocessed visible light image. Thus, preliminary enhancement processing of the visible light image can be performed. Then, based on the preprocessed visible light image and the non-visible light image, an image fusion operation is performed to generate an enhanced non-visible light image. This improves the quality of the non-visible light image, resulting in a high-quality non-visible light image. Finally, based on the enhanced non-visible light image, the preprocessed visible light image is brightened to further enhance it, resulting in an enhanced visible light image. Thus, by leveraging the reflectivity and texture information of the non-visible light image in low-light scenes, which complements the color information of the visible light image, the visible light image can be effectively enhanced, generating a clear visible light image. This avoids the loss of necessary features in the captured image and prevents the enhanced image from failing to clearly display low-light scene information. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 is a schematic diagram of an application scenario of the image processing method of some embodiments of this application;

[0016] Figure 2 is a flowchart of some embodiments of the image processing method according to this application;

[0017] Figure 3 is a flowchart of some other embodiments of the image processing method according to this application;

[0018] Figure 4 is a schematic diagram of the structure of some embodiments of the image processing apparatus according to this application;

[0019] Figure 5 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of this application.

[0020] Implementation methods of this application

[0021] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0022] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0023] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Figure 1 is a schematic diagram of an application scenario of the image processing method of some embodiments of this application.

[0028] In the application scenario shown in Figure 1, firstly, the computing device 101 can acquire a visible light image 102 of the target low-brightness scene and a non-visible light image 104 of the same scene. Secondly, the computing device 101 can preprocess the visible light image 102 to generate a preprocessed visible light image 103. Then, the computing device 101 can perform an image fusion operation based on the preprocessed visible light image 103 and the non-visible light image 104 to generate an enhanced non-visible light image 105. Finally, the computing device 101 can brighten the preprocessed visible light image 103 based on the enhanced non-visible light image 105 to enhance the preprocessed visible light image 103, resulting in an enhanced visible light image 106.

[0029] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0030] It should be understood that the number of computing devices shown in Figure 1 is merely illustrative. Any number of computing devices can be used depending on the implementation requirements.

[0031] Referring again to FIG2, a flow 200 of some embodiments of the image processing method according to this application is shown. The image processing method includes the following steps:

[0032] Step 201: Obtain the visible light image of the target low-brightness scene and the non-visible light image of the same scene.

[0033] In some embodiments, the execution entity of the image processing method (e.g., the computing device 101 shown in FIG. 1) can acquire a visible light image of the target low-brightness scene and a non-visible light image of the same scene. In practice, the execution entity can acquire the visible light image of the target low-brightness scene and the non-visible light image of the same scene in various ways. The non-visible light image may include at least one of infrared image, near-infrared image, and multispectral image.

[0034] As an example, visible light images of the target low-brightness scene and non-visible light images of the same scene can be obtained by pre-deployed shooting devices and non-visible light devices capable of capturing the target low-brightness scene.

[0035] Step 202: Preprocess the visible light image to generate a preprocessed visible light image.

[0036] In some embodiments, the execution entity may preprocess the visible light image to generate a preprocessed visible light image. In practice, the preprocessing may involve resizing and denoising the visible light image.

[0037] In the process of adopting technical solutions to solve the above-mentioned technical problems, the following technical problem 2 often occurs: visible light images have a lot of noise and color distortion, which makes it impossible to recognize the information displayed in the visible light images even after enhancement.

[0038] In some optional implementations of certain embodiments, the aforementioned execution entity may preprocess the visible light image through the following steps:

[0039] The first step is to perform a logarithmic transformation on the above visible light image to generate the first transformed image. In practice, the logarithmic transformation of the above visible light image can be performed using the following formula:

[0040] .

[0041] in, This represents the normalized visible light image. This represents the image after the first transformation.

[0042] The second step involves performing an exponential transform on the aforementioned visible light image to generate the second transformed image. This exponential transform can be performed on the visible light image using the following formula:

[0043] .

[0044] in, This represents the image after the second transformation. This represents the normalized visible light image.

[0045] The third step involves nonlinearly fusing the first transformed image and the second transformed image to generate a fused transformed image. This nonlinear fusing can be achieved using the following formula:

[0046] .

[0047] in, This indicates the transformed image after fusion. This parameter represents the degree of fusion.

[0048] The fourth step involves performing a nonlinear mapping process on the fused transformed image to generate a mapped transformed image. In practice, this nonlinear mapping process can be performed on the fused transformed image using the following formula:

[0049] .

[0050] in, This indicates the image transformation after mapping. This represents the mapping coefficient. This represents the arctangent function.

[0051] The fifth step involves inputting the mapped and transformed image into the error function to generate the adjusted and transformed image. In practice, the adjusted and transformed image can be generated using the following formula:

[0052] .

[0053] in, This indicates the image after adjustment and transformation.

[0054] The sixth step is to normalize the adjusted image to generate a normalized adjusted image as the preprocessed visible light image. Here, the adjusted image can be normalized using the following formula:

[0055] .

[0056] in, This represents the preprocessed visible light image. This represents the stability term, used to ensure numerical stability during the normalization process.

[0057] The first to sixth steps described above are an inventive point of this application, solving the technical problem that "visible light images have a lot of noise and color cast, making it impossible to recognize the information displayed in the visible light image even after enhancement." The reasons for this inability to recognize the information displayed in the visible light image are as follows: the visible light image has a lot of noise and color cast, making it impossible to recognize the information displayed in the visible light image even after enhancement. Solving these problems would allow for effective recognition of the information displayed in the visible light image, improving security in low-brightness scenes. To achieve this effect, this application firstly performs a logarithmic transformation on the visible light image to generate a first transformed image. This allows for brightening of areas with low intensity using the properties of the logarithmic function. Secondly, it performs an exponential transformation on the visible light image to generate a second transformed image. This allows for improving the contrast of high-brightness areas using the properties of the exponential function. Thirdly, it performs a nonlinear fusion process on the first and second transformed images to generate a fused transformed image. This allows for the fusion of the logarithmically and exponentially transformed images. Fourth, the fused transformed image is subjected to nonlinear mapping processing to generate a mapped transformed image. This improves the overall image brightening performance. Fifth, the mapped transformed image is input into an error function to generate an adjusted transformed image. This improves the smoothness of the overall image and suppresses bright and dark areas. Sixth, the adjusted transformed image is normalized to generate a normalized adjusted transformed image as the preprocessed visible light image. Thus, the preprocessing of the visible light image is completed, improving the contrast of high-brightness areas, brightening low-intensity areas, improving the smoothness of the overall image, and suppressing bright and dark areas, thereby effectively extracting information from low-brightness scenes in the visible light image.

[0058] Step 203: Based on the preprocessed visible light image and the non-visible light image, perform an image fusion operation to generate an enhanced non-visible light image.

[0059] In some embodiments, the execution entity may perform an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image.

[0060] In practice, an image fusion operation can be performed based on the preprocessed visible light image and the non-visible light image described above to generate an enhanced non-visible light image through the following steps:

[0061] The first step involves calculating the gradients of the preprocessed visible light image and the non-visible light image to obtain the first and second gradient values. In practice, the Sobel operator can be used to determine the first and second gradient values.

[0062] , .

[0063] in, This represents the first gradient value. This represents the first gradient value.

[0064] The second step is to normalize the first gradient value and the second gradient value to generate normalized first gradient value and normalized second gradient value.

[0065] The third step involves performing gradient fusion processing on the preprocessed visible light image and the non-visible light image based on the normalized first gradient value and the normalized second gradient value, so as to generate a gradient-fused image as an enhanced non-visible light image.

[0066] In practice, gradient fusion processing can be performed on the preprocessed visible light image and the non-visible light image using the following sub-steps:

[0067] The first sub-step generates a visible light gradient image and a non-visible light gradient image based on the above-mentioned normalized first gradient value and the above-mentioned normalized second gradient value.

[0068] The second sub-step involves determining the gradient fusion weight values ​​corresponding to the preprocessed visible light image and the non-visible light image based on the normalized first gradient value and the non-visible light image fusion weight value, thus obtaining the first gradient fusion weight value and the second gradient fusion weight value. In practice, the first gradient fusion weight value and the second gradient fusion weight value can be obtained using the following formula:

[0069] .

[0070] in, and These represent the first gradient fusion weight value and the second gradient fusion weight value, respectively. This represents the non-visible light image fusion weights, which are pre-set weights. Indicates a stable term. This represents the normalized first gradient value. This represents the normalized second gradient value.

[0071] The third sub-step involves performing image fusion processing on the visible light gradient image and the non-visible light gradient image based on the first gradient fusion weight value and the second gradient fusion weight value, to generate a fused non-visible light image as an enhanced non-visible light image. Here, the enhanced non-visible light image can be generated using the following formula:

[0072] .

[0073] in, This indicates enhancement of non-visible light images. This represents a visible light gradient image. This represents a non-visible light gradient image.

[0074] Step 204: Based on the above-mentioned enhanced non-visible light image, the above-mentioned preprocessed visible light image is brightened to enhance the above-mentioned preprocessed visible light image and obtain an enhanced visible light image.

[0075] In some embodiments, the execution entity may perform brightening processing on the preprocessed visible light image based on the enhanced non-visible light image to enhance the preprocessed visible light image and obtain an enhanced visible light image.

[0076] In practice, enhanced visible light images can be obtained through the following steps:

[0077] The first step is to determine the nonlinear transformation parameters for image enhancement of the preprocessed visible light image based on the enhanced non-visible light image and the preprocessed visible light image.

[0078] Here, the nonlinear transformation parameters for image enhancement of the preprocessed visible light image can be determined through the following sub-steps:

[0079] The first sub-step involves performing feature extraction processing on the preprocessed visible light image and the enhanced non-visible light image to generate visible light features and non-visible light features, respectively.

[0080] The second sub-step involves merging the aforementioned visible light features and the aforementioned non-visible light features to generate merged image features.

[0081] The third sub-step involves inputting the merged image features into a pre-trained parameter prediction model to obtain prediction results for the nonlinear transformation parameters. This parameter prediction model can be a pre-trained convolutional neural network model. Here, the parameter prediction model is trained using a pre-set fusion image structure loss function and exposure loss function.

[0082] .

[0083] in, This represents the loss function for the fused image structure. This represents the average intensity map of an enhanced visible light image. This represents the average intensity map of a visible light image. This represents a non-visible light image. Indicates the current pixel. This represents the four pixels that are adjacent to the current pixel in the top, bottom, left, and right directions. This indicates the number of parts into which the image is divided.

[0084] .

[0085] in, This represents the exposure loss function. This indicates the number of parts the image is divided into; here, 4 is used. 4. Average pooling is used to downsample the image.

[0086] The second step involves brightening the preprocessed visible light image based on the aforementioned nonlinear transformation parameters, thereby enhancing the image and obtaining an enhanced visible light image. This brightening process can be achieved by performing a nonlinear transformation on the preprocessed visible light image. In practice, a pre-designed nonlinear transformation formula can be used, as shown in the following equation:

[0087] .

[0088] in, The nonlinear transformation parameters are determined using the aforementioned parameter prediction model.

[0089] The above embodiments of this application have the following beneficial effects: the image processing methods of some embodiments of this application avoid the loss of necessary features in the captured image and the inability of the enhanced image to clearly display low-light scene information. Specifically, the reasons for the loss of necessary features in the captured image and the inability of the enhanced image to clearly display low-light scene information are: the black and white image generated by infrared light imaging lacks color information, resulting in the loss of necessary features in the captured image; and when enhancing a single full-color RGB image using RGB image enhancement, the lack of prior exposure information amplifies the noise and color cast of the original image, resulting in unclear outlines in the enhanced image, which in turn leads to the inability of the enhanced image to clearly display low-light scene information. Based on this, the image processing methods of some embodiments of this application first acquire a visible light image of the target low-brightness scene and a non-visible light image of the same scene. Thus, RGB visible light and non-visible light images of the low-brightness scene can be acquired. Second, the visible light image is preprocessed to generate a preprocessed visible light image. Thus, preliminary enhancement processing of the visible light image can be performed. Then, based on the preprocessed visible light image and the non-visible light image, an image fusion operation is performed to generate an enhanced non-visible light image. This improves the quality of the non-visible light image, resulting in a high-quality non-visible light image. Finally, based on the enhanced non-visible light image, the preprocessed visible light image is brightened to further enhance it, resulting in an enhanced visible light image. Thus, by leveraging the reflectivity and texture information of the non-visible light image in low-light scenes, which complements the color information of the visible light image, the visible light image can be effectively enhanced, generating a clear visible light image. This avoids the loss of necessary features in the captured image and prevents the enhanced image from failing to clearly display low-light scene information.

[0090] Please continue to refer to Figure 3, which illustrates a flow 300 of another embodiment of the image processing method. Flow 300 of this image processing method includes the following steps:

[0091] Step 301: Obtain the visible light image of the target low-brightness scene and the non-visible light image of the same scene.

[0092] In some embodiments, the specific implementation of step 301 and its resulting technical effects can be referred to step 201 in the embodiment corresponding to Figure 2, and will not be repeated here.

[0093] Step 302: Map the visible light image to a first color space or a second color space to generate a first mapped visible light image or a second mapped visible light image, which serves as the mapped visible light image.

[0094] In some embodiments, the executing entity may map the visible light image to a first color space or a second color space to generate a first mapped visible light image or a second mapped visible light image, which serves as the mapped visible light image. The first color space may be the YUV color space. In the YUV color space, the Y channel represents luminance, and the U and V channels represent chrominance. The second color space may be the HSV color space. In the HSV color space, H (Hue) represents hue, S (Saturation) represents saturation, and V (Value) represents luminance.

[0095] As an example, the above visible light image is mapped to the HSV color space. For each pixel in the visible light image, R'=R / 255, G'=G / 255, B'=B / 255, Cmax=max(R', G', B'), Cmin=min(R', G', B'), and Δ=Cmax-Cmin.

[0096] Step 303: Decouple the mapped visible light image to generate a luminance channel.

[0097] In some embodiments, the execution entity may decouple the mapped visible light image to generate a luminance channel. Here, after decoupling, in addition to the luminance channel, other related channels are also included.

[0098] As an example, if the visible light image after the above mapping is a YUV image, then the luminance channel can be the Y channel, and the other related channels can be the U channel and the V channel. If the visible light image after the above mapping is an HSV image, then the luminance channel can be the V channel, and the other related channels can be the H channel and the S channel.

[0099] Step 304: Perform a weighted summation of the luminance channel and the non-visible light image to generate an enhanced visible light image.

[0100] In some embodiments, the execution entity may perform a weighted summation of the brightness channel and the non-visible light image to generate an enhanced visible light image.

[0101] As an example, in response to the above mapping resulting in a YUV image, the Y channel can be replaced with the above non-visible light image. In response to the above mapping resulting in a Visible Light image, the V channel can be replaced with the above non-visible light image.

[0102] Step 305: Generate an RGB image based on the channels included in the mapped visible light image.

[0103] In some embodiments, the implementing entity described above can generate an RGB image based on the channels included in the mapped visible light image. In practice, the channels can be merged and converted into an RGB image.

[0104] As can be seen from Figure 3, compared with the description of some embodiments corresponding to Figure 2, the flow 300 of the image processing method in some embodiments corresponding to Figure 3 describes another specific step in generating an enhanced visible light image. Thus, the scheme described in these embodiments enhances the visible light image by using a luminance channel weighted summation method. This avoids image enhancement through a deep learning model, thereby eliminating the need for model training and reducing the time required for image enhancement.

[0105] Please continue to refer to Figure 4. As an implementation of the methods shown in the above figures, this application provides some embodiments of an image processing apparatus. These apparatus embodiments correspond to the method embodiments shown in Figure 2. The image processing apparatus can be specifically applied to various electronic devices.

[0106] As shown in Figure 4, an image processing apparatus 400 in some embodiments includes: an acquisition unit 401, a preprocessing unit 402, an execution unit 403, and a brightening unit 404. The acquisition unit 401 is configured to acquire a visible light image of a target low-brightness scene and a non-visible light image of the same scene; the preprocessing unit 402 is configured to preprocess the visible light image to generate a preprocessed visible light image; the execution unit 403 is configured to perform an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image; and the brightening unit 404 is configured to brighten the preprocessed visible light image based on the enhanced non-visible light image to enhance the preprocessed visible light image, thereby obtaining an enhanced visible light image.

[0107] It is understood that the units described in the image processing apparatus 400 correspond to the various steps in the method described with reference to FIG2. Therefore, the operations, features, and beneficial effects described above for the method also apply to the image processing apparatus 400 and the units contained therein, and will not be repeated here.

[0108] Referring now to FIG5, a schematic diagram of the structure of an electronic device 500 (computing device 101 as shown in FIG1) suitable for implementing some embodiments of the present application is shown. The electronic device shown in FIG5 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present application.

[0109] As shown in Figure 5, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0110] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 shows an electronic device 500 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Each box shown in Figure 5 may represent one device, or multiple devices may be represented as needed.

[0111] In particular, according to some embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of some embodiments of this application.

[0112] It should be noted that, in some embodiments of this application, the computer-readable medium described may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0113] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0114] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a visible light image of a target low-brightness scene and a non-visible light image of the same scene; preprocess the visible light image to generate a preprocessed visible light image; perform an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image; and brighten the preprocessed visible light image based on the enhanced non-visible light image to enhance the preprocessed visible light image, thereby obtaining an enhanced visible light image.

[0115] Computer program code for performing operations of some embodiments of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0117] The units described in some embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a preprocessing unit, an information selection gradient fusion unit, and a brightening unit. The names of these units do not necessarily limit the specific unit; for example, a preprocessing unit may also be described as "a unit that preprocesses a visible light image to generate a preprocessed visible light image."

[0118] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0119] The above description is merely a selection of preferred embodiments of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this application.

Claims

1. An image processing method, wherein, include: Acquire visible light images of the target low-brightness scene and non-visible light images of the same scene; The visible light image is preprocessed to generate a preprocessed visible light image; Based on the preprocessed visible light image and the non-visible light image, an image fusion operation is performed to generate an enhanced non-visible light image; Based on the enhanced non-visible light image, the preprocessed visible light image is brightened to enhance the preprocessed visible light image, resulting in an enhanced visible light image.

2. The method according to claim 1, wherein, The non-visible light image includes at least one of infrared image, near-infrared image, and multispectral image.

3. The method according to claim 1, wherein, After acquiring the visible light image of the target low-brightness scene and the non-visible light image of the same scene, the method further includes: The visible light image is mapped to a first color space or a second color space to generate a first mapped visible light image or a second mapped visible light image, which serves as the mapped visible light image. The mapped visible light image is decoupled to generate a luminance channel; The brightness channel and the non-visible light image are weighted and summed to generate an enhanced visible light image; An RGB image is generated based on the channels included in the mapped visible light image.

4. The method according to claim 1, wherein, The step of performing an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image includes: Gradient calculations are performed on the preprocessed visible light image and the non-visible light image to obtain a first gradient value and a second gradient value; The first gradient value and the second gradient value are normalized to generate a normalized first gradient value and a normalized second gradient value; Based on the normalized first gradient value and the normalized second gradient value, gradient fusion processing is performed on the preprocessed visible light image and the non-visible light image to generate a gradient-fused image as an enhanced non-visible light image.

5. The method according to claim 1, wherein, The step of brightening the preprocessed visible light image based on the enhanced non-visible light image to enhance the preprocessed visible light image and obtain an enhanced visible light image includes: Based on the enhanced non-visible light image and the preprocessed visible light image, determine the nonlinear transformation parameters for image enhancement of the preprocessed visible light image; Based on the nonlinear transformation parameters, the preprocessed visible light image is brightened to enhance it, resulting in an enhanced visible light image.

6. The method according to claim 5, wherein, The step of determining the nonlinear transformation parameters for image enhancement of the preprocessed visible light image based on the enhanced non-visible light image and the preprocessed visible light image includes: Feature extraction processing is performed on the preprocessed visible light image and the enhanced non-visible light image respectively to generate visible light features and non-visible light features; The visible light features and the non-visible light features are merged to generate merged image features; The merged image features are input into a pre-trained parameter prediction model to obtain prediction results for nonlinear transformation parameters.

7. The method according to claim 6, wherein, The parameter prediction model is trained using a pre-set fusion image structure loss function and exposure loss function.

8. The method according to claim 4, wherein, The step of performing gradient fusion processing on the preprocessed visible light image and the non-visible light image based on the normalized first gradient value and the normalized second gradient value to generate a gradient-fused image as an enhanced non-visible light image includes: Based on the normalized first gradient value and the normalized second gradient value, a visible light gradient image and a non-visible light gradient image are generated. Based on the normalized first gradient value and the non-visible light image fusion weight value, determine the gradient fusion weight values ​​corresponding to the preprocessed visible light image and the non-visible light image, and obtain the first gradient fusion weight value and the second gradient fusion weight value. Based on the first gradient fusion weight value and the second gradient fusion weight value, the visible light gradient image and the non-visible light gradient image are subjected to image fusion processing to generate a fused non-visible light image as an enhanced non-visible light image.

9. The method according to claim 1, wherein, The steps for preprocessing visible light images include: The visible light image is subjected to a logarithmic transformation to generate a first transformed image; The visible light image is subjected to an exponential transformation to generate a second transformed image; The first transformed image and the second transformed image are subjected to nonlinear fusion processing to generate a fused transformed image; The fused transformed image is subjected to nonlinear mapping processing to generate a mapped transformed image; The mapped and transformed image is input into the error function to generate the adjusted and transformed image; The adjusted transformed image is normalized to generate a normalized adjusted transformed image, which is then used as the preprocessed visible light image.

10. The method according to claim 4, wherein, The first and second gradient values ​​are determined using the Sobel operator.

11. The method according to claim 4, wherein, The first gradient fusion weight value and the second gradient fusion weight value are calculated using the following formula: ； in, and These represent the first gradient fusion weight value and the second gradient fusion weight value, respectively. This represents the non-visible light image fusion weights, which are pre-set weights. Indicates a stable term. This represents the normalized first gradient value. This represents the normalized second gradient value.

12. The method according to claim 5, wherein, The brightening process involves performing a nonlinear transformation on the preprocessed visible light image.

13. The method according to claim 12, wherein, A pre-designed nonlinear transformation formula is used for nonlinear transformation processing. The nonlinear transformation formula is shown in the following equation: ； in, The nonlinear transformation parameters are represented and determined by the parameter prediction model.

14. The method according to claim 7, wherein, The exposure loss function uses 4 4. Average pooling is used to downsample the image.

15. The method according to claim 3, wherein, The first color space is the YUV color space, and the second color space is the HSV color space.

16. The method according to claim 13, wherein, The mapped visible light image is decoupled and other related channels are generated. When the mapped visible light image is a YUV image, the luminance channel is the Y channel, and the other related channels are the U channel and the V channel.

17. The method according to claim 13, wherein, The mapped visible light image is decoupled and other related channels are generated. When the mapped visible light image is an HSV image, the luminance channel is the V channel, and the other related channels are the H channel and the S channel.

18. An image processing apparatus, wherein, include: The acquisition unit is configured to acquire a visible light image of the target low-brightness scene and a non-visible light image of the same scene; A preprocessing unit is configured to preprocess the visible light image to generate a preprocessed visible light image; The execution unit is configured to perform an image fusion operation based on the preprocessed visible light image and the non-visible light image to generate an enhanced non-visible light image; The brightening unit is configured to brighten the preprocessed visible light image based on the enhanced non-visible light image, so as to enhance the preprocessed visible light image and obtain an enhanced visible light image.

19. An electronic device, wherein, include: One or more processors; A storage device on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 17.

20. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 17.

Images