Image processing method and apparatus, and electronic device and computer-readable medium

Abstract

Disclosed in the present application are an image processing method and apparatus, and a device and a medium. The method comprises: acquiring a visible light image and a non-visible light image; on the basis of the visible light image and the non-visible light image, generating a fused image feature; on the basis of the fused image feature, performing prediction processing on an enhancement factor corresponding to the visible light image, so as to generate a predicted enhancement factor; and on the basis of the predicted enhancement factor, performing at least one instance of image enhancement processing on the visible light image, so as to generate an enhanced visible light image.

Description

Image processing methods, apparatuses, electronic devices and computer-readable media

[0001] This application claims priority to Chinese Patent Application No. CN202510100988.7, filed on January 23, 2025, entitled "Image Processing Method, Apparatus, Electronic Device and Computer-Readable Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application relate to the field of computer technology, and more specifically to image processing methods, apparatus, electronic devices, and computer-readable media. Background Technology

[0003] Advances in computer vision technology have made high-quality image synthesis possible. However, the following technical problems often arise when synthesizing images:

[0004] In low-light or black-light scenarios, image synthesis is computationally inefficient and cannot be processed in real time in embedded systems or other resource-constrained environments.

[0005] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Technical issues

[0006] This application provides an image processing method, apparatus, electronic device, and computer-readable medium that can improve the computational efficiency of image synthesis in low-light and black-light scenes, and ensure real-time processing in embedded systems or other resource-constrained environments. Technical solutions

[0007] The summary section of this application is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0008] Some embodiments of this application provide image processing methods, apparatuses, electronic devices, and computer-readable media to solve one or more of the technical problems mentioned in the background section above.

[0009] In a first aspect, some embodiments of this application provide an image processing method, the method comprising: acquiring a visible light image and a corresponding non-visible light image in the same scene; performing feature fusion processing on the visible light image and the non-visible light image to generate image fusion features; performing prediction processing on an enhancement factor corresponding to the visible light image based on the image fusion features to generate a predicted enhancement factor, wherein the predicted enhancement factor includes a nonlinear transformation parameter; and performing at least one image enhancement processing on the visible light image based on the predicted enhancement factor to generate an enhanced visible light image.

[0010] Secondly, some embodiments of this application provide an image processing apparatus, comprising: an acquisition unit configured to acquire a visible light image and a corresponding non-visible light image in the same scene; a feature fusion unit configured to perform feature fusion processing on the visible light image and the non-visible light image to generate image fusion features; a prediction unit configured to perform prediction processing on an enhancement factor corresponding to the visible light image based on the image fusion features to generate a predicted enhancement factor, wherein the predicted enhancement factor includes a nonlinear transformation parameter; and an image enhancement unit configured to perform at least one image enhancement processing on the visible light image based on the predicted enhancement factor to generate an enhanced visible light image.

[0011] Thirdly, some embodiments of this application provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect above.

[0012] Fourthly, some embodiments of this application provide a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect above. Beneficial effects

[0013] The above embodiments of this application have the following beneficial effects: the image processing methods of some embodiments of this application improve the computational efficiency of image synthesis in low-light and black-light scenes, ensuring real-time processing in embedded systems or other resource-constrained environments. Specifically, the reason for the low computational efficiency of image synthesis is that the computational efficiency of image synthesis is low in low-light or black-light scenes, making real-time processing impossible in embedded systems or other resource-constrained environments. Based on this, the image processing methods of some embodiments of this application first acquire a visible light image and a corresponding non-visible light image in the same scene. Thus, visible light and non-visible light images of the same scene can be acquired. Second, feature fusion processing is performed on the visible light image and the aforementioned non-visible light image to generate image fusion features. Thus, the features of the visible light image and the infrared spectrum image can be fused at multiple levels. Then, based on the aforementioned image fusion features, the enhancement factor corresponding to the aforementioned visible light image is predicted to generate a predicted enhancement factor. Thus, the degree factor of enhancement of the visible light image can be determined through the fused features. Finally, based on the aforementioned predicted enhancement factor, the aforementioned visible light image is subjected to at least one image enhancement processing to generate an enhanced visible light image. Therefore, by decoupling the prediction of enhancement factors and the image enhancement process, the computational efficiency and scalability of the method are improved, the deployment difficulty is reduced, and the computational efficiency of image synthesis in low-light and black-light scenarios is improved, ensuring real-time processing in embedded systems or other resource-constrained environments. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this application will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0015] Figure 1 is a schematic diagram of an application scenario of the image processing method of some embodiments of this application;

[0016] Figure 2 is a flowchart of some embodiments of the image processing method according to this application;

[0017] Figure 3 is a flowchart of some other embodiments of the image processing method according to this application;

[0018] Figure 4 is a flowchart of some further embodiments of the image processing method according to this application;

[0019] Figure 5 is a schematic diagram of the structure of some embodiments of the image processing apparatus according to this application;

[0020] Figure 6 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of this application. Embodiments of the present invention

[0021] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0022] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0023] It should be noted that the concepts of "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0024] It should be noted that the terms "a" and "a plurality of" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of this application are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0026] The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] Figure 1 is a schematic diagram of an application scenario of the image processing method of some embodiments of this application.

[0028] In the application scenario shown in Figure 1, firstly, the computing device 101 can acquire a visible light image 102 and a corresponding non-visible light image 103 in the same scene. Secondly, the computing device 101 can perform feature fusion processing on the visible light image 102 and the non-visible light image 103 to generate an image fusion feature 104. Then, based on the image fusion feature 104, the computing device 101 can perform prediction processing on the enhancement factor corresponding to the visible light image 102 to generate a predicted enhancement factor 105. Finally, based on the predicted enhancement factor 105, the computing device 101 can perform at least one image enhancement processing on the visible light image 102 to generate an enhanced visible light image 106.

[0029] It should be noted that the aforementioned computing device 101 can be either hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is software, it can be installed within the hardware devices listed above. It can be implemented as, for example, multiple software programs or software modules used to provide distributed services, or as a single software program or software module. No specific limitations are made here.

[0030] It should be understood that the number of computing devices shown in Figure 1 is merely illustrative. Any number of computing devices can be used depending on the implementation requirements.

[0031] Referring again to FIG2, a flow 200 of some embodiments of the image processing method according to this application is shown. The image processing method includes the following steps:

[0032] Step 201: Obtain the visible light image and the corresponding non-visible light image in the same scene.

[0033] In some embodiments, the entity executing the image processing method (e.g., the computing device 101 shown in FIG. 1) can acquire a visible light image and a corresponding non-visible light image of the same scene. Here, the visible light image and infrared spectrum image of the target scene can be acquired in real time by an imaging device, or the visible light image and infrared spectrum image can be captured by the imaging device and then transmitted through an associated terminal. The target scene can represent a low-brightness scene.

[0034] In some embodiments of this application, after step 201, the above-mentioned visible light image is mapped to a preset color space to generate a mapped visible light image.

[0035] In some embodiments, the execution entity can map the visible light image to a preset color space to generate a mapped visible light image. The preset color space can be the RGB color space, used to divide the visible light image into different channels.

[0036] Step 202: Perform feature fusion processing on the visible light image and the non-visible light image to generate image fusion features.

[0037] In some embodiments, the aforementioned execution entity may perform feature fusion processing on visible light images and non-visible light images to generate image fusion features.

[0038] In some embodiments of this application, before step 202, the visible light image and the non-visible light image are normalized to generate a normalized visible light image and a normalized non-visible light image.

[0039] In some embodiments, the execution entity may perform normalization processing on the visible light image and the non-visible light image to generate a normalized visible light image and a normalized non-visible light image. Here, since the value range of each pixel in the eight-bit stored RGB image is 0 to 255, the normalization process is to adjust the pixel value of each pixel in the visible light image to 1 / 255 of the original pixel value.

[0040] Step 203: Based on image fusion features, perform prediction processing on the enhancement factors corresponding to the visible light image to generate predicted enhancement factors.

[0041] In some embodiments, the execution entity may perform prediction processing on the enhancement factor corresponding to the visible light image based on the image fusion features to generate a predicted enhancement factor.

[0042] In practice, the enhancement factor corresponding to a visible light image can be predicted using the following steps:

[0043] The first step is to perform convolution processing on the above image fusion features to generate convolutional image fusion features.

[0044] The second step is to input the image fusion features after convolution into the hyperbolic tangent activation function to predict the enhancement factor and obtain the predicted enhancement factor.

[0045] In some alternative implementations of certain embodiments, the overall nonlinear transformation parameters for each channel of the visible light image can be pre-set as a prediction enhancement factor. The nonlinear transformation parameters can be obtained in advance based on different environmental and lighting conditions.

[0046] In some alternative implementations of certain embodiments, the visible light image can be downsampled to predict the transformation parameters at a lower resolution, and then upsampled to the original resolution to obtain the final nonlinear transformation parameters as a prediction enhancement factor.

[0047] Step 204: Based on the predicted enhancement factor, perform at least one image enhancement process on the visible light image to generate an enhanced visible light image.

[0048] In some embodiments, the execution entity may perform at least one image enhancement process on the visible light image based on the prediction enhancement factor to generate an enhanced visible light image.

[0049] In practice, based on the aforementioned prediction enhancement factor, each pixel in the visible light image can be iteratively processed to generate an iteratively processed visible light image as an enhanced visible light image. Here, the visible light image can be enhanced using the following formula:

[0050] ,

[0051] in, Represents the pixels of a visible light image. This represents the predicted enhancement factor. In response to the completion of a single image enhancement process, the processed visible light image is used as the visible light image again for image enhancement, until the preset number of image enhancement processes are completed.

[0052] In some embodiments of this application, after step 204, the following steps are also included:

[0053] The first step is to input the enhanced visible light image into a pre-trained image recognition model to obtain the image recognition result.

[0054] In some embodiments, the aforementioned executing entity may input the enhanced visible light image into a pre-trained image recognition model to obtain an image recognition result. The image recognition model may be a pre-trained model for content recognition of visible light images of a target scene. As an example, the image recognition model may be a pre-trained large-scale security recognition model.

[0055] The second step is to control the associated devices to perform the corresponding device operations in response to the image recognition results meeting the preset control conditions.

[0056] In some embodiments, the execution entity may control an associated device to perform a corresponding device operation in response to the image recognition result satisfying a preset control condition. The preset control condition indicates that the content displayed by the visible light image satisfies the control conditions of an associated device.

[0057] As an example, in response to the image recognition result indicating a fire, the device can be a fire sprinkler, and the preset control condition can be to identify the fire and control the fire sprinkler to perform a spraying operation for fire extinguishing.

[0058] The above embodiments of this application have the following beneficial effects: the image processing methods of some embodiments of this application improve the computational efficiency of image synthesis in low-light and black-light scenes, ensuring real-time processing in embedded systems or other resource-constrained environments. Specifically, the reason for the low computational efficiency of image synthesis is that the computational efficiency of image synthesis is low in low-light or black-light scenes, making real-time processing impossible in embedded systems or other resource-constrained environments. Based on this, the image processing methods of some embodiments of this application first perform feature fusion processing on a visible light image and its corresponding infrared spectral image to generate image fusion features. This allows for multi-level fusion of features between the visible light image and the infrared spectral image. Second, based on the aforementioned image fusion features, a prediction processing is performed on the enhancement factor corresponding to the visible light image to generate a predicted enhancement factor. This allows the degree of enhancement of the visible light image to be determined through the fused features. Finally, based on the aforementioned predicted enhancement factor, at least one image enhancement processing is performed on the visible light image to generate an enhanced visible light image. Therefore, by decoupling the prediction of enhancement factors and the image enhancement process, the computational efficiency and scalability of the method are improved, the deployment difficulty is reduced, and the computational efficiency of image synthesis in low-light and black-light scenarios is improved, ensuring real-time processing in embedded systems or other resource-constrained environments.

[0059] Please continue to refer to Figure 3, which illustrates a flow 300 of another embodiment of the image processing method. Flow 300 of this image processing method includes the following steps:

[0060] Step 301: Obtain the visible light image and the corresponding non-visible light image in the same scene.

[0061] In some embodiments, the specific implementation of step 301 and its resulting technical effects can be referred to step 201 in the embodiment corresponding to Figure 2, and will not be repeated here.

[0062] Step 302: Perform feature extraction processing on the visible light image and the non-visible light image respectively to generate the first image feature and the second image feature.

[0063] In some embodiments, the execution entity may perform feature extraction processing on the visible light image and the non-visible light image respectively to generate a first image feature and a second image feature.

[0064] Step 303: Input the first image features and the second image features into the pre-trained feature fusion model to obtain image fusion features.

[0065] In some embodiments, the execution entity can input the first image features and the second image features into a pre-trained feature fusion model to obtain image fusion features. The feature fusion model can be a depthwise separable convolutional model. The feature fusion model can include a first convolutional layer and a second convolutional layer. The first convolutional layer can be a 3x3 depthwise convolutional layer. The second convolutional layer can be a 1x1 pointwise convolutional layer. The stride and padding parameters of the 3x3 convolutional kernel are both set to 1, ensuring that the feature map size remains unchanged during calculation, and that the final predicted enhancement factor corresponds to the input image pixels, while also ensuring the alignment of infrared image features and RGB image features.

[0066] In practice, image fusion features can be obtained through the following steps:

[0067] The first step, based on the first image features and the second image features, is to perform the following fusion steps:

[0068] In the first fusion step, the first image features and the second image features are respectively input into the first convolutional layer of the feature fusion model to obtain the first convolutional features and the second convolutional features. Here, the calculation formula for the first convolutional layer can be expressed by the following formula:

[0069] ,

[0070] in, Indicates the pixel position index. Indicates the location of the channel. This indicates the length and width of the convolution kernel. This represents the input feature map. Indicates the action on the first A convolution kernel with 1 channel.

[0071] The second fusion step involves fusing the first and second convolutional features to generate fused convolutional features. This fusion process can be pixel-level channel-dimensional feature fusion.

[0072] The third fusion step involves inputting the fused convolutional features and the second convolutional features into the second convolutional layer of the feature fusion model to obtain the third and fourth convolutional features. Here, the calculation formula for the second convolutional layer can be expressed as follows:

[0073] ,

[0074] in, Indicates the channel position index of the output result. This represents a 1x1 convolution kernel matrix.

[0075] The fourth fusion step involves fusing the third and fourth convolutional features to generate a second-fusion convolutional feature.

[0076] The fifth fusion step updates the number of times the above fusion steps have been executed.

[0077] In the sixth fusion step, in response to the above-mentioned number of executions meeting the preset execution number threshold, the above-mentioned convolutional features after secondary fusion are determined as image fusion features.

[0078] The second step is to respond to the fact that the number of executions does not meet the preset execution threshold, and to use the convolutional features after secondary fusion as the first image features and the second convolutional features as the second image features, and to execute the fusion step again.

[0079] In the process of adopting technical solutions to solve the above-mentioned technical problems, the following technical problems often arise: during the training of the feature fusion model, due to the presence of a lot of noise and color cast in the visible light image, the fused features still have noise and color cast problems, which leads to poor performance of the trained feature fusion model.

[0080] In some embodiments of this application, the feature fusion model described above can be trained using the following loss functions:

[0081] Image structure loss function for fusion:

[0082] .

[0083] in, This represents the loss function for the fused image structure. This represents the average intensity map of an enhanced visible light image. This represents the average intensity map of a visible light image. Represents an infrared image. Indicates the current pixel. This represents the four pixels that are adjacent to the current pixel in the top, bottom, left, and right directions. This indicates the number of parts into which the image is divided.

[0084] Exposure loss function:

[0085] .

[0086] in, This represents the exposure loss function. This indicates the number of parts the image is divided into; here, 4 is used. 4. Average pooling is used to downsample the image.

[0087] Color loss function:

[0088] .

[0089] in, This represents the average value of a certain channel in the enhanced image.

[0090] Smoothing loss function:

[0091] .

[0092] in, and These represent the gradients of the generated nonlinear transformation parameter matrix in the x and y directions, respectively.

[0093] The aforementioned loss functions, as an inventive point of this application, combined with the steps described below in some embodiments of this application, solve the technical problem that "during the training of the feature fusion model, due to the presence of significant noise and color cast in visible light images, the fused features still suffer from noise and color cast, resulting in poor performance of the trained feature fusion model." The reasons for the poor performance of the feature fusion model are as follows: during the training of the feature fusion model, the presence of significant noise and color cast in visible light images results in the fused features still suffering from noise and color cast, thus leading to poor performance of the trained feature fusion model. Solving these factors can improve the fusion effect of the feature fusion model. To achieve this effect, this application trains the feature fusion model by setting four loss functions. Here, the image structure loss function constrains the generated image to retain high-frequency texture information from the two input images; the exposure loss function utilizes the reflection intensity information of the infrared image, providing a smoother brightening effect for RGB images in black light scenes compared to the set average exposure value; the color loss function controls the continuity of the color channels of the enhanced image, improving color cast issues; and the smoothing loss function controls the smoothness of the enhancement factor predicted by the network, suppressing noise in the enhancement result. In conjunction with the steps described below in some embodiments of this application, the enhanced visible light image is input into a pre-trained image recognition model to obtain an image recognition result; in response to the image recognition result satisfying preset control conditions, the associated device is controlled to perform the corresponding device operation. Thus, by using the above loss functions, the presence of excessive noise and color cast in visible light images can be suppressed, thereby improving the performance of the trained feature fusion model. This allows for effective recognition of visible light images, enabling the control of associated devices to perform corresponding device operations and avoiding waste of device operating resources.

[0094] Step 304: Based on image fusion features, perform prediction processing on the enhancement factors corresponding to the visible light image to generate predicted enhancement factors.

[0095] Step 305: Based on the predicted enhancement factor, perform at least one image enhancement process on the visible light image to generate an enhanced visible light image.

[0096] In some embodiments, the specific implementation of steps 303-304 and the resulting technical effects can be referred to steps 202-203 in the embodiment corresponding to Figure 2, and will not be repeated here.

[0097] In some embodiments of this application, after step 304, the following steps are further included:

[0098] The first step is to input the enhanced visible light image into a pre-trained image recognition model to obtain the image recognition result.

[0099] In some embodiments, the aforementioned executing entity may input the enhanced visible light image into a pre-trained image recognition model to obtain an image recognition result. The image recognition model may be a pre-trained model for content recognition of visible light images of a target scene. As an example, the image recognition model may be a pre-trained large-scale security recognition model.

[0100] The second step is to control the associated devices to perform the corresponding device operations in response to the image recognition results meeting the preset control conditions.

[0101] In some embodiments, the execution entity may control an associated device to perform a corresponding device operation in response to the image recognition result satisfying a preset control condition. The preset control condition indicates that the content displayed by the visible light image satisfies the control conditions of an associated device.

[0102] As an example, in response to the image recognition result indicating a fire, the device can be a fire sprinkler, and the preset control condition can be to identify the fire and control the fire sprinkler to perform a spraying operation for fire extinguishing.

[0103] As can be seen from Figure 3, compared with the description of some embodiments corresponding to Figure 2, the flowchart 300 of the image processing method in some embodiments corresponding to Figure 3 more clearly emphasizes the specific steps of performing feature fusion processing on the aforementioned visible light image and the corresponding infrared spectral image to generate image fusion features. Therefore, the schemes described in these embodiments generate multi-level feature fusion image features, thereby combining the characteristics of infrared images having reflectance intensity information and texture information, and decoupling the two stages of enhancement factor prediction and image enhancement, improving the computational efficiency and scalability of the method, and reducing deployment difficulty.

[0104] Please continue to refer to Figure 4, which illustrates a flow 400 of another embodiment of the image processing method. Flow 400 of this image processing method includes the following steps:

[0105] Step 401: Obtain the visible light image and the corresponding non-visible light image in the same scene.

[0106] Step 402: Perform feature fusion processing on the visible light image and the non-visible light image to generate image fusion features.

[0107] In some embodiments, the specific implementation of steps 401-402 and the resulting technical effects can be referred to steps 201-202 in the embodiment corresponding to Figure 2, and will not be repeated here.

[0108] Step 403: Based on the visible light image, determine the video frame sequence corresponding to the visible light image.

[0109] In some embodiments, the executing entity may determine a video frame sequence corresponding to the visible light image based on the visible light image. The video frame sequence may be a video containing the visible light image.

[0110] Step 404: Perform keyframe extraction processing on the video frame sequence to generate a keyframe sequence.

[0111] In some embodiments, the execution entity may perform keyframe extraction processing on the video frame sequence to generate a keyframe sequence.

[0112] Step 405: Determine whether the visible light image is a keyframe.

[0113] In some embodiments, the execution entity may determine whether the visible light image is a keyframe.

[0114] Step 406: In response to determining the visible light image as a keyframe, the visible light image is input into a pre-trained nonlinear transformation parameter prediction model to obtain the predicted nonlinear transformation parameters as prediction enhancement factors.

[0115] In some embodiments, the execution entity may, in response to determining that the visible light image is a keyframe, input the visible light image into a pre-trained nonlinear transformation parameter prediction model to obtain the predicted nonlinear transformation parameters as prediction enhancement factors.

[0116] In some embodiments of this application, after step 406, the following steps are further included:

[0117] The first step is to determine the first image corresponding to the visible light image in response to the determination that the visible light image is not a keyframe.

[0118] In some embodiments, the execution entity may determine a first image corresponding to the visible light image in response to determining that the visible light image is not a keyframe. The first image may be the keyframe in the keyframe sequence that is closest to the visible light image.

[0119] The second step is to use the nonlinear transformation parameters corresponding to the first image as the prediction enhancement factor.

[0120] In some embodiments, the execution entity may use the nonlinear transformation parameters corresponding to the first image as a prediction enhancement factor.

[0121] Step 407: Based on the predicted enhancement factor, perform at least one image enhancement process on the visible light image to generate an enhanced visible light image.

[0122] In some embodiments, the specific implementation of step 407 and its resulting technical effects can be referred to step 204 in the embodiment corresponding to Figure 2, and will not be repeated here.

[0123] Please continue to refer to Figure 5. As an implementation of the methods shown in the above figures, this application provides some embodiments of an image processing apparatus. These apparatus embodiments correspond to the method embodiments shown in Figure 2. The image processing apparatus can be specifically applied to various electronic devices.

[0124] As shown in Figure 5, an image processing apparatus 500 in some embodiments includes: an acquisition unit 501, a feature fusion unit 502, a prediction unit 503, and an image enhancement unit 504. The acquisition unit 501 is configured to acquire a visible light image and a corresponding non-visible light image within the same scene; the feature fusion unit 502 is configured to perform feature fusion processing on the visible light image and the corresponding infrared spectral image to generate image fusion features; the prediction unit 503 is configured to perform prediction processing on an enhancement factor corresponding to the visible light image based on the image fusion features to generate a predicted enhancement factor; and the image enhancement unit 504 is configured to perform at least one image enhancement processing on the visible light image based on the predicted enhancement factor to generate an enhanced visible light image.

[0125] It is understood that the units described in the image processing apparatus 500 correspond to the various steps in the method described with reference to FIG2. Therefore, the operations, features, and beneficial effects described above for the method also apply to the image processing apparatus 500 and the units contained therein, and will not be repeated here.

[0126] Referring now to FIG6, a schematic diagram of the structure of an electronic device 600 (computing device 101 as shown in FIG1) suitable for implementing some embodiments of the present application is shown. The electronic device shown in FIG6 is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present application.

[0127] As shown in Figure 6, the electronic device 600 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the electronic device 600. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0128] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG. 6 shows electronic device 600 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Each box shown in FIG. 6 may represent one device, or multiple devices may be represented as needed.

[0129] According to some embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of some embodiments of this application.

[0130] It should be noted that, in some embodiments of this application, the computer-readable medium described may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0131] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0132] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire a visible light image and a corresponding non-visible light image of the same scene; perform feature fusion processing on the visible light image and the aforementioned non-visible light image to generate image fusion features; based on the aforementioned image fusion features, perform prediction processing on the enhancement factor corresponding to the aforementioned visible light image to generate a predicted enhancement factor, wherein the predicted enhancement factor includes nonlinear transformation parameters; and based on the aforementioned predicted enhancement factor, perform at least one image enhancement processing on the aforementioned visible light image to generate an enhanced visible light image.

[0133] Computer program code for performing operations of some embodiments of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0135] The units described in some embodiments of this application can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an acquisition unit, a feature fusion unit, a prediction unit, and an image enhancement unit. The names of these units do not necessarily limit the specific unit; for example, the acquisition unit may also be described as "a unit that acquires a visible light image and a corresponding non-visible light image of the same scene."

[0136] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0137] The above description is merely a selection of preferred embodiments of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this application.

Images