Image fusion method and system based on deep learning
By fusing features from infrared, visible light, and X-ray images using deep learning methods and leveraging CAM cross-attention and self-attention mechanisms, this approach solves the problem of traditional image fusion algorithms failing to retain key information, achieving high-quality image fusion results suitable for industrial inspection.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- TECHIK INSTR SHANGHAI
- Filing Date
- 2025-04-10
- Publication Date
- 2026-06-25
AI Technical Summary
Traditional image fusion algorithms cannot fully retain the key information of the input image, resulting in blurred details and poor fusion effect. Furthermore, they lack automation and adaptability, and cannot meet the detection needs in complex industrial environments.
A deep learning-based approach is adopted, which uses the CAM cross-attention mechanism and decoder to fuse feature maps of infrared, visible light and X-ray images. The encoder extracts features and uses self-attention and coordinate attention mechanisms to enhance feature interaction. The loss function is combined to optimize model parameters and generate high-quality fused images.
It improves the image fusion effect, retains key information of the original image, enhances the detail and accuracy of the fused image, and is more adaptable to detection in complex industrial environments.
Smart Images
Figure CN2025088226_25062026_PF_FP_ABST
Abstract
Description
A Deep Learning-Based Image Fusion Method and System Technical Field
[0001] This invention relates to the field of image processing technology. More specifically, this invention relates to an image fusion method and system based on deep learning. Background Technology
[0002] In modern industrial inspection, ensuring product quality and safety is of paramount importance. Traditionally, X-ray, visible light, and infrared imaging are widely used for defect detection. X-ray images provide detailed views of internal structures, identifying internal defects such as cracks, bubbles, and voids; visible light images reveal the surface condition of a product, including features like scratches, dirt, and uneven coloring; while infrared images effectively detect heat distribution and material properties, identifying problems such as overheating and insulation defects. Although each of these three imaging methods can provide its own information, significant limitations exist when used individually. For example, X-ray images cannot accurately display surface features, while visible light and infrared images cannot reveal internal structures. In some cases, internal defects may be visible in X-ray images but appear normal in visible light and infrared images, and vice versa. Therefore, using only one of these three imaging methods to inspect product quality may miss important defect information, leading to misjudgments of the overall condition and inaccurate inspection results. Furthermore, analyzing three different images separately when inspecting product quality requires more time and resources, especially when processing high-resolution images, which may reduce defect detection efficiency.
[0003] Therefore, product quality inspection typically requires the use of fusion algorithms to fuse three images. Traditional image fusion algorithms, such as weighted averaging and wavelet transform, while achieving image fusion to some extent, have significant shortcomings. These methods often fail to fully retain all key information from the input images, resulting in blurred details and poor fusion effects. Furthermore, traditional algorithms usually rely on manually designed features, lacking automation and adaptability, and cannot meet the inspection needs of complex industrial environments. For example, Chinese invention patent application CN111861957B discloses an image fusion method and apparatus that uses wavelet transform to fuse multiple images, but the fused image fails to fully retain all key information from the input images, resulting in blurred details and poor fusion effects. Summary of the Invention
[0004] To address the technical problem that existing image fusion methods cannot fully retain all the key information in the input image after fusion, resulting in blurred details and poor fusion effects, this invention provides solutions in the following aspects.
[0005] In a first aspect, the present invention provides a deep learning-based image fusion method, comprising: inputting an infrared image into a first encoder to obtain an infrared feature map; inputting a visible light image into a second encoder to obtain a visible light feature map; inputting an X-ray image into a third encoder to obtain an X-ray feature map; performing preliminary fusion of the infrared feature map, the visible light feature map, and the X-ray feature map using a CAM cross-attention mechanism to obtain an initial fused feature map; inputting the visible light feature map, the infrared feature map, the X-ray feature map, and the initial fused feature map into a decoder to generate a fused image; the decoder is used to fuse the visible light feature map, the infrared feature map, the X-ray feature map, and the initial fused feature map to obtain a final fused feature map, and classifying the pixels of the final fused feature map to generate a fused image; the training method of the CAM cross-attention mechanism and the decoder includes: after the first encoder, the second encoder, and the third encoder have all been trained, using the first encoder, the second encoder, the third encoder, the CAM cross-attention mechanism, and the decoder to obtain the fused image; The loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder is calculated using a second loss function; the gradient of the second loss function with respect to the model parameters is calculated using a backpropagation algorithm; the parameters of the CAM cross-attention mechanism and the decoder are updated using an optimization algorithm; the parameters of the CAM cross-attention mechanism and the decoder are iteratively updated until a preset number of training rounds is reached.
[0006] Preferably, the first encoder includes: a convolutional layer, a pooling layer, and four convolutional blocks, wherein the first convolutional layer is used to extract shallow features of the image, the output of the first convolutional layer is connected to the input of the pooling layer, the output of the pooling layer is connected to the input of the first convolutional block, the output of the first convolutional block is connected to the input of the second convolutional block, the output of the second convolutional block is connected to the input of the third convolutional block, and the output of the third convolutional block is connected to the input of the fourth convolutional block; the first encoder, the second encoder, and the third encoder have the same structure.
[0007] Preferably, the convolution block consists of three convolutions, and the three convolutions are connected by residuals.
[0008] Preferably, the preliminary fusion of the infrared feature map, the visible light feature map, and the X-ray feature map includes: acquiring the features to be fused from the infrared feature map, the visible light feature map, and the X-ray feature map, respectively, and adding the above three features to obtain an initial fused feature map; acquiring the features to be fused from a feature map of a mode includes: inputting the feature map of the mode into a first self-attention mechanism to obtain a first feature map; performing horizontal and vertical movement operations on the feature positions of the first feature map, and inputting the first feature map after movement into a second self-attention mechanism to obtain a second feature map; resetting the feature positions of the second feature map and inputting the reset second feature map into a coordinate attention mechanism to obtain the features to be fused from the feature map of that mode.
[0009] Preferably, the decoder is a four-layer convolutional structure, and an upsampling layer is set between adjacent convolutional layers. The decoder is connected to the first encoder, the second encoder and the third encoder in a skip connection.
[0010] Preferably, the first encoder is a neural network model, and its training method includes: inputting an infrared image into the first encoder to obtain an infrared feature map; calculating the difference between the infrared feature map and the infrared image input into the first encoder using a first loss function; calculating the gradient of the loss function with respect to the model parameters using a backpropagation algorithm; updating the parameters of the first encoder using an optimization algorithm; and iteratively updating the parameters of the first encoder until a preset number of training rounds is reached.
[0011] Preferably, the expression for the first loss function is: in, I represents the difference between the infrared feature map and the infrared image input to the first encoder. c This indicates that the infrared image input from the first encoder is being processed. The infrared feature map is represented. It is the Frobenius norm, used to measure the difference between the input image and the reconstructed image at the pixel level. It is a structural similarity index, w s It is a balance parameter.
[0012] Preferably, the expression for the second loss function is: L cam =L int +w g L gra ; In the formula, L cam Indicates the first loss, w g It is a tradeoff parameter used to balance the weights of the two loss terms, L.int L represents the strength loss. gra Let F represent the gradient loss, and M represent the fused image. ir M vi and M xi These are intensity masks for infrared images, visible light images, and X-ray images, respectively; I ir I vi and I xi These are the infrared image, the visible light image, and the X-ray image, respectively. It is the Frobenius norm, used to measure the difference between the input image and the reconstructed image at the pixel level.
[0013] In a second aspect, the present invention provides a deep learning-based image fusion system, comprising: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the deep learning-based image fusion method of the present invention is implemented.
[0014] In summary, the beneficial effects of the present invention are that the method of the present invention can effectively improve the fusion effect of visible light images, infrared images and X-ray images. Attached Figure Description
[0015] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of the invention are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:
[0016] Figure 1 is a schematic flowchart illustrating a deep learning-based image fusion method according to an embodiment of the present invention;
[0017] Figure 2 is a schematic flowchart illustrating a deep learning-based image fusion method according to an embodiment of the present invention.
[0018] Figure 3 schematically illustrates a deep learning-based image fusion system according to an embodiment of the present invention. Detailed Implementation
[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0020] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.
[0021] Example of a deep learning-based image fusion method:
[0022] As shown in Figures 1 and 2, the deep learning-based image fusion method of the present invention includes:
[0023] S101. Obtaining infrared feature map, visible light feature map and X-ray feature map, specifically: inputting the infrared image into the first encoder to obtain the infrared feature map; inputting the visible light image into the second encoder to obtain the visible light feature map; inputting the X-ray image into the third encoder to obtain the X-ray feature map;
[0024] The first, second, and third encoders are used to extract features from infrared, visible light, and X-ray images, respectively, to generate corresponding feature maps. Extracted features may include edges, textures, and target contours. Before inputting the infrared, visible light, and X-ray images into their respective encoders, preprocessing operations such as denoising, contrast enhancement, and normalization are performed to improve image quality and consistency. The images are also adjusted to the same size and resolution to facilitate subsequent processing and fusion.
[0025] S102. Obtain the initial fused feature map, specifically by using the CAM cross-attention mechanism to initially fuse the infrared feature map, the visible light feature map, and the X-ray feature map to obtain the initial fused feature map. The cross-attention mechanism is a mechanism that establishes a correlation and calculates attention weights between two different input sequences. It allows the model to pay attention to elements in another input sequence while processing elements of one input sequence, thereby capturing the dependency between the two sequences. In image fusion, the cross-attention mechanism allows the model to focus on features of another modality while processing features of one modality, thus achieving information interaction between different modalities. In image feature fusion tasks, the CAM cross-attention mechanism can simultaneously process image data in different representations (e.g., infrared images, visible light images, and X-ray images) and capture complementary information between these images. Through information interaction between images of different modalities, the model can generate more comprehensive and accurate fused features.
[0026] S103. Generating the fused image specifically involves: inputting the visible light feature map, the infrared feature map, the X-ray feature map, and the initial fused feature map into a decoder to generate the fused image; the decoder is used to fuse the visible light feature map, the infrared feature map, the X-ray feature map, and the initial fused feature map to obtain the final fused feature map, and to classify the pixels of the final fused feature map to generate the fused image.
[0027] Skip connections can be set between the decoder and the three encoders to fuse the visible light feature map, the infrared feature map, the X-ray feature map, and the initial fusion feature map.
[0028] Traditional image fusion algorithms, such as weighted averaging and wavelet transform, while achieving image fusion to some extent, have significant shortcomings. These methods often fail to fully preserve all key information from the input image, resulting in blurred details and poor fusion quality. The deep learning-based image fusion method of this invention utilizes three encoders to extract features from infrared, visible light, and X-ray images, respectively, and then employs a CAM cross-attention mechanism to fuse these features. Since some original feature information is lost during fusion, this invention, after obtaining the initial fusion feature map, further fuses the initial fusion feature map, the original visible light image, the original infrared image, and the original X-ray image together to generate the final fused image. This process preserves as much of the original feature information (including deep / shallow features) of the visible light, infrared, and X-ray images as possible, significantly improving the quality of the generated fused image. In summary, the method of this invention can effectively improve the fusion effect of visible light, infrared, and X-ray images.
[0029] In one embodiment, the first encoder includes a convolutional layer, a pooling layer, and four convolutional blocks. The first convolutional layer is used to extract shallow features of the image. The output of the first convolutional layer is connected to the input of the pooling layer. The output of the pooling layer is connected to the input of the first convolutional block. The output of the first convolutional block is connected to the input of the second convolutional block. The output of the second convolutional block is connected to the input of the third convolutional block. The output of the third convolutional block is connected to the input of the fourth convolutional block. The first encoder, the second encoder, and the third encoder have the same structure.
[0030] After the infrared image is input into the first encoder, it first passes through the first convolutional layer to obtain shallow features of the infrared image. These shallow features are then pooled by a pooling layer to obtain pooled shallow feature information. The pooled shallow feature information is then passed through four convolutional blocks, thereby retaining more useful information from the multi-scale features. As the encoder layers become deeper, the deep features begin to concentrate on salient targets. Therefore, by using the first encoder of this invention, more useful information can be extracted from the infrared image, thereby further improving the fusion effect of visible light images, infrared images, and X-ray images.
[0031] In one embodiment, the convolutional block consists of three convolutions connected by residuals.
[0032] In one embodiment, the preliminary fusion of the infrared feature map, the visible light feature map, and the X-ray feature map includes: acquiring the features to be fused from the infrared feature map, the visible light feature map, and the X-ray feature map, respectively, and adding the above three features to obtain an initial fused feature map; acquiring the features to be fused from a feature map of a certain modality includes:
[0033] S201. Input the feature map of the modality into the first self-attention mechanism to obtain the first feature map; perform horizontal and vertical movement operations on the first feature map, and input the first feature map after the movement into the second self-attention mechanism to obtain the second feature map;
[0034] Self-attention, also known as intra-attention, is an attention mechanism that associates different positions within a single sequence to compute a representation of the same sequence. In self-attention, the model automatically learns the relationships between different positions in a sequence and generates a weighted vector for each position as its representation in the current context.
[0035] The self-attention mechanism generates different weights for each feature map location; these weights are dynamically calculated based on the current content of the feature map. This allows the model to adaptively focus on important feature parts, thereby improving the model's representational power. Furthermore, the self-attention mechanism achieves global information interaction by calculating the correlation between any two locations in the feature map. This helps the model acquire more contextual information, thus improving the accuracy of image understanding. Feeding the feature map of this modality into the first self-attention mechanism can enhance its internal features.
[0036] Feature position movement can change the spatial layout of elements in a feature map, thereby promoting the interaction and fusion between different features. This interaction and fusion helps the model capture more complex and abstract feature patterns. Therefore, after obtaining the first feature map, the feature positions are moved horizontally and vertically, and then input into the second self-attention mechanism. The second self-attention mechanism can further capture the global dependencies between features based on these moved feature maps, thereby further refining and combining features to form a more advanced and accurate feature map.
[0037] S202, Reset the feature positions of the second feature map and input the reset second feature map into the coordinate attention mechanism to obtain the feature map to be fused for this modality.
[0038] The coordinate attention mechanism can be used to introduce spatial coordinate information to enhance the expressive power of feature maps. The feature maps processed by the coordinate attention mechanism contain channel information and precise spatial location information, providing richer and more accurate feature representations for subsequent image understanding tasks.
[0039] In this embodiment, when acquiring the features to be fused in the feature map, the method first enhances the internal features of the feature map through a first self-attention mechanism. Then, after shifting the feature map with enhanced internal features, it inputs it into a second self-attention mechanism to obtain a more advanced and accurate feature map. Furthermore, it uses a coordinate attention mechanism to further process the feature map, introducing spatial coordinate information into the feature map to enhance its expressive power. This improves the effect and quality of the acquired initial fused feature map.
[0040] In one embodiment, the decoder is a four-layer convolutional structure, with an upsampling layer between adjacent convolutional layers, and the decoder is connected to the first encoder, the second encoder, and the third encoder in a skip connection.
[0041] In one embodiment, the first encoder is a neural network model, and its training method includes:
[0042] S301. Input the infrared image into the first encoder to obtain the infrared feature map;
[0043] Before training, a sufficient number of X-ray, infrared, and visible light images need to be acquired and labeled to form a dataset. It is essential to ensure that these images were captured of the same object within the same scene or target.
[0044] S302. Calculate the difference between the infrared feature map and the infrared image input to the first encoder using the first loss function;
[0045] S303. Use the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters;
[0046] S304. Use an optimization algorithm to update the parameters of the first encoder;
[0047] S305. Iteratively update the parameters of the first encoder until the preset number of training rounds is reached.
[0048] In one embodiment, the expression for the first loss function is:
[0049]
[0050] in, I represents the difference between the infrared feature map and the infrared image input to the first encoder. cThis indicates the infrared image input from the first encoder. Infrared feature map It is the Frobenius norm, used to measure the difference between the input image and the reconstructed image at the pixel level. It is a structural similarity index, w s It is a balance parameter.
[0051] In existing technologies, squared loss is typically used when calculating the difference between two images. The first loss function in this embodiment takes into account both pixel-level loss and the structural similarity between the two images when calculating the difference between the infrared feature map output by the first encoder and the infrared image input to the first encoder. Considering pixel-level loss can avoid the problem of large global error. Considering the structural similarity between the two images rather than just the difference in pixel values, the structural similarity index can better reflect the perceptual characteristics of the human eye compared to squared loss. Using the first loss function of this embodiment to train the encoder can effectively improve the quality of the infrared feature map output by the first encoder.
[0052] In one embodiment, the training method for the CAM cross-attention mechanism and the decoder includes:
[0053] S401. Input the infrared image, visible light image and X-ray image into the trained first encoder, trained second encoder and trained third encoder respectively to obtain infrared feature map, visible light feature map and X-ray feature map;
[0054] S402. Input the infrared feature map, visible light feature map and X-ray feature map into the CAM cross-attention mechanism to obtain the initial fusion feature map;
[0055] S403. Input the visible light feature map, the infrared feature map, the X-ray feature map and the initial fusion feature map into the decoder to generate the fused image;
[0056] S404. Calculate the first loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder using the second loss function.
[0057] S405. Use the backpropagation algorithm to calculate the gradient of the second loss function with respect to the model parameters;
[0058] S406. Use an optimization algorithm to update the parameters of the CAM cross-attention mechanism and the decoder;
[0059] S407. Iteratively update the parameters of the CAM cross-attention mechanism and the decoder until the preset number of training rounds is reached.
[0060] In one embodiment, the expression for the second loss function is:
[0061] L cam =L int +w g L gra ;
[0062]
[0063] In the formula, L cam Indicates the first loss, w g It is a tradeoff parameter used to balance the weights of the two loss terms, L. int L represents the strength loss. gra Let F represent the gradient loss, and M represent the fused image. ir M vi and M xi These are intensity masks for infrared images, visible light images, and X-ray images, respectively; I ir I vi and I xi These are infrared images, visible light images, and X-ray images, respectively. It is the Frobenius norm, used to measure the difference between the input image and the reconstructed image at the pixel level; L cam This represents the loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder.
[0064] Intensity loss is used to preserve key components of the fused image, such as brightness and contours. This information typically does not appear in only a single modality. Therefore, an intensity mask is introduced when calculating the intensity loss to ensure that critical information from different modalities is preserved.
[0065] By introducing intensity loss and gradient loss into the second loss function, not only are the unique features from each modality preserved, but redundant information is also reduced, improving the quality of the fused image. Introducing an intensity mask optimizes the fusion process, reduces redundancy, and enhances the expressiveness of the fused image. Introducing gradient loss ensures that the fused image appears more visually natural and better preserves the contours and details of the target.
[0066] An embodiment of a deep learning-based image fusion system: This invention also provides a deep learning-based image fusion system. As shown in Figure 3, the deep learning-based image fusion system includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement the deep learning-based image fusion method described in the above embodiment.
[0067] The deep learning-based image fusion system also includes other components well known to those skilled in the art, such as communication buses and communication interfaces. Their settings and functions are known in the art and will not be described in detail here.
[0068] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device. Any application or module described in this invention can be implemented using computer-readable / executable instructions that can be stored or otherwise maintained by such a computer-readable medium.
[0069] While this specification has shown and described numerous embodiments of the invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and essence of the invention. It should be understood that various alternatives to the embodiments of the invention described herein may be employed in the practice of this invention.
Claims
1. A deep learning-based image fusion method, characterized in that, include: The infrared image is input into the first encoder to obtain the infrared feature map; The visible light image is input into the second encoder to obtain the visible light feature map; The X-ray image is input into the third encoder to obtain the X-ray feature map; The infrared feature map, the visible light feature map, and the X-ray feature map are initially fused using the CAM cross-attention mechanism to obtain an initial fused feature map; The visible light feature map, the infrared feature map, the X-ray feature map, and the initial fused feature map are input into the decoder to generate the fused image; The decoder is used to fuse the visible light feature map, the infrared feature map, the X-ray feature map and the initial fusion feature map to obtain the final fusion feature map, and to classify the pixels of the final fusion feature map to generate the fused image; The training method for the CAM cross-attention mechanism and the decoder includes: after the first encoder, the second encoder, and the third encoder have all been trained, the fused image is obtained using the first encoder, the second encoder, the third encoder, the CAM cross-attention mechanism, and the decoder; The first loss of the fused image relative to the infrared image input to the first encoder, the visible light feature map input to the second encoder, and the X-ray image input to the third encoder is calculated using the second loss function; the gradient of the second loss function with respect to the model parameters is calculated using the backpropagation algorithm. The parameters of the CAM cross-attention mechanism and the decoder are updated using an optimization algorithm; The parameters of the CAM cross-attention mechanism and the decoder are iteratively updated until a preset number of training rounds are reached.
2. The image fusion method based on deep learning as described in claim 1, characterized in that, The first encoder includes a convolutional layer, a pooling layer, and four convolutional blocks. The first convolutional layer is used to extract shallow features of the image. The output of the first convolutional layer is connected to the input of the pooling layer. The output of the pooling layer is connected to the input of the first convolutional block. The output of the first convolutional block is connected to the input of the second convolutional block. The output of the second convolutional block is connected to the input of the third convolutional block. The output of the third convolutional block is connected to the input of the fourth convolutional block. The first encoder, the second encoder, and the third encoder have the same structure.
3. The image fusion method based on deep learning as described in claim 2, characterized in that, The convolutional block consists of three convolutions connected by residuals.
4. The image fusion method based on deep learning as described in claim 1, characterized in that, The preliminary fusion of the infrared feature map, the visible light feature map, and the X-ray feature map includes: acquiring the features to be fused from the infrared feature map, the visible light feature map, and the X-ray feature map, respectively, and adding the above three features to obtain an initial fused feature map; acquiring the features to be fused from a feature map of a mode includes: inputting the feature map of the mode into a first self-attention mechanism to obtain a first feature map; performing horizontal and vertical movement operations on the first feature map, and inputting the first feature map after the movement into a second self-attention mechanism to obtain a second feature map; The feature positions of the second feature map are reset, and the reset second feature map is input into the coordinate attention mechanism to obtain the feature map to be fused for this modality.
5. The image fusion method based on deep learning as described in claim 1, characterized in that, The decoder is a four-layer convolutional structure, with an upsampling layer between adjacent convolutional layers. The decoder is connected to the first encoder, the second encoder, and the third encoder in a skip connection.
6. The image fusion method based on deep learning as described in claim 1, characterized in that, The first encoder is a neural network model, and its training method includes: The infrared image is input into the first encoder to obtain an infrared feature map; The difference between the infrared feature map and the infrared image input to the first encoder is calculated using a first loss function; The backpropagation algorithm is used to calculate the gradient of the loss function with respect to the model parameters; Use an optimization algorithm to update the parameters of the first encoder; The parameters of the first encoder are updated iteratively until a preset number of training rounds is reached.
7. The image fusion method based on deep learning as described in claim 6, characterized in that, The expression for the first loss function is: in, I represents the difference between the infrared feature map and the infrared image input to the first encoder. c This indicates that the infrared image input from the first encoder is being processed. The infrared feature map is represented. It is the Frobenius norm, used to measure the difference between the input image and the reconstructed image at the pixel level. It is a structural similarity index, w s It is a balance parameter.
8. The image fusion method based on deep learning as described in claim 1, characterized in that, The expression for the second loss function is: L cam =L int +w g L gra ; In the formula, L cam Indicates the first loss, w g It is a tradeoff parameter used to balance the weights of the two loss terms, L. int L represents strength loss. gra Let F represent the gradient loss and M represent the fused image. ir M vi and M xi These are intensity masks for infrared images, visible light images, and X-ray images, respectively. I ir I vi and I xi These are the infrared image, the visible light image, and the X-ray image, respectively. It is the Frobenius norm, used to measure the difference between the input image and the reconstructed image at the pixel level.
9. A deep learning-based image fusion system, comprising: A processor and a memory, the memory storing computer program instructions, characterized in that, when the computer program instructions are executed by the processor, the image fusion method based on deep learning as described in any one of claims 1 to 8 is implemented.