264 results about "Multimodality image fusion" patented technology

The invention relates to the technical field of road crack detection, in particular to a road crack detection method and system based on a fused image. The method comprises the following steps: acquiring road multi-source monitoring data including a visible light image, infrared thermal imaging data and laser radar point cloud data, and performing multi-modal image fusion and road three-dimensional point cloud reconstruction to generate a fused road image and road three-dimensional modeling data; performing crack curvature analysis based on the fused road image to generate crack curvature data; performing reflection crack contour recognition and positioning on the fused road image through the crack curvature data to generate reflection crack initial positioning data; obtaining road base material data; and performing reflection crack stress field reconstruction on the road area according to the reflection crack initial positioning data to obtain a reflection crack stress field. According to the invention, through multi-modal fusion, curvature identification, stress field modeling and crack channel analysis, the accuracy and strain of road reflection crack detection are improved.

The invention discloses an end-side cloud collaborative intelligent fire-fighting fire monitoring system based on image fusion, and relates to the technical field of intelligent fire-fighting, the system is composed of a plurality of functional modules, and the system comprises a multi-modal image fusion module which generates a dynamic scanning priority map based on prior data, distinguishes a natural heat source from an abnormal fire by using a dual-light fusion algorithm, and sends an image fusion result to a cloud server; a scanning area is divided according to the thermal risk grade, and the thermal imaging resolution is dynamically adjusted; the distributed edge computing module is used for carrying out space-time synchronization on cross-modal data through a multi-modal feature alignment network, and carrying out dynamic allocation on a CUDA core and CPU resources through adaptive computing scheduling; an improved artificial bee colony algorithm is adopted, the bandwidth of the multi-sensor data flow is dynamically allocated through a time-sharing multiplexing protocol, and three-dimensional path planning is carried out; and the end-side cloud collaborative decision module constructs a federated learning driven model sharing network, and each edge node trains a lightweight YOLOv5s pruning model based on local data.

The invention belongs to the technical field of image processing, and discloses a multi-modal image fusion method and model based on global and local text perception, which are used for respectively processing global and local information of images by combining two visual language models, namely CLIP and BLIP, and realizing effective fusion of the images under complex scenes and severe weather conditions. The global text sensing module utilizes CLIP features to enhance the understanding of the model for the whole scene, and the local text sensing module utilizes BLIP features to improve the processing capability for local details. According to the double text perception mechanism, the model can more comprehensively utilize the advantages of the visual language model, and the problem that only simple text prompt is relied on or local details are excessively emphasized is avoided. The advantages of a visual language model can be fully utilized, global and local information processing is considered, and the adaptability and generalization performance of the model under complex scenes and severe weather conditions are improved.

The invention relates to the technical field of visual positioning, and provides a visual positioning method and system for precise connector assembly. Performing multi-mode HDR fusion and distortion correction on the original connector image to obtain a multi-layer fusion image matrix; and performing feature region coarse positioning in combination with a composite convolutional neural network to obtain a connector coordinate set, performing geometric feature fine positioning according to the connector coordinate set and the multilayer fusion image matrix to obtain a key point coordinate set, and performing hand-eye calibration and dynamic compensation on the key point coordinate set to generate a pose instruction. And performing vision-force control hybrid assembly processing on the pose instruction according to the sensor feedback data to obtain assembly completion state data. Through multi-modal image fusion, deep learning coarse positioning, precise geometric registration and dynamic cooperation of vision and force control, the speed, precision and stability of precise connector assembly are improved.

An image-guided diffusion network has two Convolution Neural Networks (CNNs). A RGB image and an IR image are concatenated with a Gaussian noise image and input to a denoising neural network that merges information from the RGB and IR images as noise is removed over many iterations. Then an enhancement neural network up-samples for Super Resolution (SR) and convolutes to generate a condition vector that controls Global Feature Modulation (GFM) at three convolution layers to generate a SRGFM enhanced fusion image. Timesteps are embedded using adaptive group normalization blocks within Adaptive Bottleneck Residual (ABR) blocks in the denoising network, which is a UNet having many levels of ABRs, and in the enhancement network before feature modulation. Global image features are detected by triple convoluting the image input to the enhancement network to generate the condition vector that controls feature modulation blocks at three layers of convolution.

The invention provides a multi-modal image fusion method based on modal self-adaption and modal interaction compensation, and the method comprises the following steps: S1, obtaining a multi-modal image fusion data set, and obtaining a training data set through preprocessing; S2, analyzing the modal difference characteristics of infrared and visible light images, and evaluating the correlation characteristics of image pairs in different scenes; s3, capturing a cross-modal feature dependency relationship through a self-attention mechanism; s4, a differential feature extraction strategy is adopted, model parameters are optimized through iterative training, and multi-modal image fusion is completed; s5, a modal interaction compensation module is additionally arranged, unit dynamic balance common features and modal exclusive features are fused, feature complementation is achieved in channel and space dimensions, parameters of the modal interaction compensation module are optimized, the model is made to learn the optimal fusion weight of the multi-modal features in a self-adaptive mode, and multi-modal fusion image generation optimization is achieved through the model; according to the invention, multi-modal image fusion can be accurately and effectively carried out.

The invention discloses a medical image recognition processing system and method based on multi-modal image fusion, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a medical image, and correcting the medical image to obtain an image data set; inputting the image data set into a parallel neural network, extracting to obtain tracheal tissue structure features and biological metabolism features, and fusing to generate a comprehensive image feature map; and comparing the fusion features of different organs in the comprehensive image feature map with features in a preset typical case feature library to generate risk assessment values of corresponding organs, and associating the risk assessment values with corresponding organs in a preset AR visual image. According to the method, the trachea tissue structure features and the biological metabolism features are extracted through the parallel neural network and fused to generate the comprehensive image feature map, then the comprehensive image feature map is compared with the typical case feature library to generate the risk assessment value, finally the risk assessment value is associated with the AR visual image, and effective fusion and utilization of multi-modal image data are achieved.

The invention provides a field medical sharp instrument recovery management method and system based on a module anti-permeation structure, and relates to the technical field of image analysis, and the method comprises the steps: recognizing a field medical sharp instrument through multi-modal image fusion and a deep learning model, extracting key features, and carrying out the precise positioning, segmentation and semantic analysis, classifying by combining a causal reasoning network to obtain category and danger level information, and generating and optimizing a recovery scheduling strategy by utilizing an improved space-time diagram neural network and a hierarchical reinforcement learning algorithm and combining an improved auction mechanism and a security constraint model based on the classification information and the danger level information. An optimal recovery route is planned through an improved ant colony algorithm and a heuristic search strategy in combination with a Pareto optimization algorithm.

The invention discloses a power transmission line foreign matter detection method and system based on multi-modal image fusion, and relates to the technical field of intelligent operation and maintenance and state monitoring of a power system, a lightweight Ev-Mama architecture is introduced into a backbone network part of YOLOv13, the model keeps relatively low calculation complexity, and meanwhile, the power transmission line foreign matter detection efficiency is improved. And the modeling capability of the method on the long-range dependency relationship and the global semantic information is obviously enhanced. Besides, by using the CDIDF module, the EVCS module and the MHSAA module, on the basis of increasing a small amount of calculation, the scale sensing ability, the space structure modeling ability and the context understanding ability of the model are effectively improved, and the performance bottleneck of a traditional YOLO series network in the aspects of processing small targets, shielding targets and cross-scale information fusion is effectively relieved.

The invention relates to the technical field of electrical equipment defect monitoring, in particular to a high-voltage equipment defect positioning system and method based on multi-modal image fusion, and the system comprises an ultraviolet triggering module, a modal synchronization module, a space positioning module, a structure recognition module and a fusion diagnosis module. According to the method, automatic calibration of a suspected defect area can be realized by performing quantitative threshold judgment on a corona discharge signal in an ultraviolet image, the consistency of three-mode data in space and time dimensions is ensured, and the accuracy of the defect area is improved by comparing three types of coordinates of a lead hot spot, boundary geometry and a spot center in the image and performing registration offset correction. The image feature alignment precision is improved, the insulator chain light spot distribution path is extracted, the continuous distribution length is calculated and corresponds to the pollution level standard, automatic identification of the pollution level is achieved, a multi-modal image fusion feature block is constructed, and a risk sorting label is given. And the defect positioning accuracy and the comprehensive evaluation capability on the defect property and the influence degree are effectively improved.

The invention discloses a multi-modal image fusion and identification method, which belongs to the technical field of image fusion, and obviously enhances the ability of a model to capture a complex dependency relationship between modals by introducing a multi-head attention mechanism. In a multi-modal image fusion and identification task, image data of different modals often contain complementary information, such as rich details of a visible light image, thermal sensitivity of an infrared image and penetrating power of a radar image. The multi-head attention mechanism can pay attention to the features of multiple modes at the same time, and the internal relation between the features of the different modes is learned through the self-attention mechanism. By means of the design, the model can fuse multi-modal information more flexibly, and the richness and accuracy of feature representation are effectively improved. For example, in a complex scene, the model can assist in target recognition in the visible light image by means of the heat sensitive information of the infrared image, so that the recognition accuracy and robustness are improved.

The invention discloses a road apparent disease real-time identification method based on visible light and infrared image multi-modal fusion, and belongs to the technical field of road apparent disease real-time detection based on multi-modal image fusion. In order to solve the problems of difficult road disease detection and low image depth fusion efficiency in a complex scene, the method comprises the following steps: carrying out feature decomposition extraction on a visible light image to obtain a corresponding base layer and a detail layer, and carrying out feature decomposition extraction on an infrared image to obtain a corresponding base layer and a detail layer; obtaining fusion information FB based on the base layer of the visible light image and the base layer of the infrared image, and obtaining fusion information FD based on the detail layer of the visible light image and the detail layer of the infrared image; and a fusion image F is obtained based on the FB and the FD, and real-time identification of road apparent diseases is realized by adopting a neural network model.

The invention discloses a poultry behavior abnormity real-time monitoring system based on multi-modal image fusion, particularly relates to the technical field of intelligent breeding behavior recognition, and is used for solving the problem of poor behavior monitoring accuracy under feather shielding. The method comprises the following steps: firstly, through combined perception of a visible light image and an infrared image, extracting a claw track interruption point and an anus temperature gradient direction, and realizing analysis of a motion state of a sheltered area; then, in combination with the heat conduction delay characteristic and the group movement direction, the flexion and extension angle of the covered leg joint is inverted, and a complete gait sequence is generated; thirdly, multi-source features such as gaits, temperature differences and body postures are fused, and a dynamic deviation model of the individuals relative to the mass center of the group is constructed; and finally, generating a stress behavior threshold curve according to the ground temperature and the ammonia gas concentration, outputting an abnormal behavior type and confidence, and realizing intelligent distinguishing of mechanical obstacles and adaptive behaviors.

The invention provides a fluorescence-magnetic particle image fusion method which can be applied to the technical field of medical image processing. The method comprises the following steps: performing two-dimensional convolution processing on a registered two-dimensional near-infrared fluorescence image to obtain a multi-channel near-infrared fluorescence extended feature map, and performing initial fusion on the multi-channel near-infrared fluorescence extended feature map and a registered three-dimensional magnetic particle cross-sectional image to obtain a three-dimensional near-infrared fluorescence extended feature map; performing multi-round three-dimensional convolution processing on the three-dimensional near-infrared fluorescence extended feature map to obtain a multi-scale near-infrared fluorescence feature map, and performing multi-round three-dimensional convolution processing on the registered three-dimensional magnetic particle cross-sectional image to obtain a multi-scale magnetic particle feature map; and based on an adaptive cross attention mechanism, performing multi-round feature map fusion on the multi-scale near-infrared fluorescence feature map and the multi-scale magnetic particle feature map on the same scale, and performing filtering convolution processing on a multi-round feature map fusion result to obtain a three-dimensional fluorescence-magnetic particle fusion image.

The invention belongs to the field of agricultural automation and intelligent equipment, and discloses a portable measuring method and device for pork intramuscular fat content. By screening the characteristic wave bands related to the fat content and selecting the corresponding narrow-band LEDs, the accuracy and effectiveness of the multispectral image are improved. And meanwhile, through multi-modal image fusion analysis, in combination with the RGB image, the black-and-white image and the multi-spectral image, the precision of target segmentation and content prediction is improved. And establishing an end-to-end content prediction model by adopting a mode of combining a CNN and an MLP network. Meanwhile, the FCNN network is used for processing the spectral data, the tedious spectral data preprocessing process is omitted, the advantages of the spectral data and the RGB image are fully played, and a new thought and method are provided for the field of spectral modeling. The technical scheme has the advantages of portability, high efficiency, accuracy and the like, provides powerful support for pork nutrition evaluation, and can provide reference for determination of the fat content of other livestock and poultry meat.

The invention discloses a photovoltaic module surface defect online detection method based on multi-modal image fusion, and relates to the technical field of photovoltaic module detection.The method comprises the steps that a visible light line-scan digital camera, a short-wave infrared camera, an EL line-scan digital camera and a laser speckle imaging module are installed to collect real-time data, an optical image is subjected to dynamic window median filtering, and the surface defect of the photovoltaic module is obtained; the method comprises the following steps: carrying out time moving average combined dynamic threshold segmentation on an infrared image and an EL image, carrying out image-guided outlier filtering on 3D point cloud data, judging a pixel defect type based on gray-temperature correlation and calculating a fusion weight, respectively inputting the optical image, the infrared image, the EL image and 3D point cloud into GhostNet, lightweight ViT, CNN and PointNet-Lite to extract features, and carrying out image fusion on the 3D point cloud data. And after global average pooling and splicing, inputting into a channel attention module to generate a modal weight, and after weighted fusion, obtaining a defect type through a full connection layer. According to the invention, the accuracy of defect detection is effectively improved.

The invention particularly relates to a ship multi-modal image fusion recognition method based on graph neural structure alignment, and the method comprises the following steps: 1, obtaining a ship multi-modal image, and constructing a backbone neural network to extract the features of the multi-modal image; 2, constructing graph nodes of the graph neural network, and generating an adjacent edge relationship; 3, for graph structures constructed in different modes, adopting a two-level graph attention mechanism to complete structure alignment; step 4, utilizing an optimal transmission mechanism to realize structure alignment between the infrared and visible light modal diagrams; step 5, feature re-injection is carried out to fuse space coordinates and global information, and the positioning and expression ability of node features is improved; and step 6, training the constructed ship multi-modal image fusion recognition network by adopting local feature alignment loss, graph-level semantic consistency loss and classification supervision loss. According to the method, the problem of alignment errors caused by inconsistency of infrared and visible light modal images is solved, the structure and semantic information of the infrared and optical images are fully fused, the accuracy and robustness of cross-modal target recognition are effectively improved, and the method is suitable for complex scenes such as multi-modal ship recognition.

The invention discloses a photoacoustic temperature imaging method based on coherent factor weighted fusion. The method comprises the following steps: step 1, hardware composition and configuration of a multi-bandwidth photoacoustic imaging system based on photoacoustic and ultrasonic dual modes; step 2, ultrasonic modal signal preprocessing; and step 3, calculating a coherence factor and weighting a reconstructed image. According to the invention, high-precision non-intrusive temperature measurement capability is shown, and a more advanced non-intrusive temperature monitoring means is provided for photothermal therapy of tumors. According to the method, functional modules of imaging, segmentation, temperature measurement, treatment and the like are integrated, the reconstructed image is weighted through the coherence factor, high artifacts and multiple side lobes in the FBP reconstructed image are removed, the resolution of temperature imaging is remarkably improved, and the method has remarkable advantages and huge application potential in the application of photo-thermal treatment. And a multi-modal image fusion method is assisted, so that the multi-modal information of the target region can be obtained in the treatment process, more intuitive visual tumor state feedback is provided for workers, and the treatment effect is improved.

The invention discloses a deep learning-based multimodal image fusion method for soft tissue photoacoustic / ultrasound imaging. Steps: an ultrasound-photoacoustic imaging device acquires photoacoustic and ultrasound images of human soft tissue and performs size normalization processing; an input spatial transformation module converts the images to the YCbCr space; an input pre-convolution module modifies the number of data channels; an input multi-scale feature extraction module extracts salient features from the source images; an input filter prediction module derives multi-scale filters; and an input filter fusion and adaptive enhancement module combines the input source images to obtain the final fused result. The invention has superior fusion performance compared to several traditional fusion methods and deep learning-based fusion methods, and more importantly, it exhibits excellent real-time performance. Furthermore, various modes of photoacoustic / ultrasound fusion extension experiments have verified the effectiveness of the method proposed in the invention.

The invention relates to the technical field of multi-modal image fusion perception and scene understanding, in particular to a plug-and-play prompt-driven visible light-thermal infrared image semantic segmentation method and system, and the method comprises the steps: respectively extracting the feature representation of an input visible light image and a thermal infrared image through a dual-branch LoRA fine-tuning image encoder; converting a segmentation mask generated by the existing visible light-thermal infrared image semantic segmentation model into unified prompt information, including bounding box prompt or point prompt; using a prompt encoder to encode prompt information into prompt embedding; the prompt-based mask decoder fuses the image feature representation and the prompt embedded feature through a spatial channel cross attention mechanism, and generates a final segmentation mask through a classification head; in the training stage, parameterized random disturbance is added to the prompt of the true value mask, and error distribution of the prediction prompt is simulated. According to the method, high-precision semantic segmentation adaptive to various segmentation models can be realized without retraining, and the segmentation robustness in a complex scene is remarkably improved.

The invention relates to the technical field of medical image processing, in particular to a lung tumor CT image 3D segmentation method and system based on multi-modal image fusion. The method comprises the following steps: acquiring a lung tumor CT image; determining a first texture feature based on the lung tumor CT image; identifying a lung tumor boundary by using the first texture feature; detecting a nodule protrusion area in the boundary of the lung tumor; obtaining tissue infiltration data from the nodule protrusion area; determining a second texture feature according to the tissue infiltration data; determining a tumor heterogeneity feature according to the first texture feature and the second texture feature; evaluating the potential malignancy degree by utilizing tumor heterogeneity characteristics; and dividing a tumor risk area of the lung tumor CT image based on the potential malignancy degree. According to the invention, accurate heterogeneity identification and risk region division of the lung tumor CT image are realized based on a medical image processing technology, and the accuracy of lung tumor 3D segmentation is improved.

The invention discloses a power transmission line forest fire early warning method based on multi-modal image fusion, and aims to solve the problems that when an existing forest fire early warning method is in butt joint with the actual operation condition of a power transmission line, multi-source information complementation is difficult to consider at the same time, false detection and missing detection are often caused by environmental influence, and the early warning timeliness sometimes cannot meet the real-time requirement of power grid operation. According to the method, a power transmission line forest fire early warning model based on multi-modal image fusion is constructed, and the model comprises preprocessing, multi-scale feature extraction, fire point detection and positioning, risk level definition and early warning level division. And training the model according to the visible light image, the infrared image and the radar image at each moment in the process from the beginning to the end of the mountain fire combustion. The invention belongs to the technical field of power transmission line forest fire early warning.

The invention discloses a difference feature guided spatial channel multi-modal image fusion method and system, and the method specifically comprises the steps: 1, carrying out the fusion of a visible light multi-modal image and an infrared light multi-modal image at a plurality of layers, and carrying out the complementation of the missing information between two modals; obtaining image features of the visible light multi-modal image and image features of the infrared light multi-modal image; step 2, performing interaction and fusion on the two image features in the step 1 on a channel level to obtain features corresponding to the fused visible light multi-modal image and features corresponding to the fused infrared light multi-modal image; 3, performing spatial fusion on the features obtained in the step 2 to obtain fused spatial features; according to the invention, interactive fusion can be realized between the visible light mode and the infrared light mode, so that the quality of the fused image is improved. According to the method, the disadvantage of a single-mode image in a downstream task can be effectively solved, and complementary information of two modes can be better utilized.

The invention discloses a multi-modal image fusion model construction method, and particularly relates to the technical field of image fusion model construction. Interactive modeling of a modal contribution imbalance coefficient and a confidence coefficient estimation anomaly coefficient is introduced in a multi-modal image fusion process; a causal association between confidence prediction fluctuation and modal weight extreme is converted into a quantifiable cross-modal complementarity degeneration index, and the cross-modal complementarity degeneration index is used as a core adjustment mechanism to dynamically optimize a fusion weight and training regularization, so that the problem that a single mode is excessively amplified or mistakenly weakened is effectively inhibited on a fusion strategy level; according to the method, the structure retention capability of the model in edge details and texture regions is remarkably improved, and artifacts and information loss caused by modal complementarity degradation are avoided.

The invention discloses a robot automatic grabbing path planning method based on visual identification, and relates to the technical field of intelligent grabbing. The method comprises the following steps: acquiring multi-view visual data of a target scene, and generating a scene three-dimensional compact reconstruction model through a multi-modal image fusion algorithm; performing target detection and feature extraction on the model, and screening an optimal capture point in combination with a visual attention mechanism; constructing a dynamic environment obstacle probability map, updating an obstacle state through time sequence visual tracking, and quantifying an interference weight; an initial grabbing path is planned based on an improved fast expansion random tree algorithm, path smoothness constraints and robot joint movement limit parameters are introduced, and path nodes are optimized through a Bezier curve; visual servo feedback and path deviation prediction are fused, path parameters are corrected in real time, and a continuous movement track is generated. The method effectively adapts to the dynamic environment, gives consideration to path safety, smoothness and mechanical arm motion characteristics, and remarkably improves the grabbing success rate and operation reliability.

The invention relates to the technical field of orthopedic image processing, and particularly discloses an orthopedic disease diagnosis assisting method and system based on multi-modal image fusion, and the method comprises the steps: S1, collecting skeleton image data and soft tissue image data through imaging equipment; s2, performing cross-modal feature interaction evaluation based on the skeleton image data and the soft tissue image data after preprocessing and feature extraction; s3, performing feature dimensionality reduction on the image data to be subjected to dimensionality reduction, evaluating and optimizing a feature dimensionality reduction result, and marking the image data to be subjected to dimensionality reduction after feature dimensionality reduction as low-dimensional image data; and S4, performing disease category probability prediction based on the low-dimensional image data, and generating a disease category probability distribution report. The problems of excessive redundant information, poor feature interaction effect and low disease category probability prediction accuracy in the existing multi-modal image fusion technology are solved.

The invention discloses an underground structure leakage intelligent detection system based on multi-modal image fusion and deep learning recognition. The system comprises an image acquisition and preprocessing module, a significance guide image fusion module, a bimodal target detection module and a feature fusion and output module. Compared with the prior art, the method has the following advantages: saliency guidance, channel attention and multi-modal joint training are combined, an information closed loop is constructed by a triple mechanism, and the image fusion quality is improved; bimodal parallel recognition and ANN fusion judgment are adopted to adapt to the image degradation condition in a complex environment, and the recognition accuracy of a weak signal area is improved; the system can be deployed in various underground structure scenes such as subways, tunnels, underground garages and pipe galleries, and is compatible with various hardware terminals; a lightweight feature extraction and rapid fusion module is provided, and the real-time processing requirement of edge computing nodes is met; a closed-loop detection system of image acquisition, fusion enhancement, depth identification and intelligent output is formed, the overall efficiency is high, and the false detection rate is low.

The invention discloses a multi-modal image fusion method and system based on cross-modal interactive perception, and the method comprises the steps: constructing and training a multi-modal image fusion model, inputting a to-be-fused multi-modal image into the trained multi-modal image fusion model for processing, receiving the to-be-fused multi-modal image through an encoder module, carrying out the layer-by-layer encoding of the to-be-fused multi-modal image, and carrying out the layer-by-layer encoding of the to-be-fused multi-modal image. A plurality of layers of feature maps with different scales are output, a channel-level correction module receives the plurality of layers of feature maps with different scales, performs weighted correction and outputs modal features after the plurality of layers of weighted correction, and a dynamic cross-modal interaction module receives and processes the modal features after the plurality of layers of weighted correction to obtain a plurality of layers of fusion features. And the decoder module receives the plurality of layers of fusion features, performs layer-by-layer decoding and fusion processing, and outputs a fusion image corresponding to the to-be-fused multi-modal image. According to the method, through the channel-level correction module and the dynamic cross-modal interaction module, related information among multi-modal data can be effectively interacted.

The invention discloses a method and system for constructing a homography estimation model based on cross Mangbar dynamic feature fusion optimization, and the method comprises the following steps: building an overall framework of a network which is composed of neural network layers and comprises an input module feature pyramid, a rough matching module and a fine matching module; setting a structure of a crossed Mangbar model; setting the number, the connection mode and the type of crossed Mangbar models in the rough matching module; training the network to obtain a homography estimation network and a Mangbar network for multi-modal image fusion, better exploring intra-modal and inter-modal features, and dynamically enhancing detail texture information of a source image and specific information of each modal; for a feature extraction and reconstruction part of the network, a Mangbar model suitable for a fusion task is designed by combining a visual state space model with dynamic convolution and channel attention, so that the performance and the global modeling capability of the Mangbar model are kept, the channel redundancy is reduced, and the local feature extraction capability is enhanced.

The invention relates to the technical field of medical image and artificial intelligence crossing, and discloses a cross-modal-based VMama medical image fusion method and system combining grouped ACmix convolution and selective clustering, and the system comprises an input module, a preprocessing module, a feature extraction module, a multi-scale fusion module, and an output reconstruction module. According to the scheme, cross-modal attention is introduced into a visual state space model for the first time, a new multi-modal image fusion framework LMACV is obtained, the framework performs linkage optimization on ACmix and VMamba structures in a cross-modal medical image for the first time, feature reconstruction efficiency is enhanced through a selective clustering mechanism, and MSE and PSNR are remarkably superior to existing methods such as MPCT, FATFusion and MATR. By fusing a convolutional network and state space modeling, the network can effectively capture local texture details, and meanwhile, long-distance semantic association is reserved; besides, linear state updating and cross-modal dynamic alignment of attention guidance are effectively realized, and compared with the latest MPCT algorithm, the fusion speed is improved by 37.5%.