4803 results about "Multi modal fusion" patented technology

A multi-modal information tagging method, apparatus and device, and a storage medium and a product. An information tagging effect is effectively improved by means of determining visual feature information of visual information by means of a multi-modal tagging system, performing multi-modal fusion processing on the basis of the visual feature information, the visual information, recognized text information and descriptive text information to obtain image-text fusion features, determining image-text correlation information between the visual information and both the recognized text information and the descriptive text information, on the basis of the image-text fusion features and the image-text correlation information, determining a tagging result for information to be tagged, and performing multidimensional information tagging on the basis of multi-modal information fusion and cross-modal correlations.

Disclosed are a high-performance loosely-coupled multi-modal data fusion system for a smart driving environmental perception system and a vehicle-mounted device, comprising: a fusion detection model based on a modality-independent feature interaction strategy, which is configured for converting a LiDAR point cloud, a camera image, and a millimeter-wave radar point cloud into a unified bird's-eye view representation, and performing multi-modal fusion; and a fusion tracking model based on a motion-appearance feature cascaded coupling data association strategy, which is configured for performing subsequent trajectory tracking and matching according to multi-modal fusion feature information. A VoD data set and a K-Radar data set are selected for training, verifying, and testing the comprehensive performance of the models, and a TensorRT accelerated inference model is applied, then quantized, and deployed to a vehicle-mounted computational testing platform. The present invention is compatible with mainstream sensor deployment solutions, and achieves the efficient complementary fusion of multi-source heterogeneous sensor information, significantly improving the reliability, accuracy, and adaptability of vehicle-mounted perception systems, thereby effectively responding to extreme operating conditions such as complex traffic scenarios and inclement weather.

The invention relates to the technical field of LED display, solves the problem that the existing LED display defect detection and parameter adjustment technology is lack of multi-modal information fusion and intelligent process control capability and is difficult to meet the quality control requirement of a high-precision display product, and provides an LED display defect prediction and process adjustment method and system based on multi-modal fusion. The method comprises the following steps: performing multi-modal data fusion processing on optical image data, electrical test data and thermal infrared imaging data corresponding to a to-be-tested LED display screen to obtain fused data; inputting the fused data into a pre-trained defect recognition model to obtain a defect recognition result; according to a process parameter adjustment strategy corresponding to the defect identification result, adjusting the original process parameter to obtain a target process parameter; and according to the target process parameters, process flow correction processing is carried out, and a qualified LED display screen is produced. According to the method, the defect identification precision is improved, and the quality control requirement of high-precision LED display screen production is met.

The invention discloses an unmanned aerial vehicle route automatic planning method for electric power inspection, and relates to the technical field of unmanned aerial vehicle inspection. Comprising the following steps: starting an unmanned aerial vehicle, carrying out environment perception initialization, calculating the total cruise mileage, evaluating the interference risk, carrying out real-time obstacle avoidance, dynamically optimizing an inspection route, carrying out energy monitoring management, generating a return flight strategy, recording an inspection task and carrying out adaptive learning. Through dynamic electromagnetic interference modeling, multi-modal fusion perception, self-adaptive risk decision and cloud collaborative learning, the problem of insufficient adaptability of a traditional electric power inspection unmanned aerial vehicle in a complex electromagnetic environment and a dynamic obstacle scene is solved, the safety and the inspection efficiency are improved, the robustness is enhanced, and the method is suitable for popularization and application. Intelligent upgrading is carried out through continuous learning and multi-machine cooperation, the overall operation and maintenance cost of the system is reduced, a high-reliability and full-automatic inspection solution is provided for intelligent power grid construction, and the industrial application value is remarkable.

The invention provides an intelligent image signal processing method and system based on multi-modal fusion. According to the invention, a multi-mode input signal is received, is divided into a spatial distribution feature extraction region and a dynamic change trajectory capture region, and is decomposed into a penetrability feature layer and a substance reflection feature layer; carrying out association mapping on missing pixel information in the dynamic change trajectory capture region and energy distribution of the penetrability feature layer to generate enhanced dynamic trajectory data, identifying a determined reflection mode in the substance reflection feature layer, and carrying out frequency domain superposition on corresponding frequency band response and the spatial distribution feature extraction region; generating a composite spatial feature, then constructing a multi-modal joint optimization model, then adjusting the contribution ratio of the two data, and generating a fused image signal; the technical scheme provided by the invention not only solves the problems of detail loss, artifact generation and poor dynamic adaptability caused by single-mode limitation, but also improves the spatial resolution and tracking precision of the image signal.

The invention discloses a multi-agent cooperation system and method based on spatial calculation and multi-modal AI fusion, and the method comprises the steps: collecting all real-time data of a construction site through a multi-modal data collection and fusion unit, and carrying out the processing and integration of all collected data through a multi-modal AI algorithm, and generating a unified semantic association model; a three-dimensional virtual environment of a construction site is constructed through a space calculation unit, real-time synchronous mapping is formed for the construction site, and dynamic changes in the construction process are simulated and predicted; task allocation and behavior planning are carried out on multiple agents through an agent behavior management unit; and the intelligent agents are configured to perform real-time information sharing and task cooperation through a communication protocol through the multi-intelligent-agent cooperation unit. According to the scheme, the problems of difficulty in multi-source data integration, low cross-department cooperation efficiency, insufficient construction dynamic adjustment and the like in intelligent construction in the building industry can be solved.

The invention discloses an intelligent psychological intervention system based on multi-modal fusion, which is characterized in that a three-dimensional evaluation system is constructed by integrating speech sentiment analysis, keyboard dynamics monitoring and physiological signal acquisition, and time sequence alignment and feature weighted fusion of multi-source data are realized by adopting a cross-modal Transform model. The core of the system comprises an adaptive intervention engine which defines a multi-dimensional state space based on a hierarchical reinforcement learning architecture, optimizes an intervention strategy through a PPO algorithm, and realizes dynamic emotion interaction in AR and VR scenes in combination with a digital twin training module; according to the clinical decision support system, physiological behavior characteristics and psychological assessment trends are integrated by using a multi-time scale risk prediction model, and a personalized early warning threshold system is constructed, so that the psychological state recognition accuracy is improved, the intervention intensity self-adaptive adjustment response time is shortened, and the high-risk signal early warning timeliness reaches the minute level; and the problems of evaluation hysteresis and strategy stiffness of traditional psychological intervention are obviously improved.

The invention discloses a photovoltaic power station unmanned inspection method and system based on multi-mode fusion detection, and relates to the technical field of intelligent operation and maintenance of photovoltaic power stations. The method mainly comprises the following steps: (1) collecting multi-modal data of a photovoltaic module and carrying out preprocessing operation; (2) uniformly mapping the heterogeneous data into discrete marks by using each pre-trained modal data marker; (3) training a multi-modal fusion detection network based on a Transform encoder-decoder architecture by using the training sample, and adjusting network parameters to obtain a trained model; and (4) outputting a defect type, a position bounding box and a severity score by utilizing the trained model according to the obtained multi-modal data or single-modal data of the photovoltaic module. The problems that traditional single-mode detection is high in omission ratio and multi-mode fusion is low in efficiency are solved, the inspection collection module, the data processing module, the multi-mode fusion detection module and the fault decision module are provided in a targeted mode, and full-process automation from data collection to intelligent decision is achieved. The operation and maintenance cost of the photovoltaic power station is obviously reduced; and the fault response efficiency is improved.

The invention discloses an inspection method, device, equipment and medium based on a digital airspace system, and the method comprises the steps: carrying out the feature extraction and calibration of multi-source sensing monitoring data of a to-be-inspected region through a pre-trained neural network model; obtaining a structured semantic description and a time sequence measurement value to be combined with external meteorological spatial data to be input into a digital airspace expert model based on an MoE framework for multi-modal fusion; carrying out enhancement processing on the generated multi-modal fusion representation through a variational auto-encoder so as to input the multi-modal fusion representation into a multi-modal transformer decoder for decoding, and obtaining an entity triple set to construct a digital spatial domain knowledge graph; node characterization in the knowledge graph is enhanced by adopting an attention enhancement mechanism based on a graph neural network, the node characterization is input into a conditional generative adversarial network model for multi-task prediction, and an inspection decision suggestion is generated to be executed by the unmanned aerial vehicle, so that the utilization rate of multi-source sensing data is effectively improved, and the inspection efficiency of the unmanned aerial vehicle is improved.

The invention relates to the technical field of medical artificial intelligence and knowledge graph construction, in particular to a rare disease knowledge graph construction method based on modal injection and multi-modal fusion. Comprising the following steps: S1, collecting multi-modal medical information including texts, images and genes; s2, standardization processing is carried out, and a three-layer metadata structure is constructed; s3, complementing missing modal data, and performing feature extraction and unified dimension conversion on the modal data to realize representation alignment in a shared semantic space; s4, performing multi-level semantic fusion to obtain a unified fusion semantic vector; and S5, constructing a double-layer structure system rare disease knowledge graph comprising an ontology layer and an instance layer. According to the method, multi-modal medical information of texts, images and genes is selected to construct the knowledge graph of the rare disease, the application range, coverage and accuracy of the knowledge graph are improved, correspondence adaptation of rare cases during clinical diagnosis and treatment of the rare disease can be achieved, and the method has high recognition capacity.

The invention discloses an intelligent legal data processing method and system based on big data, and the method comprises the steps: S1, constructing a legal data collection framework, and carrying out the unified coding and representation of different data forms through a multi-modal fusion technology; s2, performing hierarchical semantic analysis on the cross-domain legal clauses to generate a high-dimensional semantic representation vector with context sensing capability; s3, performing node embedding, association analysis and reasoning optimization on the constructed legal knowledge graph to generate a dynamically updated multi-domain legal association network; s4, mapping and optimization among terms are realized through semantic comparison and multi-level rule verification; s5, performing modeling analysis on the dynamic trend and the potential risk factors of the legal data, and outputting a risk prediction and early warning strategy of the legal event; and S6, improving cross-data-source cooperative computing capability and data analysis efficiency through a distributed optimization strategy. The method has the advantages of high semantic understanding depth, high risk prediction accuracy and high privacy protection and cooperative computing efficiency.

The invention discloses a submarine cable risk dynamic assessment method and system based on multi-modal deep learning, and belongs to the field of marine infrastructure operation and maintenance. Aiming at the problems of incomplete data coverage, unreal generated scene, low evaluation reliability and the like in the prior art, the method comprises the following steps of: 1) constructing a multi-source heterogeneous data set containing six types of data including geology, ocean, ships, biology and the like, and realizing data alignment by adopting space-time grid coding; 2) designing a physical constraint generative adversarial network, and generating risk scene data conforming to a fluid mechanics law through a Navier-Stokes equation constraint; 3) creating a hierarchical space-time fusion network (HST-Transform), and combining CNN spatial feature extraction, a time sequence attention mechanism and a dynamic memory module to realize multi-modal fusion; according to the method, the detection rate of rare risk events is increased by 62%, the evaluation accuracy rate reaches 91.7%, the false alarm rate is reduced by 34% compared with a traditional method, and submarine cable breakage accidents can be effectively prevented.

A distributed intelligent authentication method based on dynamic multi-modal fusion relates to the field of network security, and adopts an alliance chain + DAG hybrid block chain architecture, combines a threshold signature to realize secret key fragment management, and switches among PBFT, Raft and probabilistic algorithms through a dynamic consensus mechanism to improve authentication efficiency. The multi-mode authentication module is based on a dynamic weight distribution algorithm, integrates biological characteristics, behavior analysis, equipment fingerprints and environmental factors, and combines an LSTM-GAN model and a quantum random number driven challenge-response mechanism to realize zero-trust verification under environmental perception. The session management module generates a session key by using a chaotic mapping algorithm. In the aspect of privacy protection, CKKS homomorphic encryption, zero-knowledge proof and attribute-based encryption are fused. According to the method, the block chain technology, the secure multi-party computing technology, the machine learning technology and the quantum cryptography technology are fused, and a high-performance, high-security and strong-privacy-protection distributed authentication solution is provided.

The invention discloses a real-time health risk prediction method and system based on a dynamic knowledge graph, and relates to the technical field of medical information. The method comprises the following steps: carrying out multi-modal fusion and privacy protection preprocessing on medical and nursing heterogeneous data, and realizing semantic consistency of cross-mechanism data based on an entity alignment method of a cross-modal graph neural network; based on a hierarchical federated learning framework, local model parameters are subjected to hierarchical encryption and aggregation through a secure multi-party computing protocol to generate an initial global model, and prediction distribution of the global model is optimized in combination with knowledge distillation of differential privacy constraints; designing a gradient difference dynamic updating trigger mechanism of noise robustness, smoothing noise interference through a sliding window mean value, and realizing adaptive threshold calibration through linkage model performance verification; and light-weight deployment real-time reasoning is realized based on redundant edge pruning of confidence and 8-bit symmetric quantization. On the premise of protecting data privacy, the real-time performance and accuracy of health risk prediction are remarkably improved, and the method is suitable for a cross-institution medical care collaborative decision-making scene.

The invention discloses an adaptive teaching real-time feedback method based on multi-modal fusion, and the method comprises the following steps: synchronously collecting text modal information, voice modal information and image modal information generated by students in a teaching process, forming multi-modal original data information, and extracting historical student interaction behavior data; preprocessing the multi-modal original data information, and respectively generating corresponding text, voice and image sequence features; a visual feature encoder and a sequence feature encoder are adopted to encode each modal sequence feature to obtain a high-dimensional feature; inputting the modal high-dimensional features into a cross-modal fusion network for deep fusion; parameters of the feedback model are optimized through a model-independent element learning feedback regulation and control algorithm, and a personalized feedback strategy is generated; generating comprehensive feature representation according to the fusion features, and outputting personalized teaching feedback; and the interaction information is updated based on the feedback behavior data to realize closed-loop optimization.

An intelligent analysis method based on medical document structure perception and multi-modal fusion comprises the following steps: carrying out structure topology modeling on a medical document, extracting visual layout, text meta-information, space coordinates and semantic keyword features, constructing a semantic topological graph and dynamically shielding irrelevant contents; selecting an extraction path according to a document type, performing deep semantic analysis and entity recognition on a text-type document, and performing visual enhancement OCR recognition on a scanning-type document; the features are injected into a medical knowledge graph, and feature fusion, semantic verification, relation reasoning and information completion are achieved through a graph neural network; a three-stage strategy optimization model of basic pre-training, domain adaptation and online reinforcement learning is adopted; and large-scale processing is realized through a dynamically aggregated distributed architecture. The method is used for intelligent analysis and structured conversion of documents of hospitals, medical insurance and medical scientific research. The problems that heterogeneous medical document analysis adaptability is poor, multi-modal fusion is difficult, medical knowledge utilization is insufficient, and large-scale processing efficiency is low are solved.

The invention discloses a medical image enhancement method and system based on multi-modal fusion, and the method comprises the steps: obtaining medical image data which comprises a CT image, an MRI image, a PET image and an ultrasonic image; calculating a mutual information value of each semantic tag in the cross-modal feature vocabulary, and generating an inter-modal feature correlation matrix; performing non-rigid space registration on the unified dimension image set to generate a geometrically consistent multi-modal image set; extracting skeleton region features, soft tissue region features and high metabolism region features from the multi-modal image set, and generating a core feature set; a deep learning algorithm is adopted to train the core feature set, and a segmented image set containing a segmentation mask is generated; and dynamically adjusting the fusion weight according to the regional features of the segmented image set, and generating a fusion enhanced image. According to the invention, through feature extraction, spatial registration and deep learning fusion of the multi-modal medical image, effective integration of different-modal medical image information is realized.

The invention relates to the technical field of information retrieval, and discloses a multi-modal fusion and reinforcement learning collaborative retrieval enhancement generation method and system. The method comprises the following steps: receiving an original query input by a user, and generating a sub-query based on a large language model in combination with a multi-modal context of a current iteration step; forming a current state in combination with the sub-query and the multi-modal context, modeling a retrieval enhancement generation task as a Markov decision process, and adaptively selecting an optimal action from a predefined action set in the current state by utilizing a large language model according to a decision strategy; executing a corresponding multi-modal retrieval operation according to the optimal action, fusing the obtained multi-modal information, generating an intermediate answer or a final answer of the sub-query, and updating a multi-modal context by using the intermediate answer; off-line training optimization is carried out on the large language model through imitation learning and a calibration chain, and decision strategies and sub-queries are inferred online through the model after fine adjustment. According to the invention, more efficient and accurate complex query processing is realized.

The invention provides an infrared thermal imaging building facade defect intelligent diagnosis method based on multi-modal fusion, and relates to the technical field of building detection.The method comprises the steps that infrared thermal imaging, visible light images and three-dimensional point cloud data are synchronously collected to construct a multi-modal data set; segmenting a hot spot region by adopting an improved morphological watershed algorithm and extracting contour and temperature features; recognizing a surface crack and peeling area based on a double-branch attention network to generate a texture defect feature map; curvature distribution and thermal deformation gradient are calculated through space registration constrained by a heat conduction equation; multi-source features are fused to calculate a hot spot form dispersion TSMD and a structure risk quantification factor SRQF; constructing a defect risk decision matrix to output defect types, positions and risk levels; and superposing the diagnosis result to a BIM model to generate a three-dimensional visual report and predicting a thermodynamic evolution trend. The multi-modal data collaborative analysis is realized, the defect risk is accurately quantified, and the problems of poor anti-interference performance, inaccurate segmentation and large registration error of a traditional method are solved.

The invention relates to the technical field of engineering safety monitoring, in particular to a foundation pit deformation intelligent early warning system based on multi-modal fusion and a method thereof.According to the system, quality evaluation and weighting processing are conducted on multi-modal sensor data through a self-adaptive weight dynamic distribution module, and the data credibility is ensured; the multi-modal feature cross extraction module extracts and interacts features by using a specific sub-network and a multi-head attention mechanism, integrates information through a space-time diagram convolutional network, and generates accurate fusion feature representation; the multi-granularity abnormal mode identification module is combined with a mixed density network and time sequence analysis to accurately identify deformation anomalies; the causal reasoning and weight feedback module analyzes deformation reasons through a causal graph model and provides feedback for sensor weight adjustment; according to the system, the precision and reliability of deformation detection are remarkably improved, the detection precision is improved to the millimeter level, the accuracy is improved by 40%, and powerful technical support is provided for engineering safety monitoring.

The invention discloses an intelligent interactive enterprise management simulation system and a method thereof. The method comprises the following steps: constructing a multi-level state causal graph model with a multi-dimensional feature tag; fusing interaction data of the user, constructing a behavior intention tensor, and identifying a situation transition intention through a multi-modal fusion reasoning algorithm; a personalized interaction strategy is dynamically generated by a context-driven interaction optimization generation algorithm; a context semantic tensor is constructed based on a semantic context dynamic matching algorithm, a semantic compression response is realized in combination with a current intention and a historical information path, and a future strategy plan is actively generated through a causal relationship backstepping inference device; and establishing a situation feedback learning and weight updating mechanism, and dynamically adjusting a state causal graph, an intention tensor and an interaction strategy parameter to realize self-evolution closed-loop optimization of the model. Context changes can be perceived in real time, and the decision intention of the user can be deduced deeply.

The invention relates to the field of medical informatization. The invention discloses a thyroid cancer electronic medical record system based on multi-modal data fusion. The thyroid cancer electronic medical record system comprises a multi-modal data acquisition module which acquires patient texts, ultrasonic images, genes, biochemical indexes and clinical data and performs standardized calibration to generate standard data; the multi-modal feature extraction module extracts semantic, structure, mutation, change and fluctuation features of each standard data through multiple technologies; the single-mode prediction model construction module constructs single-mode prediction models of texts, images and the like based on the features and outputs results; and the multi-modal fusion prediction module fuses the single-modal model based on the deep learning framework to output a multi-modal fusion prediction result. According to the invention, multi-modal data are integrated, and the accuracy and comprehensiveness of thyroid cancer diagnosis are improved. The system ensures consistency through standardized data processing, and assists doctors to accurately judge pathological types, recommend therapeutic schedules and evaluate prognosis by means of a multi-modal feature extraction and fusion mechanism.

The invention relates to a multi-modal named entity recognition method based on cross-modal guide interactive fusion, which comprises the following steps: constructing a data set, designing a cross-modal contrast aggregation mechanism, respectively extracting image features and text features, and constructing a contrast learning mechanism to screen out image features with high association degree with text semantics for dynamic aggregation; a DINO model is introduced to extract image features, a dynamic similarity matching method is constructed, a dynamic similarity matching weight is generated based on text features and an image feature correlation matrix, a dynamic gating mechanism is utilized to adaptively select image features related to text feature context, and a cross-modal fusion and guide interaction strategy is constructed. Outputting an enhanced semantic representation vector, and mapping the semantic representation vector after multi-modal fusion into a final entity tag sequence by adopting a conditional random field decoder to complete entity recognition; the method has the advantage of remarkably improving the robustness and accuracy of multi-modal entity recognition.

The invention relates to the technical field of computer vision and three-dimensional reconstruction, and discloses an exhibition hall three-dimensional modeling intelligent optimization system based on multi-modal data fusion, and the system comprises a data collection module which is configured to synchronously obtain laser radar point cloud data, a multispectral image sequence and inertial measurement unit data; the preprocessing module is used for receiving the output of the data acquisition module, aligning a multi-source sensor coordinate system through a space-time calibration algorithm, and separating a static scene from a dynamic interference element by using a dynamic segmentation network; and the multi-modal fusion module is used for receiving the preprocessed data and carrying out adaptive weighted fusion on the geometric features of the laser radar and the visual texture features through a cross-modal attention mechanism. According to the invention, through multi-modal data fusion and a dynamic scene adaptive mechanism, the modeling precision and the real-time updating capability in a complex exhibition hall environment are significantly improved.

The invention discloses a large-model complaint intention recognition method based on sentiment analysis, and the method comprises the steps: obtaining call voice data of a customer, and converting the call voice data into text data; preprocessing the text data, and removing noise and marking components; performing emotion feature analysis on the text data, extracting an emotion feature value, and generating an emotion vector; the emotion vectors and the text data are input into a large model together, and the large model combines the emotion feature values and context semantics to generate intention feature vectors; constructing an emotion-intention state vector, inputting the emotion-intention state vector into an asynchronous dominant actor commentator algorithm model, and generating a corresponding complaint intention probability value; judging a complaint intention probability, and generating risk early warning; and preferentially distributing high-risk customers and responding to customer demands. Through combination of voice data preprocessing, text semantic feature extraction, emotion intensity quantitative analysis and a multi-modal fusion algorithm and efficient risk assessment based on an asynchronous dominant actor reviewer model, the early warning and response capabilities of customer complaint risks are significantly improved.

The invention discloses an enterprise data asset intelligent analysis method and system based on AI recognition, and the method comprises the steps: receiving an enterprise multi-source heterogeneous data stream, carrying out the joint feature extraction and semantic alignment through a pre-trained multi-modal fusion recognition model, and generating a structured data asset recognition result; constructing a dynamic enterprise data asset atlas according to the structured data asset identification result in combination with the data access trajectory and authority metadata collected in real time; performing spatio-temporal evolution analysis on the dynamic enterprise data asset map, and extracting potential data value density features and risk exposure features; inputting the data value density features and the risk exposure features into a self-organizing mapping network to generate a data asset grading topological graph; and based on the data asset grading topological graph, through strategy constraint reinforcement learning, generating an executable data governance action sequence. According to the embodiment of the invention, the identification precision and real-time analysis capability of special assets of enterprises can be improved.

The invention discloses a screening and evaluating system and method for cognitive impairment of old people, and belongs to the technical field of intelligent medical treatment. According to the system, electroencephalogram, VR / AR interaction, physiology and gait data are collected through the multi-modal acquisition module, the data processing module carries out data processing, dynamically selects a scale and carries out fusion analysis, the man-machine interaction module realizes natural interaction, and the output module generates a visual report and carries out early warning. The core algorithm comprises dynamic scale selection, multi-modal fusion evaluation and the like. The evaluation method comprises the steps of data acquisition, scale selection, data fusion, confidence verification, output and the like. Compared with a traditional method, the method overcomes the defects that subjectivity is high, and static evaluation, data splitting and base layer application are limited. The cognitive impairment assessment method can accurately and dynamically assess cognitive impairment, improves assessment accuracy, realizes full utilization of data, is simple and convenient to operate, is suitable for various scenes, provides efficient and reliable support for diagnosis, monitoring and intervention of cognitive impairment, and promotes development of intelligent medical treatment.

The invention discloses an intelligent data analysis method and system based on an industry large model, and relates to the technical field of data analysis, and the method comprises the steps: collecting business event data, carrying out the feature vector extraction according to the data type, calculating a zoom dot product attention score matrix through employing a self-attention mechanism, and carrying out the weighted output of a multi-modal fusion vector, constructing an industry knowledge database to determine an embedded vector; and calculating a relevancy value, screening a combination of the multi-modal fusion vector and the knowledge fragment, and forming a gating input vector to carry out fusion vector calculation. According to the method, unified expression and correlation mining of heterogeneous information such as structured data, text information and images can be realized through a multi-modal feature fusion and knowledge fragment retrieval scheme, the attention degree of each modal feature can be adaptively adjusted according to a service scene through a self-attention mechanism and a model, information complementation is realized, and the accuracy of information retrieval is improved. And automatically highlighting the multi-modal dimension most related to the current task.

The invention discloses a multi-modal fusion exhibition building tall atrium natural ventilation evaluation method. The method comprises the steps that multi-modal data are obtained, wherein the multi-modal data comprise meteorological time sequence parameters, building three-dimensional point cloud and opening geometric parameters and air conditioner cooling parameters; inputting the multi-modal data into a multi-modal fusion neural network model to obtain a natural ventilation evaluation result, the natural ventilation evaluation result comprising vertical temperature distribution of the atrium and the ventilation quantity of each opening of the atrium; wherein the multi-modal fusion neural network model comprises a meteorological time sequence feature extraction module, a building geometric coding module, an air conditioning system parameter coding module, a cross-modal feature fusion module, a vertical temperature field prediction module and a ventilation quantity prediction module. According to the method, a space-time cooperative sensing mechanism of meteorological time sequence dynamic, building space topology and air conditioner parameter features is established, and the technical bottleneck of a traditional method in the aspects of cross-modal feature space alignment and interaction efficiency is broken through.

The invention discloses a dissimilar metal laser welding device based on swing light beams and molten pool state monitoring. The dissimilar metal laser welding device aims at improving the welding quality and the joint stability. The device comprises a laser welding head with a light beam swinging function, and the laser welding head can implement nonlinear energy scanning in a welding area according to a preset track type, frequency and amplitude; the acquisition module can synchronously acquire a visual image, an excitation spectrum and an infrared thermal imaging signal of the molten pool at a high frame rate, and extracts interface diffusion and metal mixing characteristics through multi-modal fusion; the calculation module performs dynamic feature modeling according to the collected physical feature information and the swing parameters, and quantifies multi-dimensional state indexes of the welding quality; the control module carries out combined adjustment on the laser power, the welding speed and the swing parameters according to the state indexes, closed-loop feedback control is constructed, and therefore real-time stable regulation and control and defect suppression in the dissimilar metal welding process are achieved.