Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

6007 results about "Multi modal fusion" patented technology

Controllable video generation method and system based on multi-modal fusion

The invention discloses a controllable video generation method and system based on multi-modal fusion. The method comprises the following steps: acquiring multi-modal data and preprocessing the multi-modal data; extracting feature vectors, normalizing the feature vectors, and combining the feature vectors to form a multi-modal feature matrix; converting the multi-modal feature matrix into a feature sequence; performing dynamic weight distribution on each modal feature in the feature sequence; performing multi-level deep fusion on the weighted feature sequence to obtain fusion feature representation; inputting the fused feature representation into a multi-scale space-time attention mechanism to obtain enhanced feature representation; processing by using a space attention module, a time attention module and a cross attention module in sequence to obtain global enhanced feature representation, and generating multi-frame hidden space representation; and decoding to generate a high-resolution video frame and carrying out post-processing to obtain a final high-quality video sequence. According to the method, the space-time coherence of the video sequence is kept, and the video content with higher quality and stronger controllability is generated.
Owner:NANJING ARTIFICIAL INTELLIGENCE CHIPS RES INST OF AUTOMATION CHINESE ACAD OF SCI

Live broadcast room content identification and intelligent distribution method and system based on multi-modal fusion

The invention provides a live broadcast room content identification and intelligent distribution method and system based on multi-modal fusion, and relates to the technical field of machine learning, and the method comprises the steps: collecting video, audio and bullet screen data of a live broadcast room, and carrying out the feature extraction; performing cross-modal interaction, feature alignment and feature reconstruction on the multi-modal features by using the multi-modal feature fusion model to generate cross-modal fusion features; and carrying out content identification based on a pre-trained content classification model to obtain a category label and a time sequence feature. Furthermore, a live broadcast room content portrait and an association network are constructed, and a live broadcast room real-time score is generated and a live broadcast room display weight and a push strategy are dynamically adjusted in combination with user historical watching behavior data and a deep reinforcement learning model. Through multi-modal data fusion and deep reinforcement learning, accurate identification and personalized recommendation of the live broadcast room content are realized, and the user experience and the platform operation efficiency are effectively improved.
Owner:BEIJING YIHUI ZMENG NETWORK TECH

Natural environment bird monitoring method based on multi-modal fusion deep learning and computer device

The invention relates to the field of bird monitoring, in particular to a natural environment bird monitoring method based on multi-modal fusion deep learning and a computer device. The scheme comprises the following steps: establishing a multi-modal bird data set; labeling the multi-modal bird data set; carrying out preprocessing on the marked data; the method comprises the following steps of: constructing a multi-modal trans-attention Transform network model; the method comprises the following steps of: training and optimizing a multi-modal trans-attention Transform network model; sound and image data in the multi-mode bird data set are subjected to conjoint analysis, a YOLOv8 target detection model is used, marked image data are trained, and rapid detection of a bird target is achieved; in combination with a DeepSORT tracking algorithm, the detected target is tracked, and the motion trail of the birds is obtained; and performing space-time synchronization on the sound source positioning information and the visual tracking result, and verifying the identity and the position of the bird. The bird monitoring device is suitable for monitoring birds in the natural environment.
Owner:SICHUAN AGRI UNIV

Multi-modal information tagging method, apparatus and device, and storage medium and product

A multi-modal information tagging method, apparatus and device, and a storage medium and a product. An information tagging effect is effectively improved by means of determining visual feature information of visual information by means of a multi-modal tagging system, performing multi-modal fusion processing on the basis of the visual feature information, the visual information, recognized text information and descriptive text information to obtain image-text fusion features, determining image-text correlation information between the visual information and both the recognized text information and the descriptive text information, on the basis of the image-text fusion features and the image-text correlation information, determining a tagging result for information to be tagged, and performing multidimensional information tagging on the basis of multi-modal information fusion and cross-modal correlations.
Owner:BIGO TECH PTE LTD +1

High-performance loosely-coupled multi-modal data fusion system for smart driving environmental perception system and vehicle-mounted device

Disclosed are a high-performance loosely-coupled multi-modal data fusion system for a smart driving environmental perception system and a vehicle-mounted device, comprising: a fusion detection model based on a modality-independent feature interaction strategy, which is configured for converting a LiDAR point cloud, a camera image, and a millimeter-wave radar point cloud into a unified bird's-eye view representation, and performing multi-modal fusion; and a fusion tracking model based on a motion-appearance feature cascaded coupling data association strategy, which is configured for performing subsequent trajectory tracking and matching according to multi-modal fusion feature information. A VoD data set and a K-Radar data set are selected for training, verifying, and testing the comprehensive performance of the models, and a TensorRT accelerated inference model is applied, then quantized, and deployed to a vehicle-mounted computational testing platform. The present invention is compatible with mainstream sensor deployment solutions, and achieves the efficient complementary fusion of multi-source heterogeneous sensor information, significantly improving the reliability, accuracy, and adaptability of vehicle-mounted perception systems, thereby effectively responding to extreme operating conditions such as complex traffic scenarios and inclement weather.
Owner:JIANGSU UNIV

Commodity information processing and querying method and system

The invention provides a commodity information processing and querying method and system. Belongs to the technical field of information processing. The method comprises the steps of collecting text information of commodities, preprocessing the text information of the commodities through natural language processing information, and constructing a structured commodity text database; based on the high-definition image of the commodity, key features of the commodity are extracted through machine vision and are associated with information in the text database to form multi-modal commodity information representation; the method comprises the following steps: encoding a commodity text by adopting a BERT model based on Transform, and obtaining a deep semantic representation of the text; performing feature extraction on the commodity image by using a pre-trained convolutional neural network to obtain an advanced visual feature of the image; through natural language processing and machine vision technologies, the text information and the image information of the commodity are subjected to structured and multi-modal fusion, so that the processing efficiency of the commodity information is greatly improved.
Owner:万联易达物流科技有限公司

Content recommendation method and system based on semantic recognition

ActiveCN119089398AAccurate semantic representation of dataEfficient learning processSemantic analysisBiological modelsPersonalizationProcessing
The invention discloses a content recommendation method and system based on semantic recognition, and the method comprises the steps: analyzing the text, voice or video input of a user through a natural language processing technology, and generating user semantic representation data; the method comprises the following steps: dynamically constructing a user interest graph by combining user historical behavior data, and performing representation learning of nodes and edges through a multilayer graph convolutional network to generate time sequence user interest graph representation data; a self-attention mechanism and a multi-head attention mechanism of a Transform model are utilized, the matching degree of user interest representation and recommended content is optimized in combination with a contrast learning module, and personalized recommended content data are generated; collecting multi-modal information of the user, fusing the multi-modal information with the personalized recommendation content data, and generating optimized multi-modal recommendation content data through a multi-modal fusion algorithm; and collecting user feedback feature data, and dynamically adjusting and optimizing a generation strategy of recommended content through an adaptive learning algorithm. According to the invention, efficient, accurate and personalized recommendation of the recommendation system is realized, and the user experience is improved.
Owner:HANGZHOU SCENE INNOVATION CENTER CO LTD

LED display defect prediction and process adjustment method and system based on multi-modal fusion

The invention relates to the technical field of LED display, solves the problem that the existing LED display defect detection and parameter adjustment technology is lack of multi-modal information fusion and intelligent process control capability and is difficult to meet the quality control requirement of a high-precision display product, and provides an LED display defect prediction and process adjustment method and system based on multi-modal fusion. The method comprises the following steps: performing multi-modal data fusion processing on optical image data, electrical test data and thermal infrared imaging data corresponding to a to-be-tested LED display screen to obtain fused data; inputting the fused data into a pre-trained defect recognition model to obtain a defect recognition result; according to a process parameter adjustment strategy corresponding to the defect identification result, adjusting the original process parameter to obtain a target process parameter; and according to the target process parameters, process flow correction processing is carried out, and a qualified LED display screen is produced. According to the method, the defect identification precision is improved, and the quality control requirement of high-precision LED display screen production is met.
Owner:XIAMEN PROD QUALITY SUPERVISION & INSPECTION INST +1

Automatic route planning method of unmanned aerial vehicle for electric power inspection

The invention discloses an unmanned aerial vehicle route automatic planning method for electric power inspection, and relates to the technical field of unmanned aerial vehicle inspection. Comprising the following steps: starting an unmanned aerial vehicle, carrying out environment perception initialization, calculating the total cruise mileage, evaluating the interference risk, carrying out real-time obstacle avoidance, dynamically optimizing an inspection route, carrying out energy monitoring management, generating a return flight strategy, recording an inspection task and carrying out adaptive learning. Through dynamic electromagnetic interference modeling, multi-modal fusion perception, self-adaptive risk decision and cloud collaborative learning, the problem of insufficient adaptability of a traditional electric power inspection unmanned aerial vehicle in a complex electromagnetic environment and a dynamic obstacle scene is solved, the safety and the inspection efficiency are improved, the robustness is enhanced, and the method is suitable for popularization and application. Intelligent upgrading is carried out through continuous learning and multi-machine cooperation, the overall operation and maintenance cost of the system is reduced, a high-reliability and full-automatic inspection solution is provided for intelligent power grid construction, and the industrial application value is remarkable.
Owner:SUZHOU TIANJING YUNHU INTELLIGENT TECH CO LTD

Nondestructive inspection method for enhancing defect detection efficiency through deep learning

The invention relates to the technical field of non-destructive testing, and discloses a non-destructive testing method for enhancing defect detection efficiency through deep learning, which comprises the following steps: step 1, acquiring multi-modal data of a target material, including data acquired through X-ray imaging, ultrasonic detection and infrared thermal imaging; 2, signal denoising processing is carried out on the obtained multi-modal data, signal denoising is carried out through wavelet transformation, and the wavelet transformation comprises the steps that multi-scale decomposition is carried out on signals, and high-frequency noise is removed through a soft threshold method; and step 3, carrying out image enhancement processing on the de-noised multi-modal data. The internal structural features of the material are captured from multiple physical dimensions by fusing multi-modal data of X-ray imaging, ultrasonic detection and infrared thermal imaging. The multi-modal fusion can make up for the deficiency of a single detection technology, so that the defect identification in complex structures and materials is more comprehensive and accurate, thereby significantly improving the detection precision.
Owner:SUZHOU TITAN WIND POWER BLADE TECH CO LTD

Multi-modal analysis method, system and equipment for industrial inspection scene and medium

ActiveCN119128810ADetailed video image featuresRealize intelligent analysisInference methodsEngineeringVideo image
The invention discloses a multi-modal analysis method, system and device for an industrial inspection scene and a medium. The method comprises the following steps: acquiring multi-modal data of an industrial inspection area in real time, wherein the multi-modal data comprises a video image and voice data; inputting the currently collected multi-modal data into a scene type identification model, and identifying a scene type; matching a corresponding multi-modal fusion mode based on the determined scene type; according to the matched multi-modal fusion mode, performing feature extraction on the video image by using a deep learning algorithm, performing feature extraction on the text converted from the voice data by using a natural language processing technology, and performing multi-modal fusion on the extracted features; performing joint reasoning on the multi-modal fusion features by using a deep learning algorithm, and identifying potential risks or equipment faults in the industrial inspection area; and generating an inspection report and decision based on the identified potential risk or equipment fault. According to the invention, intelligent analysis of the industrial inspection scene can be realized, and the inspection efficiency and accuracy are improved.
Owner:NANJING MAGIC GRP INFORMATION TECH CO LTD

Intelligent image signal processing method and system based on multi-modal fusion

The invention provides an intelligent image signal processing method and system based on multi-modal fusion. According to the invention, a multi-mode input signal is received, is divided into a spatial distribution feature extraction region and a dynamic change trajectory capture region, and is decomposed into a penetrability feature layer and a substance reflection feature layer; carrying out association mapping on missing pixel information in the dynamic change trajectory capture region and energy distribution of the penetrability feature layer to generate enhanced dynamic trajectory data, identifying a determined reflection mode in the substance reflection feature layer, and carrying out frequency domain superposition on corresponding frequency band response and the spatial distribution feature extraction region; generating a composite spatial feature, then constructing a multi-modal joint optimization model, then adjusting the contribution ratio of the two data, and generating a fused image signal; the technical scheme provided by the invention not only solves the problems of detail loss, artifact generation and poor dynamic adaptability caused by single-mode limitation, but also improves the spatial resolution and tracking precision of the image signal.
Owner:BEIJING ZHAOKE HENGXING SCI & TECH CO LTD

Breast image classification evaluation method and system based on deep learning

The invention relates to the technical field of image classification, and discloses a breast image classification evaluation method and system based on deep learning, and the method comprises the steps: carrying out the preprocessing of sample breast image data, and obtaining comprehensive feature data; performing feature enhancement based on the comprehensive feature data to obtain enhanced feature data; performing multi-modal bidirectional feature fusion on the enhanced feature data to obtain multi-modal fusion feature data; based on the multi-modal fusion feature data, constructing and training a semi-supervised local feature reservation dense connection context-aware graph convolutional network to obtain a graph volume integral class model; according to the multi-modal fusion feature data, constructing and training a multi-branch residual attention reconstruction network to obtain a multi-scale feature fusion model; through the image volume integral class model and the multi-scale feature fusion model, classification evaluation and prediction result integration are carried out on the real-time mammary gland image data to obtain a target mammary gland image classification result, and the method improves the accuracy and reliability of mammary gland image classification.
Owner:THE FIRST AFFILIATED HOSPITAL OF JINAN UNIV

Multi-agent cooperation system and method based on spatial calculation and multi-modal AI fusion

The invention discloses a multi-agent cooperation system and method based on spatial calculation and multi-modal AI fusion, and the method comprises the steps: collecting all real-time data of a construction site through a multi-modal data collection and fusion unit, and carrying out the processing and integration of all collected data through a multi-modal AI algorithm, and generating a unified semantic association model; a three-dimensional virtual environment of a construction site is constructed through a space calculation unit, real-time synchronous mapping is formed for the construction site, and dynamic changes in the construction process are simulated and predicted; task allocation and behavior planning are carried out on multiple agents through an agent behavior management unit; and the intelligent agents are configured to perform real-time information sharing and task cooperation through a communication protocol through the multi-intelligent-agent cooperation unit. According to the scheme, the problems of difficulty in multi-source data integration, low cross-department cooperation efficiency, insufficient construction dynamic adjustment and the like in intelligent construction in the building industry can be solved.
Owner:CHINA CONSTR EIGHT ENG DIV CORP LTD

Intelligent psychological intervention system based on multi-modal fusion

The invention discloses an intelligent psychological intervention system based on multi-modal fusion, which is characterized in that a three-dimensional evaluation system is constructed by integrating speech sentiment analysis, keyboard dynamics monitoring and physiological signal acquisition, and time sequence alignment and feature weighted fusion of multi-source data are realized by adopting a cross-modal Transform model. The core of the system comprises an adaptive intervention engine which defines a multi-dimensional state space based on a hierarchical reinforcement learning architecture, optimizes an intervention strategy through a PPO algorithm, and realizes dynamic emotion interaction in AR and VR scenes in combination with a digital twin training module; according to the clinical decision support system, physiological behavior characteristics and psychological assessment trends are integrated by using a multi-time scale risk prediction model, and a personalized early warning threshold system is constructed, so that the psychological state recognition accuracy is improved, the intervention intensity self-adaptive adjustment response time is shortened, and the high-risk signal early warning timeliness reaches the minute level; and the problems of evaluation hysteresis and strategy stiffness of traditional psychological intervention are obviously improved.
Owner:JIANGSU ZHUODUN INFORMATION TECH CO LTD

Photovoltaic power station unmanned inspection method and system based on multi-mode fusion detection

The invention discloses a photovoltaic power station unmanned inspection method and system based on multi-mode fusion detection, and relates to the technical field of intelligent operation and maintenance of photovoltaic power stations. The method mainly comprises the following steps: (1) collecting multi-modal data of a photovoltaic module and carrying out preprocessing operation; (2) uniformly mapping the heterogeneous data into discrete marks by using each pre-trained modal data marker; (3) training a multi-modal fusion detection network based on a Transform encoder-decoder architecture by using the training sample, and adjusting network parameters to obtain a trained model; and (4) outputting a defect type, a position bounding box and a severity score by utilizing the trained model according to the obtained multi-modal data or single-modal data of the photovoltaic module. The problems that traditional single-mode detection is high in omission ratio and multi-mode fusion is low in efficiency are solved, the inspection collection module, the data processing module, the multi-mode fusion detection module and the fault decision module are provided in a targeted mode, and full-process automation from data collection to intelligent decision is achieved. The operation and maintenance cost of the photovoltaic power station is obviously reduced; and the fault response efficiency is improved.
Owner:ZHEJIANG UNIV

Emergency power supply state monitoring method and system based on multi-modal data

The invention discloses an emergency power supply state monitoring method and system based on multi-modal data, and relates to the technical field of power supply monitoring, and the method comprises the steps: obtaining the power supply attribute data of an emergency power supply; based on the sensor group, collecting multi-channel time sequence sampling data of the emergency power supply corresponding to a preset sampling time period in real time; performing data preprocessing on the multi-channel time sequence sampling data to obtain corresponding online monitoring time sequence data; extracting a multi-modal fusion feature group corresponding to the online monitoring time sequence data; and inputting the power supply attribute data and the multi-modal fusion feature group into a power supply state identification model to determine the operation health state of the emergency power supply, the operation health state including any one of the following state types: a normal state, an early warning state, a limited operation state and a fault state. Therefore, technical means such as multi-modal parameter acquisition, feature fusion analysis and model reasoning are comprehensively adopted, and real-time and accurate monitoring of the state of the emergency power supply is realized.
Owner:ZHUHAI TENGDUN TECH CO LTD

Inspection method, device and equipment based on digital airspace system and medium

The invention discloses an inspection method, device, equipment and medium based on a digital airspace system, and the method comprises the steps: carrying out the feature extraction and calibration of multi-source sensing monitoring data of a to-be-inspected region through a pre-trained neural network model; obtaining a structured semantic description and a time sequence measurement value to be combined with external meteorological spatial data to be input into a digital airspace expert model based on an MoE framework for multi-modal fusion; carrying out enhancement processing on the generated multi-modal fusion representation through a variational auto-encoder so as to input the multi-modal fusion representation into a multi-modal transformer decoder for decoding, and obtaining an entity triple set to construct a digital spatial domain knowledge graph; node characterization in the knowledge graph is enhanced by adopting an attention enhancement mechanism based on a graph neural network, the node characterization is input into a conditional generative adversarial network model for multi-task prediction, and an inspection decision suggestion is generated to be executed by the unmanned aerial vehicle, so that the utilization rate of multi-source sensing data is effectively improved, and the inspection efficiency of the unmanned aerial vehicle is improved.
Owner:STATE GRID ECONOMIC TECH RES INST CO LTD +1

Rare disease knowledge graph construction method based on modal injection and multi-modal fusion

The invention relates to the technical field of medical artificial intelligence and knowledge graph construction, in particular to a rare disease knowledge graph construction method based on modal injection and multi-modal fusion. Comprising the following steps: S1, collecting multi-modal medical information including texts, images and genes; s2, standardization processing is carried out, and a three-layer metadata structure is constructed; s3, complementing missing modal data, and performing feature extraction and unified dimension conversion on the modal data to realize representation alignment in a shared semantic space; s4, performing multi-level semantic fusion to obtain a unified fusion semantic vector; and S5, constructing a double-layer structure system rare disease knowledge graph comprising an ontology layer and an instance layer. According to the method, multi-modal medical information of texts, images and genes is selected to construct the knowledge graph of the rare disease, the application range, coverage and accuracy of the knowledge graph are improved, correspondence adaptation of rare cases during clinical diagnosis and treatment of the rare disease can be achieved, and the method has high recognition capacity.
Owner:湖南工商大学

Intelligent law data processing method and system based on big data

The invention discloses an intelligent legal data processing method and system based on big data, and the method comprises the steps: S1, constructing a legal data collection framework, and carrying out the unified coding and representation of different data forms through a multi-modal fusion technology; s2, performing hierarchical semantic analysis on the cross-domain legal clauses to generate a high-dimensional semantic representation vector with context sensing capability; s3, performing node embedding, association analysis and reasoning optimization on the constructed legal knowledge graph to generate a dynamically updated multi-domain legal association network; s4, mapping and optimization among terms are realized through semantic comparison and multi-level rule verification; s5, performing modeling analysis on the dynamic trend and the potential risk factors of the legal data, and outputting a risk prediction and early warning strategy of the legal event; and S6, improving cross-data-source cooperative computing capability and data analysis efficiency through a distributed optimization strategy. The method has the advantages of high semantic understanding depth, high risk prediction accuracy and high privacy protection and cooperative computing efficiency.
Owner:GUIZHOU DAIMA TECH CO LTD

Submarine cable risk dynamic assessment method and system based on multi-modal deep learning

The invention discloses a submarine cable risk dynamic assessment method and system based on multi-modal deep learning, and belongs to the field of marine infrastructure operation and maintenance. Aiming at the problems of incomplete data coverage, unreal generated scene, low evaluation reliability and the like in the prior art, the method comprises the following steps of: 1) constructing a multi-source heterogeneous data set containing six types of data including geology, ocean, ships, biology and the like, and realizing data alignment by adopting space-time grid coding; 2) designing a physical constraint generative adversarial network, and generating risk scene data conforming to a fluid mechanics law through a Navier-Stokes equation constraint; 3) creating a hierarchical space-time fusion network (HST-Transform), and combining CNN spatial feature extraction, a time sequence attention mechanism and a dynamic memory module to realize multi-modal fusion; according to the method, the detection rate of rare risk events is increased by 62%, the evaluation accuracy rate reaches 91.7%, the false alarm rate is reduced by 34% compared with a traditional method, and submarine cable breakage accidents can be effectively prevented.
Owner:GUANGDONG POWER GRID CO LTD

Distributed intelligent authentication method based on dynamic multi-modal fusion

A distributed intelligent authentication method based on dynamic multi-modal fusion relates to the field of network security, and adopts an alliance chain + DAG hybrid block chain architecture, combines a threshold signature to realize secret key fragment management, and switches among PBFT, Raft and probabilistic algorithms through a dynamic consensus mechanism to improve authentication efficiency. The multi-mode authentication module is based on a dynamic weight distribution algorithm, integrates biological characteristics, behavior analysis, equipment fingerprints and environmental factors, and combines an LSTM-GAN model and a quantum random number driven challenge-response mechanism to realize zero-trust verification under environmental perception. The session management module generates a session key by using a chaotic mapping algorithm. In the aspect of privacy protection, CKKS homomorphic encryption, zero-knowledge proof and attribute-based encryption are fused. According to the method, the block chain technology, the secure multi-party computing technology, the machine learning technology and the quantum cryptography technology are fused, and a high-performance, high-security and strong-privacy-protection distributed authentication solution is provided.
Owner:JINLING INST OF TECH

Multi-modal classroom emotion recognition method and system based on modal adaptive learning

The invention belongs to but not limited to the technical field of emotion recognition, and discloses a multi-modal classroom emotion recognition method and system based on modal adaptive learning, and the method comprises the steps: building a multi-modal data collection system, collecting the multi-modal data, such as texts, audios and videos, in a classroom teaching process in real time, and carrying out the real-time collection of the multi-modal data; and preprocessing and feature extraction are carried out on data of different modes. For emotion expression feature differences of different classroom scenes, a multi-modal data fusion method based on adaptive modal scoring (AMS) is provided, and a time sequence modeling and emotion classification model is constructed. The multi-modal fusion and time sequence modeling mechanism simulates the multi-dimensional perception of the teacher on the emotional states of the students in the class, and enhances the interpretability of the emotion recognition system. According to the system, weighted fusion is carried out on data features from texts, audios and videos through fusion of multi-modal data, and complete emotion feature representation is formed.
Owner:HUAZHONG NORMAL UNIV

Real-time health risk prediction method and system based on dynamic knowledge graph

The invention discloses a real-time health risk prediction method and system based on a dynamic knowledge graph, and relates to the technical field of medical information. The method comprises the following steps: carrying out multi-modal fusion and privacy protection preprocessing on medical and nursing heterogeneous data, and realizing semantic consistency of cross-mechanism data based on an entity alignment method of a cross-modal graph neural network; based on a hierarchical federated learning framework, local model parameters are subjected to hierarchical encryption and aggregation through a secure multi-party computing protocol to generate an initial global model, and prediction distribution of the global model is optimized in combination with knowledge distillation of differential privacy constraints; designing a gradient difference dynamic updating trigger mechanism of noise robustness, smoothing noise interference through a sliding window mean value, and realizing adaptive threshold calibration through linkage model performance verification; and light-weight deployment real-time reasoning is realized based on redundant edge pruning of confidence and 8-bit symmetric quantization. On the premise of protecting data privacy, the real-time performance and accuracy of health risk prediction are remarkably improved, and the method is suitable for a cross-institution medical care collaborative decision-making scene.
Owner:GERIATRIC HOSPITAL AFFILIATED TO WUHAN UNIVERSITY OF SCIENCE & TECHNOLOGY

Adaptive teaching real-time feedback method based on multi-modal fusion

The invention discloses an adaptive teaching real-time feedback method based on multi-modal fusion, and the method comprises the following steps: synchronously collecting text modal information, voice modal information and image modal information generated by students in a teaching process, forming multi-modal original data information, and extracting historical student interaction behavior data; preprocessing the multi-modal original data information, and respectively generating corresponding text, voice and image sequence features; a visual feature encoder and a sequence feature encoder are adopted to encode each modal sequence feature to obtain a high-dimensional feature; inputting the modal high-dimensional features into a cross-modal fusion network for deep fusion; parameters of the feedback model are optimized through a model-independent element learning feedback regulation and control algorithm, and a personalized feedback strategy is generated; generating comprehensive feature representation according to the fusion features, and outputting personalized teaching feedback; and the interaction information is updated based on the feedback behavior data to realize closed-loop optimization.
Owner:JIANGSU LINGSHU YOUZHI TECHNOLOGY CO LTD

Intelligent analysis method based on medical document structure perception and multi-modal fusion

An intelligent analysis method based on medical document structure perception and multi-modal fusion comprises the following steps: carrying out structure topology modeling on a medical document, extracting visual layout, text meta-information, space coordinates and semantic keyword features, constructing a semantic topological graph and dynamically shielding irrelevant contents; selecting an extraction path according to a document type, performing deep semantic analysis and entity recognition on a text-type document, and performing visual enhancement OCR recognition on a scanning-type document; the features are injected into a medical knowledge graph, and feature fusion, semantic verification, relation reasoning and information completion are achieved through a graph neural network; a three-stage strategy optimization model of basic pre-training, domain adaptation and online reinforcement learning is adopted; and large-scale processing is realized through a dynamically aggregated distributed architecture. The method is used for intelligent analysis and structured conversion of documents of hospitals, medical insurance and medical scientific research. The problems that heterogeneous medical document analysis adaptability is poor, multi-modal fusion is difficult, medical knowledge utilization is insufficient, and large-scale processing efficiency is low are solved.
Owner:NORTHWEST UNIV

Multi-modal medical image fusion and disease prediction method, computer program and terminal

The invention discloses a multi-modal medical image fusion and prediction method, a computer program and a terminal, and belongs to the technical field of image processing. A feature fusion model and an explicit classifier guided diffusion model are constructed; training a diffusion model guided by an explicit classifier; and extracting fusion feature information of different modal medical images, inputting the fusion feature information into the trained explicit classifier guided diffusion model, and removing noise by using a noise prediction network to generate a multi-modal fusion image. The diffusion model guided by the explicit classifier is applied to the field of medical image fusion, guidance information of the classifier is explicitly utilized in the data recovery process, it is ensured that the generated image strictly follows the preset conditions in all stages, the problem that an implicit diffusion model cannot ensure that the image consistent with the expected condition is generated is solved, and the image fusion efficiency is improved. Therefore, the consistency and coherence of the generated image are ensured, and a clear multi-modal fusion image is generated.
Owner:UNIV OF ELECTRONICS SCI & TECH OF CHINA

Medical image enhancement method and system based on multi-modal fusion

The invention discloses a medical image enhancement method and system based on multi-modal fusion, and the method comprises the steps: obtaining medical image data which comprises a CT image, an MRI image, a PET image and an ultrasonic image; calculating a mutual information value of each semantic tag in the cross-modal feature vocabulary, and generating an inter-modal feature correlation matrix; performing non-rigid space registration on the unified dimension image set to generate a geometrically consistent multi-modal image set; extracting skeleton region features, soft tissue region features and high metabolism region features from the multi-modal image set, and generating a core feature set; a deep learning algorithm is adopted to train the core feature set, and a segmented image set containing a segmentation mask is generated; and dynamically adjusting the fusion weight according to the regional features of the segmented image set, and generating a fusion enhanced image. According to the invention, through feature extraction, spatial registration and deep learning fusion of the multi-modal medical image, effective integration of different-modal medical image information is realized.
Owner:CHANGSHU INSTITUTE OF TECHNOLOGY

Industrial equipment communication method and system based on industrial internet

The invention discloses an industrial equipment communication method and system based on the industrial internet, and the method comprises the following steps: S1, collecting the multi-modal data of industrial equipment, and generating a preliminary multi-modal fusion feature; s2, performing event-driven analysis on the preliminary multi-modal fusion features to generate event sensing data; s3, constructing a semantic-driven communication protocol, and generating a semantic data queue with a priority identifier; s4, compressing the semantic data queue and carrying out time consistency calibration; s5, optimizing a communication frequency and bandwidth resource allocation strategy in combination with a mixed firefly optimization algorithm and a dynamic differential evolution algorithm; s6, constructing an industrial equipment communication topology model, and predicting path delay and bandwidth occupation indexes of the communication network in combination with the space-time multi-task network; and S7, constructing a zero-sum game model to analyze the potential confrontation relationship in the communication process, and outputting a secure communication scheme. According to the invention, semantic-driven communication, a hybrid optimization algorithm and the like are adopted, and intelligent communication of industrial equipment is realized.
Owner:刘涌滨

Cognitive state interpretable method based on brain-language-vision large model

The invention belongs to the technical field of learner cognitive state interpretation, and particularly discloses a brain-language-vision large model-based cognitive state interpretable method, which comprises the following steps of performing multi-modal data collection for the cognitive state of a learner; performing learner cognitive state characterization based on a multi-scale map convolutional network; brain-vision-language mixed multi-modal time sequence feature fusion based on hierarchical mutual information maximization is carried out; embedding knowledge graph nodes for enhancing the complex task understanding ability of the multi-modal large model; training a brain-vision-language multi-modal large model guided by knowledge; recognizing a learner cognitive state based on a multi-modal large model; the cognitive state of the learner can be explained and analyzed under the support of multi-mode owned reasoning. According to the cognitive state interpretable method based on the brain-language-vision large model, the input multi-modal fusion features are converted into high-quality prompts so as to guide the large language model to extract implicit knowledge, and important support is provided for reasoning generation.
Owner:ZHEJIANG NORMAL UNIV