777 results about "Semantic space" patented technology

The invention discloses a judicial scene-oriented multi-modal data fusion method and system. The method comprises the following steps: 1) collecting multi-modal data in a judicial application scene and converting the multi-modal data into data in a uniform format; 2) mapping the data to a unified semantic space to realize cross-modal alignment; then associating the multi-modal data to obtain the text description of the same case and the corresponding image evidence as the multi-modal features of the corresponding case; 3) constructing a knowledge graph of a judicial application scene based on the multi-modal features; driving multi-modal data fusion based on the knowledge graph and constructing a multi-modal evidence chain of each entity; 4) quantifying the integrity of the knowledge graph and the reliability of the multi-modal evidence chain according to a preset index, marking abnormal nodes in the knowledge graph according to a quantification result, and adjusting the abnormal nodes; 5) generating a structured knowledge graph according to the knowledge graph and creating a dynamic desensitization report; edges in the structured knowledge graph represent relationships between legal entities, and each relationship is bound with a multi-modal evidence chain.

The invention discloses a multi-label electrocardiogram classification method based on self-supervised pre-training and multi-modal semantic alignment, which belongs to the technical field of artificial intelligence, and comprises the following steps: realizing self-supervised pre-training of unlabeled data through a single-modal contrast enhancement network, generating global and local contrast views by adopting a multi-scale random cutting strategy, and classifying the global and local contrast views in a multi-scale random cutting mode; in combination with a teacher-student network architecture, the potential invariance features of the ECG signals are learned while negative sample dependence is avoided, the problem of annotation data scarcity is effectively relieved, and the feature robustness is improved. A multi-modal fusion mechanism based on label semantic guidance is provided, a time domain signal and a frequency domain time-frequency graph are mapped to a unified semantic space through fine-grained semantic alignment, local feature enhancement and cross-modal complementary information fusion are realized by using a cross attention mechanism, and the problem of semantic difference caused by modal heterogeneity in a traditional method is overcome. A multi-label comparison loss function based on a disease co-occurrence relation is proposed, a category discrimination boundary is dynamically optimized by modeling a label co-occurrence probability, the feature separability of a tail category is improved while the head category discrimination ability is enhanced, and the problem of sample category imbalance in a multi-label scene is remarkably relieved.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes such as financial science and technology and medical health, and discloses a decision-making method and device guided by a multi-modal semantic map, equipment and a medium. Extracting a visual feature vector, a language feature vector and an action feature vector, splicing to generate a multi-modal initial feature, mapping the multi-modal initial feature to a shared semantic space, constructing a multi-modal semantic map, and inputting a map-guided attention mechanism to generate a cross-modal alignment feature; the cross-modal alignment features and task targets are input into a meta-learner to generate task adaptability features, the task adaptability features are input into a parallel reasoning network to execute subtasks in parallel, and a gating fusion network integrates output results to generate a global decision. According to the method, cross-modal semantic association and task adaptability are enhanced through the combination of shared semantic space mapping, map guiding attention and a meta learning device, and the accuracy and efficiency of multi-modal decision making are improved through the combination of parallel reasoning and gating fusion.

According to the multi-modal automatic knowledge graph construction method based on the large language model, a multi-modal data stream is preprocessed, features are extracted, and the multi-modal data stream is mapped to a unified semantic space through a cross-modal alignment network after being processed through the large language model, a visual converter and a time sequence neural network. In the space, entities and categories are recognized based on a large language model, a triple is generated by combining a multi-modal feature judgment entity relationship, mapping fusion is performed through an ontology alignment algorithm driven by a graph neural network and a predefined domain ontology, finally knowledge is stored in a graph database, and dynamic updating is performed by means of incremental learning and online reasoning. Standardized APIs and visualization components are provided. According to the method, the construction efficiency and the automation degree of the knowledge graph are remarkably improved, the cross-modal information fusion and knowledge maintenance capability is enhanced, and the application requirements of intelligent retrieval, recommendation, decision support and the like are met.

The invention discloses an intelligent planning method and system for a weak current system in a smart park, and belongs to the technical field of weak current intelligent design. The method comprises the steps of performing feature extraction on the weak current multi-source data of the smart park to form a weak current feature set; a multi-dimensional semantic space is constructed, semantic association features are obtained, and node features, topological relations and constraint rules of the weak current system are determined; generating a weak current knowledge graph based on the information, and performing semantic alignment on the basic information of the park to obtain a final scene demand representation; performing graph reasoning and constraint calculation according to the representation to obtain a feasible region and constraint satisfaction condition, and generating a candidate construction scheme; and screening out an optimal construction scheme from the candidate schemes according to a preset comprehensive optimization strategy and sending the optimal construction scheme to a control center. According to the scheme, the weak current scheme is promoted from demand understanding to scheme optimization, and a coherent and verifiable automatic process is formed; therefore, the manual intervention is less, the design judgment is more accurate, and the finally output construction scheme has higher engineering reliability.

The invention discloses a VLM model intelligent decision-making-based driving method and device and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: extracting key visual features from continuous multi-frame driving scene images based on a preset visual feature extraction algorithm; inputting a user instruction and a time sequence visual Token corresponding to the key visual features into a preset VLM model for multi-modal alignment, and generating a target planning Token; inputting the target planning Token and the time sequence vision Token into a preset trajectory generation model to obtain a predicted trajectory; and converting the predicted trajectory into a control signal, and controlling the mobile device to complete a moving action based on the control signal. The problem of modal difference between a semantic space and an action space is solved.

The invention relates to a knowledge graph agent construction method and system based on multi-modal fusion, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring multi-modal data, and mapping different modal data to a unified semantic space through a cross-modal embedding technology to obtain multi-modal reference data; extracting features of the multi-modal reference data through an attention mechanism to obtain multi-modal joint features; constructing and updating a dynamic knowledge graph based on the multi-modal joint features to obtain a time sequence dynamic dependency knowledge graph; and according to the time sequence dynamic dependency knowledge graph, context-aware reasoning is carried out through a cross-modal reasoning engine to obtain a user input mode, and an interaction strategy is dynamically adjusted based on the user input mode to generate a multi-modal response. And the knowledge graph agent construction based on multi-modal fusion is realized.

The invention discloses an intelligent digital human training method and system based on multi-modal interaction, and belongs to the technical field of semantic indexing.The method specifically comprises the steps that voice, vision and text data are analyzed and converted into high-dimensional feature vectors through a modal exclusive encoder, the high-dimensional feature vectors are projected to a unified semantic space through a cross-modal semantic mapping model, and the high-dimensional feature vectors are obtained; generating a semantic primitive containing a modal identifier, a core semantic tag and a feature weight; semantic primitives are used as nodes, directed edges and edge weight table association strength are established based on semantic similarity, typical scene node connection weights are strengthened, and a mesh map containing intra-modal hierarchy and inter-modal cross association is formed; constructing a double-layer index on the basis of the mesh map; semantic primitives are extracted from newly added data, the position of a new node in an association graph is determined through a graph matching algorithm, an association edge with an existing node is automatically established, and a lower-layer modal exclusive index is synchronously updated.

The invention belongs to the technical field of network security, and provides an abnormal traffic detection and attack recognition method and system based on deep learning, and the method comprises the steps: data preprocessing and feature extraction, cross-modal semantic alignment and knowledge graph construction, causal enhancement association reasoning, intelligent engine optimization, cloud edge collaborative resource scheduling, and result output. According to the method, statistical features and signature features are mapped to a unified semantic space through a cross-modal semantic alignment and knowledge graph construction module, a semantic barrier between heterogeneous features is broken through, time sequence causal discovery and transfer entropy calculation are introduced, a simple correlation and a reliable causal can be distinguished, and the method has a good application prospect. According to the method, the accuracy and credibility of attack chain reasoning are improved, the false alarm rate is reduced, online self-evolution of a detection model and dynamic optimal allocation of system resources are realized through intelligent engine optimization and cloud edge collaborative resource scheduling modules, and the overall adaptability, robustness and practicability of the system are enhanced.

The invention discloses a listed company operation risk early warning method based on multi-source auditing and text semantic fusion, and relates to the technical field of auditing, and the method comprises the steps: S1, crawling and converging multi-source heterogeneous data of listed company financial newspapers, auditing suggestions, supervision announcements, inquiry letters, news public opinions and market transactions; according to the method, unstructured texts are subjected to cleaning, blocking and semantic vectorization processing, each text segment is embedded into a high-dimensional semantic space, a vector index is established, a bottom-layer knowledge base of an RAG framework is formed, in the stage, it is ensured that the data structure is uniform, the source is traceable, standardized input is provided for subsequent semantic retrieval and modeling, and the reliability of the system is improved. S2, a query expression is constructed based on a target company, a time window and a risk topic, dense semantic retrieval and sparse BM25 retrieval methods are comprehensively used, a time decay and source credibility weighting mechanism is introduced, and the problems that a traditional method is single in data dimension and information is split are solved.

The invention provides a multi-modal pre-training model construction method and system for monitoring videos, and the method comprises the steps: automatically constructing a high-quality multi-modal alignment data set through a single-modal description generation model and a large-scale language model, and remarkably reducing the marking cost; a special coding network and a shared projection layer are adopted to realize feature extraction and uniform semantic space alignment of video, audio and text modes; performing dynamic semantic fusion by using a modal collaborative attention mechanism; designing cross-modal contrast learning, mask prediction and time sequence consistency tasks to carry out multi-task pre-training; an external knowledge base is introduced, and the semantic reasoning ability is enhanced through a microretrieval mechanism; and optimizing model parameters by adopting a multi-task joint loss function and an end-to-end training strategy. According to the method, efficient and automatic construction, deep semantic alignment and fusion and intelligent reasoning of knowledge enhancement of monitoring video multi-modal data are realized, and the understanding and generalization ability of the model in a complex scene is effectively improved.

The invention discloses a file retrieval method, device and equipment based on multi-modal AI and a storage medium, relates to the technical field of data retrieval, and aims to solve the problem that traditional file retrieval is low in efficiency and precision. The method comprises the following steps: performing content analysis on a multi-modal file including a picture file, a video file and a document file of text and / or visual information, and extracting a text semantic feature, a picture visual feature and a video time sequence feature; mapping the text semantic feature, the picture visual feature and the video time sequence feature to a cross-modal semantic space used for representing a high-dimensional vector associated with different modal features, and generating a cross-modal association vector; according to the modal type of the retrieval information, a corresponding retrieval module in a mixed retrieval engine is called to conduct retrieval in a cross-modal semantic space, an initial retrieval result is obtained, and the mixed retrieval engine comprises a text retrieval module, a visual retrieval module and a cross-modal fusion module; and performing dynamic weight distribution sorting on the initial retrieval result, and outputting a target matching result.

The invention relates to the technical field of data fusion, and discloses an animal scene-oriented adaptive multi-modal data fusion method, which comprises the following steps of: extracting spatio-temporal characteristics from multi-source heterogeneous data such as visual sense, auditory sense and physiological sensing, constructing an animal-environment-group ternary spatio-temporal relation graph, and constructing an animal-environment-group ternary spatio-temporal relation graph; a pilot frequency sampling problem is solved through an adaptive interpolation algorithm, cross-modal projection alignment is completed in a public semantic space, unified space-time representation is output, and confidence coefficient weight is dynamically calculated based on uncertainty measurement of each modal feature. According to the method, accurate alignment of multi-modal data is realized through the cross-modal space-time attention network, the multi-modal feature alignment error is reduced compared with that of a traditional LSTM method, the training data volume of a federated element migration reinforcement learning framework is reduced compared with that of a traditional migration learning method, and the cross-species generalization performance of the model is improved. A multi-level causal inference engine quantitatively reveals causal association between environmental factors and animal diseases, and in combination with a dynamic decision tree visualization technology, the decision recognition degree is improved.

The invention discloses a multi-modal content intelligent auditing and violation detection method and system, and relates to the technical field of information processing. The method comprises the following steps: carrying out audio-picture separation on a video stream, and carrying out parallel processing on audio-to-text and visual key frame extraction; a space-time encoder is constructed to record the corresponding relation of the time stamps of all the modes; constructing a multi-modal resource target dictionary, and forming an inter-entity knowledge graph by using relation categories; calculating a confidence coefficient difference index between modals by comparing, learning and training the shared semantic space; and carrying out violation judgment, triggering a sensitive characteristic threshold value for any mode, and starting multi-mode evidence cross validation. According to the method, audio and picture separation is carried out on the video stream, parallel processing is carried out on audio-to-text and visual key frame extraction, a multi-modal resource target dictionary is constructed, violation judgment is carried out according to confidence coefficient difference indexes among modals, and false information auditing efficiency and detection efficiency are improved.

The invention discloses a multi-modal data fusion and intelligent analysis platform and method for an unmanned aerial vehicle inspection system for a wind power plant. The platform comprises a multi-modal data acquisition module, a feature extraction and standardization module, a multi-modal information fusion module, a joint learning and optimization module, a domain knowledge injection module and an intelligent decision and application module. The system processes multi-source heterogeneous data through an integrated learning and deep learning fusion strategy, projects features to a shared semantic space by using joint training and comparative learning to enhance the anomaly discrimination ability, and performs verification and semantic enhancement on a supervised retrieval result in combination with a knowledge base in the wind power field. And finally, outputting a high-reliability diagnosis report and a maintenance suggestion. According to the invention, accurate identification and positioning of the fan fault are realized, and the inspection efficiency and the system decision reliability are significantly improved.

The invention provides a multi-modal large model incremental training data screening method, and relates to the technical field of data processing, and the method comprises the steps: executing modal structure analysis on newly added multi-modal data, extracting each modal vector, calculating a semantic matching degree, and removing samples lower than a preset first threshold value; calculating a multi-level semantic distance between a sample embedding vector and a historical clustering center in a unified semantic space, and dividing a core semantic region sample, a boundary semantic region sample and a discrete semantic region sample according to the change rate of the multi-level semantic distance; performing semantic fine-grained alignment on the boundary semantic region samples, when multimodal unstable distribution is detected, executing local context reconstruction to repair semantic deviation, and if the multimodal unstable distribution is still unstable, removing the semantic deviation; performing multiple rounds of small-batch reasoning, calculating a semantic stability coefficient based on a semantic prediction result, and when the semantic stability coefficient is lower than a preset second threshold value, determining that the sample is a potential drift sample and removing the potential drift sample; constructing an incremental training data set; according to the method, the autonomy and accuracy of incremental training data screening are improved.

The invention discloses a large model industry knowledge question-answering method and system supporting multi-modal input, and the method comprises the steps: obtaining a multi-modal query, and converting the multi-modal query into a corresponding feature set; importing the feature set into a preset semantic alignment and enhancement network to obtain enhanced semantic representation; the preset semantic alignment and enhancement network performs semantic alignment and fusion on features of different modals in a shared semantic space through a cross-modal attention mechanism, a multi-modal consistency constraint and a modal complementarity constraint; matching corresponding multi-modal knowledge fragments from the multi-modal industry knowledge base on the basis of enhanced semantic representation; and importing the multi-modal query, the enhanced semantic representation and the multi-modal knowledge fragment into a pre-training large model to generate a question and answer result. According to the method, semantic alignment and deep fusion are performed on the multi-modal query by establishing the semantic alignment and enhancement network, and the professional knowledge range is expanded by introducing the multi-modal industry knowledge base, so that the retrieval efficiency can be improved, and the question and answer generation quality can be improved.

The invention discloses a multi-channel advertisement effect prediction and optimization method and system based on artificial intelligence. The method comprises the following steps: obtaining and preprocessing advertisement putting data, user behavior data and external environment data; performing feature extraction on the preprocessed data, mapping different types of content feature vectors to a unified semantic space through a cross-channel feature projection matrix, and generating an advertisement feature set; fusing a prediction model of Transform and a graph neural network, and synchronously predicting a conversion rate, a delivery return rate and a user interaction index of the target advertisement in each channel based on the advertisement feature set; constructing a strategy optimization network based on reinforcement learning, taking a prediction result output by the prediction model as state input, and dynamically generating a budget allocation matrix and a channel selection weight through a strategy gradient algorithm; and pushing the adjusted advertisement putting strategy to an advertisement management platform according to the optimization strategy, and executing advertisement putting. The accuracy and stability of advertisement effect prediction can be improved.

The invention discloses a joint modeling method based on a knowledge graph and large model retrieval enhancement. The joint modeling method comprises the following steps: S1, constructing an aviation electric fitting knowledge graph; s2, performing semantic indexing on the knowledge graph based on a pre-training language model and a graph neural network, generating an embedded representation fusing node attributes and a high-order neighborhood relationship, and establishing a knowledge vector database; s3, based on a self-adaptive retrieval strategy, dynamically judging whether to trigger knowledge graph retrieval or not according to the question type; s4, performing correlation reordering on the retrieved knowledge sub-graphs by adopting an attention mechanism; and S5, mapping the structured knowledge graph to a semantic space through an encoder and a projection layer, generating a soft prompt, splicing the soft prompt to an LLM input sequence, and generating a structured enhanced precise response.

The invention discloses a text and image fused online comment toxicity detection and filtering method and system, and relates to the technical field of natural language processing, and the method comprises the steps: extracting a semantic vector of a comment text through a RoBERTa-Marge model; the visual features of the image are extracted through an OfficientNet-V2 model; aligning heterogeneous modal features by adopting a double-flow contrast loss function; self-adaptive decision making of culture sensitivity: loading a regional sensitive rule table according to a user IP address, and dynamically adjusting symbolic semantics; calculating an intimacy correction factor based on the social relationship between the publisher and the receiver; and performing context weighted toxicity scoring, calculating a user historical behavior weight, and outputting a final toxicity probability. According to the method, a deep dynamic mapping mechanism of text and visual features is constructed, heterogeneous features are extracted through RoBERTa-Large and OfficientNet-V2 double-flow architectures, semantic space alignment is forced by utilizing comparative learning, and the problem of image-text splitting detection in the traditional technology is solved.

The invention discloses a knowledge base knowledge association fusion method based on a knowledge graph, and the method comprises the steps: integrating structured, semi-structured and non-structured data through a cross-modal alignment technology, and constructing a multi-source heterogeneous data association network of a unified semantic space; newly added external data are fused to a multi-source heterogeneous data association network in real time by using a dynamic attention mechanism, and entity conflicts are eliminated by combining rule reasoning and a machine learning model; hidden association among entities in the multi-source heterogeneous data association network is mined based on the graph neural network, and a knowledge graph logic chain is complemented; based on the knowledge graph, intelligent question and answer and risk assessment decision scenes are supported through a hybrid retrieval architecture and an inference engine; and automatically updating and associating the knowledge base of the network extension knowledge graph by adopting an incremental learning technology. Natural language questions and answers are supported, accurate answers are generated through knowledge reasoning, and user experience is remarkably enhanced.

The invention provides a network situation awareness method based on a heterogeneous graph neural network, and the method comprises the steps: carrying out the cleaning and standardization of structured data, constructing a unified semantic space based on topic prior, and forming a robust sample representation through employing a cross-modal alignment and attention fusion method; time, entities and link relations of different platform data are aligned, a'trigger-argument-role 'triple is extracted, and explicit / implicit propagation evidences are fused to construct a directed propagation graph; candidate paths are obtained through causal consistency, key links are screened through multi-path reliability and significance test, and provenance positioning and cross-platform association are achieved. According to the invention, network situation awareness with a localizable origin, a traceable path and an explainable result can be realized in a multi-source heterogeneous and high-noise scene; robustness is improved through significance control and multi-path fusion, decision reliability is improved through calibration and uncertainty propagation, and the method can be applied to scenes such as public safety monitoring, public opinion research and judgment and compliance risk control.

The invention discloses a fusion semantic vector space mapping-based translation ambiguity term accurate matching method, which comprises the following steps of: S1, obtaining source language ambiguity terms, context texts and a target language candidate translation list, and extracting domain tags and term matching features to form a multi-modal data set; s2, using improved XLM-R model coding to generate term-level, sentence-level and translation-level semantic vectors; s3, training a dynamic mapping matrix based on a bilingual parallel corpus, and aligning source side vectors to a shared semantic space; s4, fusing the source-side basic vector and the multi-dimensional features through a double-channel attention fusion network, and generating source-side and translation-side comprehensive semantic vectors; s5, introducing term-context attention weight to correct cosine similarity; and S6, outputting an optimal translation through normalized sorting and part-of-speech secondary judgment. According to the method, multi-field ambiguous term accurate matching is realized, the term translation precision and efficiency in professional fields are improved, and the requirements of high reliability of term translation in the fields of medicine, machinery, computers and the like are met.

The invention discloses a data security risk assessment method and system based on a large model, and relates to the field of artificial intelligence large models. The method comprises the following steps: firstly, preprocessing a benchmarking and checking material, and then encoding the preprocessed benchmarking and checking material and a standard specification into a unified semantic space by utilizing a pre-training language model to form a benchmarking and checking vector database; then, on the basis of standard specifications, a data security risk assessment vertical large model is utilized to construct a compliance and security risk analysis assessment item prompt, and a structured assessment problem library is formed after manual verification; and then respectively calling each evaluation item prompt in the structured evaluation problem library, carrying out semantic retrieval in the benchmarking check vector database to generate an enhanced prompt, and then carrying out multi-stage reasoning to obtain a data security risk evaluation result. Through the pre-training language model and the data security risk assessment vertical large model, automatic and intelligent data security risk assessment is realized.

The invention discloses an equipment safety early warning method and equipment based on multi-modal data and a medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: collecting industrial equipment multi-modal data, and preprocessing the multi-modal data to obtain standard multi-dimensional equipment data; performing feature extraction on the standard multi-dimensional equipment data to obtain a multi-dimensional equipment feature vector; based on joint semantic coding, mapping the multi-dimensional device feature vectors to a unified semantic space, and performing weighted fusion on the multi-dimensional device feature vectors by using a gating mechanism to obtain device fusion features; based on the equipment fusion features, constructing a risk association map to obtain a predicted risk conduction path; and based on the risk conduction path, triggering graded early warning, and generating a visual decision report. According to the method, the risk conduction path is predicted by constructing the risk association map, and the risk of the industrial equipment is accurately predicted and actively prevented and controlled.

The invention relates to a scientific research path generation method and system of a big language model based on knowledge graph enhancement, and belongs to the field of artificial intelligence. The method comprises the steps that a literature data set is analyzed based on a large language model, and a heterogeneous knowledge graph fusing knowledge triples and evidence metadata is constructed; performing self-supervised training on the atlas through a heterogeneous graph neural network to generate a knowledge embedding matrix; designing a semantic aligner to embed the map and align the map with the semantic space of the large language model; searching seed nodes according to user query and extracting context sub-graphs; converting the sub-graph into a graph lexical element sequence; and constructing a mixed prompt input large language model in combination with a natural language instruction, and generating a structured scientific research path. According to the method and the system, the quality and the credibility of a scientific research path can be accurately found, a literature reading sequence and an experiment reproduction sequence are clarified, and a more efficient technical engine is provided for knowledge discovery.

The invention provides an osteoporosis auxiliary diagnosis method fusing CT image features and a semantic knowledge graph, and the method is characterized in that the method comprises the steps: multi-source heterogeneous data collection and standardization processing; constructing a modal exclusive depth coding network; mapping and aligning a unified semantic space; carrying out multi-level cross-modal attention fusion; self-adaptive segmentation and feature enhancement of anatomical perception are carried out; semantic reasoning guided by the knowledge graph; performing multi-task collaborative diagnosis output; and training optimization and interference elimination. Through automatic multi-modal analysis and intelligent diagnosis, the workload of doctors in the imaging department is remarkably relieved, and the diagnosis time of a single example is shortened to be within 3 minutes from the average 15-20 minutes. The standardized diagnosis process of the system improves the diagnosis consistency among different doctors, reduces the diagnosis deviation caused by experience difference, and helps primary hospitals to improve the osteoporosis diagnosis and treatment level.

The invention discloses a real-time data analysis method and system based on multi-modal semantic mapping, and the method comprises the steps: obtaining input event information containing a multi-modal data flow, and carrying out the preprocessing of the multi-modal data flow; performing space-time alignment on each piece of modal data in the preprocessed multi-modal data stream, extracting modal features of each piece of modal data by using a pre-trained multi-modal encoder, and mapping the modal features to a unified semantic space; dynamically adjusting the weight of each modal feature by using a routing network according to the event information, and fusing each modal feature based on the weight to obtain a fused feature; and dynamically analyzing the cluster structure change of the fusion feature distribution by using a clustering algorithm, carrying out anomaly detection by using an outlier analysis method according to the cluster structure change, and if an anomaly triggering condition is met, generating an alarm signal.

The invention provides a cross-modal joint contrast learning method and device and electronic equipment, and the method comprises the steps: constructing a sample data set which covers a plurality of task types, such as a text retrieval image, an image retrieval text, a text retrieval text, an image retrieval image, an image-text joint retrieval image and an image-text joint retrieval text; and generating prompt words for identifying task types for each task sample, splicing the prompt words with sample data to form task input, and determining modal types of a retrieval object and a retrieval target. A retrieval object and a retrieval target are respectively input into encoders of corresponding modes to extract features, the features are mapped to the same semantic space through a unified projection layer to obtain an embedded vector, and the parameters of the encoders and the projection layer are optimized by utilizing a contrast learning loss function based on the similarity of the retrieval object and the retrieval target, so that multi-task unified training is realized. Various cross-modal retrieval tasks can be supported in a unified semantic space at the same time, and the overall retrieval effect is improved on the premise of ensuring multi-task performance balance.

The invention discloses a multi-mode federated learning method and system, computer equipment and a readable storage medium, and belongs to the technical field of federated learning. The multi-modal federated learning method comprises the following steps: on each client node, mapping local data of various modals into a plurality of vectors in a unified semantic space, determining an incidence matrix of the data of the various modals, and fusing the plurality of vectors according to the incidence matrix to obtain a local semantic vector; training a local model by using the local semantic vector to obtain local model parameters, and uploading the local model parameters to a server; on the server, identifying the difference degree between the data distribution condition of each client node and the global data distribution condition, and determining the node weight vector of each client node; and performing weighted aggregation on the corresponding local model parameters by using the node weight vector of each client node to generate global model parameters for next federated learning. Therefore, the performance of the training model can be improved.