310 results about "Modal data" patented technology

The invention discloses a heterogeneous data conversion method and system based on a multi-modal large model, and aims to solve the problems of single-modal limitation, high migration cost and the like in traditional heterogeneous data processing. The method comprises the following steps: firstly, carrying out format adaptation and classification preprocessing on multi-source heterogeneous data such as images, texts, audios and videos; then, through a ViT, BERT and Transform cross attention mechanism, deep analysis and cross-modal semantic association of visual, text and audio features are realized, and visual-text-audio semantic mapping is established; and based on a dynamic prompt project, guiding the multi-modal large model to generate structured data according to a preset Schema zero sample. And then error correction and format normalization are carried out through a Drools rule engine to ensure that the data meet the business standard. And finally, supporting multi-format output of JSON / XML / database table and the like, and meanwhile, associating original data storage to realize data traceability. According to the method, the limitation of single-modal processing is broken through, and cross-modal data deep association is realized.

The invention discloses an integrated sensor multi-modal data edge computing system optimized by a large model. The integrated sensor multi-modal data edge computing system comprises a data preprocessing module which obtains pre-training parameters of each modal data through the large model; the modal specific coding module optimizes attention weight through an encoder branch to determine key information of data of each modal, and advanced feature representation of each modal is obtained; the fusion strategy selection and execution module is used for calculating the correlation between the rest modes and the final core mode, and inputting the advanced feature representation of the mode with the strongest correlation in the rest modes and the advanced feature representation of the final core mode into a cross-mode encoder to generate the fusion feature representation of the fused modes; continuing to perform modal fusion operation until all modalities are fused, and generating final fusion feature representation; and the cross-modal coding and task processing module finally generates a multi-modal fusion model prediction result through cross-modal feature representation. According to the invention, the performance and adaptability of multi-modal data processing and the data processing efficiency are improved.

The invention discloses a multi-modal data labeling method and system based on large model pre-labeling, and the method comprises the steps: S1, receiving to-be-labeled multi-modal original data and labeling task definition, and generating a structured task instruction signal; s2, inputting the structured task instruction signal into a multi-modal large model, and generating a pre-labeling result signal containing a preliminary label and a corresponding confidence coefficient thereof; s3, scheduling a manual verification task based on the confidence coefficient in the pre-labeled result signal; s4, according to the manual verification signal, performing parameter fine tuning or prompt optimization on the multi-modal large model, and generating a model optimization signal; and S5, pre-labeling the new multi-modal original data by using the multi-modal large model updated by the model optimization signal, and fusing an artificial verification signal. According to the multi-modal data annotation method and system based on large model pre-annotation, the problems that traditional multi-modal data annotation is low in efficiency, high in cost and difficult to unify in quality can be solved.

A system and method for compressing and restoring multi-modal data utilizing a variational autoencoder to enable homomorphic compression techniques is disclosed. Multi-modal input data, comprising at least two different data types, is compressed into a unified latent space using an encoder network of a multi-modal variational autoencoder. Homomorphic operations are performed on compressed data in the latent space. The latent space compressed data is decompressed using a decoder network of the multi-modal variational autoencoder. The system utilizes modality-specific layers and cross-modal attention mechanisms to effectively process diverse data types. The homomorphic operations enable performing computations while the data is in a compressed form, preserving results of those operations in the decompressed output. This approach allows for efficient storage, transmission, and analysis of multi-modal data while maintaining privacy and data integrity across different modalities.

The invention discloses a multi-modal data sensing processing method and system for a sound intelligent film, and relates to the technical field of data processing. By constructing the modal state transition diagram and the spatial position matrix, the abnormal state and the interference area can be accurately identified, and the processing strategy is dynamically adjusted, so that the response speed and the processing flexibility of the system to the abnormal modal data are effectively improved. Particularly, under the condition that interference data is frequently transmitted or bandwidth resources are limited, data transmission and calculation burdens are greatly reduced through a modal compression and delay decoding mechanism, and continuity and stability of a main processing stream are guaranteed. Meanwhile, the method has high adaptive capacity, cache management and path priority can be dynamically adjusted according to real-time state changes, and data congestion and delay diffusion are effectively prevented.

The invention discloses a multi-modal attack identification method fusing BMama and difference to guide trans-attention, which comprises the following steps: simulating a false data injection attack, a denial of service attack, an address resolution protocol spoofing attack and a domain name system spoofing attack, collecting physical layer sensor data and network layer flow data, and preprocessing multi-modal data; bMama is constructed to perform dynamic time modeling on multi-modal data, a graph neural network is combined to adversariate a variational auto-encoder, features of a power grid system topology and a communication topology structure are fused, and robustness of potential representation is enhanced through adversarial training; the method comprises the following steps of: guiding feature complementary fusion by using modal difference through a difference guide iteration cross-attention fusion mechanism, improving the capability of distinguishing complex attacks, finally carrying out attack detection and classification on fused modals, and executing end-to-end optimization according to a weighted combination of loss of each part. The method can effectively detect and classify the multi-modal attack in the smart power grid, and enhances the safety and reliability of a complex system.

The invention relates to the technical field of data processing, in particular to an intelligent data augmentation and cleaning system and method based on multi-modal consistency detection, and the method comprises the following steps: generating multi-modal data; carrying out multi-mode consistency detection; performing data cleaning and anomaly detection; performing intelligent auxiliary labeling; quality control and feedback optimization; compliance and data traceability management; the method has the beneficial effects that multi-modal data (such as an image-text pair and an image-audio pair) with consistent semantics is generated through a generative adversarial network (GAN), a variational auto-encoder (VAE) and a cross-modal feature fusion technology, and content matching of the generated data among different modals is ensured. According to the consistency guarantee, the quality of a multi-modal data set is remarkably improved, richer and more real training data is provided for a large model, and model performance reduction caused by modal mismatching is reduced.

The invention relates to the technical field of manufacturing equipment control, in particular to an intelligent manufacturing equipment control method and system based on a large diffusion model. The method comprises the following steps: acquiring multi-mode type data of the intelligent manufacturing equipment; the method comprises the following steps: performing time series data acquisition based on multi-modal type data of intelligent manufacturing equipment to generate an initial multi-modal data set; performing operation mode fusion on the initial multi-mode data set to generate manufacturing equipment multi-mode operation data; equipment virtual operation processing is carried out on the manufacturing equipment multi-mode operation data based on a preset diffusion large model, and an equipment operation virtual state distribution diagram is generated; and performing physical constraint optimization on the initialized diffusion large model based on the equipment operation virtual state distribution diagram, and generating an equipment physical characteristic matching diffusion model. According to the method, through combination of the diffusion large model, multi-modal data fusion, incremental learning and differential privacy protection, the control accuracy and adaptability of the intelligent manufacturing equipment are improved.

The invention discloses a multi-mode fusion data processing method and device, a medium, electronic equipment and a program product, belongs to the technical field of computers, and can reduce the data processing complexity and meet the service requirement of high timeliness when responding to application service to integrate multi-mode data. The multi-mode fusion data processing method comprises the following steps: receiving a multi-mode data retrieval statement, analyzing the multi-mode data retrieval statement, and translating the multi-mode data retrieval statement into single-mode data retrieval statements corresponding to a plurality of modes respectively; according to each single-mode data retrieval statement, performing data retrieval in a corresponding single-mode database in the multi-mode database to obtain target data of different modalities; wherein single-mode databases corresponding to a plurality of modes are maintained in the multi-mode database, and independent data management is carried out on each single-mode database; and performing feature fusion and modal alignment processing on the retrieved target data of different modalities to generate a unified multi-modal data representation, and feeding back the unified multi-modal data representation as a retrieval result.

A model training method includes obtaining first and second modal data sets including pieces of data, wherein the pieces of data include sub-modal data pieces, and wherein pieces of the first modal data correspond to pieces of the second modal data; obtaining masked data sets by masking a piece of the first and second sub-modal data, respectively; performing feature prediction on the first masked data set and the second modal data set and the second masked data set and the first modal data set, based on ta feature extraction model, to obtain first and second global features and first and second global recovery features; and generating a trained feature extraction model by optimizing the feature extraction model based on the first global recovery features, the first global features, the second global recovery features, and the second global features.

The invention discloses a task processing method based on a multi-modal world model, a storage medium and equipment, and the method comprises the steps: obtaining multi-modal data which comprises data of at least one of the following modals: image data, voice data, text data, structured data and action data; encoding the various modal data in the multi-modal data by using encoders corresponding to the various modal data in the multi-modal world model to obtain features of the various modal data; mixing the features of the various modal data by using a position encoder in the multi-modal world model to obtain a first mixed feature; using a multi-modal feature conversion model in the multi-modal world model to predict a second mixed feature based on the first mixed feature, the second mixed feature including features of modal data not included in the multi-modal data; and decoding the features of the corresponding modes in the second mixed features by using a decoder of the corresponding task to obtain a task processing result of the corresponding task.

The invention provides a multi-modal data desensitization method, system and device and a medium, and belongs to the technical field of information security. The method comprises the following steps: firstly, acquiring multi-modal original data based on preset access information and acquisition frequency, and after standardized preprocessing, positioning sensitive information by using a classification recognition algorithm and generating an analysis report. Then, according to a report, matching a strategy from a desensitization strategy library bound with the sensitivity level, adjusting a dynamic confusion algorithm parameter to desensitize, integrating and generating desensitized multi-modal data, and recording and storing a mapping relationship between the original data and the desensitized data; and if a restoration request is received, after authorization verification is passed, reverse desensitization restoration data is carried out by using the mapping relation. Besides, full-process feedback information is collected, the desensitization effect is quantitatively evaluated according to a preset index, the desensitization strategy and algorithm are iteratively optimized accordingly, and effective desensitization and safety management of multi-modal data are achieved.

The invention relates to the technical field of data processing, and discloses a multi-modal data fusion method, device and equipment, a storage medium and a computer program product, and the method comprises the steps: after receiving a data fusion request, mapping multi-modal data collected by each sensor in a target region into a three-dimensional twin model of a power plant, and obtaining a plurality of fuzzy regions; for each fuzzy region, determining target modal data of each fuzzy region in a preset time period from the multi-modal data; projecting the target modal data to the corresponding time nodes to obtain a data sequence under a unified time scale; and selecting a target time node corresponding to the data fusion request from the data sequence according to the data fusion request, and fusing target modal data in the target time node to obtain a data fusion result. Target data of sensors at different installation positions are projected to a unified time scale, so that time alignment of multi-sampling-frequency data is realized, and the precision of equipment state evaluation in an industrial scene is improved.

The invention discloses a high-speed data processing and exchanging method and system based on Zynq UltraScale + MPSoCs, and belongs to the technical field of data processing.The method comprises the steps that nanosecond-level synchronous pulses are generated through a PL end of Zynq, initial multi-modal data are collected, the transmission rate and integrity are analyzed through a double-SERDES controller of the PL end, and a dynamic data processing matrix is constructed; pL and PS end data flow distribution is simulated and optimized through a preset multi-data granularity scale, and an exchange instruction set is generated; in combination with an on-chip bus topology, a first data exchange path is formed through DMA-OCM interaction and hardware accelerator adaptation, and a second data exchange path is obtained by adjusting sub-channel allocation and bus priority; and the operation state parameters of the second data exchange path are written back to the data frame state attribute bar corresponding to the dynamic data processing matrix in real time, so that efficient processing and dynamic optimization of the multi-modal data are realized, and the system stability and the transmission efficiency are improved.

The invention relates to a cross-platform UI automatic testing method based on a multi-modal large model, and belongs to the technical field of UI automatic testing. The method comprises the following steps: acquiring data of multiple modes from UI interfaces of different platforms, and preprocessing the acquired multi-mode data; the preprocessed multi-modal data are input into a multi-modal large model, feature fusion is carried out, and multi-modal feature representation is formed; automatically generating a cross-platform test script according to a test requirement and the output of the multi-modal large model, and simulating user operation through interaction with an agent program or a driving program of each platform; performing intelligent analysis on a test result by using the multi-modal large model, including comprehensive evaluation on multi-modal data generated in a test process; abnormal conditions and potential problems in the test are automatically identified, and an analysis result is obtained; and a detailed test report is automatically generated, so that the problems of lack of visual verification, poor cross-platform compatibility, high maintenance cost and the like in the traditional U I test are solved.

The invention provides an electric power information multi-modal data dynamic integration method and system, relates to the technical field of electric power information data integration, and realizes multi-modal data fusion through a technical chain of asynchronous receiving, feature extraction, elastic alignment, dynamic verification and intelligent compensation. The method comprises the following steps: acquiring a sensor numerical value, an infrared image and an environment audio heterogeneous data stream in real time; extracting each modal feature and calculating a historical deviation coefficient; sensor data is used as a reference axis, and a multi-modal maximum feature coincidence interval is determined through sliding window matching; performing cross-modal feature cross validation in the extended dynamic interaction interval, and marking credible fusion features; redundant feature compensation is triggered for the modals which do not pass verification, and an integrated feature vector is generated; and if continuous failure occurs, marking the data source as a low-confidence data source. Through elastic time alignment and a dynamic verification mechanism, the problem of native asynchronization of power multi-modal data is solved, the feature fusion reliability is improved, and accurate fault information can be provided.

The invention provides an intelligent decision-making method and system based on adaptive cross-modal attention, and relates to the technical field of data processing, and the method comprises the steps: obtaining multi-modal data; performing preprocessing on the multi-mode data; extracting original semantic features in each mode from the preprocessed multi-mode data through a deep neural network; calculating the cross-modal attention of each original semantic feature; according to the cross-modal attention, performing cross-modal fusion on each original semantic feature to obtain a fusion feature; and according to the fusion features, intelligent decision making is carried out through a full-connection network. According to the cross-modal attention mechanism, information of different modals can be dynamically weighted according to the importance of each modal, and when the data quality of a certain modal is poor or is completely missing, the weight of the modal can be adaptively reduced, so that the interference of irrelevant or repeated information is avoided, the feature fusion effect is enhanced, and the decision accuracy is improved.

The invention discloses a data analysis and prediction method and system based on multi-modal data fusion, and the method comprises the steps: collecting multi-modal original data, and carrying out the preprocessing of the multi-modal original data, and generating normalized data; setting sliding window parameters, and performing sliding fragmentation on each modal data to obtain sliding window fragment data; extracting modal features, and fusing the modal features into sliding window level fusion feature representation; inputting the improved variational information bottleneck model, and performing feature compression and reconstruction; all fusion features are aggregated, a prediction result is output through modeling reasoning, and the model is optimized. According to the method, the improved sliding window algorithm and the variational information bottleneck model are fused, so that efficient analysis and accurate prediction of multi-modal data are realized.

The invention discloses a detection method and system capable of explaining video hatred content based on multi-agent cooperation. The detection method comprises the following steps: S1, data preprocessing: carrying out multi-modal data processing on input video content; s2, performing opposite-view-angle agent debate: setting a supporter agent and a counter agent, wherein the supporter agent and the counter agent are simultaneously used for capturing semantic clues in video contents and generating explanatory evidences; s3, evidence synthesis: setting a comprehensive agent to integrate the explanatory evidence, and generating comprehensive explanation; s4, reason increase prediction: fusing the multi-modal features and reason features, and performing prediction through a two-layer and multi-layer perceptron; s5, calculating a hatred score: calculating the hatred score according to the predicted content, and determining whether the video content contains hatred content according to a preset threshold value; according to the method, through multi-agent cooperation and a reason enhancement prediction mechanism, the private hatred content in the video content can be effectively identified, and meanwhile, credible and comprehensive explanation is provided.

The invention discloses an electric power dispatching method and device fusing multi-modal data and dual verification, electronic equipment and a storage medium, and belongs to the field of intelligent dispatching of an electric power system.The method comprises the steps that multi-modal dynamic data, including equipment images, load time sequences, logs, dispatching instructions, meteorological early warning and electricity price data, of an electric power system are obtained; visual, time sequence and semantic features are extracted, cross-modal alignment is carried out, and a joint representation vector is generated; based on the vector and the real-time data, an initial scheduling strategy is generated through a reinforcement learning model; and through historical scheme matching and physical constraint verification, determining a target scheduling strategy and executing scheduling. Therefore, by implementing the method and the device, the problems of incomplete power grid state perception caused by insufficient multi-modal data fusion and low reliability caused by lack of automatic credibility verification of the AI scheduling strategy in the prior art can be solved.

The invention discloses a multi-modal data fusion method and device based on an electric power scene and a storage medium, and the method comprises the steps: obtaining multi-modal data of the electric power scene, carrying out the coding processing of the multi-modal data, obtaining multi-modal features, and converting the multi-modal features into feature vectors of the same dimension; adding coding information to the feature vector to obtain a coded feature vector; generating a plurality of mask matrixes based on feature blocks in the encoded feature vectors, and combining all the mask matrixes into a complete mask matrix; and feature blocks in the encoded feature vectors are spliced into a unified feature matrix, and the unified feature matrix is input into a Transform encoder, so that the Transform encoder carries out fusion processing on the multi-modal features in the unified feature matrix according to the complete mask matrix, and fusion data are obtained. According to the method, the potential relationship between different modals can be deeply mined, so that the multi-modal data fusion effect can be effectively improved.

The invention provides a multi-modal data enhanced storage system based on a Parquet format. Unified storage and optimization of multi-modal data are achieved through native transformation and expansion of the Parquet format. The system comprises a modal perception format expansion component and a modal optimization coding strategy component, wherein the modal perception format expansion component realizes deep fusion of multi-modal semantics and a storage structure by expanding a Parquet logic type system and constructing a four-layer hierarchical metadata architecture from a file level to a page level; and performing instruction level fusion with a Parquet compression algorithm, executing targeted storage optimization based on semantic descriptors and modal characteristic parameters, and solving the problems of unified storage and efficient management of multi-modal data.

The invention relates to the technical field of Internet of Things data processing, in particular to a multi-modal data real-time processing method for an Internet of Things gateway, and the method comprises the steps: obtaining multi-source operation data, and packaging the multi-source operation data into a standard operation data set; obtaining high-bandwidth modal data in the standard operation data set; when the maximum link delay time is greater than a preset delay abnormal threshold value, obtaining a version effective fingerprint of the target model and a local model fingerprint; when a model version abnormity diagnosis result is obtained, calculating information redundancy; obtaining correction delay time, and comparing the correction delay time with a preset normal reference delay interval; and when the correction delay time is greater than or equal to the upper limit value of the normal reference delay interval, adjusting the current decision period. By monitoring delay in real time, diagnosing consistency of model versions, eliminating redundant information and executing link adjustment, accurate processing of high-bandwidth modal data is realized, and real-time performance and reliability of Internet of Things gateway data processing are improved.

The invention relates to the technical field of data processing, in particular to a multi-modal data processing fusion method and system oriented to an IPU chip. The method comprises the following steps: acquiring multi-modal space-time sensing data; according to the multi-mode space-time sensing data, configuring the multi-mode processing parallelism degree of the IPU chip, and initializing a multi-mode parallel processing strategy of the IPU chip; based on an IPU chip multi-modal parallel processing strategy, performing multi-modal data parallel processing on the multi-modal time-space sensing data by using an IPU chip to obtain multi-dimensional sensing interaction feature data; identifying a candidate operation point set according to the multi-dimensional perception interaction feature data, and screening effective operation points; formulating a self-adaptive execution strategy according to the effective operation points; according to the method, the parallel computing capability of the IPU chip is utilized, so that the multi-modal data processing efficiency and real-time performance are remarkably improved.

A distributed operation and maintenance data management method based on a multi-modal log aggregation framework relates to the field of electrical digital data processing, and comprises the following steps: calculating a node load state index; determining node acquisition parameters including an acquisition period and an acquisition trigger condition; when the log file change amount of the target node is larger than the collection triggering condition, event collection is executed, and event collection data flow is obtained; when the log file change amount is smaller than or equal to the collection triggering condition, polling collection is executed according to the collection period, and a periodic collection data stream is obtained; generating a data compensation instruction and executing incremental acquisition to obtain a compensation data stream; merging to generate complete log data, and carrying out type distinguishing to generate a multi-modal data packet; and allocating the data to corresponding storage engines to obtain multi-modal log aggregation data. By implementing the method, the integrity of log data acquisition in a large-scale distributed system can be improved.

The invention provides a full-modal harmful information identification method and device and electronic equipment, and the method comprises the steps: obtaining multi-modal to-be-identified data, and distributing the to-be-identified data to a corresponding modal processing branch according to a corresponding modal type; for each modal processing branch, extracting data modal features corresponding to the to-be-identified data, inputting the data modal features to at least one harmful information identification model meeting a preset task requirement, and determining a harmful information identification result, each harmful information identification model being based on an identity, or based on content, or based on an attribute category; or performing harmful information identification capability based on semantics; and determining a modal, task and confidence recognition result triple corresponding to each modal harmful information recognition result, performing confidence fusion and voting fusion for the full-modal recognition result triple, and determining a harmful information fusion judgment result. And high-precision harmful information identification can be realized.

Some examples of training a large language model (LLM) to detect an anomaly comprise receiving a plurality of modalities of training data, each modality of training data comprising a different data type. The training data comprises labels indicating a presence of an anomaly. Each modality of training data is transformed with a respective transformation to obtain respective vector embeddings. The respective vector embeddings of the training data are input into a pretrained LLM. Weights of the pre-trained LLM are fine-tuned using the training data via supervised learning such that the pre-trained LLM is able to detect anomalies in multi-modal data.

The invention discloses a label printing equipment abnormity diagnosis method based on multi-modal fusion, and the method comprises the steps: collecting multi-modal data, and generating a label path phase coding sequence; based on label path phase coding, locking alignment is executed on the multi-modal data, and unification is performed until a phase coordinate system is synchronized; inputting synchronous multi-modal data, inverting the slip amount, the elongation rate and the speed difference of the coiled material, and outputting implicit physical variables; constructing a three-domain hybrid diagnosis network which comprises phase, time and visual branches, and extracting a multi-modal feature vector; setting phase-physical consistency gating, performing sparse selection and routing on the sub-experts, and outputting a candidate feature set; and a phase alignment attention mechanism is adopted, candidate features are fused, and abnormal types, confidence coefficients and station positioning are output. Through multi-modal data phase alignment, physical constraint inversion and cross-domain fusion, high-precision anomaly diagnosis and intelligent deviation correction of the label printing equipment are realized.

The invention discloses a multi-modal data transmission method and system based on dynamic priority, and the method comprises the steps: respectively extracting image, voice and text feature data from image, voice and text data in collected multi-modal data of a service, respectively adding corresponding modal embedding vectors, and then carrying out the splicing; inputting the spliced feature data into a cross attention mechanism, and adding the attention scores of the feature data of each modal to obtain the attention score of the multi-modal fusion feature data of the service; performing joint evaluation on the attention score of the multi-modal fusion feature data of the service and a priority preset for the service to obtain a dynamic priority of the service; and preferentially transmitting the data of the service with high dynamic priority. According to the invention, transmission of data with high dynamic priority can be guaranteed preferentially, and key and important data can be guaranteed to be transmitted preferentially.

The invention provides a mobile Internet of Things operation data management method and system, and relates to the technical field of mobile Internet of Things data management, and the method comprises the following steps: collecting and preprocessing multi-modal data, and obtaining the preprocessed data; calculating a data change rate and dividing the data change rate into data in different stable states based on the preprocessed data, and performing hierarchical compression on the data in different stable states to obtain compressed data; calculating a data synchronization priority, and performing data synchronization on the compressed data according to the data synchronization priority to obtain synchronized data; verifying the synchronized data, and calculating a retransmission index to carry out data retransmission to obtain the data after data retransmission; and storing the data after data retransmission, and constructing a composite index. The method has the advantages that the compression strategy and the data synchronization strategy are dynamically applied, the data compression and data transmission efficiency is high, the composite index comprises the data compression feature and the time-space feature, and cross-modal data rapid positioning is effectively achieved.