431 results about "Modal data" patented technology

The disclosure provides a cross-modal data processing method and apparatus, a device, a storage medium, and a program product. The method comprises: obtaining first modal data to be processed; obtaining a first modal data feature by performing feature extraction based on the first modal data; and obtaining second modal data based on the first modal data feature and a cross-modal processing model, the first modal data and the second modal data having different modalities, wherein the cross-modal processing model needs to be pre-trained based on a concatenated training sample, and the concatenated training sample comprises a concatenated image sample and a corresponding concatenated text sample.

The invention discloses an equipment state monitoring method and system based on multi-modal data fusion. According to the method, numerical data of equipment temperature, strain and vibration acceleration and operation record text data are acquired, an index table is established based on a maintenance event to realize heterogeneous data time sequence alignment, equipment dynamic features are extracted by combining Fourier transform and wavelet transform, and text semantic features are extracted by using an LSTM network. And designing a multi-head cross attention mechanism to realize deep feature fusion, constructing a state evaluation model by adopting CatBoost, and introducing KL divergence detection to dynamically update the model. The technical problems that in equipment state monitoring, multi-source heterogeneous data is difficult to align, numerical features and text features are difficult to effectively fuse, and a model is difficult to adapt to data distribution changes are solved, and the accuracy and reliability of state evaluation are improved.

The invention discloses a heterogeneous data conversion method and system based on a multi-modal large model, and aims to solve the problems of single-modal limitation, high migration cost and the like in traditional heterogeneous data processing. The method comprises the following steps: firstly, carrying out format adaptation and classification preprocessing on multi-source heterogeneous data such as images, texts, audios and videos; then, through a ViT, BERT and Transform cross attention mechanism, deep analysis and cross-modal semantic association of visual, text and audio features are realized, and visual-text-audio semantic mapping is established; and based on a dynamic prompt project, guiding the multi-modal large model to generate structured data according to a preset Schema zero sample. And then error correction and format normalization are carried out through a Drools rule engine to ensure that the data meet the business standard. And finally, supporting multi-format output of JSON / XML / database table and the like, and meanwhile, associating original data storage to realize data traceability. According to the method, the limitation of single-modal processing is broken through, and cross-modal data deep association is realized.

The invention discloses a cross-modal question and answer processing method and device and a storage medium, and belongs to the technical field of natural language processing. The method comprises the steps that in response to multi-modal data input by a user, modal data features corresponding to all data modals in the multi-modal data are extracted; based on an interactive attention mechanism, determining an information weight corresponding to each data mode; according to the information weight, fusing the modal data features corresponding to the data modals, and generating fused data features corresponding to the multi-modal data; and obtaining response data matched with the fused data features, and outputting the response data. According to the method and the device, the information weight between different modalities is dynamically adjusted, so that the most relevant modal information can be more accurately focused in the question and answer process, and the question and answer retrieval accuracy is effectively improved.

The invention discloses an integrated sensor multi-modal data edge computing system optimized by a large model. The integrated sensor multi-modal data edge computing system comprises a data preprocessing module which obtains pre-training parameters of each modal data through the large model; the modal specific coding module optimizes attention weight through an encoder branch to determine key information of data of each modal, and advanced feature representation of each modal is obtained; the fusion strategy selection and execution module is used for calculating the correlation between the rest modes and the final core mode, and inputting the advanced feature representation of the mode with the strongest correlation in the rest modes and the advanced feature representation of the final core mode into a cross-mode encoder to generate the fusion feature representation of the fused modes; continuing to perform modal fusion operation until all modalities are fused, and generating final fusion feature representation; and the cross-modal coding and task processing module finally generates a multi-modal fusion model prediction result through cross-modal feature representation. According to the invention, the performance and adaptability of multi-modal data processing and the data processing efficiency are improved.

The invention relates to a multi-modal data fusion method and device, computer equipment and a storage medium. The method comprises the following steps: carrying out high-order feature extraction on multi-modal data to be fused by utilizing a special modal encoder to obtain high-order embedding features of the multi-modal data; the high-order embedded features are decoupled in combination with expert knowledge constraints and a multi-head attention mechanism, and high-frequency features representing local information and low-frequency features representing global information are obtained respectively; and aligning the low-frequency features by using a cosine alignment algorithm, performing orthogonal projection on the high-frequency features by using an orthogonal projection technology, and performing multi-level fusion on the aligned low-frequency features and the orthogonally projected high-frequency features through an adaptive deep fusion network to obtain fused multi-modal joint features. According to the method, the problems of heterogeneity and isomerism of the data are solved, effective fusion of different modal data is ensured, and the accuracy and individuation capability of disease risk early warning can be remarkably improved.

The invention discloses a missing perception prompting method and system based on modal specific and general information, and belongs to the field of multi-modal learning. Performing modal missing processing on the initial features of the original video sample; unifying feature dimensions and sizes of modal missing features; wherein after unifying the feature dimensions, splicing to obtain a multi-modal joint feature; performing size unification and then performing fusion to obtain a modal universal prompt; using a gating fusion mechanism to control the fusion of the learnable matrix corresponding to the modal to obtain a modal specific prompt; fusing the modal general prompt and the modal specific prompt to obtain a specific-general prompt as a missing perception prompt; and the specific-general prompt and the multi-modal joint features are spliced to serve as input of a multi-modal model, and a full connection layer is introduced to complete a multi-modal downstream task. According to the method, the learnable specific-universal prompt of mode missing perception is introduced into the multi-mode model, so that cross-mode application in a data mode incomplete scene is realized, and the performance of a multi-mode downstream task is improved.

The invention relates to an intelligent verification code generation and anti-refreshing method and system and computer equipment. The method comprises the following steps: performing multi-modal generation processing on verification code content to obtain original verification code data; performing multi-dimensional feature extraction on the user interaction behavior to obtain user behavior feature data; inputting the original verification code data and the user behavior feature data into a double attention network for fusion processing to obtain fused feature data; performing dynamic difficulty assessment calculation based on the fused feature data to obtain verification code difficulty adjustment parameters; performing difficulty adjustment on the original verification code data according to the verification code difficulty adjustment parameter to obtain target verification code data; and based on the fused feature data and the target verification code data, performing multi-dimensional anti-refreshing rule analysis, and generating a self-adaptive anti-refreshing rule set. According to the method, the multi-modal data can be comprehensively utilized, the difficulty is dynamically adjusted, and the intelligent verification code of the anti-refreshing rule is adaptively generated.

The invention discloses a multi-modal data labeling method and system based on large model pre-labeling, and the method comprises the steps: S1, receiving to-be-labeled multi-modal original data and labeling task definition, and generating a structured task instruction signal; s2, inputting the structured task instruction signal into a multi-modal large model, and generating a pre-labeling result signal containing a preliminary label and a corresponding confidence coefficient thereof; s3, scheduling a manual verification task based on the confidence coefficient in the pre-labeled result signal; s4, according to the manual verification signal, performing parameter fine tuning or prompt optimization on the multi-modal large model, and generating a model optimization signal; and S5, pre-labeling the new multi-modal original data by using the multi-modal large model updated by the model optimization signal, and fusing an artificial verification signal. According to the multi-modal data annotation method and system based on large model pre-annotation, the problems that traditional multi-modal data annotation is low in efficiency, high in cost and difficult to unify in quality can be solved.

The invention relates to the technical field of data processing, in particular to a multi-modal recommendation method based on an information bottleneck principle. The method comprises the following steps: constructing a data set # imgabs0 # according to a user-article historical interaction record in an Amazon data set; constructing a user-article graph for each mode according to the data set, and performing multi-layer graph convolution calculation on the embedding of the user and the article to obtain the final expression of the user and the fusion embedding of the article; an article-article similarity graph is constructed for each mode of the articles, neighbors of the first k articles with the highest similarity are reserved, article semantic signals are grouped according to the contribution degree of the article modes to downstream recommendation tasks, and final expressions of the articles are obtained; and performing optimization through denoising, loss calculation and an Adam optimizer. According to the method, key information can be extracted from multi-modal data, redundancy and noise can be removed, and the contribution degree of article characterization to downstream recommendation tasks in different modals can be calculated.

A system and method for compressing and restoring multi-modal data utilizing a variational autoencoder to enable homomorphic compression techniques is disclosed. Multi-modal input data, comprising at least two different data types, is compressed into a unified latent space using an encoder network of a multi-modal variational autoencoder. Homomorphic operations are performed on compressed data in the latent space. The latent space compressed data is decompressed using a decoder network of the multi-modal variational autoencoder. The system utilizes modality-specific layers and cross-modal attention mechanisms to effectively process diverse data types. The homomorphic operations enable performing computations while the data is in a compressed form, preserving results of those operations in the decompressed output. This approach allows for efficient storage, transmission, and analysis of multi-modal data while maintaining privacy and data integrity across different modalities.

The invention discloses a multi-modal data sensing processing method and system for a sound intelligent film, and relates to the technical field of data processing. By constructing the modal state transition diagram and the spatial position matrix, the abnormal state and the interference area can be accurately identified, and the processing strategy is dynamically adjusted, so that the response speed and the processing flexibility of the system to the abnormal modal data are effectively improved. Particularly, under the condition that interference data is frequently transmitted or bandwidth resources are limited, data transmission and calculation burdens are greatly reduced through a modal compression and delay decoding mechanism, and continuity and stability of a main processing stream are guaranteed. Meanwhile, the method has high adaptive capacity, cache management and path priority can be dynamically adjusted according to real-time state changes, and data congestion and delay diffusion are effectively prevented.

The invention discloses a multi-modal attack identification method fusing BMama and difference to guide trans-attention, which comprises the following steps: simulating a false data injection attack, a denial of service attack, an address resolution protocol spoofing attack and a domain name system spoofing attack, collecting physical layer sensor data and network layer flow data, and preprocessing multi-modal data; bMama is constructed to perform dynamic time modeling on multi-modal data, a graph neural network is combined to adversariate a variational auto-encoder, features of a power grid system topology and a communication topology structure are fused, and robustness of potential representation is enhanced through adversarial training; the method comprises the following steps of: guiding feature complementary fusion by using modal difference through a difference guide iteration cross-attention fusion mechanism, improving the capability of distinguishing complex attacks, finally carrying out attack detection and classification on fused modals, and executing end-to-end optimization according to a weighted combination of loss of each part. The method can effectively detect and classify the multi-modal attack in the smart power grid, and enhances the safety and reliability of a complex system.

The invention provides a multi-modal large model confrontation and defense method and device, and the method comprises the steps: obtaining to-be-detected initial multi-modal data; judging whether the initial multi-modal data is malicious data or not by adopting a pre-trained detection model; if the initial multi-modal data is malicious data, a pre-trained recovery model is adopted to recover the malicious data into target multi-modal data, and the recovery model is used for recovering the malicious data into normal data; and inputting the target multi-modal data into a multi-modal large model to obtain an output result. According to the embodiment of the invention, the pre-trained detection model and the pre-trained recovery model are adopted, the detection and recovery of the adversarial attack of the input data of the multi-modal large model are realized, the robustness of the model is enhanced, the brought time consumption is reduced as much as possible, and the safety and the accuracy of the output result of the multi-modal large model are ensured.

The invention discloses a software operation anomaly detection method and system, a medium, a product and equipment. The method comprises the steps that multi-modal data generated by software to be detected in the operation period is acquired; performing feature extraction on data of each mode in the multi-mode data to obtain feature information corresponding to each mode; fusing the feature information corresponding to each mode through a self-attention mechanism to obtain a fused feature vector; using a supervised learning module and an unsupervised learning module contained in an anomaly detection model to perform anomaly detection based on the fused feature vector to obtain an anomaly detection result, thereby comprehensively considering the performance of the software in different modes during operation, comprehensively, quickly and accurately identifying the fault existing during the operation of the software, and improving the fault detection accuracy. And the accuracy and reliability of software operation anomaly detection are improved.

The invention relates to the technical field of data processing, in particular to an intelligent data augmentation and cleaning system and method based on multi-modal consistency detection, and the method comprises the following steps: generating multi-modal data; carrying out multi-mode consistency detection; performing data cleaning and anomaly detection; performing intelligent auxiliary labeling; quality control and feedback optimization; compliance and data traceability management; the method has the beneficial effects that multi-modal data (such as an image-text pair and an image-audio pair) with consistent semantics is generated through a generative adversarial network (GAN), a variational auto-encoder (VAE) and a cross-modal feature fusion technology, and content matching of the generated data among different modals is ensured. According to the consistency guarantee, the quality of a multi-modal data set is remarkably improved, richer and more real training data is provided for a large model, and model performance reduction caused by modal mismatching is reduced.

The invention relates to the technical field of manufacturing equipment control, in particular to an intelligent manufacturing equipment control method and system based on a large diffusion model. The method comprises the following steps: acquiring multi-mode type data of the intelligent manufacturing equipment; the method comprises the following steps: performing time series data acquisition based on multi-modal type data of intelligent manufacturing equipment to generate an initial multi-modal data set; performing operation mode fusion on the initial multi-mode data set to generate manufacturing equipment multi-mode operation data; equipment virtual operation processing is carried out on the manufacturing equipment multi-mode operation data based on a preset diffusion large model, and an equipment operation virtual state distribution diagram is generated; and performing physical constraint optimization on the initialized diffusion large model based on the equipment operation virtual state distribution diagram, and generating an equipment physical characteristic matching diffusion model. According to the method, through combination of the diffusion large model, multi-modal data fusion, incremental learning and differential privacy protection, the control accuracy and adaptability of the intelligent manufacturing equipment are improved.

The invention discloses a multi-mode fusion data processing method and device, a medium, electronic equipment and a program product, belongs to the technical field of computers, and can reduce the data processing complexity and meet the service requirement of high timeliness when responding to application service to integrate multi-mode data. The multi-mode fusion data processing method comprises the following steps: receiving a multi-mode data retrieval statement, analyzing the multi-mode data retrieval statement, and translating the multi-mode data retrieval statement into single-mode data retrieval statements corresponding to a plurality of modes respectively; according to each single-mode data retrieval statement, performing data retrieval in a corresponding single-mode database in the multi-mode database to obtain target data of different modalities; wherein single-mode databases corresponding to a plurality of modes are maintained in the multi-mode database, and independent data management is carried out on each single-mode database; and performing feature fusion and modal alignment processing on the retrieved target data of different modalities to generate a unified multi-modal data representation, and feeding back the unified multi-modal data representation as a retrieval result.

A model training method includes obtaining first and second modal data sets including pieces of data, wherein the pieces of data include sub-modal data pieces, and wherein pieces of the first modal data correspond to pieces of the second modal data; obtaining masked data sets by masking a piece of the first and second sub-modal data, respectively; performing feature prediction on the first masked data set and the second modal data set and the second masked data set and the first modal data set, based on ta feature extraction model, to obtain first and second global features and first and second global recovery features; and generating a trained feature extraction model by optimizing the feature extraction model based on the first global recovery features, the first global features, the second global recovery features, and the second global features.

The invention discloses a multi-hardware mixed large model training method and system and a related device, which are used for simplifying a model structure and improving model performance. The method comprises the following steps: firstly, collecting multi-modal data, constructing a multi-modal model architecture, and carrying out benchmark test on a multi-modal large model to obtain first performance indexes of multiple pieces of hardware; thirdly, decomposing the received calculation task into a plurality of sub-tasks, dividing the multi-modal large model into a plurality of sub-models, and matching the sub-models and the sub-tasks to proper hardware for training according to a first performance index; and then pruning operation is carried out on the information transmission link according to the calculated comprehensive weight value. The second performance index is evaluated, a performance evaluation report is obtained, a dynamic adjustment strategy is applied to pruning operation, subsequent pruning operation is executed, and after the pruning operation is executed, reinitialization and local fine adjustment are conducted on the sub-model. And finally, comprehensively evaluating the multi-modal large model by using the test data set.

The invention discloses a test question data generation method, electronic equipment, a storage medium and electronic equipment, and relates to the field of large model technology and text data processing. The method comprises the steps that in response to a first input instruction acting on an operation interface, a target document corresponding to the input instruction is displayed on the operation interface, and the target document comprises multi-modal data; in response to a conversion instruction acting on the operation interface, displaying target text data corresponding to the conversion instruction on the operation interface; in response to a second input instruction acting on the operation interface, a generation condition corresponding to the second input instruction is displayed on the operation interface, and the generation condition comprises at least one of a target type, a target difficulty level and a target dimension; and in response to a generation instruction acting on the operation interface, displaying the target test question data on the operation interface. The technical problem of low efficiency and diversity of test question data generation in related technologies is solved.

The invention discloses a task processing method based on a multi-modal world model, a storage medium and equipment, and the method comprises the steps: obtaining multi-modal data which comprises data of at least one of the following modals: image data, voice data, text data, structured data and action data; encoding the various modal data in the multi-modal data by using encoders corresponding to the various modal data in the multi-modal world model to obtain features of the various modal data; mixing the features of the various modal data by using a position encoder in the multi-modal world model to obtain a first mixed feature; using a multi-modal feature conversion model in the multi-modal world model to predict a second mixed feature based on the first mixed feature, the second mixed feature including features of modal data not included in the multi-modal data; and decoding the features of the corresponding modes in the second mixed features by using a decoder of the corresponding task to obtain a task processing result of the corresponding task.

The invention provides an equipment information management method and system based on a multi-modal large model in the technical field of equipment information management. The method comprises the following steps: S1, acquiring historical multi-modal data to construct a data set, and dividing the data set into a training set, a verification set and a test set; s2, creating a multi-modal large model based on a Vi T module, a Bert module, a numerical feature construction module, a feature fusion module and an output module, and setting hyper-parameters; s3, training and evaluating the multi-modal large model through the training set, the verification set and the test set; step S4, compressing the multi-modal large model which passes the evaluation; and S5, collecting real-time multi-modal data of installed equipment in the management area by each area server, and inputting the real-time multi-modal data into the multi-modal large model to obtain an equipment information analysis report. The method has the advantages that the efficiency, the quality, the traceability and the resource utilization rate of equipment information management are greatly improved, and then the reliability of equipment operation and maintenance is greatly improved.

The invention provides a multi-modal data desensitization method, system and device and a medium, and belongs to the technical field of information security. The method comprises the following steps: firstly, acquiring multi-modal original data based on preset access information and acquisition frequency, and after standardized preprocessing, positioning sensitive information by using a classification recognition algorithm and generating an analysis report. Then, according to a report, matching a strategy from a desensitization strategy library bound with the sensitivity level, adjusting a dynamic confusion algorithm parameter to desensitize, integrating and generating desensitized multi-modal data, and recording and storing a mapping relationship between the original data and the desensitized data; and if a restoration request is received, after authorization verification is passed, reverse desensitization restoration data is carried out by using the mapping relation. Besides, full-process feedback information is collected, the desensitization effect is quantitatively evaluated according to a preset index, the desensitization strategy and algorithm are iteratively optimized accordingly, and effective desensitization and safety management of multi-modal data are achieved.

The invention provides a multi-modal data fusion method and system, a readable storage medium and computer equipment, and the method comprises the steps: obtaining modal data corresponding to each equipment node, the modal data comprising text data, image data, video data and audio data, and each equipment node corresponding to one modal data; performing gradient learning on the initial local model on each equipment node according to the modal data to obtain a final local model on each equipment node; and fusing the final local models on the equipment nodes by adopting a weighted average algorithm to obtain an initial fusion model, and carrying out iterative convergence on the initial fusion model to obtain a final fusion model. According to the method, multi-modal and cross-modal federated learning can be realized, so that correlation information among different modals can be better captured, the performance and generalization ability of the model are improved, and more comprehensive data understanding and application are realized.

The invention relates to the technical field of data processing, and discloses a multi-modal data fusion method, device and equipment, a storage medium and a computer program product, and the method comprises the steps: after receiving a data fusion request, mapping multi-modal data collected by each sensor in a target region into a three-dimensional twin model of a power plant, and obtaining a plurality of fuzzy regions; for each fuzzy region, determining target modal data of each fuzzy region in a preset time period from the multi-modal data; projecting the target modal data to the corresponding time nodes to obtain a data sequence under a unified time scale; and selecting a target time node corresponding to the data fusion request from the data sequence according to the data fusion request, and fusing target modal data in the target time node to obtain a data fusion result. Target data of sensors at different installation positions are projected to a unified time scale, so that time alignment of multi-sampling-frequency data is realized, and the precision of equipment state evaluation in an industrial scene is improved.

The invention discloses a high-speed data processing and exchanging method and system based on Zynq UltraScale + MPSoCs, and belongs to the technical field of data processing.The method comprises the steps that nanosecond-level synchronous pulses are generated through a PL end of Zynq, initial multi-modal data are collected, the transmission rate and integrity are analyzed through a double-SERDES controller of the PL end, and a dynamic data processing matrix is constructed; pL and PS end data flow distribution is simulated and optimized through a preset multi-data granularity scale, and an exchange instruction set is generated; in combination with an on-chip bus topology, a first data exchange path is formed through DMA-OCM interaction and hardware accelerator adaptation, and a second data exchange path is obtained by adjusting sub-channel allocation and bus priority; and the operation state parameters of the second data exchange path are written back to the data frame state attribute bar corresponding to the dynamic data processing matrix in real time, so that efficient processing and dynamic optimization of the multi-modal data are realized, and the system stability and the transmission efficiency are improved.

The invention relates to a cross-platform UI automatic testing method based on a multi-modal large model, and belongs to the technical field of UI automatic testing. The method comprises the following steps: acquiring data of multiple modes from UI interfaces of different platforms, and preprocessing the acquired multi-mode data; the preprocessed multi-modal data are input into a multi-modal large model, feature fusion is carried out, and multi-modal feature representation is formed; automatically generating a cross-platform test script according to a test requirement and the output of the multi-modal large model, and simulating user operation through interaction with an agent program or a driving program of each platform; performing intelligent analysis on a test result by using the multi-modal large model, including comprehensive evaluation on multi-modal data generated in a test process; abnormal conditions and potential problems in the test are automatically identified, and an analysis result is obtained; and a detailed test report is automatically generated, so that the problems of lack of visual verification, poor cross-platform compatibility, high maintenance cost and the like in the traditional U I test are solved.

The invention provides an electric power information multi-modal data dynamic integration method and system, relates to the technical field of electric power information data integration, and realizes multi-modal data fusion through a technical chain of asynchronous receiving, feature extraction, elastic alignment, dynamic verification and intelligent compensation. The method comprises the following steps: acquiring a sensor numerical value, an infrared image and an environment audio heterogeneous data stream in real time; extracting each modal feature and calculating a historical deviation coefficient; sensor data is used as a reference axis, and a multi-modal maximum feature coincidence interval is determined through sliding window matching; performing cross-modal feature cross validation in the extended dynamic interaction interval, and marking credible fusion features; redundant feature compensation is triggered for the modals which do not pass verification, and an integrated feature vector is generated; and if continuous failure occurs, marking the data source as a low-confidence data source. Through elastic time alignment and a dynamic verification mechanism, the problem of native asynchronization of power multi-modal data is solved, the feature fusion reliability is improved, and accurate fault information can be provided.

The present invention provides a method for improving the accuracy of event detection based on multi-model fusion, comprising the following steps: performing multimodal data preprocessing on a monitoring video stream to generate a standardized feature matrix; performing feature-level fusion of the outputs of a YOLOv5 improved model, a ResNet Transformer hybrid model, and an LSTM spatiotemporal analysis model through an adaptive weighted fusion module; performing multidimensional event detection based on the fused feature matrix to generate a preliminary detection result set; establishing a three-level logical judgment mechanism to verify the confidence of the detection result; and outputting the verified final event detection result to a monitoring system. The present invention can improve the accuracy of event detection and is suitable for monitoring event detection in complex environments.