236 results about "Modal data" patented technology

The invention discloses a heterogeneous data conversion method and system based on a multi-modal large model, and aims to solve the problems of single-modal limitation, high migration cost and the like in traditional heterogeneous data processing. The method comprises the following steps: firstly, carrying out format adaptation and classification preprocessing on multi-source heterogeneous data such as images, texts, audios and videos; then, through a ViT, BERT and Transform cross attention mechanism, deep analysis and cross-modal semantic association of visual, text and audio features are realized, and visual-text-audio semantic mapping is established; and based on a dynamic prompt project, guiding the multi-modal large model to generate structured data according to a preset Schema zero sample. And then error correction and format normalization are carried out through a Drools rule engine to ensure that the data meet the business standard. And finally, supporting multi-format output of JSON / XML / database table and the like, and meanwhile, associating original data storage to realize data traceability. According to the method, the limitation of single-modal processing is broken through, and cross-modal data deep association is realized.

The invention discloses a multi-modal data labeling method and system based on large model pre-labeling, and the method comprises the steps: S1, receiving to-be-labeled multi-modal original data and labeling task definition, and generating a structured task instruction signal; s2, inputting the structured task instruction signal into a multi-modal large model, and generating a pre-labeling result signal containing a preliminary label and a corresponding confidence coefficient thereof; s3, scheduling a manual verification task based on the confidence coefficient in the pre-labeled result signal; s4, according to the manual verification signal, performing parameter fine tuning or prompt optimization on the multi-modal large model, and generating a model optimization signal; and S5, pre-labeling the new multi-modal original data by using the multi-modal large model updated by the model optimization signal, and fusing an artificial verification signal. According to the multi-modal data annotation method and system based on large model pre-annotation, the problems that traditional multi-modal data annotation is low in efficiency, high in cost and difficult to unify in quality can be solved.

The invention discloses a multi-modal attack identification method fusing BMama and difference to guide trans-attention, which comprises the following steps: simulating a false data injection attack, a denial of service attack, an address resolution protocol spoofing attack and a domain name system spoofing attack, collecting physical layer sensor data and network layer flow data, and preprocessing multi-modal data; bMama is constructed to perform dynamic time modeling on multi-modal data, a graph neural network is combined to adversariate a variational auto-encoder, features of a power grid system topology and a communication topology structure are fused, and robustness of potential representation is enhanced through adversarial training; the method comprises the following steps of: guiding feature complementary fusion by using modal difference through a difference guide iteration cross-attention fusion mechanism, improving the capability of distinguishing complex attacks, finally carrying out attack detection and classification on fused modals, and executing end-to-end optimization according to a weighted combination of loss of each part. The method can effectively detect and classify the multi-modal attack in the smart power grid, and enhances the safety and reliability of a complex system.

The invention provides a multi-modal data desensitization method, system and device and a medium, and belongs to the technical field of information security. The method comprises the following steps: firstly, acquiring multi-modal original data based on preset access information and acquisition frequency, and after standardized preprocessing, positioning sensitive information by using a classification recognition algorithm and generating an analysis report. Then, according to a report, matching a strategy from a desensitization strategy library bound with the sensitivity level, adjusting a dynamic confusion algorithm parameter to desensitize, integrating and generating desensitized multi-modal data, and recording and storing a mapping relationship between the original data and the desensitized data; and if a restoration request is received, after authorization verification is passed, reverse desensitization restoration data is carried out by using the mapping relation. Besides, full-process feedback information is collected, the desensitization effect is quantitatively evaluated according to a preset index, the desensitization strategy and algorithm are iteratively optimized accordingly, and effective desensitization and safety management of multi-modal data are achieved.

The invention discloses a high-speed data processing and exchanging method and system based on Zynq UltraScale + MPSoCs, and belongs to the technical field of data processing.The method comprises the steps that nanosecond-level synchronous pulses are generated through a PL end of Zynq, initial multi-modal data are collected, the transmission rate and integrity are analyzed through a double-SERDES controller of the PL end, and a dynamic data processing matrix is constructed; pL and PS end data flow distribution is simulated and optimized through a preset multi-data granularity scale, and an exchange instruction set is generated; in combination with an on-chip bus topology, a first data exchange path is formed through DMA-OCM interaction and hardware accelerator adaptation, and a second data exchange path is obtained by adjusting sub-channel allocation and bus priority; and the operation state parameters of the second data exchange path are written back to the data frame state attribute bar corresponding to the dynamic data processing matrix in real time, so that efficient processing and dynamic optimization of the multi-modal data are realized, and the system stability and the transmission efficiency are improved.

The invention provides an intelligent decision-making method and system based on adaptive cross-modal attention, and relates to the technical field of data processing, and the method comprises the steps: obtaining multi-modal data; performing preprocessing on the multi-mode data; extracting original semantic features in each mode from the preprocessed multi-mode data through a deep neural network; calculating the cross-modal attention of each original semantic feature; according to the cross-modal attention, performing cross-modal fusion on each original semantic feature to obtain a fusion feature; and according to the fusion features, intelligent decision making is carried out through a full-connection network. According to the cross-modal attention mechanism, information of different modals can be dynamically weighted according to the importance of each modal, and when the data quality of a certain modal is poor or is completely missing, the weight of the modal can be adaptively reduced, so that the interference of irrelevant or repeated information is avoided, the feature fusion effect is enhanced, and the decision accuracy is improved.

The invention discloses a data analysis and prediction method and system based on multi-modal data fusion, and the method comprises the steps: collecting multi-modal original data, and carrying out the preprocessing of the multi-modal original data, and generating normalized data; setting sliding window parameters, and performing sliding fragmentation on each modal data to obtain sliding window fragment data; extracting modal features, and fusing the modal features into sliding window level fusion feature representation; inputting the improved variational information bottleneck model, and performing feature compression and reconstruction; all fusion features are aggregated, a prediction result is output through modeling reasoning, and the model is optimized. According to the method, the improved sliding window algorithm and the variational information bottleneck model are fused, so that efficient analysis and accurate prediction of multi-modal data are realized.

The invention discloses an electric power dispatching method and device fusing multi-modal data and dual verification, electronic equipment and a storage medium, and belongs to the field of intelligent dispatching of an electric power system.The method comprises the steps that multi-modal dynamic data, including equipment images, load time sequences, logs, dispatching instructions, meteorological early warning and electricity price data, of an electric power system are obtained; visual, time sequence and semantic features are extracted, cross-modal alignment is carried out, and a joint representation vector is generated; based on the vector and the real-time data, an initial scheduling strategy is generated through a reinforcement learning model; and through historical scheme matching and physical constraint verification, determining a target scheduling strategy and executing scheduling. Therefore, by implementing the method and the device, the problems of incomplete power grid state perception caused by insufficient multi-modal data fusion and low reliability caused by lack of automatic credibility verification of the AI scheduling strategy in the prior art can be solved.

The invention provides a multi-modal data enhanced storage system based on a Parquet format. Unified storage and optimization of multi-modal data are achieved through native transformation and expansion of the Parquet format. The system comprises a modal perception format expansion component and a modal optimization coding strategy component, wherein the modal perception format expansion component realizes deep fusion of multi-modal semantics and a storage structure by expanding a Parquet logic type system and constructing a four-layer hierarchical metadata architecture from a file level to a page level; and performing instruction level fusion with a Parquet compression algorithm, executing targeted storage optimization based on semantic descriptors and modal characteristic parameters, and solving the problems of unified storage and efficient management of multi-modal data.

The invention relates to the technical field of Internet of Things data processing, in particular to a multi-modal data real-time processing method for an Internet of Things gateway, and the method comprises the steps: obtaining multi-source operation data, and packaging the multi-source operation data into a standard operation data set; obtaining high-bandwidth modal data in the standard operation data set; when the maximum link delay time is greater than a preset delay abnormal threshold value, obtaining a version effective fingerprint of the target model and a local model fingerprint; when a model version abnormity diagnosis result is obtained, calculating information redundancy; obtaining correction delay time, and comparing the correction delay time with a preset normal reference delay interval; and when the correction delay time is greater than or equal to the upper limit value of the normal reference delay interval, adjusting the current decision period. By monitoring delay in real time, diagnosing consistency of model versions, eliminating redundant information and executing link adjustment, accurate processing of high-bandwidth modal data is realized, and real-time performance and reliability of Internet of Things gateway data processing are improved.

A distributed operation and maintenance data management method based on a multi-modal log aggregation framework relates to the field of electrical digital data processing, and comprises the following steps: calculating a node load state index; determining node acquisition parameters including an acquisition period and an acquisition trigger condition; when the log file change amount of the target node is larger than the collection triggering condition, event collection is executed, and event collection data flow is obtained; when the log file change amount is smaller than or equal to the collection triggering condition, polling collection is executed according to the collection period, and a periodic collection data stream is obtained; generating a data compensation instruction and executing incremental acquisition to obtain a compensation data stream; merging to generate complete log data, and carrying out type distinguishing to generate a multi-modal data packet; and allocating the data to corresponding storage engines to obtain multi-modal log aggregation data. By implementing the method, the integrity of log data acquisition in a large-scale distributed system can be improved.

The invention discloses a label printing equipment abnormity diagnosis method based on multi-modal fusion, and the method comprises the steps: collecting multi-modal data, and generating a label path phase coding sequence; based on label path phase coding, locking alignment is executed on the multi-modal data, and unification is performed until a phase coordinate system is synchronized; inputting synchronous multi-modal data, inverting the slip amount, the elongation rate and the speed difference of the coiled material, and outputting implicit physical variables; constructing a three-domain hybrid diagnosis network which comprises phase, time and visual branches, and extracting a multi-modal feature vector; setting phase-physical consistency gating, performing sparse selection and routing on the sub-experts, and outputting a candidate feature set; and a phase alignment attention mechanism is adopted, candidate features are fused, and abnormal types, confidence coefficients and station positioning are output. Through multi-modal data phase alignment, physical constraint inversion and cross-domain fusion, high-precision anomaly diagnosis and intelligent deviation correction of the label printing equipment are realized.

The invention provides a mobile Internet of Things operation data management method and system, and relates to the technical field of mobile Internet of Things data management, and the method comprises the following steps: collecting and preprocessing multi-modal data, and obtaining the preprocessed data; calculating a data change rate and dividing the data change rate into data in different stable states based on the preprocessed data, and performing hierarchical compression on the data in different stable states to obtain compressed data; calculating a data synchronization priority, and performing data synchronization on the compressed data according to the data synchronization priority to obtain synchronized data; verifying the synchronized data, and calculating a retransmission index to carry out data retransmission to obtain the data after data retransmission; and storing the data after data retransmission, and constructing a composite index. The method has the advantages that the compression strategy and the data synchronization strategy are dynamically applied, the data compression and data transmission efficiency is high, the composite index comprises the data compression feature and the time-space feature, and cross-modal data rapid positioning is effectively achieved.

The invention discloses an intelligent control method and device, electronic equipment and a readable storage medium, and belongs to the technical field of intelligent control. Under the condition that the gesture instruction does not conflict with the voice instruction, performing multi-dimensional normalization on the gesture feature vector to obtain a to-be-fused gesture vector; performing normalization processing on the voice feature vector based on context semantic perception to obtain a to-be-fused voice vector; fusing the to-be-fused gesture vector and the to-be-fused voice vector to obtain a fusion vector, and determining a fusion instruction corresponding to the fusion vector as a target execution instruction; and under the condition that the gesture instruction conflicts with the voice instruction, selecting a target execution instruction according to a first confidence coefficient corresponding to the gesture instruction and a second confidence coefficient corresponding to the voice instruction. The self-adaptive normalization strategy is designed for different modal data, the data fusion accuracy is improved, meanwhile, the gesture and voice instruction conflict can be intelligently and flexibly processed, and the accuracy of determining the real intention of the user is improved.

The invention relates to an interaction instruction identification method and device and an electronic whiteboard cooperation system, and the method comprises the steps: obtaining the interaction data of each mode in a target scene; based on a multi-modal cross attention mechanism, carrying out data fusion on the interaction data of each modal to obtain fused modal data, and combining the fused modal data and the interaction data of each modal into target modal data; judging whether the user intentions corresponding to the target modal data have intention conflicts or not, and if yes, determining a to-be-executed intention from different user intentions based on a preset conflict resolution rule in the target scene; and generating an execution operation instruction according to the to-be-executed intention. Based on a multi-modal cross attention mechanism, accurate fusion of interaction data of different modals can be completed, the data isomerism of different modals and the expression form difference of the same intention are solved, and the accuracy of interaction instruction recognition is finally improved in combination with a conflict resolution rule.

The invention discloses a multi-modal video recognition and low-code complex time sequence intelligent algorithm arrangement method, which comprises the following steps of: firstly, jointly constructing a safety production basic algorithm library by using a multi-modal video recognition technology and experience accumulated in previous projects, then analyzing requirements, determining a requirement function and a mixed algorithm requirement, and constructing a safety production basic algorithm library; a reasonable large model architecture is designed; a perfect time sequence algorithm model is established by deeply analyzing the time sequence dependency relationship under various scenes; machine learning and artificial intelligence technologies are utilized to realize dynamic optimization of an algorithm execution sequence; and finally, a friendly interface of the main body is developed and provided by using a low-code technology, so that the main body can complete the arrangement of a complex time sequence algorithm through simple operation. The invention aims to improve the accuracy of video recognition through multi-modal data fusion, simplify the arrangement process of a complex time sequence intelligent algorithm by using a low-code technology, and accelerate the development and deployment of the algorithm.

The invention relates to the technical field of computer data processing, and discloses a multi-modal data labeling method based on a large model technology, and the method comprises the steps: 1, carrying out the data collection of multi-modal data, and carrying out the data preprocessing of the collected multi-modal data; 2, single-mode feature extraction is carried out on each kind of single-mode data in the multi-mode data, cross-mode feature fusion is carried out, and a fusion feature vector capable of being co-processed is generated; 3, performing model training according to the fusion feature vector to obtain a joint training model, and labeling the multi-modal data based on the joint training model to obtain a preliminary labeling result; and 4, carrying out manual intervention and data storage on the preliminary labeling result, and outputting a labeling result. By means of the scheme, accurate and efficient annotation of various types of data such as texts, images, voices and videos can be achieved, and deep application and development of multi-modal data in the field of artificial intelligence are promoted.

The invention relates to a local-global association learning-based multi-modal power data restoration method, which comprises the steps of obtaining initial data and performing preprocessing to obtain a power data abnormal value, the initial data comprising power operation data, external association data and multi-modal data; based on the preprocessed power operation data and the power data abnormal value, local association learning is carried out, local data association is mined, and a local restoration scheme of the power data abnormal value is obtained; performing global association learning based on the preprocessed initial data and the abnormal value of the power data, mining a cross-time and cross-source global association rule, and obtaining a reasonable correction range of the abnormal value of the power data; and generating a power operation data restoration result in combination with the local restoration scheme and the reasonable correction range. Compared with the prior art, the method has the advantages that the data exception can be accurately recognized and repaired, and the data repair accuracy is greatly improved.

The invention provides a video evidence obtaining method based on Transform and quantum features, and belongs to the technical field of crossing of multimedia content security and computer vision, and the method comprises the steps: S1, collecting multi-modal original data; s2, preprocessing the multi-modal data; s3, single-mode authenticity preliminary detection is carried out; s4, multi-modal feature fusion of quantum optimization is carried out; and S5, multi-modal authenticity comprehensive judgment is carried out. Through multi-mode cooperation and quantum technology innovation, the problems of insufficient robustness, inaccurate positioning and the like of a traditional evidence obtaining technology are effectively solved, and an efficient, accurate and feasible technical scheme is provided for authenticity verification of video content.

The invention discloses a hotspot sensing hierarchical storage and computing resource scheduling method oriented to multi-modal data, and belongs to the technical field of data center resource management. The core of the method is as follows: firstly, monitoring characteristics, access modes and computing node loads of multi-modal data in real time; secondly, a comprehensive popularity evaluation model is used for dynamically calculating the comprehensive popularity value of the data object, and the model comprehensively considers the data type, the access frequency, the data size and the task priority; then, intelligently scheduling the data to different performance levels of the hierarchical storage system according to the popularity value; and finally, when a calculation task is scheduled, synchronously sensing the heat and the position of required data, and carrying out collaborative scheduling by taking optimal data locality and load balance as targets. According to the method, storage and computing resources are managed cooperatively through the unified heat index, the problem of resource scheduling disjunction in a multi-modal data environment is effectively solved, and the system throughput rate, the response speed and the resource utilization efficiency are remarkably improved.

The invention relates to the technical field of data processing, and discloses a multi-modal data processing method, device, equipment and medium, which can be applied to the financial and medical fields, the method comprises the following steps: preprocessing multi-modal input data to obtain standardized multi-modal data, and constructing a training data set for simulating missing modalities; carrying out feature generation on existing missing modalities to construct a complete modal feature set; and inputting each modal feature in the complete modal feature set into the corresponding independent processing sub-network, and fusing to obtain a multi-modal data processing result. In the invention, aiming at the problem that available modal information cannot be effectively utilized to supplement a missing modal in a complex scene in which multi-modal data are simultaneously missing in the prior art, a generative completion model can be utilized to perform feature generation on the existing missing modal, and the feature is input to a corresponding independent processing sub-network; and finally, outputting a multi-modal data processing result, so that the decision accuracy of the model in a modal missing scene is improved.

The invention discloses a data analysis method and device based on cross-modal data fusion, equipment and a medium, and relates to the field of data processing. The method comprises the following steps: acquiring multi-modal data associated with a data analysis target; performing time alignment on each modal data, and performing space alignment based on each modal data after time alignment; obtaining feature information corresponding to each mode in the multi-mode data after spatial alignment, and performing associated feature information identification between the feature information of each two modes to obtain an associated feature information identification result between the multi-mode data; and screening the feature information of each modal based on an associated feature information identification result, performing weighted fusion processing on the screened feature information of each modal to obtain a fusion result, and performing data analysis based on the fusion result to obtain an analysis result corresponding to the data analysis target. The efficiency of data fusion and the comprehensiveness and accuracy of data analysis are improved.

The invention relates to the technical field of bridge state evaluation, in particular to a bridge state evaluation method and system based on a large model, and the method comprises the following steps: collecting historical multi-modal data of different regions on a bridge, and then generating a standardized multi-modal data set; extracting multi-modal features from the standardized data set by using a pre-training field large model, and generating a data state tag as a weak supervision signal; constructing a first-level bridge state evaluation model based on a Transform architecture according to the multi-modal features and the weak supervision signals, wherein the first-level bridge state evaluation model is used for evaluating the regional state of the bridge and generating a regional state vector; and constructing a Transform-based secondary bridge state evaluation model, and further obtaining an overall state evaluation result of the bridge according to the regional state vector and the space coordinate code. According to the method, through pre-training the large model and the cross-modal attention mechanism, the data annotation cost can be reduced, and precise assessment from the local state to the overall state of the bridge is realized in combination with a two-stage Transform architecture.

The invention provides an intelligent question and answer method and device, electronic equipment and a computer readable storage medium. The method comprises the steps of obtaining question data input by a user; decomposing the problem data to determine a plurality of subtasks corresponding to the problem data; for each subtask, determining a processing data type of the subtask, and matching an intelligent agent corresponding to the processing data type; generating a target code for solving the subtask through the intelligent agent, and executing the target code to obtain target information corresponding to the subtask; and fusing the target information to obtain target output data corresponding to the problem data. According to the method, the subtasks are allocated to the intelligent agents of the corresponding types on the basis of the types of the processed data, so that the multi-modal data can be supported, meanwhile, the target codes are generated on the basis of the specific subtasks, the subtasks are completed through the target codes, the limitation that only specific data can be supported is solved, and the performability of intelligent question answering is greatly expanded.

The invention discloses a cross-factory equipment fault collaborative diagnosis method and device, equipment and a medium, and relates to the technical field of computers. Comprising the following steps: verifying related authentication data provided by a participant, and accessing the participant to a preset zero-trust authentication system based on a verification result; the method comprises the following steps: acquiring multi-modal data of target equipment, and preprocessing the multi-modal data to obtain multi-modal data features; performing federated learning training on the initial diagnosis model based on the multi-modal data features to obtain a cross-factory joint diagnosis model; obtaining a historical fault case, and constructing a target diagnosis knowledge graph related to the target equipment based on the historical fault case; and if the target equipment has a fault, performing combined diagnosis by using the cross-factory combined diagnosis model and the target diagnosis knowledge graph based on a preset zero-trust authentication system to obtain a target diagnosis result. Therefore, safe, efficient and interpretable cross-factory equipment fault collaborative diagnosis can be realized.

The invention discloses an emotion analysis method and device, computer equipment and a storage medium. The method comprises the steps of obtaining first modal data and second modal data having emotion relevance with the first modal data; obtaining first modal representation data according to the first modal data, and obtaining second modal representation data according to the second modal data; performing cross-modal fusion on the first modal representation data and the second modal representation data to obtain cross-modal association representation data; performing feature mining on the basis of the cross-modal association representation data to obtain at least two pieces of characteristic feature data which contribute to sentiment analysis; fusing the at least two pieces of characteristic feature data to obtain target fusion representation data; and performing sentiment analysis according to the target fusion representation data to obtain a sentiment analysis result. Therefore, efficient fusion and feature mining are carried out on the multi-modal representation through the multi-level fusion mechanism, and the sentiment analysis accuracy is effectively improved.

The invention provides a content recommendation method, a server and a display device, and the method comprises the steps: receiving multi-modal data obtained when the display device merges at least one modal data of different platforms, carrying out the multi-modal feature fusion of the multi-modal data, obtaining a multi-modal interest feature, then receiving a recommendation request sent by the display device, and carrying out the recommendation of the multi-modal interest feature. And querying media asset recommendation data in a database according to the multi-modal interest characteristics, and then sending the media asset recommendation data to a display device, so that the display device displays a picture corresponding to the media asset recommendation data. By combining behavior operation data of the user on modes of videos, texts, audios and the like in different application programs, the media asset recommendation data finally pushed to the display equipment better conforms to interest preferences of the user, and the accuracy of pushing the media asset recommendation data is improved.

The invention relates to an intelligent home recommendation method and system based on cross-modal fusion. The method comprises the steps that user modal data, environment modal data and equipment behavior modal data are collected in real time; converting the environment modal data into quantum state amplitude features through a quantum encoder, and obtaining user behavior phase features through a time sequence feature extractor; performing cross-modal fusion on the quantum state amplitude features and the behavior phase features based on quantum entanglement operation to generate a dynamic fusion vector; inputting the dynamic fusion vector into a hybrid decision engine, outputting a recommendation instruction set by the hybrid decision engine, and verifying the physical feasibility of the recommendation instruction set based on an equipment state knowledge graph; and generating and executing a final recommendation instruction according to the verification result. According to the invention, intelligent matching of the job seeker and the recruitment demand is realized, and the recruitment efficiency and accuracy are improved.

The invention relates to a dynamic data fusion-based virtual-real synchronization system and method for a digital twin system, and the system comprises a multi-modal data collection layer, a virtual-real mapping engine, a self-adaptive model updating module and a bidirectional control interface, and collects the multi-source sensor data of a physical entity in real time through an edge calculation node; high-precision dynamic matching of the physical coordinate system and the virtual model is realized by using a spatial three-point alignment algorithm; adaptively adjusting the simulation step length and the updating frequency of the digital twinborn model based on a reinforcement learning algorithm; closed-loop control of the virtual layer on the physical layer is realized through a bidirectional control interface; according to the method, the problems of model updating delay, difficulty in heterogeneous data integration, large transmission delay and the like are effectively solved, the synchronous delay is reduced to be within 50ms from 200ms, the model prediction accuracy is improved, real-time processing of 100,000-level concurrent data points is supported, and the real-time performance, the precision and the intelligent level of the digital twin system are remarkably improved.

The invention discloses a method for efficiently caching off-line state data of a water affair edge intelligent gateway based on multi-modal data, which relates to the technical field of water affair edge intelligent gateways, and comprises the following steps of: under the condition of identifying that a sensor is temporarily restarted, constructing a modal acquisition reference matrix by using a multi-modal data structure of a historical sampling period; a modal difference vector is generated in combination with the actual collection content of the current period and is used for determining the modal data gap state under the condition that the sensor is temporarily restarted; and based on the determined modal data gap state, carrying out matching analysis on the image data acquisition record at the current time point and the modal data gap vector at the current time point, and identifying modal inconsistency of the image and the sensor in the same time point cache. According to the invention, the problem that the modal gap of the multi-modal data cannot be identified and marked in an offline state is solved, and modal consistency identification and structural integrity guarantee are realized.