168 results about "Text mode" patented technology

The invention discloses a large model enhanced cross-modal retrieval method and system and related equipment. The cross-modal retrieval method comprises the following steps: acquiring an image-text pair; based on the image-text pair, obtaining large model enhancement information of the image-text pair, combining an original image, a text and the enhancement information, adopting a multi-branch encoder of a pre-training CLIP model to obtain a plurality of feature vectors, constructing a multi-stage collaborative alignment loss function, and performing collaborative alignment on an image mode and a text mode in a public semantic space; and training a model through the multi-stage collaborative alignment loss function and a pre-constructed training database, and performing retrieval through the trained model. According to the method, collaborative learning is carried out on the image-text features obtained by the encoder, auxiliary semantic enhancement information is added, and collaborative alignment is carried out on the image modality and the text modality in the public semantic space, so that a better retrieval network is trained, and the accuracy of image-text retrieval is improved.

The invention provides a financial analysis method and system based on big data, and the method comprises the steps: analyzing the dependency relationship between a non-standard code combination and a fuzzy text mode in a material purchasing scene based on a credible rule library, extracting a new attribute combination feature, and forming an optimization rule feature set if the new attribute combination feature can improve the matching accuracy; performing rule-based key field and attribute set verification matching on the real-time non-standard transaction data according to the optimization rule feature set in combination with a dependency relationship between a non-standard code combination and a fuzzy text mode, and outputting a matched transaction record set; and comparing the difference between the matched transaction record set and the historical manual matching record, extracting and judging whether the newly added non-standard attribute combination is applicable to the non-standard transaction data which is not covered or not after a matching rule is adjusted by adopting an incremental learning algorithm, and if so, forming a dynamic matching table.

The invention is applicable to the technical field of artificial intelligence, and provides a cross-modal retrieval method and device, electronic equipment and a storage medium, and the method comprises the following steps: encoding an image modal through an image encoder, encoding a text modal through a text encoder, and generating a vector representation of a high-dimensional space; inputting the vector of the image mode and the vector of the text mode into the same shared self-attention layer for alignment, and extracting features through respective feedforward network layers; performing cross-modal cross fusion processing on the vector of the image modal and the vector of the text modal through a cross attention mechanism to obtain a cross-modal representation vector of the image modal and a cross-modal representation vector of the text modal; and based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, outputting the similarity of the image modality and the text modality. According to the method, the cross-modal retrieval precision can be improved.

The invention provides a multi-modal pre-training model construction method and system for monitoring videos, and the method comprises the steps: automatically constructing a high-quality multi-modal alignment data set through a single-modal description generation model and a large-scale language model, and remarkably reducing the marking cost; a special coding network and a shared projection layer are adopted to realize feature extraction and uniform semantic space alignment of video, audio and text modes; performing dynamic semantic fusion by using a modal collaborative attention mechanism; designing cross-modal contrast learning, mask prediction and time sequence consistency tasks to carry out multi-task pre-training; an external knowledge base is introduced, and the semantic reasoning ability is enhanced through a microretrieval mechanism; and optimizing model parameters by adopting a multi-task joint loss function and an end-to-end training strategy. According to the method, efficient and automatic construction, deep semantic alignment and fusion and intelligent reasoning of knowledge enhancement of monitoring video multi-modal data are realized, and the understanding and generalization ability of the model in a complex scene is effectively improved.

The invention belongs to the technical field of image processing, and discloses a multi-image processing method based on multi-modal entity alignment, which comprises the following steps: acquiring a plurality of images; retrieving rich semantic information of the entity from an external knowledge base; selecting the most representative image for each entity in the multi-image scene by using semantic information; encoding the original input of each mode; performing first-layer fusion between a visual mode and a text mode by using cross-divergent attention, then performing second-layer interaction with a structured mode, and finally aligning an entity representation of an image by using contrast loss; and outputting multiple images fusing the text modality and the visual modality. According to the invention, hierarchical interaction fusion is utilized to enhance multi-modal interaction; enhancing entity text representation by integrating the external attribute value and the context information; and the most representative image is selected by using the semantic text, so that the influence of uncorrelated images is minimized.

The invention discloses a continuous sign language recognition method based on multi-granularity cross-modal comparative learning. The method comprises the following steps: firstly, fully tapping a hidden semantic mapping relationship between a hidden sign language video and a corresponding sign language text; then, a multi-granularity cross-modal comparison mechanism is used for relieving potential influences of redundant frames and semantic irrelevant identification words on cross-modal alignment, and through a coarse-granularity video-sentence comparison strategy and a fine-granularity frame-word comparison strategy, visual and text modal features are effectively aligned from global and local angles; semantic interference caused by redundant frames and semantic irrelevant identification words is relieved through cross-granularity video-word comparison and cross-granularity frame-sentence comparison strategies; and finally, obtaining sign language text sentences through a decoder by utilizing visual embedding obtained in a visual encoder, and taking the sign language text sentences as predicted continuous sign language recognition results. The hidden semantic mapping relation between the sign language video and the corresponding sign language text is fully excavated, the semantic understanding of the sign language model on the visual context is enhanced, and the applicability and accuracy of sign language recognition are improved.

The invention discloses a multi-modal multi-label emotion recognition method and system based on modal contribution assessment, and the method comprises the steps: obtaining original feature sequences of a text mode, a visual mode and an audio mode, mapping the original feature sequences through an encoder, and obtaining high-dimensional feature embedding of the three modes; calculating a marginal contribution degree of each mode to emotion prediction at a sample level by utilizing emotion prediction values embedded by high-dimensional features of the three modes; according to the marginal contribution degree of each modal, fusing the features of each modal to form a fused feature, and processing the fused feature through a multi-modal encoder to obtain a multi-modal integrated feature; and through a trainable label embedding sequence, learning a dependency relationship between emotion labels, embedding and inputting the multi-modal integrated features and the labels into an emotion decoder to obtain a multi-modal emotion representation of mixed emotion, and inputting the multi-modal emotion representation into a multi-label classifier to obtain a predicted emotion label. According to the invention, the efficiency of multi-modal feature fusion and the accuracy of emotion recognition can be improved.

The invention provides a text data automatic labeling method and system integrating a pre-training NLP model, and relates to the technical field of artificial intelligence, the method comprises the following steps: obtaining original text data to be labeled, the text data comprising customer consultation, complaint or feedback dialogue records, evaluations and work orders; performing word segmentation, stop word filtering and part-of-speech tagging processing on the original text data to generate structured text data; inputting the structured text data into a pre-trained NLP model, and correcting semantic features output by the model through a domain adaptation layer to obtain corrected semantic features; and carrying out few-sample learning on the corrected semantic features, enhancing the recognition capability on a low-frequency text mode through a contrast learning mechanism, and generating an enhanced feature vector resistant to long-tail interference. According to the method, automatic annotation of the text data is realized by automatically processing data acquisition and preprocessing and integrating the pre-training NLP model.

The invention discloses a dialogue emotion recognition method and device based on a multi-modal large model, and relates to the field of emotion recognition, and the method comprises the steps: obtaining all statement sets in a dialogue in a current scene; each statement comprises an audio mode, a video mode and a text mode; constructing a dialogue emotion recognition model; the dialogue emotion recognition model comprises a feature extraction layer, a bidirectional gating unit, a multi-modal large model, a BERT language model, a modal information complementation module, a semantic graph-based multilayer residual image convolutional network and a full connection layer; and according to all statement sets in the dialogue, adopting the trained dialogue emotion recognition model to obtain an emotion recognition result. The dialogue emotion recognition method and device can improve the accuracy and robustness of dialogue emotion recognition.

The invention provides a multi-modal sentiment analysis method for contrast enhancement non-isomorphic modal fusion, and relates to the technical field of sentiment analys.According to the method, a visual modal feature and an audio modal feature are fused to make up for the difference of information density with text modalities, and joint contrast learning is designed to make up for the difference of information density with text modalities. The loss in the samples, the loss between the samples and the contrast learning in the samples are obtained through the contrast learning in the samples and the contrast learning between the samples, so that the consistency between the modes and the contrast learning between the samples are enhanced, the independence and the uniqueness of each mode feature are guaranteed, and information loss caused by excessive fusion is avoided. According to the method, complementarity and difference between modes are considered, the difference of information density between the visual and audio modes and the text mode is made up, and the performance and generalization ability of the sentiment analysis model are improved.

The invention belongs to the crossing field of computer vision and city design, and provides a multi-mode cooperative driving DiT city layout generation method, so as to solve the defects of an existing generation model in the aspects of spatial rationality and planning controllability. According to the method, a transformer-based diffusion generation model is innovatively constructed, and two additional control signals are integrated in a DiT model to realize controlled generation. On the aspect of model architecture design, a DiT framework with multi-modal condition fusion is constructed, and a road network sketch (image control) and planning semantics (text control) dual guide mechanism is integrated. In the text mode, a dynamic semantic fusion module based on a pre-training language model is designed, and text information is deeply embedded into the generation process. In the aspect of image control, a mixed attention regulation and control mechanism is put forward, multi-scale fusion of road network structural features is realized in combination with cross attention and AdaLN technologies, and the problem of spatial layout distortion is solved. Experiments show that the method can effectively integrate text and image information, and the generated city layout image is superior to that generated by an existing method.

The invention discloses a speech emotion recognition method based on multi-modal feature fusion, and the method comprises the steps: designing a multi-modal speech emotion recognition model, enabling an audio to serve as a main mode, employing the function of wav2vec2.0, and combining a text mode with a visual mode to enhance the performance of emotion recognition based on the audio; the self-attention mechanism, the GRU and the LVC are combined to extract global and local information from a visual mode, the global information is extracted only from text features, visual clues and text clues related to emotion can be comprehensively understood, the extracted text and visual features are integrated into a wav2vec2.0 model, multi-mode representation of multi-level audio information is formed, and the multi-mode representation of the multi-level audio information is realized. And meanwhile, the discrimination capability of the model is enhanced by utilizing multi-modal homogeneous feature difference learning, and the proposed model is evaluated on two data sets, so that the model is proved to have excellent robustness and generalization compared with the existing most advanced model.

The invention discloses a hatred model factor identification method and system based on dual granularity symmetry and context perception alignment. The method comprises the following steps: firstly, respectively extracting a text mode and a visual mode from input image data containing text information, and carrying out feature coding and embedding so as to obtain text mode features and visual mode features; according to the method, a mode of constructing same-granularity and cross-granularity dual symmetric factors in visual and text modal representation is adopted to control modal internal feature fusion, so that a function of generating high-semantic-consistency modal internal feature representation is realized; in the inter-modal representation fusion process, a context perception cross-modal alignment strategy is introduced, and dual symmetric modal information is used as a prompt, so that the understanding of memetic context information is enhanced; whether aggressive or discrimination information aiming at individuals or groups is transmitted or not can be accurately identified by excavating potential emotional tendency and value orientation, and the identification accuracy and efficiency of the hatred model factor are improved.

The invention is suitable for the technical field of sentiment analysis, and particularly relates to a multi-modal sentiment analysis method based on pre-training and text modal guidance, and the method is implemented in two stages: the pre-training stage comprises the steps of obtaining initial features of each modal of multi-modal data, and decomposing each modal feature into similar features and different features; constructing positive and negative pairs in and between samples, calculating comparison loss and single-modal prediction errors, extracting module and projector parameters, and inputting a short video to obtain each modal feature; high-scale text representation is learned through initial text similar features, super-modal features are adaptively updated through a multi-head attention mechanism, and audio and video redundant information is suppressed by using a weight module guided by a text mode; through the data sampler and the feature comparison learning method, multi-mode fusion with the text mode as an anchor point is achieved, the emotional tendency in a complex scene can be recognized more accurately, and the method has wide value in the fields of emotion classification, social media content recommendation and the like.

The invention discloses an emotion analysis method and device for promoting multi-mode information fusion, and the method comprises the steps: building an acoustic vocabulary and a visual vocabulary based on a K-Means algorithm, querying the features of a non-text mode, obtaining a word index sequence of the non-text mode, and embedding the word index sequence into the initial features of the text mode to build a non-language displacement vector; adding the non-language displacement vector into the initial feature of the text mode for displacement operation to obtain a multi-mode vector; and performing context sensing and sentiment analysis on the multi-modal vector to obtain and output a sentiment analysis result. According to the method, the feature sequence of the non-text mode is converted into the word index sequence which is the same as the text mode through the K-Means algorithm, the distribution gap between the language text mode and the non-language text mode during fusion is reduced, and subsequent feature fusion is facilitated; during feature fusion, the expression of a language text mode is enhanced through a non-language displacement vector, so that the accuracy of sentiment analysis is improved.

The invention provides a knowledge graph construction method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a script text and a deductive video corresponding to the script text; constructing a plurality of first nodes of a text mode based on the script text, and constructing a plurality of second nodes of an image mode and a plurality of third nodes of an audio mode based on the deductive video; based on the content information corresponding to the fourth node and the fifth node, determining a semantic consistency score between the fourth node and the fifth node; determining a time sequence synchronization score between the fourth node and the fifth node based on the time sequence information corresponding to the fourth node and the fifth node; in response to the fact that the semantic consistency score is larger than a first threshold value and the time sequence synchronism score is larger than a second threshold value, constructing a first relationship between the fourth node and the fifth node; and constructing the knowledge graph based on the plurality of first nodes, the plurality of second nodes, the plurality of third nodes and the plurality of first relationships.

The invention relates to the technical field of big data management, and particularly provides a data rule design method and device for intelligent visual online testing, and the method comprises the following steps: S1, defining a data rule: defining the rule from three aspects of custom quality dimension, a basic rule base and a business rule base; s2, the JavaScript script is verified, and the JavaScript S3, performing matching by using a regular expression, and searching and replacing a complex text mode; s4, online testing and formatting are carried out, regular expressions or JS scripts are selected to carry out online testing and editing on data rule contents, meanwhile, the rule contents are displayed as verification or processing results of the data, in addition, the JS scripts carry out parameter analysis on rule methods, and attribute classification and filling are carried out on parameters; and S5, task scheduling and optimization. Compared with the prior art, the method can provide data meanings and business rule standards which are commonly obeyed by an upper business level.

The invention discloses an emotion feature fused bimodal forged information detection method, which comprises the following steps of: extracting emotion features, respectively extracting emotion categories of a text and an image through a large language model, respectively giving emotion intensity of the categories, and scoring the emotion; analyzing the emotion consistency of the mixed modal features of the text and the image; the hybrid expert network calculates domain probabilities to which a text mode, an image mode and a hybrid mode belong respectively through a text domain encoder, an image domain encoder and an image text hybrid feature domain encoder based on domain class processing features; the hybrid expert network learns feature information of a text mode, an image mode and a hybrid mode according to the domain information; the AdaIN network adjusts modal distribution, the AdaIN network adjusts feature distribution according to contribution of each modal, the adjusted features are fused and input into the classifier, a final classification result is obtained, and the information detection accuracy is greatly improved.

The invention relates to an image processing technology, in particular to a multi-modal large model reasoning acceleration method fusing image feature compression and self-adaptive pruning, which comprises the following steps of: determining the optimal image block number of an input image according to the aspect ratio of the input image and the resolution of a reference image so as to dynamically match and adjust the image resolution; processing the input image after the image resolution is adjusted through a visual encoder and an attention module to obtain image feature tokens, analyzing the redundancy and semantic information contribution degree between the image feature tokens, retaining the image feature token which contributes to the image information amount greatly, and realizing image feature compression; encoding the input text to obtain a text feature token; according to the importance scores of the image feature tokens and the text feature tokens, single-mode redundancy features and cross-mode redundancy features of the visual mode and the text mode are recognized and removed, and a redundancy model structure is cut off. The model reasoning speed is increased as much as possible while effective information is reserved.

The invention provides an infrared and visible light image fusion method based on multi-semantic deep collaboration. The method mainly solves the problem that an existing method cannot fully integrate text modals and global consistency between image fusion and downstream tasks. Comprising the following steps: 1) constructing a dual-task parallel network structure, and efficiently establishing a deep correlation between image fusion and a downstream segmentation task; 2) designing a multi-semantic deep collaboration module, realizing effective fusion of multi-modal information by deeply integrating text features, pixel-level features and segmented semantic features, and meeting semantic requirements of downstream tasks; 3) guiding an image fusion and segmentation task by using deeper and fine-grained semantic information in a text mode, and enhancing semantic consistency between a fusion result and a downstream task; and 4) inputting the obtained multiple semantic features into a fusion decoder to generate a final image fusion result. The semantic comprehension and visual perception capabilities of the model can be effectively enhanced, and the multi-modal image fusion performance is improved.

The invention discloses a multi-modal federated learning method based on self-attention, and relates to the field of data processing, and the method comprises the steps: firstly, designing a multi-modal feature coding module at a client, carrying out the deep semantic modeling of image and text modals based on a Transform architecture, and capturing the internal structure information of each modality; secondly, a local cross-modal self-attention modeling module is introduced, fine-grained semantic alignment between images and texts is modeled through a bidirectional cross-attention mechanism, and the local cross-modal understanding ability is enhanced; a multi-head collaborative fusion optimization module is further combined, inter-modal deep semantic collaborative modeling is carried out under a multi-attention perspective, and a fusion effect is optimized through a gating mechanism, so that the problems of inter-modal conflicts and inconsistent expressions are effectively relieved.

The invention discloses an incomplete multi-mode dialogue emotion recognition method and system based on a speaker and time sequence information joint graph network, and the method comprises the steps: obtaining the deep features of a text mode, a voice mode and a visual mode in a dialogue through a feature extraction module, and guaranteeing the high expression capability of the features through a pre-training model; the random mode missing simulation module effectively simulates the data incomplete condition in a real scene, and the robustness of the model is improved; a bi-directional long-short term memory network (Bi-LSTM) is combined with a time sequence diagram network (TGNN) to capture context and time dynamic characteristics of a dialogue, and meanwhile, an interaction relationship between speakers is modeled through a speaker influence matrix, so that joint modeling of a time sequence and speaker information is realized; the deep features are further extracted through the graph convolutional network, and the emotion discrimination of the features is enhanced; finally, the modal reconstruction and emotion classification module significantly improves the accuracy and robustness of incomplete multi-modal dialogue emotion recognition through reconstruction of missing modals and multi-class emotion prediction, and is suitable for man-machine interaction and emotion analysis application in a complex real scene.

The invention provides a flood event multi-mode retrieval method and device based on a semantic intensifier, and relates to the technical field of data retrieval, and the method comprises the steps: carrying out multi-stage semantic enhancement on keywords through a global flood event knowledge base and internet news information included in the semantic intensifier, then quality priority retrieval and preliminary screening are carried out to obtain a to-be-matched multi-source remote sensing image, and corresponding optical and radar image features and context semantic features of a semantic enhancement text are extracted from an image mode and a text mode respectively; and then fusing the features of the image modality and the text modality to obtain a multi-modality fusion feature, mapping to obtain the similarity between the semantic enhancement text and the to-be-matched multi-source remote sensing image, and finally determining a retrieval result of the keyword. According to the method and the device, the problems of low retrieval matching degree and low retrieval efficiency caused by lack of semantics and difficulty in acquisition of event-related space-time ranges in flood event remote sensing data retrieval are solved.

The invention discloses a multi-modal rumor detection system based on multi-field perception, relates to the technical field of rumor detection, and is used for solving the technical problem of low multi-field rumor detection precision in the prior art. The rumor detection system comprises a multi-view feature extraction module, a multi-domain perception module, a multi-level knowledge fusion module and a final classifier. Wherein the multi-view feature extraction module is used for respectively extracting corresponding single-mode features from a text mode and an image mode, and acquiring aligned image and text features as multi-mode view features; the multi-domain sensing module is used for learning cross-domain shared knowledge and domain specific knowledge at a text view angle, an image view angle and a multi-mode view angle; the multi-level knowledge fusion module is used for fusing cross-domain shared knowledge and domain specific knowledge of a text view angle, an image view angle and a multi-mode view angle; and the final classifier is used for obtaining a prediction result according to the cross-domain shared knowledge and the domain specific knowledge obtained by the multi-level knowledge fusion module.

The invention provides a reference video target segmentation method and system based on zero sample learning, and belongs to the technical field of target identification. And processing the acquired reference video data by using a pre-trained reference video target segmentation model to obtain a final target segmentation result. According to the reference video target segmentation deep learning network model based on multi-granularity feature fusion of zero sample learning, a multi-granularity visual feature extraction module is introduced, and a target granularity visual feature extraction stage, a frame granularity visual feature extraction stage and a video granularity feature extraction stage are included; the target granularity visual feature extraction stage integrates global semantic information of video frames, the visual feature extraction stage of frame granularity can further align a visual mode and a text mode, the visual feature extraction stage of video granularity can effectively extract motion clues in a video, visual feature representation is enhanced, and the visual feature extraction efficiency is improved. The multi-granularity visual features effectively improve the segmentation performance.

The invention relates to the technical field of computers, and provides a regional attribute recognition method and related device.The method comprises the steps that feature extraction is conducted on at least two kinds of modal data extracted based on multimedia resources, and corresponding initial features are obtained; performing cross-modal feature fusion on the obtained at least two initial features to obtain a fused feature; for the initial feature of each non-text mode, performing global feature extraction on a plurality of sub-features in one initial feature to obtain corresponding non-text global features; each sub-feature represents one item of sub-data in the corresponding non-text modal data; based on the fusion feature, combining the obtained at least one non-text global feature to obtain a region recognition result of the multimedia resource; and when the region identification result shows that the multimedia resource is regional, identifying a target region name associated with a region event corresponding to the multimedia resource based on the text modal data, thereby accurately identifying the region attribute of the multimedia resource.

The invention relates to the technical field of natural language processing, and discloses a cross-border consumption behavior dynamic analysis method and device based on a large language model.The method comprises the steps that multi-modal original data, obtained from multiple data sources, in a target cross-border consumption scene is obtained, the multi-modal original data at least comprises text data and visual data related to the target commodity or the target service; performing feature extraction and fusion processing on the multi-modal original data to generate a unified semantic representation containing text semantics and visual semantics; identifying a scene type to which the target cross-border consumption scene belongs, and determining contribution weights corresponding to a text mode and a visual mode in the unified semantic representation based on the scene type; and analyzing the weighted unified semantic representation according to the contribution weight, and generating a consumer behavior intention label corresponding to the target cross-border consumption scene. The cross-border consumption behavior intention analysis method and device can improve the suitability and accuracy of cross-border consumption behavior intention analysis, and meet the refined operation requirements of cross-border e-commerce.

Methods, apparatuses, devices and media are provided for generating a video in a text mode in an information sharing application. In a method, a request is received for generating the video from a user of the information sharing application. An initial page is displayed for generating the video in the information sharing application, the initial page comprising an indication for entering a text. A text input is obtained from the user in response to a detection of a touch by the user in an area where the initial page locates. A video to be published in the information sharing application is generated based on the text input. In some examples, within the information sharing application, the user may directly generate a corresponding video based on a text input. In this way, a complexity of user operation may be reduced, and the user may be provided with richer publishing content.

The invention discloses a method for controlling an autonomous vehicle and related equipment, and the method for controlling the autonomous vehicle comprises the steps: responding to a natural language instruction of a user, and generating a text feature based on the natural language instruction; collecting original sensor data, and inputting the original sensor data into a visual encoder of a VLA model for processing to obtain key features representing a current environment; the key features and the text features are input into the VLA model, a decision-making thinking chain output by the VLA model in a text mode is obtained, multimedia display is conducted on the decision-making thinking chain, and the decision-making thinking chain comprises the current state of a vehicle, decision-making reasons and a decision-making process.

The invention discloses a multi-modal language model-assisted category guide image segmentation method and system, and relates to the technical field of image processing, and the method comprises the steps: respectively extracting the original image features and user interaction click features of a to-be-segmented image through an image encoder and a prompt encoder; coding the true value mask by using a convolutional network, splicing the true value mask with original image features, and then carrying out cross attention fusion to generate mask image features; extracting original text features through a text encoder, and aligning the original text features with mask image features based on comparative learning; and inputting the aligned text features, original image features and interactive click features into a mask decoder, and performing cross attention operation and normalized focus loss calculation to obtain segmentation loss and optimize an overall training target. According to the method, through normalization of focus loss and cross entropy loss functions, accurate alignment of a region-level visual mode and a text mode is ensured, and the accuracy and comprehensiveness in an image segmentation task are remarkably improved.