Patents
Literature
Patsnap Eureka AI that helps you search prior art, draft patents, and assess FTO risks, powered by patent and scientific literature data.

231 results about "Text mode" patented technology

Text mode is a computer display mode in which content is internally represented on a computer screen in terms of characters rather than individual pixels. Typically, the screen consists of a uniform rectangular grid of character cells, each of which contains one of the characters of a character set. Text mode is contrasted to all points addressable (APA) mode or other kinds of computer graphics modes.

Multi-modal systems and methods for voice-based mental health assessment with emotion stimulation

Multi-modal systems, for voice-based mental health assessment with emotion stimulation, comprising: a task construction module to construct tasks for capturing acoustic, linguistic, and affective characteristics of speech of a user; a stimulus output module comprising stimuli, basis the constructed tasks, to be presented to a user in order to elicit a trigger of one or more types of user behaviour, the triggers being in the form on input responses; response intake module to present, to a user, the stimuli, and, in response, receive corresponding responses in one or more formats from responses; an autoencoder to define relationship / s, using the fused features, between: an audio modality to output extracted high-level text features; and a text modality to output extracted high-level audio features; the autoencoder to receive extracted high-level text and audio features, in parallel, to output a shared representation feature data set for emotion classification correlative to the mental health assessment.
Owner:WONDER TECHNOLOGIES PTE LTD

Fine-grained living body face attribute editing method and system based on text guidance

The invention discloses a fine-grained living body face attribute editing method and system based on text guidance, and the method comprises the following steps: obtaining a face image data set and a text data set, and carrying out the data preprocessing; based on a face identity feature extraction module and a face key point feature extraction module, respectively extracting multi-scale face identity features and multi-scale face key point features of the face image; performing feature splicing on the multi-scale face identity features and the multi-scale face key point features based on a multi-scale feature mixing module, and performing weighted average to obtain face identity structure features; respectively embedding the face image and the original text into the conditional diffusion model as an image mode and a text mode; carrying out two-stage training; and obtaining a final text condition vector based on the original text and the guide text, inputting the final text condition vector into the trained condition diffusion model, and generating a fine-grained face attribute editing picture. According to the invention, refinement and diversification of face attribute editing are realized.
Owner:SOUTH CHINA UNIV OF TECH

Large model enhanced cross-modal retrieval method and system and related equipment

The invention discloses a large model enhanced cross-modal retrieval method and system and related equipment. The cross-modal retrieval method comprises the following steps: acquiring an image-text pair; based on the image-text pair, obtaining large model enhancement information of the image-text pair, combining an original image, a text and the enhancement information, adopting a multi-branch encoder of a pre-training CLIP model to obtain a plurality of feature vectors, constructing a multi-stage collaborative alignment loss function, and performing collaborative alignment on an image mode and a text mode in a public semantic space; and training a model through the multi-stage collaborative alignment loss function and a pre-constructed training database, and performing retrieval through the trained model. According to the method, collaborative learning is carried out on the image-text features obtained by the encoder, auxiliary semantic enhancement information is added, and collaborative alignment is carried out on the image modality and the text modality in the public semantic space, so that a better retrieval network is trained, and the accuracy of image-text retrieval is improved.
Owner:UNIV OF SCI & TECH OF CHINA

Video segmentation method, server, storage medium and program product

The invention provides a video segmentation method, a server, a storage medium and a program product, and the method comprises the steps: segmenting to-be-segmented video data into a plurality of data segments, extracting the features of a single mode of each data segment through a video theme segmentation model, including the text features of a text mode and the visual features of a visual mode; the text features and the visual features of all the data fragments are fused, multi-modal information fusion can be carried out at the middle representation level, the relation and interaction between different modals can be better captured, and the multi-modal fusion features with higher quality of all the data fragments are obtained; furthermore, according to the multi-modal fusion feature of each data fragment, whether each data fragment is the theme boundary is predicted, the theme boundary of the video data can be accurately predicted, the accuracy of theme boundary identification is improved, and thus the accuracy and quality of a video theme segmentation result are improved.
Owner:ALIBABA (CHINA) CO LTD

Financial analysis method and system based on big data

The invention provides a financial analysis method and system based on big data, and the method comprises the steps: analyzing the dependency relationship between a non-standard code combination and a fuzzy text mode in a material purchasing scene based on a credible rule library, extracting a new attribute combination feature, and forming an optimization rule feature set if the new attribute combination feature can improve the matching accuracy; performing rule-based key field and attribute set verification matching on the real-time non-standard transaction data according to the optimization rule feature set in combination with a dependency relationship between a non-standard code combination and a fuzzy text mode, and outputting a matched transaction record set; and comparing the difference between the matched transaction record set and the historical manual matching record, extracting and judging whether the newly added non-standard attribute combination is applicable to the non-standard transaction data which is not covered or not after a matching rule is adjusted by adopting an incremental learning algorithm, and if so, forming a dynamic matching table.
Owner:HANSHAN NORMAL UNIV

Cross-modal retrieval method and device, electronic equipment and storage medium

The invention is applicable to the technical field of artificial intelligence, and provides a cross-modal retrieval method and device, electronic equipment and a storage medium, and the method comprises the following steps: encoding an image modal through an image encoder, encoding a text modal through a text encoder, and generating a vector representation of a high-dimensional space; inputting the vector of the image mode and the vector of the text mode into the same shared self-attention layer for alignment, and extracting features through respective feedforward network layers; performing cross-modal cross fusion processing on the vector of the image modal and the vector of the text modal through a cross attention mechanism to obtain a cross-modal representation vector of the image modal and a cross-modal representation vector of the text modal; and based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, outputting the similarity of the image modality and the text modality. According to the method, the cross-modal retrieval precision can be improved.
Owner:STATE GRID HEBEI ELECTRIC POWER CO LTD +2

Multi-modal pre-training model construction method and system for monitoring video

The invention provides a multi-modal pre-training model construction method and system for monitoring videos, and the method comprises the steps: automatically constructing a high-quality multi-modal alignment data set through a single-modal description generation model and a large-scale language model, and remarkably reducing the marking cost; a special coding network and a shared projection layer are adopted to realize feature extraction and uniform semantic space alignment of video, audio and text modes; performing dynamic semantic fusion by using a modal collaborative attention mechanism; designing cross-modal contrast learning, mask prediction and time sequence consistency tasks to carry out multi-task pre-training; an external knowledge base is introduced, and the semantic reasoning ability is enhanced through a microretrieval mechanism; and optimizing model parameters by adopting a multi-task joint loss function and an end-to-end training strategy. According to the method, efficient and automatic construction, deep semantic alignment and fusion and intelligent reasoning of knowledge enhancement of monitoring video multi-modal data are realized, and the understanding and generalization ability of the model in a complex scene is effectively improved.
Owner:BEIJING JIAOTONG UNIV

Multi-image processing method based on multi-modal entity alignment

The invention belongs to the technical field of image processing, and discloses a multi-image processing method based on multi-modal entity alignment, which comprises the following steps: acquiring a plurality of images; retrieving rich semantic information of the entity from an external knowledge base; selecting the most representative image for each entity in the multi-image scene by using semantic information; encoding the original input of each mode; performing first-layer fusion between a visual mode and a text mode by using cross-divergent attention, then performing second-layer interaction with a structured mode, and finally aligning an entity representation of an image by using contrast loss; and outputting multiple images fusing the text modality and the visual modality. According to the invention, hierarchical interaction fusion is utilized to enhance multi-modal interaction; enhancing entity text representation by integrating the external attribute value and the context information; and the most representative image is selected by using the semantic text, so that the influence of uncorrelated images is minimized.
Owner:NAT UNIV OF DEFENSE TECH

Multi-granularity cross-modal comparative learning method and device for multi-modal sentiment analysis

The invention relates to a multi-granularity cross-modal contrast learning method and device for multi-modal sentiment analysis, and the method comprises the steps: carrying out the coding of a text mode, a visual mode, and an audio mode, and obtaining a coded text mode, a coded visual mode, and a coded audio mode; constructing in-sample positive and negative pairs and between-sample positive and negative pairs, performing in-sample comparative learning according to the in-sample positive and negative pairs, and performing between-sample comparative learning according to the between-sample positive and negative pairs; and optimizing a multi-granularity cross-modal contrast learning model according to the overall contrast loss function, the multi-modal prediction loss function and the single-modal prediction loss function, so as to obtain a target multi-modal sentiment analysis result by using the optimized multi-granularity cross-modal contrast learning model. Therefore, the problems that the modals are considered to be equal and the difference between the modals is ignored in the prior art, so that the modals are not fully interacted in the fusion process, important information is easy to lose, the accuracy of a final result is influenced and the like are solved.
Owner:WUHAN UNIV

Continuous sign language recognition method based on multi-granularity cross-modal comparative learning

The invention discloses a continuous sign language recognition method based on multi-granularity cross-modal comparative learning. The method comprises the following steps: firstly, fully tapping a hidden semantic mapping relationship between a hidden sign language video and a corresponding sign language text; then, a multi-granularity cross-modal comparison mechanism is used for relieving potential influences of redundant frames and semantic irrelevant identification words on cross-modal alignment, and through a coarse-granularity video-sentence comparison strategy and a fine-granularity frame-word comparison strategy, visual and text modal features are effectively aligned from global and local angles; semantic interference caused by redundant frames and semantic irrelevant identification words is relieved through cross-granularity video-word comparison and cross-granularity frame-sentence comparison strategies; and finally, obtaining sign language text sentences through a decoder by utilizing visual embedding obtained in a visual encoder, and taking the sign language text sentences as predicted continuous sign language recognition results. The hidden semantic mapping relation between the sign language video and the corresponding sign language text is fully excavated, the semantic understanding of the sign language model on the visual context is enhanced, and the applicability and accuracy of sign language recognition are improved.
Owner:ZHEJIANG UNIV OF TECH

Multi-modal multi-label emotion recognition method and system based on modal contribution evaluation

The invention discloses a multi-modal multi-label emotion recognition method and system based on modal contribution assessment, and the method comprises the steps: obtaining original feature sequences of a text mode, a visual mode and an audio mode, mapping the original feature sequences through an encoder, and obtaining high-dimensional feature embedding of the three modes; calculating a marginal contribution degree of each mode to emotion prediction at a sample level by utilizing emotion prediction values embedded by high-dimensional features of the three modes; according to the marginal contribution degree of each modal, fusing the features of each modal to form a fused feature, and processing the fused feature through a multi-modal encoder to obtain a multi-modal integrated feature; and through a trainable label embedding sequence, learning a dependency relationship between emotion labels, embedding and inputting the multi-modal integrated features and the labels into an emotion decoder to obtain a multi-modal emotion representation of mixed emotion, and inputting the multi-modal emotion representation into a multi-label classifier to obtain a predicted emotion label. According to the invention, the efficiency of multi-modal feature fusion and the accuracy of emotion recognition can be improved.
Owner:JIANGSU UNIV

Text data automatic labeling method and system integrating pre-training NLP model

The invention provides a text data automatic labeling method and system integrating a pre-training NLP model, and relates to the technical field of artificial intelligence, the method comprises the following steps: obtaining original text data to be labeled, the text data comprising customer consultation, complaint or feedback dialogue records, evaluations and work orders; performing word segmentation, stop word filtering and part-of-speech tagging processing on the original text data to generate structured text data; inputting the structured text data into a pre-trained NLP model, and correcting semantic features output by the model through a domain adaptation layer to obtain corrected semantic features; and carrying out few-sample learning on the corrected semantic features, enhancing the recognition capability on a low-frequency text mode through a contrast learning mechanism, and generating an enhanced feature vector resistant to long-tail interference. According to the method, automatic annotation of the text data is realized by automatically processing data acquisition and preprocessing and integrating the pre-training NLP model.
Owner:SHANGHAI XIAOLING NETWORK TECH CO LTD

Processing method and device for adjusting answer text based on adaptive retrieval enhancement mechanism

The embodiment of the invention relates to a processing method and device for adjusting an answer text based on a self-adaptive retrieval enhancement mechanism. The method comprises the following steps: receiving a question text; taking an LLM model used for processing the intelligent question and answer task as a first model; inputting the question text into a first model to generate an answer text; inputting the question text and the answer text into a first model for question complexity and answer quality evaluation; performing text mode adjustment according to the complexity and the quality evaluation result; when the adjustment mode is the first mode, directly taking the current answer text as an adjusted answer text, when the adjustment mode is the second mode, performing single-round retrieval enhancement and answer text adjustment to obtain the adjusted answer text, and when the adjustment mode is the third mode, performing multi-round retrieval enhancement and answer text adjustment to obtain the adjusted answer text; and displaying the obtained adjusted answer text. According to the invention, the processing efficiency of the LLM model can be improved, and the accuracy of the answer text is improved.
Owner:BEIJING DP TECH CO LTD

Dialogue emotion recognition method and device based on multi-modal large model

The invention discloses a dialogue emotion recognition method and device based on a multi-modal large model, and relates to the field of emotion recognition, and the method comprises the steps: obtaining all statement sets in a dialogue in a current scene; each statement comprises an audio mode, a video mode and a text mode; constructing a dialogue emotion recognition model; the dialogue emotion recognition model comprises a feature extraction layer, a bidirectional gating unit, a multi-modal large model, a BERT language model, a modal information complementation module, a semantic graph-based multilayer residual image convolutional network and a full connection layer; and according to all statement sets in the dialogue, adopting the trained dialogue emotion recognition model to obtain an emotion recognition result. The dialogue emotion recognition method and device can improve the accuracy and robustness of dialogue emotion recognition.
Owner:ZHEJIANG NORMAL UNIV +1

Text pedestrian re-identification method based on attention screening mask strategy

The invention belongs to the technical field of pedestrian re-identification, and particularly relates to a text pedestrian re-identification method based on an attention screening mask strategy. The invention designs a multi-granularity text pedestrian re-identification method based on global feature extraction and local feature extraction. The method comprises the following steps: carrying out rough multi-modal feature alignment on a text mode and an image mode by introducing an improved global feature alignment function; and then, through an attention mechanism, screening a mark with a relatively high attention score for masking, and fully interacting the unmasked text and image with the masked text and image, so that the accuracy of pedestrian re-recognition from the text to the image is improved. According to the method, testing is carried out on a public data set CUHK-PEDES, and an excellent effect is achieved.
Owner:UESTC (SHENZHEN) ADVANCED RES INST

Multi-modal sentiment analysis method for contrast enhancement of non-isomorphic modal fusion

The invention provides a multi-modal sentiment analysis method for contrast enhancement non-isomorphic modal fusion, and relates to the technical field of sentiment analys.According to the method, a visual modal feature and an audio modal feature are fused to make up for the difference of information density with text modalities, and joint contrast learning is designed to make up for the difference of information density with text modalities. The loss in the samples, the loss between the samples and the contrast learning in the samples are obtained through the contrast learning in the samples and the contrast learning between the samples, so that the consistency between the modes and the contrast learning between the samples are enhanced, the independence and the uniqueness of each mode feature are guaranteed, and information loss caused by excessive fusion is avoided. According to the method, complementarity and difference between modes are considered, the difference of information density between the visual and audio modes and the text mode is made up, and the performance and generalization ability of the sentiment analysis model are improved.
Owner:NORTHEASTERN UNIV CHINA

DiT city layout generation method based on multi-mode cooperative driving

The invention belongs to the crossing field of computer vision and city design, and provides a multi-mode cooperative driving DiT city layout generation method, so as to solve the defects of an existing generation model in the aspects of spatial rationality and planning controllability. According to the method, a transformer-based diffusion generation model is innovatively constructed, and two additional control signals are integrated in a DiT model to realize controlled generation. On the aspect of model architecture design, a DiT framework with multi-modal condition fusion is constructed, and a road network sketch (image control) and planning semantics (text control) dual guide mechanism is integrated. In the text mode, a dynamic semantic fusion module based on a pre-training language model is designed, and text information is deeply embedded into the generation process. In the aspect of image control, a mixed attention regulation and control mechanism is put forward, multi-scale fusion of road network structural features is realized in combination with cross attention and AdaLN technologies, and the problem of spatial layout distortion is solved. Experiments show that the method can effectively integrate text and image information, and the generated city layout image is superior to that generated by an existing method.
Owner:NANKAI UNIV

Speech emotion recognition method based on multi-modal feature fusion

The invention discloses a speech emotion recognition method based on multi-modal feature fusion, and the method comprises the steps: designing a multi-modal speech emotion recognition model, enabling an audio to serve as a main mode, employing the function of wav2vec2.0, and combining a text mode with a visual mode to enhance the performance of emotion recognition based on the audio; the self-attention mechanism, the GRU and the LVC are combined to extract global and local information from a visual mode, the global information is extracted only from text features, visual clues and text clues related to emotion can be comprehensively understood, the extracted text and visual features are integrated into a wav2vec2.0 model, multi-mode representation of multi-level audio information is formed, and the multi-mode representation of the multi-level audio information is realized. And meanwhile, the discrimination capability of the model is enhanced by utilizing multi-modal homogeneous feature difference learning, and the proposed model is evaluated on two data sets, so that the model is proved to have excellent robustness and generalization compared with the existing most advanced model.
Owner:ANHUI UNIV OF FINANCE & ECONOMICS

Multi-modal sentiment analysis method based on non-homogeneous hierarchical fusion

The invention discloses a multi-modal sentiment analysis method based on non-homogeneous hierarchical fusion, which comprises the following steps: S1, constructing a non-homogeneous hierarchical fusion network model which comprises a multi-modal word correction module, a Transform aggregation module, a cross attention fusion module and a function-based attention fusion module; s2, modal feature extraction is carried out through a multi-modal word correction module; s3, fusion of the visual mode and the voice mode is executed through a Transform aggregation module, and a video and voice fusion mode is obtained; s4, executing fusion of the text mode and the video and audio fusion mode through a cross attention fusion module; s5, dynamically adjusting the weight of each dimension in each modal representation through a function-based attention fusion module; and S6, predicting a final emotion regression label through regression analysis calculation. The emotion semantic information fusion method solves the problems of relatively large information density difference among voice, vision and texts, relatively high multi-mode isomerism difficulty and relatively poor effect in an emotion semantic information fusion method in the prior art.
Owner:CHINA THREE GORGES UNIV

Hatred mold factor identification method and system based on dual particle size symmetry and context perception alignment

The invention discloses a hatred model factor identification method and system based on dual granularity symmetry and context perception alignment. The method comprises the following steps: firstly, respectively extracting a text mode and a visual mode from input image data containing text information, and carrying out feature coding and embedding so as to obtain text mode features and visual mode features; according to the method, a mode of constructing same-granularity and cross-granularity dual symmetric factors in visual and text modal representation is adopted to control modal internal feature fusion, so that a function of generating high-semantic-consistency modal internal feature representation is realized; in the inter-modal representation fusion process, a context perception cross-modal alignment strategy is introduced, and dual symmetric modal information is used as a prompt, so that the understanding of memetic context information is enhanced; whether aggressive or discrimination information aiming at individuals or groups is transmitted or not can be accurately identified by excavating potential emotional tendency and value orientation, and the identification accuracy and efficiency of the hatred model factor are improved.
Owner:NANJING AUDIT UNIV

Multi-modal sentiment analysis method based on pre-training and text modal guidance

The invention is suitable for the technical field of sentiment analysis, and particularly relates to a multi-modal sentiment analysis method based on pre-training and text modal guidance, and the method is implemented in two stages: the pre-training stage comprises the steps of obtaining initial features of each modal of multi-modal data, and decomposing each modal feature into similar features and different features; constructing positive and negative pairs in and between samples, calculating comparison loss and single-modal prediction errors, extracting module and projector parameters, and inputting a short video to obtain each modal feature; high-scale text representation is learned through initial text similar features, super-modal features are adaptively updated through a multi-head attention mechanism, and audio and video redundant information is suppressed by using a weight module guided by a text mode; through the data sampler and the feature comparison learning method, multi-mode fusion with the text mode as an anchor point is achieved, the emotional tendency in a complex scene can be recognized more accurately, and the method has wide value in the fields of emotion classification, social media content recommendation and the like.
Owner:XIANGTAN UNIV

Emotion analysis method and device for promoting multi-modal information fusion

The invention discloses an emotion analysis method and device for promoting multi-mode information fusion, and the method comprises the steps: building an acoustic vocabulary and a visual vocabulary based on a K-Means algorithm, querying the features of a non-text mode, obtaining a word index sequence of the non-text mode, and embedding the word index sequence into the initial features of the text mode to build a non-language displacement vector; adding the non-language displacement vector into the initial feature of the text mode for displacement operation to obtain a multi-mode vector; and performing context sensing and sentiment analysis on the multi-modal vector to obtain and output a sentiment analysis result. According to the method, the feature sequence of the non-text mode is converted into the word index sequence which is the same as the text mode through the K-Means algorithm, the distribution gap between the language text mode and the non-language text mode during fusion is reduced, and subsequent feature fusion is facilitated; during feature fusion, the expression of a language text mode is enhanced through a non-language displacement vector, so that the accuracy of sentiment analysis is improved.
Owner:GUANGDONG UNIV OF TECH

Data processing method and device and electronic equipment

The embodiment of the invention discloses a data processing method and device and electronic equipment. The method comprises the steps of receiving a question which is input by a user and comprises a target file of a non-text mode and a question statement of a text mode, wherein the question statement is a statement for asking a question for the target file; target knowledge content matched with the question is retrieved from a knowledge base containing multiple pieces of knowledge content, and the knowledge content comprises a file of a non-text mode and description information of the text mode for describing the file; inputting the target knowledge content into a large-scale language model to obtain the large-scale language model which learns an association relationship between a file and description information in the target knowledge content; and inputting the question into the large-scale language model in which the association relationship is learned, and taking the output of the large-scale language model for the question as an answer corresponding to the question.
Owner:ANT BLOCKCHAIN TECHNOLOGY (SHANGHAI) CO LTD

Knowledge graph construction method and device, equipment, storage medium and computer program product

The invention provides a knowledge graph construction method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a script text and a deductive video corresponding to the script text; constructing a plurality of first nodes of a text mode based on the script text, and constructing a plurality of second nodes of an image mode and a plurality of third nodes of an audio mode based on the deductive video; based on the content information corresponding to the fourth node and the fifth node, determining a semantic consistency score between the fourth node and the fifth node; determining a time sequence synchronization score between the fourth node and the fifth node based on the time sequence information corresponding to the fourth node and the fifth node; in response to the fact that the semantic consistency score is larger than a first threshold value and the time sequence synchronism score is larger than a second threshold value, constructing a first relationship between the fourth node and the fifth node; and constructing the knowledge graph based on the plurality of first nodes, the plurality of second nodes, the plurality of third nodes and the plurality of first relationships.
Owner:TENCENT TECHNOLOGY (SHENZHEN) CO LTD

Out-of-distribution data detection system and method based on image-text bimodal matching

The invention discloses an out-of-distribution data detection system and method based on image-text bimodal matching, and belongs to the technical field of out-of-distribution data detection. The current text category feature and the current area visual feature of each tag category to which a to-be-recognized image possibly belongs are aggregated; performing weighted summation on the aggregation result and the current global visual feature to obtain a current image aggregation feature; matching the current image aggregation feature with the storage text mode and the storage visual mode of each label category to obtain matching scores corresponding to the current text mode and the visual mode of each label category, and performing weighted summation on the two matching scores to obtain a bimodal matching score; judging whether the image to be recognized is in-distribution data or out-of-distribution data; according to the method, the fusion information corresponding to the text modality and the visual modality of the data in the distribution is used for carrying out the out-of-distribution detection for the first time, and the out-of-distribution data in the real environment can be effectively detected at low cost.
Owner:HUAZHONG UNIV OF SCI & TECH

Data rule design method and device for intelligent visual online test

The invention relates to the technical field of big data management, and particularly provides a data rule design method and device for intelligent visual online testing, and the method comprises the following steps: S1, defining a data rule: defining the rule from three aspects of custom quality dimension, a basic rule base and a business rule base; s2, the JavaScript script is verified, and the JavaScript S3, performing matching by using a regular expression, and searching and replacing a complex text mode; s4, online testing and formatting are carried out, regular expressions or JS scripts are selected to carry out online testing and editing on data rule contents, meanwhile, the rule contents are displayed as verification or processing results of the data, in addition, the JS scripts carry out parameter analysis on rule methods, and attribute classification and filling are carried out on parameters; and S5, task scheduling and optimization. Compared with the prior art, the method can provide data meanings and business rule standards which are commonly obeyed by an upper business level.
Owner:SHANGHAI INSPUR CLOUD COMPUTING SERVICE CO LTD

Dual-mode forged information detection method fusing emotion features

The invention discloses an emotion feature fused bimodal forged information detection method, which comprises the following steps of: extracting emotion features, respectively extracting emotion categories of a text and an image through a large language model, respectively giving emotion intensity of the categories, and scoring the emotion; analyzing the emotion consistency of the mixed modal features of the text and the image; the hybrid expert network calculates domain probabilities to which a text mode, an image mode and a hybrid mode belong respectively through a text domain encoder, an image domain encoder and an image text hybrid feature domain encoder based on domain class processing features; the hybrid expert network learns feature information of a text mode, an image mode and a hybrid mode according to the domain information; the AdaIN network adjusts modal distribution, the AdaIN network adjusts feature distribution according to contribution of each modal, the adjusted features are fused and input into the classifier, a final classification result is obtained, and the information detection accuracy is greatly improved.
Owner:SHENZHEN KIM DAI INTELLIGENCE INNOVATION TECHNOLOGY CO LTD

Multi-modal large model reasoning acceleration method fusing image feature compression and adaptive pruning

The invention relates to an image processing technology, in particular to a multi-modal large model reasoning acceleration method fusing image feature compression and self-adaptive pruning, which comprises the following steps of: determining the optimal image block number of an input image according to the aspect ratio of the input image and the resolution of a reference image so as to dynamically match and adjust the image resolution; processing the input image after the image resolution is adjusted through a visual encoder and an attention module to obtain image feature tokens, analyzing the redundancy and semantic information contribution degree between the image feature tokens, retaining the image feature token which contributes to the image information amount greatly, and realizing image feature compression; encoding the input text to obtain a text feature token; according to the importance scores of the image feature tokens and the text feature tokens, single-mode redundancy features and cross-mode redundancy features of the visual mode and the text mode are recognized and removed, and a redundancy model structure is cut off. The model reasoning speed is increased as much as possible while effective information is reserved.
Owner:SOUTH CHINA UNIV OF TECH

Infrared and visible light image fusion method based on multi-semantic deep collaboration

The invention provides an infrared and visible light image fusion method based on multi-semantic deep collaboration. The method mainly solves the problem that an existing method cannot fully integrate text modals and global consistency between image fusion and downstream tasks. Comprising the following steps: 1) constructing a dual-task parallel network structure, and efficiently establishing a deep correlation between image fusion and a downstream segmentation task; 2) designing a multi-semantic deep collaboration module, realizing effective fusion of multi-modal information by deeply integrating text features, pixel-level features and segmented semantic features, and meeting semantic requirements of downstream tasks; 3) guiding an image fusion and segmentation task by using deeper and fine-grained semantic information in a text mode, and enhancing semantic consistency between a fusion result and a downstream task; and 4) inputting the obtained multiple semantic features into a fusion decoder to generate a final image fusion result. The semantic comprehension and visual perception capabilities of the model can be effectively enhanced, and the multi-modal image fusion performance is improved.
Owner:XIDIAN UNIV

Multi-modal federal learning method based on self-attention

The invention discloses a multi-modal federated learning method based on self-attention, and relates to the field of data processing, and the method comprises the steps: firstly, designing a multi-modal feature coding module at a client, carrying out the deep semantic modeling of image and text modals based on a Transform architecture, and capturing the internal structure information of each modality; secondly, a local cross-modal self-attention modeling module is introduced, fine-grained semantic alignment between images and texts is modeled through a bidirectional cross-attention mechanism, and the local cross-modal understanding ability is enhanced; a multi-head collaborative fusion optimization module is further combined, inter-modal deep semantic collaborative modeling is carried out under a multi-attention perspective, and a fusion effect is optimized through a gating mechanism, so that the problems of inter-modal conflicts and inconsistent expressions are effectively relieved.
Owner:ANHUI NORMAL UNIV