127 results about "Text mode" patented technology

The invention provides a financial analysis method and system based on big data, and the method comprises the steps: analyzing the dependency relationship between a non-standard code combination and a fuzzy text mode in a material purchasing scene based on a credible rule library, extracting a new attribute combination feature, and forming an optimization rule feature set if the new attribute combination feature can improve the matching accuracy; performing rule-based key field and attribute set verification matching on the real-time non-standard transaction data according to the optimization rule feature set in combination with a dependency relationship between a non-standard code combination and a fuzzy text mode, and outputting a matched transaction record set; and comparing the difference between the matched transaction record set and the historical manual matching record, extracting and judging whether the newly added non-standard attribute combination is applicable to the non-standard transaction data which is not covered or not after a matching rule is adjusted by adopting an incremental learning algorithm, and if so, forming a dynamic matching table.

The invention provides a multi-modal pre-training model construction method and system for monitoring videos, and the method comprises the steps: automatically constructing a high-quality multi-modal alignment data set through a single-modal description generation model and a large-scale language model, and remarkably reducing the marking cost; a special coding network and a shared projection layer are adopted to realize feature extraction and uniform semantic space alignment of video, audio and text modes; performing dynamic semantic fusion by using a modal collaborative attention mechanism; designing cross-modal contrast learning, mask prediction and time sequence consistency tasks to carry out multi-task pre-training; an external knowledge base is introduced, and the semantic reasoning ability is enhanced through a microretrieval mechanism; and optimizing model parameters by adopting a multi-task joint loss function and an end-to-end training strategy. According to the method, efficient and automatic construction, deep semantic alignment and fusion and intelligent reasoning of knowledge enhancement of monitoring video multi-modal data are realized, and the understanding and generalization ability of the model in a complex scene is effectively improved.

The invention belongs to the technical field of image processing, and discloses a multi-image processing method based on multi-modal entity alignment, which comprises the following steps: acquiring a plurality of images; retrieving rich semantic information of the entity from an external knowledge base; selecting the most representative image for each entity in the multi-image scene by using semantic information; encoding the original input of each mode; performing first-layer fusion between a visual mode and a text mode by using cross-divergent attention, then performing second-layer interaction with a structured mode, and finally aligning an entity representation of an image by using contrast loss; and outputting multiple images fusing the text modality and the visual modality. According to the invention, hierarchical interaction fusion is utilized to enhance multi-modal interaction; enhancing entity text representation by integrating the external attribute value and the context information; and the most representative image is selected by using the semantic text, so that the influence of uncorrelated images is minimized.

The invention discloses a multi-modal multi-label emotion recognition method and system based on modal contribution assessment, and the method comprises the steps: obtaining original feature sequences of a text mode, a visual mode and an audio mode, mapping the original feature sequences through an encoder, and obtaining high-dimensional feature embedding of the three modes; calculating a marginal contribution degree of each mode to emotion prediction at a sample level by utilizing emotion prediction values embedded by high-dimensional features of the three modes; according to the marginal contribution degree of each modal, fusing the features of each modal to form a fused feature, and processing the fused feature through a multi-modal encoder to obtain a multi-modal integrated feature; and through a trainable label embedding sequence, learning a dependency relationship between emotion labels, embedding and inputting the multi-modal integrated features and the labels into an emotion decoder to obtain a multi-modal emotion representation of mixed emotion, and inputting the multi-modal emotion representation into a multi-label classifier to obtain a predicted emotion label. According to the invention, the efficiency of multi-modal feature fusion and the accuracy of emotion recognition can be improved.

The invention provides a text data automatic labeling method and system integrating a pre-training NLP model, and relates to the technical field of artificial intelligence, the method comprises the following steps: obtaining original text data to be labeled, the text data comprising customer consultation, complaint or feedback dialogue records, evaluations and work orders; performing word segmentation, stop word filtering and part-of-speech tagging processing on the original text data to generate structured text data; inputting the structured text data into a pre-trained NLP model, and correcting semantic features output by the model through a domain adaptation layer to obtain corrected semantic features; and carrying out few-sample learning on the corrected semantic features, enhancing the recognition capability on a low-frequency text mode through a contrast learning mechanism, and generating an enhanced feature vector resistant to long-tail interference. According to the method, automatic annotation of the text data is realized by automatically processing data acquisition and preprocessing and integrating the pre-training NLP model.

The invention belongs to the crossing field of computer vision and city design, and provides a multi-mode cooperative driving DiT city layout generation method, so as to solve the defects of an existing generation model in the aspects of spatial rationality and planning controllability. According to the method, a transformer-based diffusion generation model is innovatively constructed, and two additional control signals are integrated in a DiT model to realize controlled generation. On the aspect of model architecture design, a DiT framework with multi-modal condition fusion is constructed, and a road network sketch (image control) and planning semantics (text control) dual guide mechanism is integrated. In the text mode, a dynamic semantic fusion module based on a pre-training language model is designed, and text information is deeply embedded into the generation process. In the aspect of image control, a mixed attention regulation and control mechanism is put forward, multi-scale fusion of road network structural features is realized in combination with cross attention and AdaLN technologies, and the problem of spatial layout distortion is solved. Experiments show that the method can effectively integrate text and image information, and the generated city layout image is superior to that generated by an existing method.

The invention discloses a hatred model factor identification method and system based on dual granularity symmetry and context perception alignment. The method comprises the following steps: firstly, respectively extracting a text mode and a visual mode from input image data containing text information, and carrying out feature coding and embedding so as to obtain text mode features and visual mode features; according to the method, a mode of constructing same-granularity and cross-granularity dual symmetric factors in visual and text modal representation is adopted to control modal internal feature fusion, so that a function of generating high-semantic-consistency modal internal feature representation is realized; in the inter-modal representation fusion process, a context perception cross-modal alignment strategy is introduced, and dual symmetric modal information is used as a prompt, so that the understanding of memetic context information is enhanced; whether aggressive or discrimination information aiming at individuals or groups is transmitted or not can be accurately identified by excavating potential emotional tendency and value orientation, and the identification accuracy and efficiency of the hatred model factor are improved.

The invention is suitable for the technical field of sentiment analysis, and particularly relates to a multi-modal sentiment analysis method based on pre-training and text modal guidance, and the method is implemented in two stages: the pre-training stage comprises the steps of obtaining initial features of each modal of multi-modal data, and decomposing each modal feature into similar features and different features; constructing positive and negative pairs in and between samples, calculating comparison loss and single-modal prediction errors, extracting module and projector parameters, and inputting a short video to obtain each modal feature; high-scale text representation is learned through initial text similar features, super-modal features are adaptively updated through a multi-head attention mechanism, and audio and video redundant information is suppressed by using a weight module guided by a text mode; through the data sampler and the feature comparison learning method, multi-mode fusion with the text mode as an anchor point is achieved, the emotional tendency in a complex scene can be recognized more accurately, and the method has wide value in the fields of emotion classification, social media content recommendation and the like.

The invention provides a knowledge graph construction method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring a script text and a deductive video corresponding to the script text; constructing a plurality of first nodes of a text mode based on the script text, and constructing a plurality of second nodes of an image mode and a plurality of third nodes of an audio mode based on the deductive video; based on the content information corresponding to the fourth node and the fifth node, determining a semantic consistency score between the fourth node and the fifth node; determining a time sequence synchronization score between the fourth node and the fifth node based on the time sequence information corresponding to the fourth node and the fifth node; in response to the fact that the semantic consistency score is larger than a first threshold value and the time sequence synchronism score is larger than a second threshold value, constructing a first relationship between the fourth node and the fifth node; and constructing the knowledge graph based on the plurality of first nodes, the plurality of second nodes, the plurality of third nodes and the plurality of first relationships.

The invention discloses an emotion feature fused bimodal forged information detection method, which comprises the following steps of: extracting emotion features, respectively extracting emotion categories of a text and an image through a large language model, respectively giving emotion intensity of the categories, and scoring the emotion; analyzing the emotion consistency of the mixed modal features of the text and the image; the hybrid expert network calculates domain probabilities to which a text mode, an image mode and a hybrid mode belong respectively through a text domain encoder, an image domain encoder and an image text hybrid feature domain encoder based on domain class processing features; the hybrid expert network learns feature information of a text mode, an image mode and a hybrid mode according to the domain information; the AdaIN network adjusts modal distribution, the AdaIN network adjusts feature distribution according to contribution of each modal, the adjusted features are fused and input into the classifier, a final classification result is obtained, and the information detection accuracy is greatly improved.

The invention relates to an image processing technology, in particular to a multi-modal large model reasoning acceleration method fusing image feature compression and self-adaptive pruning, which comprises the following steps of: determining the optimal image block number of an input image according to the aspect ratio of the input image and the resolution of a reference image so as to dynamically match and adjust the image resolution; processing the input image after the image resolution is adjusted through a visual encoder and an attention module to obtain image feature tokens, analyzing the redundancy and semantic information contribution degree between the image feature tokens, retaining the image feature token which contributes to the image information amount greatly, and realizing image feature compression; encoding the input text to obtain a text feature token; according to the importance scores of the image feature tokens and the text feature tokens, single-mode redundancy features and cross-mode redundancy features of the visual mode and the text mode are recognized and removed, and a redundancy model structure is cut off. The model reasoning speed is increased as much as possible while effective information is reserved.

The invention provides an infrared and visible light image fusion method based on multi-semantic deep collaboration. The method mainly solves the problem that an existing method cannot fully integrate text modals and global consistency between image fusion and downstream tasks. Comprising the following steps: 1) constructing a dual-task parallel network structure, and efficiently establishing a deep correlation between image fusion and a downstream segmentation task; 2) designing a multi-semantic deep collaboration module, realizing effective fusion of multi-modal information by deeply integrating text features, pixel-level features and segmented semantic features, and meeting semantic requirements of downstream tasks; 3) guiding an image fusion and segmentation task by using deeper and fine-grained semantic information in a text mode, and enhancing semantic consistency between a fusion result and a downstream task; and 4) inputting the obtained multiple semantic features into a fusion decoder to generate a final image fusion result. The semantic comprehension and visual perception capabilities of the model can be effectively enhanced, and the multi-modal image fusion performance is improved.

The invention discloses a multi-modal federated learning method based on self-attention, and relates to the field of data processing, and the method comprises the steps: firstly, designing a multi-modal feature coding module at a client, carrying out the deep semantic modeling of image and text modals based on a Transform architecture, and capturing the internal structure information of each modality; secondly, a local cross-modal self-attention modeling module is introduced, fine-grained semantic alignment between images and texts is modeled through a bidirectional cross-attention mechanism, and the local cross-modal understanding ability is enhanced; a multi-head collaborative fusion optimization module is further combined, inter-modal deep semantic collaborative modeling is carried out under a multi-attention perspective, and a fusion effect is optimized through a gating mechanism, so that the problems of inter-modal conflicts and inconsistent expressions are effectively relieved.

The invention discloses an incomplete multi-mode dialogue emotion recognition method and system based on a speaker and time sequence information joint graph network, and the method comprises the steps: obtaining the deep features of a text mode, a voice mode and a visual mode in a dialogue through a feature extraction module, and guaranteeing the high expression capability of the features through a pre-training model; the random mode missing simulation module effectively simulates the data incomplete condition in a real scene, and the robustness of the model is improved; a bi-directional long-short term memory network (Bi-LSTM) is combined with a time sequence diagram network (TGNN) to capture context and time dynamic characteristics of a dialogue, and meanwhile, an interaction relationship between speakers is modeled through a speaker influence matrix, so that joint modeling of a time sequence and speaker information is realized; the deep features are further extracted through the graph convolutional network, and the emotion discrimination of the features is enhanced; finally, the modal reconstruction and emotion classification module significantly improves the accuracy and robustness of incomplete multi-modal dialogue emotion recognition through reconstruction of missing modals and multi-class emotion prediction, and is suitable for man-machine interaction and emotion analysis application in a complex real scene.

The invention discloses a multi-modal rumor detection system based on multi-field perception, relates to the technical field of rumor detection, and is used for solving the technical problem of low multi-field rumor detection precision in the prior art. The rumor detection system comprises a multi-view feature extraction module, a multi-domain perception module, a multi-level knowledge fusion module and a final classifier. Wherein the multi-view feature extraction module is used for respectively extracting corresponding single-mode features from a text mode and an image mode, and acquiring aligned image and text features as multi-mode view features; the multi-domain sensing module is used for learning cross-domain shared knowledge and domain specific knowledge at a text view angle, an image view angle and a multi-mode view angle; the multi-level knowledge fusion module is used for fusing cross-domain shared knowledge and domain specific knowledge of a text view angle, an image view angle and a multi-mode view angle; and the final classifier is used for obtaining a prediction result according to the cross-domain shared knowledge and the domain specific knowledge obtained by the multi-level knowledge fusion module.

The invention provides a reference video target segmentation method and system based on zero sample learning, and belongs to the technical field of target identification. And processing the acquired reference video data by using a pre-trained reference video target segmentation model to obtain a final target segmentation result. According to the reference video target segmentation deep learning network model based on multi-granularity feature fusion of zero sample learning, a multi-granularity visual feature extraction module is introduced, and a target granularity visual feature extraction stage, a frame granularity visual feature extraction stage and a video granularity feature extraction stage are included; the target granularity visual feature extraction stage integrates global semantic information of video frames, the visual feature extraction stage of frame granularity can further align a visual mode and a text mode, the visual feature extraction stage of video granularity can effectively extract motion clues in a video, visual feature representation is enhanced, and the visual feature extraction efficiency is improved. The multi-granularity visual features effectively improve the segmentation performance.

The invention relates to the technical field of computers, and provides a regional attribute recognition method and related device.The method comprises the steps that feature extraction is conducted on at least two kinds of modal data extracted based on multimedia resources, and corresponding initial features are obtained; performing cross-modal feature fusion on the obtained at least two initial features to obtain a fused feature; for the initial feature of each non-text mode, performing global feature extraction on a plurality of sub-features in one initial feature to obtain corresponding non-text global features; each sub-feature represents one item of sub-data in the corresponding non-text modal data; based on the fusion feature, combining the obtained at least one non-text global feature to obtain a region recognition result of the multimedia resource; and when the region identification result shows that the multimedia resource is regional, identifying a target region name associated with a region event corresponding to the multimedia resource based on the text modal data, thereby accurately identifying the region attribute of the multimedia resource.

The invention relates to the technical field of natural language processing, and discloses a cross-border consumption behavior dynamic analysis method and device based on a large language model.The method comprises the steps that multi-modal original data, obtained from multiple data sources, in a target cross-border consumption scene is obtained, the multi-modal original data at least comprises text data and visual data related to the target commodity or the target service; performing feature extraction and fusion processing on the multi-modal original data to generate a unified semantic representation containing text semantics and visual semantics; identifying a scene type to which the target cross-border consumption scene belongs, and determining contribution weights corresponding to a text mode and a visual mode in the unified semantic representation based on the scene type; and analyzing the weighted unified semantic representation according to the contribution weight, and generating a consumer behavior intention label corresponding to the target cross-border consumption scene. The cross-border consumption behavior intention analysis method and device can improve the suitability and accuracy of cross-border consumption behavior intention analysis, and meet the refined operation requirements of cross-border e-commerce.

Methods, apparatuses, devices and media are provided for generating a video in a text mode in an information sharing application. In a method, a request is received for generating the video from a user of the information sharing application. An initial page is displayed for generating the video in the information sharing application, the initial page comprising an indication for entering a text. A text input is obtained from the user in response to a detection of a touch by the user in an area where the initial page locates. A video to be published in the information sharing application is generated based on the text input. In some examples, within the information sharing application, the user may directly generate a corresponding video based on a text input. In this way, a complexity of user operation may be reduced, and the user may be provided with richer publishing content.

The invention discloses a method for controlling an autonomous vehicle and related equipment, and the method for controlling the autonomous vehicle comprises the steps: responding to a natural language instruction of a user, and generating a text feature based on the natural language instruction; collecting original sensor data, and inputting the original sensor data into a visual encoder of a VLA model for processing to obtain key features representing a current environment; the key features and the text features are input into the VLA model, a decision-making thinking chain output by the VLA model in a text mode is obtained, multimedia display is conducted on the decision-making thinking chain, and the decision-making thinking chain comprises the current state of a vehicle, decision-making reasons and a decision-making process.

The invention discloses a multi-modal language model-assisted category guide image segmentation method and system, and relates to the technical field of image processing, and the method comprises the steps: respectively extracting the original image features and user interaction click features of a to-be-segmented image through an image encoder and a prompt encoder; coding the true value mask by using a convolutional network, splicing the true value mask with original image features, and then carrying out cross attention fusion to generate mask image features; extracting original text features through a text encoder, and aligning the original text features with mask image features based on comparative learning; and inputting the aligned text features, original image features and interactive click features into a mask decoder, and performing cross attention operation and normalized focus loss calculation to obtain segmentation loss and optimize an overall training target. According to the method, through normalization of focus loss and cross entropy loss functions, accurate alignment of a region-level visual mode and a text mode is ensured, and the accuracy and comprehensiveness in an image segmentation task are remarkably improved.

The invention discloses a document layout analysis and reconstruction method based on vector topology and conflict arbitration, and relates to the technical field of document image processing, in particular to a method for carrying out accurate identification, classification and structured reconstruction on layout elements of an unstructured electronic document containing a complex vector chart. Constructing a page vector topological graph by using the bottom vector instruction and the connected topological structure; on the basis, multi-dimensional geometric features, statistical features and text mode features are introduced to jointly participate in conflict arbitration between the table and the complex graph; and applying a forced spatial exclusive constraint to text extraction by utilizing an arbitration result, and carrying out semantic classification and rearrangement in combination with a style reference and a spatial proximity relationship. According to the document layout analysis and reconstruction method, the distinguishing capacity of a table and a complex graph is improved, the attribution judgment precision of characters and a main body text in the graph is improved, and self-adaptive recognition of a title and a main body style is achieved under different layout styles.

The invention discloses an infrared and visible light image fusion method and system based on text-guided semantic perception, and relates to the technical field of image fusion. A text prompt is obtained by using a reference image subjected to semantic segmentation, and the features of a source image are respectively obtained by using encoders with the same structure and independent parameters. A semantic correspondence alignment module is provided between an encoder and a decoder, and rich feature representations related to text features are obtained, so that the difference between a text mode and an image mode is reduced. In a decoder stage, a semantic modulation module is provided, interaction between a text and an image is fully realized, effective combination of texture details and advanced semantic information is realized in combination with shallow layer features, and finally a fused image with rich semantics and complete details is generated. According to the method, the overall quality of the fused image is remarkably improved in the aspects of semantic consistency and perception quality.

The embodiment of the invention discloses a classification method and device based on multi-modal data, equipment and a medium. The method comprises the steps of obtaining to-be-classified data of a to-be-classified object in a candidate mode; wherein the candidate modes comprise a text mode, a visual mode and an audio mode; inputting the to-be-classified data into the trained multi-modal classification model to obtain an object classification result; wherein the multi-modal classification model is obtained based on joint training of text sample data, visual sample data and audio sample data of the sample training object. And the accuracy of the classification result determined by the multi-modal classification model based on the multi-modal data is improved.

An extended reality (XR) system is described herein. The XR system includes a display system mounted to the headset for displaying a display screen including computer-generated images thereon, and a controller that includes one or more processors programmed to execute an algorithm for operating in an optical character recognition (OCR) mode to display a text selection screen by generating an image distortion zone within the text selection screen associated with a non-viewable boundary and modifying received video images to adjust a viewable image attribute of a portion of the received video images displayed within the image distortion zone, and operating in the scrolling text mode to display a text display screen by generating machine-readable text OCR and modifying the generating machine-readable text using a lightweight language model for text correction.

The invention discloses a multi-mode anomaly detection method based on neighborhood context consistency detection. The method comprises the following steps: firstly, respectively extracting a structural modal feature and a text modal feature for each node under the setting of a multi-relation graph; respectively coding two modal representations of the nodes through a graph neural network and a text encoder; secondly, feature aggregation of relation perception is carried out in a neighbor node set for each relation type, and neighborhood representation of nodes in a graph mode and a text mode is obtained; afterwards, through node self-alignment and neighborhood alignment tasks, the consistency between structural features and text features is enhanced, and comparison learning is adopted for optimization in the training process; in the reasoning stage, according to node cross-modal residual information, a node anomaly score is generated in combination with a multi-relation weighting strategy, and the node anomaly score is used for identifying camouflage abnormal nodes. By means of the process, complex hidden abnormal behaviors existing in the graph can be accurately detected.

The invention discloses a text and image multi-mode user interest identification method and system oriented to private domain communities. The system comprises a text feature extraction module, an image feature extraction module, a multi-modal fusion module, an interest probability aggregation module and a dynamic threshold judgment module. The multi-modal fusion module adopts a gating mechanism to realize self-adaptive fusion, a gating factor is automatically adjusted according to the characteristics of an input sample, and weighting is carried out between a text mode and an image mode, so that the robustness of a fusion result can still be ensured when the quality of information in different modes is unbalanced. And the interest probability aggregation module performs time sequence smoothing on the instant interest probability through an exponential weighted moving average method, and inhibits short-time noise interference. And the dynamic threshold value judgment module is used for calculating a dynamic threshold value according to the mean value and the standard deviation of the recent window, comparing the smoothed interest probability with the dynamic threshold value, and judging that the corresponding interest label is activated when the smoothed interest probability is greater than or equal to the dynamic threshold value. According to the method, the user interests can be accurately recognized in a complex and changeable private domain community environment, and the adaptability, stability and real-time performance of interest recognition are achieved.

The invention discloses a bearing fault diagnosis method for an aero-engine driven by an audio large model, and relates to the technical field of bearing fault diagnosis. The method comprises the following steps: establishing the recognition capability of a large-scale language model on audio input through multi-stage processing to obtain a pre-training basic model; constructing a paired data set of aero-engine bearing vibration signals and detailed text description of the aero-engine bearing vibration signals, and performing amplitude normalization and coding on the vibration signals to obtain audio samples; establishing connection between the audio samples and the text modals through a cross-modal attention mechanism; and taking the audio sample and the text mode which are connected together as the input of a pre-training basic model, and directly generating a text fault tag and a diagnosis explanation in an autoregression mode by utilizing the generation capability of the model. According to the invention, a system capable of directly outputting interpretable and operable fault diagnosis results is established by using the general mode recognition capability of a large-scale audio pre-training model.

The invention relates to the technical field of image processing, solves the technical problem that open set detection cannot be realized when a new unknown target appears in a detection image, and particularly relates to an open set target detection method based on fusion text information. Comprising the following steps: acquiring text template information preset according to a detection scene, user text information input by a user and corresponding to a detection requirement, and an original image; and establishing an open set target detection model for fusing the text mode for the target sample with the unknown category to realize open set detection. Rich detection information is provided for target detection for solving the problem of newly appearing unknown targets in detection images, finer-grained understanding of the images is achieved, the precision of target detection in the open world is improved, the limitation that target detection in a traditional method needs target limitation can be overcome, unknown category detection in the open world is achieved, and the detection efficiency is improved. And the method better adapts to real-world scenes.

The invention provides a training data optimization method and device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence such as machine learning. According to the specific implementation scheme, a pre-trained entity recognition model is adopted to recognize information of a text mode in training data, and an entity in the information of the text mode is obtained; obtaining the attribute information of the entity in the information of the picture mode of the training data; and updating the training data based on the attribute information of the entity. According to the technology disclosed by the invention, the content of the training data can be enriched, the quality of the training data can be improved, and the training data can be optimized, so that the accuracy of the VLM trained based on the optimized training data can be effectively improved.