229 results about "Text alignment" patented technology

The invention discloses an underwater target detection method based on vision-language model knowledge distillation. Along with continuous maturity of a vision-language large model, the powerful visual feature extraction capability and text alignment capability are widely applied to the field of deep learning. The method comprises the following steps: firstly, collecting and fusing a plurality of public underwater data sets, and cleaning and preprocessing the data to construct a rich and high-quality training data set; secondly, a vision-language model is introduced, in a large model fine tuning stage, feature extraction and fine-grained fine tuning are carried out through image and text alignment, association between the image and the text is effectively captured, and therefore more accurate feature representation is generated; and finally, in a knowledge distillation stage, the method adopts a teacher network and a student network to carry out feature migration. The teacher network obtains fine features from the backbone network after fine tuning, and transmits the knowledge to the student network through a mean square error loss function, thereby improving the generalization ability and detection performance of the student network, and finally achieving the robust detection of an underwater target. According to the method, the vision-language model and the knowledge distillation technology are fully utilized, the underwater target features are enhanced, the model can learn the robust target features, and the detection robustness is improved.

The invention discloses a data analysis problem generation method based on image input and large model combination, and the method comprises the following steps: S1, obtaining original image data, and carrying out the preprocessing of the original image data; s2, constructing a visual language model based on a Qwen-VL architecture, and performing bidirectional alignment to generate a visual feature vector; s3, constructing a cue word template, and fusing the cue word template through a cross attention mechanism to generate structured text description; s4, constructing a large language model based on a Transform architecture, and performing supervision and fine tuning by adopting a LoRA method to generate a data analysis problem candidate sequence; s5, semantic consistency detection and structural rule matching are carried out, and sequences which do not meet semantic specifications or structural constraints are removed; and S6, constructing an image-text alignment triple, and writing the image-text alignment triple into the data structure in the JSON format for coding and storage. According to the method, the image can be converted into a data analysis problem, and the text generation quality and efficiency are remarkably improved.

The invention discloses a human motion generation method and system based on text alignment, and belongs to the technical field of computer vision and deep learning. According to the method, a two-stage training mode is adopted, in the first stage, a quantized variational auto-encoder model for reconstructing human motion is trained at first, then parameter freezing is carried out on a decoder network in the quantized variational auto-encoder model and a pre-trained text-to-motion cross-modal retrieval model, the decoder network and a bidirectional mask Transform model form a training framework for training, and the quantized variational auto-encoder model and the pre-trained text-to-motion cross-modal retrieval model form a training framework for training; and after training is completed, reasoning tasks are performed. According to the method, by improving a text encoder, adding semantic alignment loss and adopting a bidirectional mask type Transform architecture, the overall motion perception ability is enhanced, semantic alignment of motion is generated, the generated human motion sequence better conforms to original data distribution, the generated human motion sequence and an input motion description text are more semantic-consistent, and the motion perception ability of a human body is improved. And the motion generation quality is improved.

The invention relates to an intelligent short message template generation method and system based on a large language AI model, and the method comprises the steps: extracting cross-cultural communication frequency, sentence pattern preference analysis results and screen size adaptation data from social media interaction data and equipment use logs of target audiences, and generating a cultural background distribution matrix; identifying regional culture features in the culture background distribution matrix to obtain a taboo matching result; generating an expression habit preference matrix according to the updated culture background distribution matrix; generating a preliminary short message template according to the expression habit preference matrix and the symbol semantic mapping table, and outputting an adjusted template set conforming to message length control; analyzing the acceptability of the adjusted template set on the language conciseness requirement and the interactive button layout, and generating a new template set; evaluating a text alignment mode of the new template set; and screening candidate short message templates according to the information definition score. According to the invention, the accuracy and acceptability of cross-culture short message communication can be effectively improved, and the participation degree of audiences is enhanced.

The invention relates to the technical field of cross-language information processing, and provides a cross-language text fusion intelligent alignment method and system.The cross-language text fusion intelligent alignment method comprises the steps that a hierarchical alignment model is constructed through preprocessing and label recognition of multi-coding-type texts, deep semantic feature extraction and labeling of a multi-language pre-training model and text semantic and format information analysis; transform is taken as a core, cross-language semantic association is enhanced through a multi-head attention mechanism, character-level and paragraph-level format collaboration is realized through label weight allocation and condition constraint, the format alignment accuracy is obviously improved in a multi-language mixed typesetting scene, and the multi-language mixed typesetting efficiency is improved. The analysis capability of the model on the structured information can be extended to layout relation processing of texts, images and tables, so that the comprehensive alignment efficiency in a multi-modal fusion scene is remarkably improved. The accuracy of text and label fusion is guaranteed, the problem of confusion of messy codes and labels is avoided, and high-precision cross-language text alignment from semantics to formats, from single mode to multiple modes and from semantics to formats is achieved.

The invention relates to the technical field of computer vision, in particular to a reloading pedestrian re-identification method based on a visual language pre-training model. The method comprises the following steps: a training stage: inputting an image to obtain a clothing mask image, and generating clothing irrelevant / relevant prompts; text encoder parameters are fixed, prompt weights are optimized, text prompts are input into an encoder to obtain features, and a classifier is constructed to achieve image-text alignment through cross entropy loss; using a visual encoder to extract mask pattern features, and constraining a class center Euclidean distance to realize image-image alignment; stripping clothes characteristics: extracting clothes area characteristics and corresponding text characteristics, optimizing by a classifier, and introducing orthogonal loss to decouple clothes correlation; in the reasoning stage, a query image is input into a trained image encoder to extract features, and cosine similarity ranking and result returning are calculated according to the features of the image library. According to the technical scheme, the recognition accuracy of the pedestrian re-recognition method under the condition of pedestrian clothing change can be improved.

The invention provides an automatic driving model based on a multi-modal large model, a training method and an automatic driving method, and relates to the technical field of computers, in particular to the technical field of automatic driving and artificial intelligence. According to the implementation scheme, a training corpus data set is obtained, wherein the training corpus data set at least comprises a visual text alignment corpus and a spatial understanding training corpus for an automatic driving scene; encoding visual data in the visual text alignment corpus by using a visual encoder to obtain encoded data; mapping the coded data by using a mapping layer; processing the mapped coded data, the mapped text data and the spatial understanding training corpus by using a generation layer to obtain a first prediction result and a second prediction result of the automatic driving model; parameters of the automatic driving model are adjusted based at least on the first prediction result and the second prediction result. The automatic driving model obtained through training in the embodiment of the invention has the understanding ability of multi-modal information and also has the reasoning ability in an automatic driving scene.

The invention discloses a text generation image diffusion model enhancement method based on multi-target preference optimization. The method comprises the following steps: firstly, determining a plurality of reward models, and constructing a sample pair training set comprising positive and negative samples; and then, generating a loss weight of each sample pair in the sample pair training set, performing fine tuning training on the text map diffusion model by using the sample pair training set, and in the fine tuning training process, calculating a loss function value of each sample pair in combination with the loss weight of each sample pair until the training is completed, thereby obtaining an aligned text map diffusion model. According to the method provided by the invention, manual data annotation is not needed, the problems of preference inconsistency and over-optimization in a multi-reward scene are effectively solved, and the image quality, the text alignment capability and the multi-target optimization performance of the text-to-image generation model are remarkably improved. The method is superior to an existing optimization method under single-reward and multi-reward setting, shows higher generation quality and robustness, and can be seamlessly applied to various picture generation models.

The invention provides a landslide classification method and system based on a visual language model and a cross attention mechanism. The method and the system specifically comprise the following steps: data preprocessing: carrying out Canny edge detection on an RGB image, and calculating terrain attributes such as a gradient and a slope direction for a DEM (Digital Elevation Model); feature extraction: capturing local features by adopting a reflection filling convolution layer and multi-scale residual connection; a visual language model is introduced, wherein semantic enhancement features are extracted through image-text alignment by means of the visual language model; cross self-attention fusion: capturing a global context through self-attention, and focusing heterogenous data complementary information by cross attention; and classifying and outputting: outputting a result by using global average pooling and a linear classifier. Through the visual language model and the cross self-attention mechanism, the landslide recognition capability under the complex terrain is effectively improved, an efficient and reliable technical means is provided for geological disaster monitoring, and the method can be widely applied to the fields of landslide recognition, risk assessment and the like.

The invention discloses an image-text processing method and device for marking a compression framework. The method comprises the following steps of: extracting visual features; a visual mark screening processing step; a text feature extraction step; and a multi-modal fusion and model processing step. The method has the beneficial effects that the inference efficiency of the MLLMs is remarkably improved under the condition that the visual mark compression frame does not need additional training; through global and local information fusion of the DVTS module and text guide supplement of the TGVC module, the number of visual marks is greatly reduced, meanwhile, key visual information is reserved, and visual-text alignment is enhanced; experiments show that in various image and video benchmark tests, compared with an existing method, the framework has the advantages that the calculation cost is greatly reduced, the model performance is maintained and even improved, and the framework has remarkable technical advantages and application potential.

Embodiments of the present disclosure relate to training-free consistent text-to-image generation. A pre-trained text-to-image diffusion model is leveraged to generate images depicting a consistent subject for diverse prompts describing scenes. Inputs to the model are a text description of at least one subject with prompts (scene text descriptions) describing scenes, where each prompt is associated with a different generated image and the text description is used for all images that depict the subject. Internal activations (intermediate data) computed by the model during generation of the different images are shared for generation of the different images. A subject-driven shared attention block and correspondence-based feature injection are incorporated into the model to promote subject consistency within each image and / or between images. Additionally, layout diversity is encouraged while maintaining subject consistency. The model achieves state-of-the-art performance on subject consistency and text alignment, without requiring any optimization and naturally extends to multi-subject scenarios.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a multi-module collaborative voice generation method, which comprises the following steps: acquiring a reference text and reference voice, converting the reference text into a phoneme sequence and generating potential features, converting the reference voice into a Mel spectrogram and extracting style features, and extracting fundamental frequency information. A phoneme sequence and a Mel spectrogram are input into a text alignment module to generate alignment information, voice duration is predicted based on the alignment information and potential features, fundamental frequency of voice is predicted through fundamental frequency information, the predicted duration, fundamental frequency and style features are input into a decoder to generate a reconstructed Mel spectrogram, and finally the reconstructed Mel spectrogram is converted into voice audio data through a vocoder. According to the method, the generated voice can better reflect the natural rhythm and tone change of the text content by combining accurate control on the voice duration, the fundamental frequency and the style characteristics, and the rhythm and expressive force of the voice are enhanced, so that the naturalness and the fidelity of voice synthesis are improved.

The invention provides an intelligent decision-making method based on a multi-modal large model and related equipment, which are applied to an artificial intelligence technology, and the method comprises the steps: collecting a current field image of a to-be-decided region, and carrying out the preprocessing of the current field image, and obtaining a target field image; inputting the target field image into a pre-trained vision-text alignment model, and performing prediction by the vision-text alignment model by using the target field image to obtain a disease and pest diagnosis scheme corresponding to the to-be-decided region; dividing the to-be-decided region into a plurality of partitions, and processing current data of each mode in current multi-mode data of the partitions to obtain a space-time sequence of each mode; inputting the space-time sequence of each modal of the subarea into a pre-trained irrigation decision model, and performing prediction by the irrigation decision model according to the space-time sequence of each modal to obtain irrigation data of the subarea; the irrigation strategy of each subarea is generated according to the disease and pest diagnosis scheme and the irrigation data of each subarea. The cost can be reduced, and the generation efficiency and accuracy of the irrigation strategy can be improved.

The invention provides a multi-modal large model training system and method applied to the traffic field. The system comprises a large language model module, a feature alignment module, a multi-modal module and a user interaction module. Wherein the large language model module adopts a decoder architecture improved in Transform, and traffic text data and non-traffic text data which are collected in advance are used for training to obtain a language model; the feature alignment module extracts high-level semantic features of an image through an image encoder, and performs image and text alignment training in combination with a language model to obtain a text image alignment model; the multi-modal module adopts multi-modal instruction data containing a text image question and answer pair and combines the text image alignment model to carry out fusion training to obtain a multi-modal large model; and the user interaction module is used for receiving a user request, performing reasoning prediction through a multi-modal large model, and providing prediction and interaction results. The method solves various complex and tedious problems in the traffic field, and has excellent expansibility and robustness.

The invention discloses a cross-modal image-text retrieval method based on pulse fusion, and belongs to the technical field of image-text retrieval. The method comprises the following steps: inputting a target image set into a target detection network to obtain an image floating point code, and inputting a target text set into a word segmentation device to obtain a word floating point code; inputting the image floating point code and the word floating point code into a pulse encoder to obtain a first image pulse code and a first text pulse code; inputting the first image pulse code and the first text pulse code into a pulse cross attention fusion module to obtain a second image pulse code and a second text pulse code; respectively carrying out weighted accumulation and average pooling on the second image pulse code and the second text pulse code to obtain an image floating point feature vector set and a text floating point feature vector set, and calculating cosine similarity to obtain an image-text alignment result; and the text retrieval result of each image in the target image set is obtained based on the image-text alignment result, so that the accuracy and efficiency of image-text retrieval are improved.

The invention provides a multi-modal data pairing method and system based on deep learning, and relates to the technical field of data processing, and the method comprises the steps: obtaining a video multi-frame sequence and a target text, and respectively extracting an overlapped frame group set and a standardized text sequence; performing spatio-temporal feature extraction and text dependency relationship coding to obtain a video time sequence vector sequence and a text vector sequence; executing cross-modal alignment search, and constructing a monotonic matching path set; calculating a semantic and action entity relationship consistency score of the paired elements on the path to obtain a comprehensive score; and determining an alignment relationship between the video and the text based on the optimal path. According to the method, accurate matching of the video and the text is realized, and the cross-modal retrieval efficiency is improved.

The invention provides a video understanding method and device of a multi-modal large model based on multi-branch visual representation, relates to the technical field of vision, and aims to realize universal, generalized and accurate fine-grained video understanding. The method comprises the steps of obtaining text feature representation corresponding to text question information; performing time sequence information coding on the to-be-understood video to obtain time sequence characteristic representation; fine-grained information extraction is carried out on the to-be-understood video to obtain visual feature representation, and the visual feature representation comprises visual-text alignment feature representation and fine-grained visual feature representation; at least performing spatial compression on the visual feature representation to obtain a compressed visual feature representation; and inputting the compressed visual feature representation, the time sequence feature representation and the text feature representation into a large language model, and generating a video understanding result corresponding to the text question information.

The invention provides an open vocabulary semantic segmentation method and device based on feature interaction and multi-modal data fusion. An SAM encoder of freezing parameters is adopted to extract RGB image and Mask image features in parallel, and boundary information enhancement is carried out through a feature fusion module. A CLIP image encoder of freezing parameters is used for carrying out multi-layer feature extraction on an RGB image, and multi-scale semantic information optimization is carried out through a feature enhancement module. And fusing image features extracted by the SAM branch and the CLIP branch through a feature interaction module to realize cross-network feature complementation. In a prediction stage, a category integration strategy combined with a temperature scaling operation is introduced to optimize prediction effects of known and unknown categories. According to the method, the advantages of SAM in the aspect of image boundary extraction and the image-text alignment capability of CLIP are combined, and the generalization capability and robustness of the model in an open vocabulary semantic segmentation task are effectively improved through multi-modal feature fusion and interaction.

Embodiments of the disclosure provide a solution for speech generation. A method includes: determining, based on a target text, a plurality of phoneme feature representations corresponding to a sequence of phonemes in the target text and respective phoneme durations for the plurality of phoneme feature representations; extending the plurality of phoneme feature representations based on the respective phoneme durations, to obtain an extended sequence of phoneme feature representations; masking at least one phoneme feature representation in the extended sequence of phoneme feature representations, to obtain a sequence of masked phoneme feature representations; and generating a target speech corresponding to the target text at least based on the sequence of masked phoneme feature representations.

The invention provides a visual-language model training method and device and related equipment, and the method comprises the steps: constructing a training data set comprising an image-text alignment task, a text-driven visual positioning task and a plain text reasoning task through determining a visual encoder and a heterogeneous language model which form a multi-modal heterogeneous recognition model; carrying out dimension dynamic alignment adaptation on the heterogeneous language model, enabling a parameter architecture of the heterogeneous language model to be matched with a visual feature dimension output by a visual encoder, and carrying out supervision and fine adjustment on the visual-language model by adopting a freezing-unfreezing two-stage training strategy based on a training data set; and performing parameter training on a cross-modal alignment module connected with the visual encoder and the heterogeneous language model so as to obtain a trained visual-language model. According to the training method, time consumed by model training is saved, and the convergence speed is increased. Through dimension dynamic alignment and hierarchical weight mapping, the pre-training language ability is reserved to the maximum extent, the plain text task performance loss is reduced, and disastrous forgetting of the model is avoided.

The invention discloses an image-text alignment model training method and device, computer equipment and a storage medium, and the method comprises the steps: carrying out the coding of input image data and text data, and obtaining an image block feature sequence and a text token feature sequence; performing bidirectional feature interaction on the image block feature sequence and the text token feature sequence to obtain fused image features and fused text prompt features; inputting the fused image features, the fused text prompt features and a preset output token into an SAM mask decoder to generate an image mask; filtering the fused image features based on the image mask, extracting image global features related to the text description, and obtaining text global features; constructing a joint loss function; and performing end-to-end training on the model based on the joint loss function until the model converges. According to the method, the limitation of coarse-grained alignment of a traditional model is effectively solved, and the pertinence and matching precision of image-text features are improved.

The invention relates to the technical field of automatic driving end-to-end perception, in particular to an end-to-end automatic driving long tail recognition method based on comparative learning pre-training, and the method comprises the steps: firstly generating synthetic image data with long tail distribution characteristics through a conditional diffusion model; a fine-grained scene classifier is adopted to carry out systematic arrangement and semantic annotation on the generated samples, and a structured multi-modal image-text alignment data set is constructed; and finally, fusing the enhanced data set with the original training set, and optimizing a vision-language joint embedding space through a multi-task contrast loss function to realize parameter updating of the pre-training model. According to the method, a closed-loop optimization mechanism of a generative data enhancement and contrast learning framework is creatively established, the problem of data scarcity in a long-tail distribution scene is effectively relieved, and the cross-modal representation capability and downstream task generalization performance of the model on low resource categories are remarkably improved.

The invention discloses a multi-modal fine-grained semantic alignment method and device based on a graph neural network, and relates to the field of multi-modal deep learning. Firstly, deep feature extraction is performed on input multi-modal original data, and then word-level text features and local image features are constructed into a cross-modal graph structure. And performing weighted aggregation on node neighborhood information of the cross-modal graph structure through the graph attention network. And finally, carrying out weighted fusion on the text alignment features and the image alignment features. According to the cross-modal feature fusion method, the word-level text features and the local image features are uniformly abstracted into the graph structure nodes for refined alignment, a more accurate cross-modal semantic corresponding relation can be captured, the heterogeneity problem in expression modes and semantic structures is effectively relieved, and the accuracy and reliability of cross-modal feature fusion are improved. The graph attention network can adaptively adjust the weight distribution of information propagation, highlights the effect of key features in the alignment process, and ensures that the model makes full use of important semantic relationships.

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal large model-based bidding and cross-bidding early warning and blocking method and system, and the method comprises the following steps: receiving and preprocessing various qualification files, including business licenses, industry licenses, auditing reports and bidding files, submitted by suppliers; applying a document layout analysis model based on joint training of a visual coding network and a text alignment network; the method has the beneficial effects that a scanned certificate, an audit report and a bidding file are automatically analyzed into structured semantic fragments; performing same semantic space alignment on the fragments, supplier basic information, historical records and real-time captured data of an authoritative authentication website; any contradiction or gap among different sources of key information such as registration capital, permission items, financial indexes and the like can be quickly found, and interpretable risk scores and processing suggestions can be given; on the premise of ensuring the private security of the data, the qualification auditing efficiency and accuracy are improved, and the workload of manual checking is reduced.

The invention discloses a vision and text alignment method and system, and belongs to the technical field of artificial intelligence and multi-modal semantic understanding. In order to solve the problem of insufficient vision and language deep fusion in the existing multi-mode question answering, the method mainly comprises the following steps: mapping visual features to a self-attention input space of a language model through a perceptron network, and introducing a fusion attention mechanism into each layer of decoder of the language model to realize layer-by-layer interaction processing of vision and texts. According to the method, deep alignment and fusion of visual information and text semantics can be realized, and the understanding and generation capabilities of a multi-modal question-answering system are improved.

The invention relates to a cross-modal retrieval technology, in particular to a cross-modal image-text pedestrian retrieval method based on a generative model, which comprises the following steps: acquiring text description information and a corresponding original pedestrian image; calling a diffusion generation model to generate an intermediate image based on the text description; original image features and text features are extracted through an image encoder and a text encoder respectively, and semantic-fused text feature representation is constructed; calculating the similarity between the image feature representation and the text feature representation to obtain an image-text matching score; introducing and generating an intermediate image as a semantic bridge, and realizing multi-modal fusion among the image, the text and the intermediate image through a cross attention mechanism to obtain fused image and text features; and using the fusion features to train a recognition model, and finally outputting a pedestrian image retrieval result corresponding to the text description. According to the method, the image-text alignment precision can be improved, the robustness of a scene with incomplete text description is enhanced, and the recognition accuracy in a cross-modal retrieval task is improved.

The invention discloses a multi-modal power sample feature migration method and system based on dual cross-modal information decoupling, electronic equipment and a storage medium, which are applied to the technical field of smart power grids, and comprise the following steps: carrying out hierarchical alignment of visual features and language features by adopting a fine-grained image-text alignment strategy, and capturing distinctive multi-modal representation; based on a double-flow attention mechanism, performing enhanced feature extraction on the multi-modal data; wherein the double-flow attention mechanism comprises an image self-adaptive attention mechanism and a text guide attention mechanism; the method comprises the following steps: performing cross-modal information decoupling processing on multi-modal data, separating to obtain semantic features representing general semantics and modal specific features representing modal features, and performing fine-grained alignment recombination processing to obtain cross-modal fusion features; and migrating the deeply fused multi-modal power sample features to a downstream task of the power system. According to the method, the understanding of the multi-modal features and the further improvement of the migration capability and the efficient operation of the power system tasks are realized.

The invention provides a cross-modal semantic alignment method, device and equipment based on fine-grained semantic decoupling, and the method comprises the steps: inputting a to-be-aligned image into a pre-training image encoder of a cross-modal semantic alignment model, and obtaining a global semantic feature, meanwhile, a semantic feature extraction module is used in a high-level encoding block of the image encoder and based on a preset query vector, high-level visual feature extraction is conducted, visual fine-grained semantic features are obtained and input into a semantic feature decoupling module to be decoupled, and fine-grained semantic features of attributes, objects and combinations are obtained; inputting prompt words corresponding to attributes, objects and combinations in the to-be-aligned image into a text encoder for text feature extraction to obtain soft prompt text features of the attributes, the objects and the combinations; and inputting the fine-grained semantic features and the soft prompt text features corresponding to the global semantic features, the attributes, the objects and the combinations into a loss calculation module for image-text alignment, and performing linear processing on an alignment result to obtain an alignment prediction result.

The invention relates to the technical field of image texts, in particular to an image text alignment method based on a multi-modal large language model, which comprises the following steps of: segmenting an image into local areas and encoding the local areas into visual feature vectors by adopting a pre-trained visual encoder; visual and text features are mapped to the same dimension space through a learning projection layer, an InfoNCE loss function is adopted to maximize the similarity of a positive sample pair, dynamic interaction between text query and an image area is achieved through a cross attention mechanism, language module parameters are frozen to pre-train a visual encoder, an overall module is combined to be fine-tuned, and text query is completed. An error case is analyzed, and a verified and improved alignment module is obtained through a data enhancement optimization module. The method solves the problems that a traditional image text alignment method is incomplete in manual feature extraction and low in precision, an existing deep learning model is limited in cross-modal semantic understanding ability, and the generalization ability is poor when facing long-tail data.

The invention discloses a multi-modal event extraction method and device, equipment, a storage medium and a program product, and the method comprises the steps: carrying out the feature extraction of text data in multi-modal data, generating a text label sequence based on a text feature extraction result, carrying out the target detection of image data in the multi-modal data, and obtaining a target detection result; obtaining image-text fine-grained information of the multi-modal data based on the text label sequence and the target detection result, generating a text entity set and a visual entity set based on the image-text fine-grained information, performing entity matching on the text entity set and the visual entity set, performing fine-grained image-text alignment on the multi-modal data based on an entity matching result, and obtaining the multi-modal data. According to the method, the multi-modal data are subjected to fine-grained image-text alignment, and the multi-modal data subjected to fine-grained image-text alignment are input into the pre-constructed event extraction model for event extraction, so that effective analysis of global and local semantic information is realized, events can be accurately extracted from the multi-modal data with complex semantic scenes, and missing of potential event values is effectively avoided.