Method, system, and apparatus for analyzing event on the basis of generative task and multimodality

Abstract

The present application discloses a method, system, and apparatus for analyzing an event on the basis of a generative task and multimodality, which are used for quickly and accurately ascertaining the progress of an event. The method of the present application comprises: generating structured event information from multi-modal data, and obtaining an event relationship; constructing an event graph; extracting multi-modal features, and using a cross-modal attention mechanism to obtain multi-modal sentiment relationship vectors; on the basis of a preset graph neural network and in view of the event graph, obtaining sentiment feature vectors; calculating a contrastive learning loss value, and obtaining a target sentiment feature on the basis of the contrastive learning loss value; using the cross-modal attention mechanism to generate a target event graph; extracting a relationship feature, and obtaining an event prediction graph in view of attention weights and a preset time series model; extracting a causal chain and performing inference, and obtaining a trigger condition and an emotion factor after the inference; and combining the event prediction graph and the target event graph with a node feature sequence, the trigger condition, and the emotion factor, and obtaining an event analysis result.

Description

An event analysis method, system, and apparatus based on generative tasks and multimodal approaches. Technical Field

[0001] This application relates to the field of computer data analysis, and in particular to a method, system and apparatus for event analysis based on generative tasks and multimodality. Background Technology

[0002] With the rise of smart city construction, cities are becoming increasingly larger and more complex, leading to stronger demands for governance in areas such as security, services, and refined management. Urban event situation monitoring and analysis have become key supporting elements, and various departments hope to integrate new technologies to govern urban operations and respond promptly to governance needs. However, driven by mobile internet and IoT technologies, urban event data exhibits complex characteristics such as polymorphism, heterogeneity, spatiotemporal nature, and social aspects. Furthermore, the diverse formats of urban event data pose significant challenges to data fusion and analysis.

[0003] In existing technologies, event analysis typically employs feature fusion techniques based on deep learning. First, the BERT model is used to extract semantic features from text data, converting the text data into vector representations to capture contextual information and semantic relationships within the text. Then, a convolutional neural network is used to extract visual features from images, and an audio processing model is employed to extract acoustic features from audio. After extraction, the extracted features from different modalities are fused, and the fused features are input into a regression model for analysis and prediction, yielding the event analysis results.

[0004] However, event analysis based on deep learning feature fusion technology, which only fuses the basic features of each modality, is difficult to accurately grasp the trend and emotional dynamics of events when faced with complex and ever-changing urban events. This affects the scientific nature and timeliness of urban governance decisions, leading to missed opportunities for emergency response and an expansion of the event's impact. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a method, system, and apparatus for event analysis based on generated tasks and multimodal approaches.

[0006] The technical solution provided in this application is described below:

[0007] The first aspect of this application provides an event analysis method based on generative tasks and multimodal analysis, the method comprising:

[0008] Acquire multimodal data, which includes text data, image data, and audio data;

[0009] Based on a preset large-scale model, a generation task is set, the multimodal data is input into the generation task, structured event information is generated, and relation extraction and semantic analysis are performed on the structured event information to obtain event relations.

[0010] Construct an event graph, using the structured event information as nodes in the event graph, and connect the nodes according to the event relationships;

[0011] Text features, image features, and audio features of the multimodal data are extracted, and a cross-modal attention mechanism is used to fuse the text features, image features, and audio features to obtain a multimodal sentiment relationship vector;

[0012] The multimodal sentiment relationship vectors are aggregated based on a preset graph neural network, and sentiment feature vectors are obtained by combining them with the event graph.

[0013] Positive and negative sample pairs are constructed based on the emotional feature vector and the multimodal data. Based on the positive and negative sample pairs, a contrastive learning loss value is calculated using a preset contrastive function. The contrastive learning loss value is then used to align and optimize the emotional feature vector and the multimodal data to obtain the target emotional feature.

[0014] The emotional feature vector is transmitted to the node of the event graph, and the attention weight of the emotional feature vector is adjusted using the cross-modal attention mechanism to generate the target event graph;

[0015] The preset graph neural network is used to extract relational features from the target event graph, and the attention weights are combined to obtain a node feature sequence. The node feature sequence is then input into a preset time series model to capture dynamic evolution patterns, and an event prediction graph is obtained based on the dynamic evolution patterns.

[0016] Extract the causal chain from the event prediction graph, and perform causal relationship reasoning based on the node feature sequence and the causal chain to obtain the triggering conditions and emotional factors.

[0017] The event prediction map is matched with the target event map, and the event analysis results are calculated by combining the node feature sequence, the triggering conditions, and the emotional factors.

[0018] Optionally, the step of extracting text features, image features, and audio features from the multimodal data, and fusing the text features, image features, and audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relation vector includes:

[0019] The text semantics and text sentiment in the multimodal data are extracted using a pre-defined language model, and text features are obtained based on the text semantics and text sentiment.

[0020] Image scenes and facial expressions are extracted from the multimodal data using ResNet or Vision Transformer, and image features are obtained based on the image scenes and facial expressions.

[0021] The Wav2Vec audio model is used to extract audio intonation, audio volume, and audio rhythm from the multimodal data, and audio features are obtained based on the audio intonation, audio volume, and audio rhythm.

[0022] The target weights of the text features, image features, and audio features are calculated using a cross-modal attention mechanism. The target weights are then combined with the text features, image features, and audio features to obtain a multimodal sentiment relation vector.

[0023] Optionally, the step of setting a generation task based on a preset large model, inputting the multimodal data into the generation task, and generating structured event information includes:

[0024] Based on a preset large-scale model, a generation task is set, and the multimodal data is input into the generation task for training to obtain a structured description.

[0025] Under the preset parameters of the current preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs.

[0026] Based on the target probability, the preset parameters are adjusted using the cross-entropy loss function, and structured event information is generated after the adjustment.

[0027] Optionally, the preset parameters are adjusted based on the target probability and the cross-entropy loss function to generate structured event information, which is expressed by the following formula:

[0028] Where L(θ) represents structured event information, N represents the number of multimodal data, and P(Y) represents the number of data points. i |X i ;θ) represents the target probability, θ represents the preset parameter, X i Y represents the multimodal data of the i-th input; i This represents the structured description of the i-th output.

[0029] Optionally, after matching the event prediction map with the target event map, and combining the node feature sequence, the triggering condition, and the emotional factors to calculate the event analysis result, the method further includes:

[0030] The data structure of the event analysis results is converted into the target data structure, and a preset visualization tool is used to generate a visualization analysis result from the converted event analysis results.

[0031] Optionally, the step of extracting causal chains from the event prediction graph, performing causal relationship inference based on the node feature sequence and the causal chains, and obtaining triggering conditions and emotional factors after inference, is expressed by the following formula:

[0032] Among them, v t Indicates the triggering condition, e t Indicates the emotional factor, v t+k This represents the triggering condition corresponding to time step k. W represents the node feature sequence. p e represents the parameters of the weight matrix. t (t) represents the causal chain.

[0033] Optionally, the contrastive learning loss value is represented by the following formula:

[0034] Among them, L contrastive Indicates the comparison loss value, The emotional feature vector representing the positive sample pair. Let τ represent the sentiment feature vector of the negative sample pair, τ represent an adjustable parameter, and N represent the amount of multimodal data. The similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair is represented by k, where k represents the traversal index variable.

[0035] A second aspect of this application provides an event analysis system based on generative tasks and multimodality, the system comprising:

[0036] An acquisition unit is used to acquire multimodal data, which includes text data, image data, and audio data;

[0037] The first generation unit is used to set a generation task based on a preset generation model, input the multimodal data into the generation task, generate structured event information, and perform relation extraction and semantic analysis on the structured event information to obtain event relations.

[0038] A construction unit is used to construct an event graph, using the structured event information as nodes in the event graph, and connecting the nodes according to the event relationships;

[0039] The fusion unit is used to extract text features, image features, and audio features from the multimodal data, and to fuse the text features, image features, and audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector.

[0040] An aggregation unit is used to aggregate the multimodal sentiment relationship vectors based on a preset graph neural network and combine them with the event graph to obtain sentiment feature vectors.

[0041] The first computing unit is used to construct positive sample pairs and negative sample pairs based on the emotional feature vector and the multimodal data, calculate the contrastive learning loss value using a preset contrast function based on the positive sample pairs and the negative sample pairs, and perform alignment optimization on the emotional feature vector and the multimodal data through the contrastive learning loss value to obtain the target emotional feature.

[0042] The second generation unit is used to transmit the sentiment feature vector to the node of the event graph, and use the cross-modal attention mechanism to adjust the attention weight of the sentiment feature vector to generate the target event graph.

[0043] The capture unit is used to extract relational features from the target event graph using the preset graph neural network, combine the attention weights to obtain a node feature sequence, input the node feature sequence into a preset time series model to capture dynamic evolution patterns, and obtain an event prediction graph based on the dynamic evolution patterns.

[0044] The reasoning unit is used to extract the causal chain in the event prediction graph, perform causal relationship reasoning based on the node feature sequence and the causal chain, and obtain the triggering condition and emotional factors after reasoning.

[0045] The second calculation unit is used to match the event prediction map with the target event map, and calculate the event analysis result by combining the node feature sequence, the triggering condition and the emotional factor.

[0046] Optionally, the fusion unit is specifically used for:

[0047] The text semantics and text sentiment in the multimodal data are extracted using a pre-defined language model, and text features are obtained based on the text semantics and text sentiment.

[0048] Image scenes and facial expressions are extracted from the multimodal data using ResNet or Vision Transformer, and image features are obtained based on the image scenes and facial expressions.

[0049] The Wav2Vec audio model is used to extract audio intonation, audio volume, and audio rhythm from the multimodal data, and audio features are obtained based on the audio intonation, audio volume, and audio rhythm.

[0050] The target weights of the text features, image features, and audio features are calculated using a cross-modal attention mechanism. The target weights are then combined with the text features, image features, and audio features to obtain a multimodal sentiment relation vector.

[0051] Optionally, the first generation unit is specifically used for:

[0052] Based on a preset large-scale model, a generation task is set, and the multimodal data is input into the generation task for training to obtain a structured description.

[0053] Under the preset parameters of the current preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs.

[0054] Based on the target probability, the preset parameters are adjusted using the cross-entropy loss function, and structured event information is generated after the adjustment.

[0055] Optionally, the preset parameters are adjusted based on the target probability and the cross-entropy loss function to generate structured event information, which is expressed by the following formula:

[0056] Where L(θ) represents structured event information, N represents the number of multimodal data, and P(Y) represents the number of data points. i |X i ;θ) represents the target probability, θ represents the preset parameter, X i Y represents the multimodal data of the i-th input; i This represents the structured description of the i-th output.

[0057] Optionally, a third generation unit may also be included, specifically for:

[0058] The data structure of the event analysis results is converted into the target data structure, and a preset visualization tool is used to generate a visualization analysis result from the converted event analysis results.

[0059] Optionally, the step of extracting causal chains from the event prediction graph, performing causal relationship inference based on the node feature sequence and the causal chains, and obtaining triggering conditions and emotional factors after inference, is expressed by the following formula:

[0060] Among them, v t Indicates the triggering condition, e t Indicates the emotional factor, v t+k This represents the triggering condition corresponding to time step k. W represents the node feature sequence. p e represents the parameters of the weight matrix. t (t) represents the causal chain.

[0061] Optionally, the contrastive learning loss value is represented by the following formula:

[0062] Among them, L contrastive Indicates the comparison loss value, The emotional feature vector representing the positive sample pair. Let τ represent the sentiment feature vector of the negative sample pair, τ represent an adjustable parameter, and N represent the amount of multimodal data. The similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair is represented by k, where k represents the traversal index variable.

[0063] A third aspect of this application provides an event analysis apparatus based on generative tasks and multimodality, the apparatus comprising:

[0064] Processor, memory, input / output units, and bus;

[0065] The processor is connected to the memory, the input / output unit, and the bus;

[0066] The memory stores a program, which the processor invokes to execute the first aspect and any one of the optional methods in the first aspect.

[0067] A fourth aspect of this application provides a computer-readable storage medium on which a program is stored, which, when executed on a computer, performs the methods of the first aspect and any one of the first aspects.

[0068] As can be seen from the above technical solutions, this application has the following advantages:

[0069] This event analysis method, based on generative tasks and multimodal approaches, overcomes the limitations of existing deep learning-based feature fusion techniques that only fuse basic features from various modalities by utilizing multimodal data, large-scale model generation tasks, and cross-modal attention mechanisms. This results in more accurate urban event analysis. First, text, image, audio, and video features are extracted from multimodal data, and weights are dynamically allocated using a cross-modal attention mechanism to form multimodal sentiment relationship vectors. This allows for a more comprehensive and in-depth exploration of potential correlations between data, enhancing the ability to capture event features. Then, structured event information is generated and an event graph is constructed. Combined with a pre-defined graph neural network, the multimodal sentiment relationship vectors are aggregated to generate sentiment feature vectors. This tightly integrates the emotional information of the event with its structure and relationships, enabling a more accurate grasp of event trends and emotional dynamics.

[0070] After generating sentiment feature vectors, the contrastive learning loss value calculated using a preset contrastive loss function is used to optimize the sentiment feature vectors and multimodal data. Then, the sentiment feature vectors are input as node attributes into the event graph. A cross-modal attention mechanism is used to track the development and changes of the event in real time, dynamically adjusting the weights of the sentiment feature vectors to generate a target event graph. Subsequently, a preset graph neural network is used to process the target event graph, extracting the relational features of the nodes. During the extraction process, the graph neural network can deeply explore the potential connections between nodes, providing reliable relational information for subsequent analysis. After extracting the relational features, the node feature sequence is obtained by combining the attention weights and inputting the node feature sequence into a preset time series model. Upon receiving the node feature sequence, the time series model iteratively calculates to capture the dynamic evolution of the event over time, generating an event prediction graph based on the dynamic evolution pattern, thereby predicting the future development trend of the event. Finally, the event prediction graph is fused with the target event graph, and the event analysis results are obtained by combining the node feature sequence. The event analysis results obtained in this application comprehensively consider the current state and future trends of the event, and can provide a more comprehensive and accurate basis for decision-making when facing complex and ever-changing urban events. This helps to improve the scientific nature and timeliness of urban governance decisions, avoid missing the best opportunity to deal with emergency events, and thus effectively control the scope of the event's impact. Attached Figure Description

[0071] Figure 1 is a schematic flowchart of an embodiment of the event analysis method based on generation tasks and multimodality provided in this application;

[0072] Figure 2 is a flowchart illustrating another embodiment of the event analysis method based on generation tasks and multimodality provided in this application;

[0073] Figure 3 is a schematic diagram of an embodiment of the event analysis system based on generative tasks and multimodality provided in this application;

[0074] Figure 4 is a schematic diagram of an embodiment of the event analysis device based on generation tasks and multimodality provided in this application. Detailed Implementation

[0075] This application provides an event analysis method based on generation tasks and multimodal analysis, which can quickly and accurately grasp the development trend of events. It should be noted that the event analysis method based on generation tasks and multimodal analysis in this application is applied to a terminal.

[0076] Referring to Figure 1, this application first provides an embodiment of an event analysis method based on generative tasks and multimodality, which includes:

[0077] S101. Acquire multimodal data, which includes text data, image data, and audio data;

[0078] In this embodiment, in order to comprehensively acquire multimodal data on urban event trends, it is necessary to filter out the source platforms of text data, image data, and audio data respectively. Comprehensive acquisition of multimodal data can provide richer information sources for subsequent analysis.

[0079] Text data is primarily obtained through web crawling technology, extracting text information related to urban events from various news websites, social media platforms, and government bulletin boards. This text includes event descriptions and public discussions. Image data is mainly obtained by capturing keyframes from surveillance cameras in real time and by capturing images related to urban events from social media platforms. These keyframes and images reflect the situation at the scene when the events occurred. Audio data is obtained through recording devices in public places and on-site interviews, capturing sound information such as shouts and alarms, providing multi-dimensional support for event assessment. By employing multiple channels and methods, rich and comprehensive multimodal data is obtained, improving the accuracy and comprehensiveness of urban situation monitoring and avoiding the limitations of single-modal data.

[0080] S102. Based on the preset large-scale model, set the generation task, input multimodal data into the generation task, generate structured event information, and perform relation extraction and semantic analysis on the structured event information to obtain event relationships.

[0081] When setting up a generation task based on a preset large generation model, the acquired multimodal data is preprocessed, including noise removal, filtering of irrelevant information, and handling of missing values. The preprocessed multimodal data is then input into the generation task, where the preset large generation model generates structured event information.

[0082] After generating structured event information, the pre-built large model will also perform relation extraction and semantic analysis on the generated structured event information to identify causal relationships, temporal relationships, etc. After the identification is completed, the event relationships are obtained. The relation extraction and semantic analysis of the structured event information further explores the intrinsic connections between events, making the understanding of the urban situation more in-depth and comprehensive.

[0083] S103. Construct an event graph, using structured event information as nodes in the event graph, and connect the nodes according to event relationships;

[0084] When constructing an event graph, the processed structured event information is used as nodes in the event graph. Each node represents a specific event and contains detailed information about the event, such as event type and key elements. Then, based on the event relationships obtained through relation extraction and semantic analysis, relationship connections are established between the nodes. The completed event graph can quickly provide an understanding of the relationships between events and the overall situation.

[0085] In practical applications, an event graph can be formalized as a directed graph G = (V, E), where V is the set of event nodes and E represents the causal relationships between events, such as node v. i and v j v i It is v j The cause is that there is an edge e. i,j ∈E, indicating from v i Point to v j The causal relationship.

[0086] S104. Extract text features, image features, and audio features from the multimodal data, and use a cross-modal attention mechanism to fuse the text features, image features, and audio features to obtain a multimodal sentiment relationship vector.

[0087] In this embodiment, text features, image features, and audio features are extracted from the acquired multimodal data. Extracting features from different modal data can fully explore the key information in each modal data.

[0088] After feature extraction, a cross-modal attention mechanism is used for feature fusion. This mechanism allows features from different modalities to reference each other. For example, text features reference information carried by image and audio features during fusion. In analyzing urban safety incidents, the text feature might describe a tense atmosphere, the image feature might show a chaotic scene, and the audio might convey noise. The cross-modal attention mechanism combines this information from different modalities, ultimately resulting in a multimodal sentiment vector that integrates information from multiple modalities. The application of this mechanism breaks down information barriers between different modalities, allowing features from different modalities to complement and fuse with each other. The resulting multimodal sentiment vector contains rich information from multiple data sources, significantly improving the understanding and representation of sentiment relationships in multimodal data. Compared to single-modal feature analysis, it can more comprehensively and accurately capture the emotional and semantic information behind the data, providing a solid data foundation for subsequent sentiment analysis and event understanding.

[0089] S105. Aggregate multimodal sentiment relationship vectors based on a preset graph neural network and combine them with an event graph to obtain sentiment feature vectors;

[0090] The pre-defined graph neural network (BRNN) takes multimodal sentiment relationship vectors as input and uses its network structure and algorithms to model the sentiment relationships between different modalities represented by these vectors. During modeling, the BRNN treats the multimodal sentiment relationship vectors as nodes and edges in a graph, aggregating them through graph convolution. This BRNN effectively aggregates multimodal sentiment relationship vectors, uncovering hidden relationships and patterns between different sentiment modalities.

[0091] Subsequently, the aggregation results are combined with the already constructed event graph, which contains rich event information and relationships between events. By integrating the aggregated multimodal sentiment relationship vectors into the event graph, the event graph not only contains information about the events themselves, but also incorporates the sentiment information reflected by the multimodal data, thus obtaining sentiment feature vectors.

[0092] In the actual analysis of urban safety incidents, the event map records information such as the time, location, and relevant personnel of the incident, while the multimodal sentiment vector reflects the public's emotional attitude towards the incident. Combining the two, the sentiment feature vector can more comprehensively reflect the incident and the emotional state related to the incident, and gain a deeper understanding of the various situations of incidents in the city. This helps to identify potential social problems and risks in advance, and provides a more comprehensive and valuable reference for urban governance and decision-making.

[0093] S106. Construct positive and negative sample pairs based on the emotional feature vector and multimodal data. Calculate the contrastive learning loss value using a preset contrast function based on the positive and negative sample pairs. Align and optimize the emotional feature vector and multimodal data using the contrastive learning loss value to obtain the target emotional feature.

[0094] Constructing positive and negative sample pairs allows us to learn the differences between different emotion categories and the similarity of multimodal data within the same emotion category. First, multimodal data within the same emotion category are combined to form positive sample pairs. For example, in urban safety incidents, all text, image, and audio data expressing the emotion of "worry" are paired together to form positive sample pairs. Then, multimodal data from different emotion categories are combined to form negative sample pairs. For instance, multimodal data expressing the emotion of "anger" is combined with multimodal data expressing the emotion of "calm" to form negative sample pairs.

[0095] After constructing positive and negative sample pairs, a pre-defined contrastive function is used to calculate the contrastive learning loss value, which quantifies the degree of difference between different sample pairs. Then, based on the contrastive learning loss value, the sentiment feature vector is aligned and optimized with the multimodal data. During the optimization process, parameters are continuously adjusted to make multimodal features of the same sentiment category closer together in the feature space, and multimodal features of different sentiment categories further apart, ultimately yielding the target sentiment feature. The obtained target sentiment feature accurately reflects the actual sentiment in the multimodal data, improving the accuracy of sentiment analysis and providing more reliable results for analyzing public sentiment attitudes in urban safety incidents.

[0096] Specifically, the contrastive learning loss value is represented by the following formula:

[0097] Among them, L contrastive Indicates the comparison loss value, This represents the sentiment feature vector of a positive sample pair. Let represent the sentiment feature vector of the negative sample pair, τ represent an adjustable parameter, and N represent the amount of multimodal data. This represents the similarity between the sentiment feature vectors of positive and negative sample pairs, where k represents the traversal index variable.

[0098] S107. Transmit the sentiment feature vector to the nodes of the event graph, adjust the attention weight of the sentiment feature vector using a cross-modal attention mechanism, and generate the target event graph.

[0099] In this embodiment, the obtained emotional feature vectors are transmitted to each node of the event graph. The nodes of the event graph represent different event elements, such as subject, time, location, behavior, etc. The emotional feature vectors carry the emotional information contained in the multimodal data. After the emotional feature vectors are integrated into the nodes, the information dimensions of the event graph are enriched, expanding from a single event structure description to a comprehensive representation that includes emotional information.

[0100] Subsequently, the cross-modal attention mechanism dynamically allocates attention weights based on the importance of different modalities in event understanding and sentiment expression. For example, when the information conveyed by the image is more important for sentiment judgment, the cross-modal attention mechanism will correspondingly increase the weight of the sentiment feature vector corresponding to the image features; when the text is more important, it will correspondingly increase the weight of the sentiment feature vector corresponding to the text features. Through this dynamic adjustment, the emotional information of key modalities can be flexibly highlighted according to the characteristics of different events, so that each node in the event graph can fuse multimodal sentiment information with the most appropriate weights, ultimately generating the target event graph.

[0101] S108. Use a preset graph neural network to extract relational features from the target event graph, combine attention weights to obtain node feature sequences, and input the node feature sequences into a preset time series model to capture dynamic evolution patterns. Based on the dynamic evolution patterns, obtain the event prediction graph.

[0102] In this embodiment, the preset graph neural network first traverses and calculates the nodes and edges in the target event graph, identifies and extracts the causal relationships, temporal relationships and semantic associations between events, and then integrates the node features by combining the attention weights adjusted by the cross-modal attention mechanism to obtain the node feature sequence.

[0103] The obtained node feature sequences are then input into a pre-defined time series model. This model analyzes and models the data as it changes over time, capturing the dynamic evolution of events over time through the processing of the node feature sequences. Finally, based on these captured dynamic evolution patterns, an event prediction map is generated. This event prediction map not only contains information about the current event but also predicts and presents potential future events based on historical data and dynamic patterns. This allows for advance understanding of the possible development and changes of events, preventing potential crises.

[0104] S109. Extract the causal chain from the event prediction graph, and perform causal relationship reasoning based on the node feature sequence and causal chain to obtain the triggering conditions and emotional factors.

[0105] Event prediction graphs contain rich event information and their predictive relationships. After identifying the causal connections between events in the event prediction graph, causal chains are extracted. These causal chains clearly present the causal logic between events. Causal reasoning is then performed based on node feature sequences and causal chains. During the reasoning process, the event attributes and sentiment information contained in the node feature sequences, as well as the logical relationships presented by the causal chains, can be comprehensively considered.

[0106] Through causal reasoning, we ultimately arrive at the triggering conditions and emotional factors. The triggering conditions clarify the preconditions and key factors that lead to an event, while the emotional factors reveal how emotions drive the evolution of the event and the impact of different emotions on the outcome.

[0107] Specifically, causal chains are extracted from the event prediction graph, and causal relationship inference is performed based on node feature sequences and causal chains. After inference, triggering conditions and emotional factors are obtained, which are represented by the following formula:

[0108] Among them, v t Indicates the triggering condition, e t Indicating emotional factors, v t+k This represents the triggering condition corresponding to time step k. W represents the node feature sequence. p e represents the parameters of the weight matrix. t (t) represents a causal chain.

[0109] S110. Match the event prediction map with the target event map, and calculate the event analysis results by combining the node feature sequence, triggering conditions and emotional factors.

[0110] In this embodiment, the process of matching the event prediction graph with the target event graph involves comparing the nodes, edges, and attribute information of the two graphs to identify similarities and differences for matching and fusion. Subsequently, a comprehensive analysis and calculation are performed, incorporating node feature sequences, triggering conditions, and emotional factors. This calculation fully utilizes diverse information resources, resulting in a more comprehensive, in-depth, and accurate analysis. The node feature sequences contain detailed event characteristics and emotional information, the triggering conditions clarify the prerequisites for the event's occurrence, and the emotional factors encompass the impact of emotions on the event.

[0111] After the calculations are completed, the event analysis results provide detailed decision-making basis for urban governance departments, enabling them to clearly understand the development dynamics of the event, potential risks, and public sentiment, which facilitates the formulation of more scientific, reasonable, and effective response strategies, improves the efficiency and effectiveness of urban governance, and better ensures the safety and stable development of the city.

[0112] This event analysis method, based on generative tasks and multimodal approaches, overcomes the limitations of existing deep learning-based feature fusion techniques that only fuse basic features from various modalities by utilizing multimodal data, large-scale model generation tasks, and cross-modal attention mechanisms. This results in more accurate urban event analysis. First, text, image, audio, and video features are extracted from multimodal data, and weights are dynamically allocated using a cross-modal attention mechanism to form multimodal sentiment relationship vectors. This allows for a more comprehensive and in-depth exploration of potential correlations between data, enhancing the ability to capture event features. Then, structured event information is generated and an event graph is constructed. Combined with a pre-defined graph neural network, the multimodal sentiment relationship vectors are aggregated to generate sentiment feature vectors. This tightly integrates the emotional information of the event with its structure and relationships, enabling a more accurate grasp of event trends and emotional dynamics.

[0113] After generating sentiment feature vectors, the contrastive learning loss value calculated using a preset contrastive loss function is used to optimize the sentiment feature vectors and multimodal data. Then, the sentiment feature vectors are input as node attributes into the event graph. A cross-modal attention mechanism is used to track the development and changes of the event in real time, dynamically adjusting the weights of the sentiment feature vectors to generate a target event graph. Subsequently, a preset graph neural network is used to process the target event graph, extracting the relational features of the nodes. During the extraction process, the graph neural network can deeply explore the potential connections between nodes, providing reliable relational information for subsequent analysis. After extracting the relational features, the node feature sequence is obtained by combining the attention weights and inputting the node feature sequence into a preset time series model. Upon receiving the node feature sequence, the time series model iteratively calculates to capture the dynamic evolution of the event over time, generating an event prediction graph based on the dynamic evolution pattern, thereby predicting the future development trend of the event. Finally, the event prediction graph is fused with the target event graph, and the event analysis results are obtained by combining the node feature sequence. The event analysis results obtained in this embodiment take into account both the current state and future trends of the event. When faced with complex and ever-changing urban events, it can provide a more comprehensive and accurate basis for decision-making, which helps to improve the scientific nature and timeliness of urban governance decisions, avoid missing the best opportunity to deal with emergency events, and thus effectively control the scope of the event's impact.

[0114] Please refer to Figure 2, which shows another embodiment of an event analysis method based on generative tasks and multimodality provided in this application. This embodiment includes:

[0115] S201. Acquire multimodal data, which includes text data, image data, and audio data;

[0116] In this embodiment, step S201 is similar to step S101 in the previous embodiment, and will not be described again here.

[0117] S202. Based on the preset large-scale model, set the generation task, input multimodal data into the generation task for training, and obtain a structured description;

[0118] When setting up a generation task based on a pre-defined large-scale generation model, the goal of the generation task is first clarified, input prompts and output templates are designed according to the needs of urban situation monitoring and the characteristics of multimodal data. Then, the acquired multimodal data is preprocessed, including noise removal, filtering of irrelevant information and handling of missing values. After preprocessing, the input prompts and multimodal data are input into the generation task, and the pre-defined large-scale generation model in the generation task analyzes and trains the input prompts and multimodal data.

[0119] During training, the pre-defined large-scale generative model performs semantic understanding and analysis on text data to extract key information; it performs feature recognition on image data to identify objects and human actions in the scene; and it performs content parsing on audio data to determine the meaning of the sounds. After training, a structured description is obtained according to the output template. For example, in an urban fire incident, the pre-defined large-scale generative model in the generation task learns and trains on text, fire scene images, and rescue audio to generate a structured description that includes the time and location of the fire, the cause of the fire, the development of the fire, and rescue operations. This structured description has higher accuracy and completeness and is more comprehensive than the description generated from single-modal data.

[0120] Specifically, the input prompts are designed to help the pre-defined large-scale model more accurately understand the goal of the generation task and generate structured event information. Examples of input prompts are as follows:

[0121] "Please extract event information from the following text and generate structured event information, including event type, time, location, participants, victims, outcome, and causal relationship."

[0122] The output template for the generated task needs to explicitly define the event type and elements, which can be described using JSON structure or natural language:

[0123] The following are examples of natural language descriptions:

[0124] "On October 10, 2023, a fire broke out in Huangpu District, Shanghai. The fire started due to a short circuit in the electrical wiring. Firefighters rushed to the scene to carry out the rescue."

[0125] S203. Under the current preset parameters of the large model, combine multimodal data and corresponding structured descriptions into sample pairs, and calculate the target probability of the structured event information generated from the input multimodal data based on the sample pairs.

[0126] Under the preset parameters of the current large-scale generative model, each set of acquired multimodal data is paired with a structured description to form a sample pair. Then, based on the sample pairs, the target probability of the structured description generated from the input multimodal data is calculated. During the calculation of the target probability, the large-scale generative model processes and analyzes the multimodal data according to the preset parameters. For example, it performs word vector conversion on text, feature extraction and matching on images, and feature analysis on audio. Then, based on the sample pairs, it calculates the percentage of structured event information derived from the processed multimodal data. The calculated percentage is the target probability of generating structured event information from the multimodal data.

[0127] The target probability can reflect the understanding and generation capabilities of the pre-defined generative model for different multimodal data, making it easier to identify the strengths and weaknesses of the pre-defined generative model, and thus improve and optimize it in a targeted manner.

[0128] S204. Based on the target probability, the preset parameters are adjusted in conjunction with the cross-entropy loss function. After adjustment, structured event information is generated, and relation extraction and semantic analysis are performed on the structured event information to obtain event relations.

[0129] In this embodiment, preset parameters are adjusted based on the calculated target probability and the cross-entropy loss function. The cross-entropy loss function is mainly used to measure the difference between the model's prediction and the actual result. First, the calculated target probability is obtained, and the loss value for generating structured event information is calculated using the cross-entropy loss function. Based on the magnitude of the loss value, the preset parameters of the preset generation model are adjusted through backpropagation using an optimization algorithm. The optimization algorithm can use stochastic gradient descent. By adjusting the preset parameters in conjunction with the cross-entropy loss function, the preset generation model can be continuously optimized, making the structured event information generated by the preset generation model more accurate and reliable.

[0130] Specifically, based on the target probability, the preset parameters are adjusted using the cross-entropy loss function. After adjustment, structured event information is generated, as expressed by the following formula:

[0131] Where L(θ) represents structured event information, N represents the amount of multimodal data, and P(Y) represents the number of data points. i |X i ;θ) represents the target probability, θ represents the preset parameter, X i Y represents the multimodal data of the i-th input;i This represents the structured description of the i-th output.

[0132] During the adjustment process, the preset large-scale generation model updates its parameters in the direction of reducing the loss value, making the structured event information generated by the preset large-scale generation model in the generation task closer to the real labels. After the adjustment is completed, structured event information is generated. Subsequently, the relationships between different elements in the structured event information are extracted, and semantic analysis is performed to deeply understand the meaning of the event information, finally obtaining the event relationships.

[0133] S205. Construct an event graph, using structured event information as nodes in the event graph, and connect the nodes according to event relationships;

[0134] In this embodiment, step S205 is similar to step S103 in the previous embodiment, and will not be described again here.

[0135] S206. Use a pre-defined language model to extract text semantics and text sentiment from multimodal data, and obtain text features based on text semantics and text sentiment;

[0136] In this embodiment, the text data from the multimodal data is first input into a pre-defined language model. The language model performs lexical analysis on the text data, breaking it down into individual words or phrases and tagging each word with its part of speech. Then, it performs syntactic analysis, parsing the grammatical structure of the sentences and determining the relationships between sentence components such as subject, verb, and object. Through lexical and syntactic analysis, the pre-defined language model further understands the key information and thematic content in the text data, thereby extracting the text semantics.

[0137] While extracting the semantics of the text, the sentiment analysis module in the pre-set language model is used to judge the sentiment tendency of the text data, and to determine whether the text data expresses positive, negative or neutral sentiment. For example, it can be judged whether the text is a praise for urban safety measures or a dissatisfaction with urban safety incidents. After the judgment is completed, the judgment result is extracted as the text sentiment.

[0138] The extracted text semantics and text sentiment are encoded and transformed into numerical text features. For example, the text semantics are mapped to a point in a high-dimensional vector space, and the text sentiment is represented by specific numerical values ​​and incorporated into the vector. Finally, the text features are obtained.

[0139] This embodiment extracts text semantics and sentiment from a pre-defined language model to obtain text features, enabling in-depth mining of the rich information contained in the text portion of multimodal data. Extracting text semantics helps to accurately understand people's descriptions and opinions of urban safety incidents, grasping the key points and core content of the events; text sentiment allows urban governance departments to understand the public's attitudes towards the events.

[0140] S207. Use ResNet or Vision Transformer to extract image scenes and image expressions from multimodal data, and obtain image features based on image scenes and image expressions;

[0141] In this embodiment, ResNet or Vision Transformer is used to extract image scenes and facial expressions from multimodal data. This effectively extracts key information from the image portion of multimodal data and obtains representative image features. The extraction of image scenes can intuitively show the environment and background of urban safety incidents, enabling a quick understanding of the on-site situation. The extraction of facial expressions can reflect people's emotional state during the incident, providing important clues for understanding the public's reaction to the incident.

[0142] When using ResNet for extraction, image data in multimodal data requires processing such as image resizing and pixel value normalization to meet the input requirements of the ResNet model. The processed image data is then fed into the ResNet model, where convolutional and pooling layers extract image scenes and facial expressions. Finally, the extracted scene and expression data are converted into image features. When using Vision Transformer for extraction, the image can be directly segmented into multiple small blocks, each mapped into a vector sequence and input into the Vision Transformer model. The Vision Transformer's self-attention mechanism processes these vector sequences, capturing the relationships between different blocks to identify image scenes and facial expressions. Finally, the extracted scene and expression data are converted into image features. Converting image scene and expression information into image features facilitates fusion with features from text and audio data in multimodal data, enabling a more comprehensive analysis of urban events.

[0143] During the extraction process using ResNet or Vision Transformer, the ResNet or Vision Transformer models capture detailed information in the image data, such as buildings, roads, and people, and then identify the scene depicted in the image data. By analyzing the facial features of people, they can determine the expressions of people in the image, such as fear, surprise, and calmness.

[0144] S208. Use the Wav2Vec audio model to extract audio intonation, audio volume, and audio rhythm from multimodal data, and obtain audio features based on audio intonation, audio volume, and audio rhythm.

[0145] In this embodiment, the audio data in the multimodal data is first preprocessed. Preprocessing includes removing noise interference, improving audio clarity and quality, and ensuring the audio data format meets the input requirements of the Wav2Vec audio model. The preprocessed audio data is then input into the Wav2Vec audio model, which performs frame-by-frame analysis. The model extracts audio intonation through convolutional, pooling, and fully connected layers, measures the volume to understand the intensity of the sound, and analyzes the rhythmic characteristics to obtain the audio rhythm. Audio intonation conveys the speaker's emotional state and attitude, volume reflects the urgency or importance of an event, and audio rhythm represents the dynamic changes in the development of an event.

[0146] After extraction, the extracted audio intonation, volume, and rhythm are encoded into numerical audio features. For example, for audio from urban safety incidents, the Wav2Vec audio model can extract the intonation of shouts, the volume of alarms, and the rhythm of rescue operations, which can then be converted into audio features.

[0147] This embodiment uses the Wav2Vec audio model to extract audio features, which can not only provide a deeper understanding of urban events from an audio perspective, but also supplement information that text and images cannot fully express. This helps urban management departments to more accurately judge the atmosphere at the scene of an event, people's emotional fluctuations, and the development trend of the event, providing strong support for the formulation of more effective response strategies.

[0148] S209. Calculate the target weights of text features, image features, and audio features using a cross-modal attention mechanism. Combine the target weights with the text features, image features, and audio features to obtain a multimodal sentiment relationship vector.

[0149] In this embodiment, the cross-modal attention mechanism is used to calculate the target weights of text features, image features, and audio features. This can effectively uncover the intrinsic relationships between different modal data and dynamically allocate the proportion of different modal features in the overall information, highlighting key information and avoiding excessive interference to the overall analysis due to incomplete or inaccurate information in a certain modality.

[0150] First, the extracted text, image, and audio features are input into a cross-modal attention mechanism model. This model compares and analyzes the features from these three different modalities. For text and image features, the model searches for correlations, such as whether scene elements mentioned in the text are reflected in the image, to measure the relevance between them. For text and audio features, the model analyzes the relationship between the sentiment expressed in the text and the tone and volume of the audio. For image and audio features, the model explores whether the image scene in the image features matches the audio rhythm and sonic atmosphere. Through these comparative analyses, the model calculates the target weight of each modal feature in the overall information representation. Finally, the target weights are weighted and fused with the corresponding text, image, and audio features to obtain a multimodal sentiment relation vector.

[0151] Multimodal sentiment relation vectors fully combine the feature advantages of text, image, and audio modalities, no longer limited to information from a single modality. They contain more comprehensive and richer sentiment and relational information, which can more accurately reflect the true situation and inherent emotional tendencies of events. This helps urban governance departments to analyze events in depth from multiple dimensions, better grasp the public's emotional attitudes and the potential relationships between events, and thus formulate response measures that are more in line with actual needs, thereby further improving the decision-making quality and effectiveness of urban safety management.

[0152] S210. Aggregate multimodal sentiment relationship vectors based on a preset graph neural network and combine them with an event graph to obtain sentiment feature vectors;

[0153] S211. Construct positive and negative sample pairs based on the emotional feature vector and multimodal data. Calculate the contrastive learning loss value using a preset contrast function based on the positive and negative sample pairs. Align and optimize the emotional feature vector and multimodal data using the contrastive learning loss value to obtain the target emotional feature.

[0154] S212. Transmit the sentiment feature vector to the nodes of the event graph, adjust the attention weight of the sentiment feature vector using a cross-modal attention mechanism, and generate the target event graph.

[0155] S213. Use a preset graph neural network to extract relational features from the target event graph, combine attention weights to obtain node feature sequences, and input the node feature sequences into a preset time series model to capture dynamic evolution patterns. Based on the dynamic evolution patterns, obtain the event prediction graph.

[0156] S214. Extract the causal chain from the event prediction graph, and perform causal relationship reasoning based on the node feature sequence and causal chain to obtain the triggering conditions and emotional factors.

[0157] S215. Match the event prediction map with the target event map, and calculate the event analysis results by combining the node feature sequence, triggering conditions and emotional factors.

[0158] In this embodiment, steps S210 to S215 are similar to steps S105 to S110 in the previous embodiment, and will not be described again here.

[0159] S216. Convert the data structure of the event analysis results into the target data structure, and use a preset visualization tool to generate visualization analysis results from the converted event analysis results.

[0160] In this embodiment, the data structure of the event analysis results is first analyzed in detail to clarify the existing data organization, data elements, and relationships between them. Data transformation rules are designed based on the actual needs of the target data structure. For example, if the original data structure is hierarchical and the target data structure is a key-value pair structure for easy querying, the original data needs to be reorganized and mapped according to the key-value pair format. Subsequently, the data in the event analysis results is converted to the target data structure according to the data transformation rules. After the conversion, preset visualization tools are used, and appropriate visualization chart types are selected based on different dimensions and needs of the event analysis results. For example, bar charts are used to compare the frequency of different events, line charts to show event trends, and maps to visually present the distribution of event locations. Finally, visualized analysis results are generated.

[0161] By using pre-set visualization tools to generate visualization analysis results, the readability and understandability of event analysis results are improved. The visualized charts can present complex and abstract event analysis data in an intuitive graphical form, enabling quick understanding of key information, development trends, and relationships between different factors.

[0162] The event analysis system based on generation tasks and multimodality provided in this application will be described in detail below. Please refer to Figure 3, which shows another embodiment of the event analysis system based on generation tasks and multimodality provided in this application. The system includes:

[0163] The acquisition unit 301 is used to acquire multimodal data, which includes text data, image data, and audio data.

[0164] The first generation unit 302 is used to set generation tasks based on a preset generation model, input multimodal data into the generation task, generate structured event information, and perform relation extraction and semantic analysis on the structured event information to obtain event relationships.

[0165] The construction unit 303 is used to construct an event graph, using structured event information as nodes in the event graph and connecting the nodes according to event relationships;

[0166] The fusion unit 304 is used to extract text features, image features and audio features from multimodal data, and uses a cross-modal attention mechanism to fuse text features, image features and audio features to obtain a multimodal sentiment relationship vector.

[0167] Aggregation unit 305 is used to aggregate multimodal sentiment relationship vectors based on a preset graph neural network and combine them with an event graph to obtain sentiment feature vectors;

[0168] The first computing unit 306 is used to construct positive sample pairs and negative sample pairs based on the emotional feature vector and multimodal data, calculate the contrast learning loss value using a preset contrast function based on the positive sample pairs and negative sample pairs, and optimize the alignment of the emotional feature vector and multimodal data through the contrast learning loss value to obtain the target emotional feature.

[0169] The second generation unit 307 is used to transmit the sentiment feature vector to the nodes of the event graph, and use a cross-modal attention mechanism to adjust the attention weight of the sentiment feature vector to generate the target event graph.

[0170] The capture unit 308 is used to extract relational features in the target event graph using a preset graph neural network, combine attention weights to obtain node feature sequences, and input the node feature sequences into a preset time series model to capture dynamic evolution patterns, and obtain an event prediction graph based on the dynamic evolution patterns.

[0171] The reasoning unit 309 is used to extract causal chains from the event prediction graph, and to perform causal relationship reasoning based on the node feature sequence and causal chains. After reasoning, the triggering conditions and emotional factors are obtained.

[0172] The second calculation unit 310 is used to match the event prediction map with the target event map, and calculate the event analysis results by combining the node feature sequence, triggering conditions and emotional factors.

[0173] Optionally, the fusion unit 304 is specifically used for:

[0174] The text semantics and sentiment in multimodal data are extracted using a pre-defined language model, and text features are obtained based on the text semantics and sentiment.

[0175] Use ResNet or Vision Transformer to extract image scenes and facial expressions from multimodal data, and obtain image features based on image scenes and facial expressions;

[0176] The Wav2Vec audio model is used to extract audio intonation, volume, and rhythm from multimodal data, and audio features are obtained based on audio intonation, volume, and rhythm.

[0177] The target weights of text features, image features, and audio features are calculated using a cross-modal attention mechanism. The target weights are then combined with the text features, image features, and audio features to obtain a multimodal sentiment relationship vector.

[0178] Optionally, the first generating unit 302 is specifically used for:

[0179] Based on a pre-defined large-scale model, a generation task is set, and multimodal data is input into the generation task for training to obtain a structured description.

[0180] Under the current preset parameters for generating large models, multimodal data and corresponding structured descriptions are combined into sample pairs, and the target probability of the structured event information generated from the input multimodal data is calculated based on the sample pairs.

[0181] Based on the target probability, the preset parameters are adjusted using the cross-entropy loss function, and structured event information is generated after the adjustment.

[0182] Optionally, based on the target probability and in conjunction with the cross-entropy loss function, preset parameters are adjusted to generate structured event information, expressed by the following formula:

[0183] Where L(θ) represents structured event information, N represents the amount of multimodal data, and P(Y) represents the number of data points. i |X i ;θ) represents the target probability, θ represents the preset parameter, X i Y represents the multimodal data of the i-th input; i This represents the structured description of the i-th output.

[0184] Optionally, a third generation unit 311 is also included, specifically for:

[0185] The data structure of the event analysis results is converted into the target data structure, and the converted event analysis results are generated into visual analysis results using a preset visualization tool.

[0186] Optionally, causal chains are extracted from the event prediction graph, and causal relationship inference is performed based on the node feature sequence and causal chains. After inference, the triggering conditions and emotional factors are obtained, which are represented by the following formula:

[0187] Among them, v t Indicates the triggering condition, e t Indicating emotional factors, v t+k This represents the triggering condition corresponding to time step k. W represents the node feature sequence. p e represents the parameters of the weight matrix. t (t) represents a causal chain.

[0188] Optionally, the contrastive learning loss value can be represented by the following formula:

[0189] Among them, L contrastive Indicates the comparison loss value, This represents the sentiment feature vector of a positive sample pair. Let represent the sentiment feature vector of the negative sample pair, τ represent an adjustable parameter, and N represent the amount of multimodal data. This represents the similarity between the sentiment feature vectors of positive and negative sample pairs, where k represents the traversal index variable.

[0190] This application also provides an event analysis apparatus based on generation tasks and multimodality. Please refer to Figure 4, which shows an embodiment of the event analysis apparatus based on generation tasks and multimodality provided in this application. The apparatus includes:

[0191] Processor 401, memory 402, input / output unit 403, bus 404;

[0192] The processor 401 is connected to the memory 402, the input / output unit 403, and the bus 404;

[0193] The memory 402 stores a program, and the processor 401 calls the program to execute any of the methods described above.

[0194] This application also relates to a computer-readable storage medium on which a program is stored, characterized in that, when the program is run on a computer, it causes the computer to perform any of the methods described above.

[0195] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0196] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0197] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0198] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0199] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An event analysis method based on generative tasks and multimodal approaches, characterized in that, The method includes: Acquire multimodal data, which includes text data, image data, and audio data; Based on a preset large-scale model, a generation task is set, the multimodal data is input into the generation task, structured event information is generated, and relation extraction and semantic analysis are performed on the structured event information to obtain event relations. Construct an event graph, using the structured event information as nodes in the event graph, and connect the nodes according to the event relationships; Text features, image features, and audio features of the multimodal data are extracted, and a cross-modal attention mechanism is used to fuse the text features, image features, and audio features to obtain a multimodal sentiment relationship vector; The multimodal sentiment relationship vectors are aggregated based on a preset graph neural network, and sentiment feature vectors are obtained by combining them with the event graph. Positive and negative sample pairs are constructed based on the emotional feature vector and the multimodal data. Based on the positive and negative sample pairs, a contrastive learning loss value is calculated using a preset contrastive function. The contrastive learning loss value is then used to align and optimize the emotional feature vector and the multimodal data to obtain the target emotional feature. The emotional feature vector is transmitted to the node of the event graph, and the attention weight of the emotional feature vector is adjusted using the cross-modal attention mechanism to generate the target event graph; The preset graph neural network is used to extract relational features from the target event graph, and the attention weights are combined to obtain a node feature sequence. The node feature sequence is then input into a preset time series model to capture dynamic evolution patterns, and an event prediction graph is obtained based on the dynamic evolution patterns. Extract the causal chain from the event prediction graph, and perform causal relationship reasoning based on the node feature sequence and the causal chain to obtain the triggering conditions and emotional factors. The event prediction map is matched with the target event map, and the event analysis results are calculated by combining the node feature sequence, the triggering conditions, and the emotional factors.

2. The method according to claim 1, characterized in that, The step of extracting text features, image features, and audio features from the multimodal data, and then fusing the text features, image features, and audio features using a cross-modal attention mechanism to obtain a multimodal sentiment vector includes: The text semantics and text sentiment in the multimodal data are extracted using a pre-defined language model, and text features are obtained based on the text semantics and text sentiment. Image scenes and facial expressions are extracted from the multimodal data using ResNet or Vision Transformer, and image features are obtained based on the image scenes and facial expressions. The Wav2Vec audio model is used to extract audio intonation, audio volume, and audio rhythm from the multimodal data, and audio features are obtained based on the audio intonation, audio volume, and audio rhythm. The target weights of the text features, image features, and audio features are calculated using a cross-modal attention mechanism. The target weights are then combined with the text features, image features, and audio features to obtain a multimodal sentiment relation vector.

3. The method according to claim 1, characterized in that, The process of setting a generation task based on a preset large model, inputting the multimodal data into the generation task, and generating structured event information includes: Based on a preset large-scale model, a generation task is set, and the multimodal data is input into the generation task for training to obtain a structured description. Under the preset parameters of the current preset large model, the multimodal data and the corresponding structured description are combined into sample pairs, and the target probability of the structured event information generated by the input multimodal data is calculated based on the sample pairs. Based on the target probability, the preset parameters are adjusted using the cross-entropy loss function, and structured event information is generated after the adjustment.

4. The method according to claim 3, characterized in that, The preset parameters are adjusted based on the target probability and the cross-entropy loss function. The adjusted parameters generate structured event information, which is expressed by the following formula: Where L(θ) represents structured event information, N represents the number of multimodal data, and P(Y) represents the number of data points. i |X i ;θ) represents the target probability, θ represents the preset parameter, X i Y represents the multimodal data of the i-th input; i This represents the structured description of the i-th output.

5. The method according to claim 1, characterized in that, After matching the event prediction map with the target event map, and combining the node feature sequence, the triggering conditions, and the emotional factors to calculate the event analysis result, the method further includes: The data structure of the event analysis results is converted into the target data structure, and a preset visualization tool is used to generate a visualization analysis result from the converted event analysis results.

6. The method according to claim 1, characterized in that, The process involves extracting causal chains from the event prediction graph, performing causal reasoning based on the node feature sequences and the causal chains, and obtaining triggering conditions and emotional factors, expressed by the following formula: Among them, v t Indicates the triggering condition, e t Indicates the emotional factor, v t+k This represents the triggering condition corresponding to time step k. W represents the node feature sequence. p e represents the parameters of the weight matrix. t (t) represents the causal chain.

7. The method according to any one of claims 1 to 6, characterized in that, The contrastive learning loss value is expressed by the following formula: Among them, L contrastive Indicates the comparison loss value, The emotional feature vector representing the positive sample pair. Let τ represent the sentiment feature vector of the negative sample pair, τ represent an adjustable parameter, and N represent the amount of multimodal data. The similarity between the sentiment feature vectors of the positive sample pair and the negative sample pair is represented by k, where k represents the traversal index variable.

8. An event analysis system based on generative tasks and multimodality, characterized in that, The system includes: An acquisition unit is used to acquire multimodal data, which includes text data, image data, and audio data; The first generation unit is used to set a generation task based on a preset generation model, input the multimodal data into the generation task, generate structured event information, and perform relation extraction and semantic analysis on the structured event information to obtain event relations. A construction unit is used to construct an event graph, using the structured event information as nodes in the event graph, and connecting the nodes according to the event relationships; The fusion unit is used to extract text features, image features, and audio features from the multimodal data, and to fuse the text features, image features, and audio features using a cross-modal attention mechanism to obtain a multimodal sentiment relationship vector. An aggregation unit is used to aggregate the multimodal emotional relationship vectors based on a preset graph neural network and combine them with the event graph to obtain an emotional feature vector. The first computing unit is used to construct positive sample pairs and negative sample pairs based on the emotional feature vector and the multimodal data, calculate the contrastive learning loss value using a preset contrast function based on the positive sample pairs and the negative sample pairs, and perform alignment optimization on the emotional feature vector and the multimodal data through the contrastive learning loss value to obtain the target emotional feature. The second generation unit is used to transmit the sentiment feature vector to the node of the event graph, and use the cross-modal attention mechanism to adjust the attention weight of the sentiment feature vector to generate the target event graph. The capture unit is used to extract relational features from the target event graph using the preset graph neural network, combine the attention weights to obtain a node feature sequence, input the node feature sequence into a preset time series model to capture dynamic evolution patterns, and obtain an event prediction graph based on the dynamic evolution patterns. The reasoning unit is used to extract the causal chain in the event prediction graph, perform causal relationship reasoning based on the node feature sequence and the causal chain, and obtain the triggering condition and emotional factors after reasoning. The second calculation unit is used to match the event prediction map with the target event map, and calculate the event analysis result by combining the node feature sequence, the triggering condition and the emotional factor.

9. An event analysis device based on generative tasks and multimodality, characterized in that, The device includes: Processor, memory, input / output units, and bus; The processor is connected to the memory, the input / output unit, and the bus; The memory stores a program, which the processor invokes to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, the program performing the method as claimed in any one of claims 1 to 7 when executed on a computer.

Images