Emotion-cause knowledge-enhanced multimodal aspect-sentiment pair extraction method and system
By using manual annotation and large language models to generate sentiment reasoning knowledge, the problems of modal semantic inconsistency and insufficient context in multimodal sentiment pair extraction are solved, and more accurate aspect-sentiment pair extraction is achieved.
Patent Information
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- INSTITUTE OF INFORMATION ENGINEERING CHINESE ACADEMY OF SCIENCES
- Filing Date
- 2025-01-17
- Publication Date
- 2026-07-09
Smart Images
Figure CN2025072919_09072026_PF_FP_ABST
Abstract
Description
A Multimodal Aspect of Emotional Cause Knowledge Enhancement: A Method and System for Emotion Pair Extraction Technical Field
[0001] This invention relates to the field of natural language processing technology, specifically to a multimodal aspect of emotion reasoning knowledge enhancement—a method and system for emotion pair extraction. Background Technology
[0002] In recent years, the widespread adoption of social media has led people to express their opinions and attitudes no longer solely through text, often supplementing them with other forms of content such as images, audio, and video. As internet users generate a large amount of unstructured content, including images and text, on social media, researchers in the field of fine-grained sentiment analysis have paid extensive attention to the Multimodal Aspect-Sentiment Pair Extraction (MASPE) task. In content combining text and images, textual information possesses inherent characteristics such as concise content and informal writing style. These unique characteristics pose challenges to traditional aspect-sentiment pair joint extraction. To fully utilize multimodal features and improve the performance of aspect-sentiment pair joint extraction tasks, numerous studies have attempted to implicitly align text and image features using strategies such as cross-attention mechanisms and their various variants, or maximizing intermodal mutual information. However, this series of methods focusing on image-text matching patterns (I+T) has two limitations: First, the feature distributions of different modalities are biased, making it difficult for the model to learn intermodal aligned representations when faced with multiple modalities. Second, the image feature extractors used in existing methods are trained on large-scale datasets such as ImageNet and COCO. The labels in these datasets mainly consist of task-specific nouns rather than aspect targets relevant to sentiment analysis tasks, resulting in a significant bias between the two types of datasets. Given these limitations caused by these two biases, some multimodal fusion methods may be less effective than the current best language models that focus solely on text.
[0003] Essentially, the MASPE task is a multimodal task, where text and images contribute unequally to the task. When images fail to provide more interpretable information for understanding text semantics, image information can even be discarded or ignored. Furthermore, research has demonstrated that introducing document-level context on top of the original text can significantly improve the performance of named entity recognition tasks. Therefore, some researchers have attempted to solve multimodal named entity recognition tasks using text-to-text matching (T+T) patterns. In these methods, images are converted into text representations, which can be achieved using techniques such as image captioning and handwritten digit recognition. Clearly, due to the absence of feature distribution differences, modeling inter-text attention mechanisms is superior to modeling cross-modal attention mechanisms. Considering that aspect and opinion item extraction in fine-grained sentiment analysis are similar to named entity recognition tasks, and aspect-opinion relationship classification is similar to entity relationship classification tasks, inspired by the aforementioned work, this invention attempts to explore the possibility of converting the existing multimodal aspect-sentiment pair extraction methods from the I+T pattern to the T+T pattern to avoid the problem of modal semantic inconsistency. However, existing text-to-text matching models still have two potential shortcomings in the field of sentiment analysis: (1) Methods that rely solely on in-sample information lack sufficient sentiment-related contextual information when the data content itself is relatively simple, and often require additional external knowledge to enhance the understanding of the text; (2) Methods that introduce external knowledge often retrieve relevant knowledge from external knowledge bases (such as Wikipedia) which is usually redundant. In some cases, the extended knowledge with low relevance may even mislead the model in understanding the text information.
[0004] Recently, the field of Large Language Models (LLMs) has been developing rapidly, with some interesting new discoveries and advancements. On the one hand, research on LLMs shows that generative models have significant limitations in sequence labeling tasks. On the other hand, LLMs have achieved surprising results in various natural language processing tasks and multimodal tasks. These LLMs, with their context learning capabilities, can be seen as encyclopedias of comprehensive world knowledge, often providing high-quality knowledge to aid in text content understanding. This leads to the hypothesis: could we activate the potential of LLMs in MASPE tasks by endowing them with reasonable heuristics? Summary of the Invention
[0005] To address the aforementioned technical problems, this invention proposes a multimodal aspect-sentiment pair extraction method and system with enhanced sentiment cause knowledge, in order to solve the problem that previous methods neglected the lack of sufficient contextual information in the Chinese text content of multimodal image and text data pairs.
[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:
[0007] A multimodal aspect-sentiment pair extraction method enhanced with knowledge of sentiment causes, the method comprising:
[0008] Generate sentiment reasoning knowledge for each image-text data pair; wherein, the image-text data pair includes: original text and original image;
[0009] With the aid of the aforementioned knowledge of emotional causes, the aspect-emotion pair of the image and text data pair is obtained.
[0010] Furthermore, the generation of sentiment reason knowledge for each image-text data pair includes:
[0011] Random samples are taken from the training set, and the selected image and text data samples are manually annotated with sentiment reason knowledge to construct a set of manually annotated examples. The content of each manually annotated example includes: the original text of the image and text data sample, the image and text description of the image and text data sample, the question and sentiment reason knowledge.
[0012] Based on the similarity between the image-text data pair and the sample of the image-text data pair, select K most similar manually labeled examples for the image-text data pair from the set of manually labeled examples;
[0013] Retrieve the original text and image description of the image-text data pair;
[0014] Construct a test example, the content of which includes: the original text of the image-text data pair, the image text description of the image-text data pair, and the question;
[0015] The test example and its K most similar manually labeled examples are embedded into the sentiment reason knowledge generation prompt template, and sentiment reason knowledge is generated for the text and image data pair based on the large language model.
[0016] Furthermore, based on the similarity between the image-text data pair and the sample image-text data pair, K most similar manually labeled examples are selected for the image-text data pair from the set of manually labeled examples, including:
[0017] Encode the image-text data pairs and each image-text data pair sample in the manually annotated example set, respectively;
[0018] Based on the encoding results, calculate the cosine similarity between the image-text data pair and the samples of each image-text data pair;
[0019] Based on the cosine similarity, the K most similar manually labeled examples of the image and text data pair are obtained.
[0020] Furthermore, the image text descriptions of the image-text data pairs are generated based on the visual-language pre-trained model BLIP-2.
[0021] Furthermore, with the assistance of the aforementioned knowledge of emotional causes, the aspect-emotion pair of the text-image data pair is obtained, including:
[0022] By combining the original text and the knowledge of the emotional reasons, a new text is obtained;
[0023] The new text is input into a Transformer-based encoder for encoding, resulting in an encoded vector;
[0024] The encoded vector is input into a conditional random field of a linear chain to predict the sequence label and obtain the label sequence probability.
[0025] Based on the probability of the label sequence, the aspect-sentiment pair of the image and text data pair is obtained.
[0026] Furthermore, the process of training the Transformer-based encoder and the conditional random field includes:
[0027] Calculate the probability of the label sequence for the image and text data of the sample;
[0028] The loss function is defined using negative log-likelihood, and the loss value is calculated based on the label sequence probability of the image and text data pair of samples and the true label sequence of the image and text data pair of samples.
[0029] Backpropagation is performed based on this loss value to update the parameters of the Transformer-based encoder and the conditional random field.
[0030] A multimodal aspect-emotion pair extraction system with enhanced knowledge of emotional causes, the system comprising:
[0031] The emotional reason knowledge generation module is used to generate emotional reason knowledge for each image-text data pair; wherein, the image-text data pair includes: original text and original image;
[0032] The aspect-emotion pair generation module is used to obtain the aspect-emotion pair of the image and text data pair with the assistance of the knowledge of the emotional causes.
[0033] An electronic device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the multimodal aspect-sentiment pair extraction method for enhancing sentiment cause knowledge as described above.
[0034] A computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the multimodal aspect-sentiment pair extraction method for enhancing sentiment cause knowledge as described above.
[0035] A computer program product, when run on a computer device, causes the computer device to perform the multimodal aspect-emotion pair extraction method with enhanced emotion cause knowledge as described above.
[0036] Compared with the prior art, the present invention has at least the following beneficial effects.
[0037] 1) This invention converts the image-text matching mode into the text-text matching mode, which can perform modality fusion without semantic differences.
[0038] 2) With the support of a large language model, this invention extracts language knowledge containing descriptions of image content and explanations of emotional reasons from image-text data pairs, providing richer contextual information for simple text content to assist in understanding the semantics of the original text, thereby improving the accuracy of the extraction task and demonstrating good practicality. Attached Figure Description
[0039] Figure 1 is a flowchart of the multimodal aspect-emotion pair extraction method for enhancing emotional cause knowledge provided in an embodiment of the present invention.
[0040] Figure 2 is a prompt template for generating sentiment reason knowledge in ChatGPT according to an embodiment of the present invention. Detailed Implementation
[0041] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings.
[0042] Figure 1 is a flowchart of the multimodal aspect-sentiment pair extraction method enhanced with sentiment cause knowledge. As shown in Figure 1, this method mainly includes two stages: heuristic generation of sentiment cause knowledge and aspect-sentiment pair extraction enhanced with sentiment cause knowledge. The entire process requires first randomly sampling a portion of samples from the training data for manual annotation, and then having the large language model annotate all data under the guidance of manual annotation prompts to obtain sentiment cause knowledge. After obtaining sentiment cause knowledge, it is used as auxiliary content and concatenated into the original input text. Then, the text encoder and sequence labeler are trained on the training data, and finally applied to the actual extraction.
[0043] (I) The stage of heuristic generation of knowledge about emotional causes.
[0044] In the heuristic generation stage of sentiment causal knowledge, this invention uses manually labeled examples to guide the generation of large language models, generating sentiment causal knowledge with multimodal semantics for each image-text data pair. In this embodiment, the image-text data pair includes a text segment and an image.
[0045] Step 1, Predefined Artificial Examples: This invention randomly samples a certain proportion (e.g., 1 / 21) of the image and text data from the training data and manually annotates the samples with sentiment reasoning knowledge, obtaining a set of artificial examples. The annotation process consists of two steps:
[0046] (1-1) Identify all target aspects mentioned in the text, including the aspects mentioned in the text and their corresponding sentiment polarities;
[0047] (1-2) Simultaneously consider the text content, image information and corresponding real labels to provide a comprehensive argument for judging each aspect and its corresponding emotional polarity.
[0048] This yields a predefined set of manually labeled examples, where each element consists of the text content, image information, and sentiment reasoning knowledge of the image data pairs.
[0049] Step 2, similarity-aware example selection: This invention first obtains the multimodal representations of each image-text data pair. Based on this, by measuring the similarity between the multimodal representations of the image-text data pairs, the K most similar examples are selected from the artificial example set for each image-text data pair to construct example demonstrations for large language model reasoning.
[0050] (2-1) Define the multimodal aspect-sentiment pair extraction model trained by predecessors as M. This model consists of a base encoder M. b and sequence labeler M c Therefore, for any multimodal image-text data pair input containing image I and text T, encoder M is used. b Encode: H=M b (T, I)
[0051] (2-2) Given a dataset D and a predefined set of manually labeled examples G, calculate the cosine similarity between each image-text data pair in dataset D and each image-text data pair in the set of manually labeled examples G, and select the K examples with the highest similarity from the set of manually labeled examples G. For the i-th image-text data pair in dataset D, calculate the set S of the K most similar examples. i :
[0052] Step 3, Heuristic Enhancement of Hint Generation: For each image-text data pair in dataset D, construct a hint for inference using the ChatGPT large language model. The hint includes a hint header, a set of contextual example demonstrations, and a test input. The hint header is natural language describing the task, and the examples in the contextual example demonstrations are constructed according to a specified template.
[0053] Text: T k Image: V k Question: Q, Answer: A k
[0054] Among them, T k It is a graph-text data pair with the text content in the input, V k It is a textual description of the image in the input, Q is the natural language used to ask questions to ChatGPT, and A is the textual description of the image. k It is a manual process for creating image-text data pairs (T) k V k The emotional reasoning knowledge is labeled. Based on this, the contextual example demonstration is obtained by combining the content of K examples in the example set S constructed according to the specified template above. The example template for the test input is:
[0055] Text: T, Image: V, Question: Q, Answer:
[0056] ChatGPT will generate new answers for the test input example, guided by examples presented in the context example, as the sentiment cause knowledge Z for that example.
[0057] (ii) Aspects of enhanced knowledge of emotional causes - Emotional pair extraction stage.
[0058] In the aspect of enhancing emotional cause knowledge—the emotional pair extraction stage—this invention uses the emotional cause knowledge from the previous stage to enrich the contextual information of the original input text content, thereby assisting in the extraction of the emotional pair.
[0059] Step 1, Construct Input: Concatenate the text content T and its corresponding sentiment reason knowledge Z in the image-text data pair to construct a new text content [T; Z], which serves as the model input for the aspect-sentiment pair extraction stage, where the text length of T is n and the text length of Z is m.
[0060] Step 2, Text Encoding: The text input [T; Z] is encoded using the Transformer-based text encoder XLM-RoBERTa to obtain the representation vector (h1, ..., h2). n , ..., h n+m );
[0061] Step 3, Label Prediction: Linear-chain Conditional Random Field (CRF) is used to predict sequence labels. Given text content T and corresponding sentiment reasoning knowledge Z, the probability calculation process for the label sequence of this text-image data pair is defined as follows:
[0062] Wherein, φ(y) i-1 y i h i ) and φ(y′ i-1 y′ i h i All of these are potential functions, and Y represents the set of all possible label sequences;
[0063] Step 4, Model Training: Define the loss function using negative log-likelihood:
[0064] Among them, y * θ represents the true label sequence, and θ represents the trainable parameters in the model.
[0065] Step 5, using the backpropagation algorithm, according to XLM-RoBERTa and CRF were trained.
[0066] In summary, to address the lack of contextual information in textual content during multimodal aspect-sentiment pair tasks, a method for extracting aspect-sentiment pairs enhanced with sentiment cause knowledge is proposed. This method extracts (aspect, sentiment) pairs mentioned in text-image data pairs. Specifically, it employs manual annotation to identify sentiment cause knowledge in a small amount of text-image data. Then, a similarity-aware example selection module selects relevant examples from the manually annotated example set for each sample in the dataset and integrates them into a prompt template to construct example samples for context learning. Subsequently, leveraging the small-sample context learning capability of a large language model, a heuristic method is used to extract auxiliary knowledge similar to the manually annotated content from the original text. Finally, with the assistance of sentiment cause knowledge, all mentioned aspect-sentiment pairs are extracted from the original text. Thus, this invention has two main advantages: first, it transforms the image-text matching mode into a text-text matching mode, enabling modal fusion without semantic differences; second, empowered by a large language model, it extracts linguistic knowledge containing image content descriptions and emotional explanations from image-text data pairs, providing richer contextual information for simple text content to aid in understanding the semantics of the original text, thereby improving the accuracy of the extraction task and demonstrating good practicality.
[0067] Another embodiment of the present invention provides a multimodal aspect-emotion pair extraction system with enhanced knowledge of emotional causes, comprising:
[0068] The emotional reason knowledge generation module is used to generate emotional reason knowledge for each image-text data pair; wherein, the image-text data pair includes: original text and original image;
[0069] The aspect-emotion pair generation module is used to obtain the aspect-emotion pair of the image and text data pair with the assistance of the knowledge of the emotional causes.
[0070] The specific implementation process of each module is described in the preceding section on the method of this invention. For example, the sentiment cause knowledge generation module first uses manual annotation to annotate a small number of image-text data pairs, including descriptions of the image content and brief explanations of the reasons related to the task. Then, through the similarity-aware example selection module, relevant examples are selected for each sample in the dataset from the manually annotated example set, and integrated into the prompt template to construct example samples for context learning. Afterwards, leveraging the small-sample context learning capability of the large language model, heuristic methods are used to extract auxiliary knowledge similar to the manually annotated content from the original text, thereby providing rich document-level information for the original text. Subsequently, the aspect-sentiment pair generation module concatenates the generated sentiment cause knowledge to the input text content as auxiliary information, constructs a new text input, encodes it using an XLM-RoBERTa encoder, and then uses a CRF sequence labeler for label prediction, finally inferring the (aspect, sentiment) pair.
[0071] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.
[0072] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.
[0073] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the concept of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A multimodal aspect-emotion pair extraction method for enhancing knowledge of emotional causes, characterized in that, The method includes: Generate sentiment reasoning knowledge for each image-text data pair; wherein, the image-text data pair includes: original text and original image; With the aid of the aforementioned knowledge of emotional causes, the aspect-emotion pair of the image and text data pair is obtained.
2. The method according to claim 1, characterized in that, The process of generating sentiment reason knowledge for each image-text data pair includes: Random samples are taken from the training set, and the selected image and text data samples are manually annotated with sentiment reason knowledge to construct a set of manually annotated examples. The content of each manually annotated example includes: the original text of the image and text data sample, the image and text description of the image and text data sample, the question and sentiment reason knowledge. Based on the similarity between the image-text data pair and the sample of the image-text data pair, select K most similar manually labeled examples for the image-text data pair from the set of manually labeled examples; Retrieve the original text and image description of the image-text data pair; Construct a test example, the content of which includes: the original text of the image-text data pair, the image text description of the image-text data pair, and the question; The test example and its K most similar manually labeled examples are embedded into the sentiment reason knowledge generation prompt template, and sentiment reason knowledge is generated for the text and image data pair based on the large language model.
3. The method according to claim 2, characterized in that, Based on the similarity between the image-text data pair and the sample images of the image-text data pair, K most similar manually labeled examples are selected for the image-text data pair from the set of manually labeled examples, including: Encode the image-text data pairs and each image-text data pair sample in the manually annotated example set, respectively; Based on the encoding results, calculate the cosine similarity between the image-text data pair and the samples of each image-text data pair; Based on the cosine similarity, the K most similar manually labeled examples of the image and text data pair are obtained.
4. The method according to claim 2, characterized in that, The image-text data pairs contain text descriptions generated based on the visual-language pre-trained model BLIP-2.
5. The method according to claim 1, characterized in that, With the assistance of the aforementioned knowledge of emotional causes, the aspect-emotion pair of the image-text data pair is obtained, including: By combining the original text and the knowledge of the emotional reasons, a new text is obtained; The new text is input into a Transformer-based encoder for encoding, resulting in an encoded vector; The encoded vector is input into a conditional random field of a linear chain to predict the sequence label and obtain the label sequence probability. Based on the probability of the label sequence, the aspect-sentiment pair of the image and text data pair is obtained.
6. The method according to claim 5, characterized in that, The process of training the Transformer-based encoder and the conditional random field includes: Calculate the probability of the label sequence for the image and text data of the sample; The loss function is defined using negative log-likelihood, and the loss value is calculated based on the label sequence probability of the image and text data pair of samples and the true label sequence of the image and text data pair of samples. Backpropagation is performed based on this loss value to update the parameters of the Transformer-based encoder and the conditional random field.
7. A multimodal aspect-emotion pair extraction system with enhanced knowledge of emotional causes, characterized in that, The system includes: The emotional reason knowledge generation module is used to generate emotional reason knowledge for each image-text data pair; wherein, the image-text data pair includes: original text and original image; The aspect-emotion pair generation module is used to obtain the aspect-emotion pair of the image and text data pair with the assistance of the knowledge of the emotional causes.
8. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the multimodal aspect-emotion pair extraction method for enhancing emotional cause knowledge as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the multimodal aspect-emotion pair extraction method for enhancing emotion cause knowledge as described in any one of claims 1-6.
10. A computer program product, characterized in that, When the computer program product is run on a computer device, the computer device performs the multimodal aspect-emotion pair extraction method with enhanced emotion cause knowledge as described in any one of claims 1-6.