Multimodal Sentiment Classification via Entity-Level Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional sentiment analysis methods are ineffective in accurately classifying entity sentiments in data types such as short-captioned image posts on social media, where sentiment often relies on contextual information in accompanying images.
Innovation Solution
A multimodal sentiment system utilizing multiple recurrent neural networks and an entity-oriented visual attention mechanism to process textual and visual contexts, generating accurate sentiment predictions by concatenating representations of the target entity, left and right contexts, and visual features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional textual-based sentiment analysis methods are used, then the analysis process is simple, but the accuracy is insufficient for multimodal posts with limited text
Solution Approach 1:
The patent combines multiple recurrent neural networks (RNNs) to simultaneously process textual context and visual features from images. The textual RNN processes caption text while the visual RNN processes image features, and their outputs are merged through concatenation and attention mechanisms to achieve accurate sentiment classification for multimodal posts, resolving the contradiction between accuracy and complexity by integrating multiple processing pathways.
Solution Approach 2:
The patent transitions from unimodal textual analysis to multimodal analysis by adding the visual dimension. It processes both textual data (captions) and visual data (images) simultaneously using separate RNNs, then integrates them through attention mechanisms. This dimensional expansion enables accurate sentiment analysis of social media posts that contain both text and images, overcoming the limitation of text-only approaches.
2Measurement precision
If textual information is sparse in social media posts, then the posts are more authentic and engaging, but sentiment analysis becomes more difficult
Solution Approach 1:
The patent introduces visual features from accompanying images as an intermediary source of sentiment information. When textual captions are sparse or insufficient, the visual RNN extracts sentiment-relevant features from images, and the attention mechanism dynamically weights this visual information to compensate for limited text, enabling accurate sentiment analysis even when textual information is minimal.
Solution Approach 2:
The system merges textual and visual information processing pathways, allowing the model to leverage both modalities simultaneously. When text is sparse, the visual pathway provides complementary sentiment signals that compensate for the information loss, maintaining high classification accuracy across posts with varying amounts of textual content.
Data Source
AI summary
Sentiment classification can be implemented by an entity-level multimodal sentiment classification neural network. The neural network can include left, right, and target entity subnetworks. The neural network can further include an image network that generates representation data that is combined and weighted with data output by the left, right, and target entity subnetworks to output a sentiment classification for an entity included in a network post.