Contextual LSTM for Image Activity Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Standard long short-term memory units use one-dimensional hidden states that do not preserve local information, which is crucial for action recognition tasks in image classification.
Innovation Solution
A convolutional neural network processes images to generate two-dimensional feature maps, which are then processed through a contextual long short-term memory unit to generate two-dimensional outputs and attention masks, preserving local information and enabling effective scene classification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard long short-term memory units with one-dimensional hidden states are used, then the model structure is simple and easy to implement, but local information is lost which reduces classification accuracy
Solution Approach 1:
The patent transforms the traditional one-dimensional hidden state into a two-dimensional feature map structure. The contextual LSTM unit processes spatial dimensions (height and width) explicitly, allowing the model to preserve local spatial information while maintaining temporal sequencing capabilities. This dimensional change enables the network to capture both local and global contextual information for improved action recognition accuracy.
2Measurement precision
If two-dimensional feature maps are processed through contextual LSTM units, then local information is preserved improving classification accuracy, but computational complexity increases
Solution Approach 1:
The patent segments the video data processing into distinct spatial and temporal components. The contextual LSTM unit first processes spatial segments (feature maps at each time step) to preserve local information, then aggregates temporal sequences. This segmentation allows efficient processing by handling spatial and temporal dimensions separately, reducing overall computational energy requirements while maintaining high classification accuracy.
3Measurement precision
If attention masks are generated based on two-dimensional outputs, then the model can focus on relevant regions improving action recognition, but the processing time increases
Solution Approach 1:
The patent generates attention masks as part of the forward propagation process through the contextual LSTM unit, rather than as a separate post-processing step. The attention mechanism is integrated into the core processing pipeline, allowing the model to compute attention weights simultaneously with feature extraction and temporal sequencing. This preliminary action reduces processing time by eliminating redundant computational steps.
Data Source
AI summary
An image including a visual capture of a scene may be accessed. The image may be processed through a convolutional neural network. The convolutional neural network may generate a set of two-dimensional feature maps based on the image. The set of two-dimensional feature maps may be processed through a contextual long short-term memory unit. The contextual long short-term memory unit may generate a set of two-dimensional outputs based on the set of two-dimensional feature maps. A set of attention-masks for the image may be generated based on the set of two-dimensional outputs and the set of two-dimensional feature maps. The set of attention-masks may define dimensional portions of the image. The scene may be classified based on the two-dimensional outputs.