Transformer Action Recognition With Temporal Similarity Matrices
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action recognition systems struggle to accurately identify new actions using a limited number of training videos, often requiring large datasets to achieve reliable performance.
Innovation Solution
A system utilizing a transformer architecture for action recognition, which generates similarity matrices based on comparisons between temporally ordered images of query and support videos, allowing for the recognition of new actions with as few as 1-5 support videos, and includes modules for vector representation, embedding, and positional encoding to enhance similarity determination.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If traditional action recognition systems use a limited number of training videos, then the training data requirement is reduced, but the recognition accuracy deteriorates
Solution Approach 1:
The patent introduces temporal similarity matrices as an intermediary representation between the limited training videos and the recognition task. These matrices capture temporal relationships and are processed by transformer architectures to enhance the information content from few-shot data, thereby maintaining high accuracy despite limited training samples
Solution Approach 2:
The system transforms the input video data into different parameter representations including temporal similarity matrices and vector representations. By changing the parameter space and using positional encodings, the system extracts more discriminative features from limited training videos, improving recognition accuracy without requiring more data
2Device complexity
If traditional parametric classifiers are used, then the system complexity is low, but the performance on new actions deteriorates
Solution Approach 1:
The patent replaces traditional parametric classifiers with transformer architectures that use attention mechanisms. This substitution enables the system to capture complex temporal dependencies and generalize better to new actions, significantly improving reliability on unseen actions despite increased computational complexity
3Measurement precision
If large datasets are used for training, then the recognition accuracy improves, but the data processing time increases
Solution Approach 1:
The patent extracts and focuses only on the most relevant temporal relationships by computing temporal similarity matrices that highlight key temporal patterns. This extraction approach allows the system to achieve high accuracy using only essential features from minimal training videos, avoiding the time cost of processing large datasets
Data Source
AI summary
A method includes: (i) receiving a query video including performance of an action; (ii) receiving a predetermined number of support videos including performance of actions, respectively, the predetermined number of support videos being less than 100 support videos; (iii) determining a similarity matrix based on a comparison of temporally ordered images of the query video with temporally ordered images of one of the support videos, respectively; (iv) determining a similarity value for the one of the support videos based on the similarity matrix; (v) repeating (iii) and (iv) for each of the support videos; (vi) identifying the highest one of the similarity values and the one of the support videos associated with the highest one of the similarity values; and (vii) setting a first indicator of the action in the query video to the same as a second indicator of the action performed in the one of the support videos.