Object-Augmented Action Recognition With Human-Object Relation Attention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action recognition systems fail to adequately consider the complex interactions between objects and humans in a scene, leading to the introduction of action-irrelevant information and compromised model performance due to a lack of relationship consideration between objects and actions.
Innovation Solution
An object-augmented action recognition system that processes initial patch tokens to extract human and object tokens, applies cross-attention mechanisms to construct Human-Object (HO) tokens with visual and spatial relationship information, and synthesizes these tokens into a single feature representation for accurate action recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If object information is introduced into action recognition models, then contextual understanding of actions is enriched, but action-irrelevant information may be introduced compromising model performance
Solution Approach 1:
The patent introduces Human-Object (HO) tokens as intermediary elements that mediate between object information and action recognition. These HO tokens selectively represent meaningful human-object interactions while filtering out irrelevant object information through cross-attention mechanisms, thus enriching contextual understanding without compromising model performance
Solution Approach 2:
The patent applies local quality by differentiating the treatment of different objects based on their relevance to human actions. Through cross-attention mechanisms, the model dynamically assigns different attention weights to different objects, focusing computational resources on locally relevant object-human interactions while ignoring irrelevant objects
2Measurement precision
If complex interactions between objects and humans are considered, then action recognition accuracy improves, but computational complexity increases
Solution Approach 1:
The patent segments the complex computation into distinct modules: initial patch token processing, human and object token extraction, HO token construction, and cross-attention mechanisms. This segmentation allows the model to handle complex interactions systematically while maintaining computational efficiency through specialized processing for each component
Solution Approach 2:
The patent employs dynamic cross-attention mechanisms that adaptively adjust computational focus based on the specific video content. The attention weights are dynamically computed to emphasize relevant human-object interactions while reducing computation on irrelevant elements, thus achieving high accuracy without constant maximum computational complexity
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method of object-augmented action recognition and an object-augmented action recognition system may be provided, the method comprises processing one or more initial patch tokens to extract one or more human tokens and one or more object tokens; applying the extracted one or more human tokens and the extracted one or more object tokens and the one or more initial patch tokens to a first cross attention mechanism to obtain a set of processed patch tokens containing object-guided information; enumerating to construct a set of Human-Object (HO) tokens, the constructed HO tokens each comprising information associated with visual relationship and information associated with spatial relationship between a human token and an object token; applying the constructed HO tokens and the processed patch tokens containing object-guided information to a second cross attention mechanism to obtain a set of resultant processed HO tokens; applying a feature aggregation process to the resultant processed HO tokens to synthesize the features of M groups of the resultant processed HO tokens into a single feature representation; obtaining a prediction result based on the set of processed patch tokens containing object-guided information; obtaining an action recognition prediction result based on a result of the feature aggregation process and the prediction result based on the set of processed patch tokens containing object-guided information; and outputting for use the action recognition prediction result.