Keypoint-Only Group Activity Recognition via Multiscale Transformer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Group Activity Recognition (GAR) methods face challenges in compositional understanding of crowded scenes, scene biases, and privacy concerns due to reliance on RGB modality, which affects model robustness and generalizability.
Innovation Solution
The COMPOSER architecture uses a Multiscale Transformer-based approach with keypoint-only modality, performing attention-based reasoning over tokens at multiple scales, clustering intermediate representations, and employing auxiliary predictions and data augmentations to maintain consistent cluster assignments and enhance model training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If RGB modality is used for Group Activity Recognition, then visual information richness is improved, but scene biases and privacy concerns increase
Solution Approach 1:
The patent extracts only the essential keypoint information from video frames, discarding the rest of the RGB image data. This extraction principle removes harmful scene biases and privacy concerns while retaining sufficient visual information for group activity recognition, directly resolving the contradiction between information richness and harmful factors.
Solution Approach 2:
The patent introduces keypoint data as an intermediary representation between RGB images and activity recognition. These keypoints serve as a mediator that captures essential motion and position information without exposing detailed visual content, thus reducing scene biases and privacy issues while maintaining recognition accuracy.
2Object-affected harmful factors
If keypoint-only modality is used, then privacy concerns and scene biases are reduced, but compositional understanding in crowded scenes becomes more difficult
Solution Approach 1:
The patent segments the group activity recognition task into multiple scales, from individual keypoints to person-level representations and finally to group-level compositions. This hierarchical segmentation makes compositional understanding tractable in crowded scenes by breaking down the complex reasoning into manageable stages, overcoming the difficulty while using only keypoint data.
Solution Approach 2:
The patent introduces a temporal dimension and multi-scale hierarchy to enhance keypoint-only representations. By incorporating time-based keypoint trajectories and organizing representations across multiple scales (keypoint, person, group), the system compensates for the limited spatial information in keypoint data, enabling effective compositional reasoning in crowded scenes.
3Adaptability or versatility
If multi-scale Transformer approach is used with keypoint data, then compositional reasoning capability is improved, but computational complexity increases
Solution Approach 1:
The patent segments the computational workload across multiple scales, where each scale processes a specific level of abstraction (keypoints, persons, groups). This segmentation allows the Transformer to focus computational resources on relevant relationships at each scale, improving compositional reasoning while managing complexity through hierarchical organization of computations.
Data Source
AI summary
A method for compositional reasoning of group activity in videos with keypoint-only modality is presented. The method includes obtaining video frames from a video stream received from a plurality of video image capturing devices, extracting keypoints all of persons detected in the video frames to define keypoint data, tokenizing the keypoint data with time and segment information, clustering groups of keypoint persons in the video frames and passing the clustering groups through multi-scale prediction, and performing a prediction to provide a group activity prediction of a scene in the video frames.