Temporal-Pooling Pyramid for Video Action Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current computer systems are unable to accurately recognize action in video due to challenges in effectively combining and analyzing visual elements and their motions across frames.
Innovation Solution
The implementation of a system that generates temporal-pooling pyramids by pooling visual features from multiple frames at different scales, using techniques like mean or maximum pooling, and encoding these features with methods such as VLAD or Fisher Vector encoding, followed by training classifiers for action recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If visual features from multiple frames are pooled at different temporal scales, then action recognition accuracy is improved, but computational complexity increases
Solution Approach 1:
The temporal pooling process is segmented into multiple levels (first-level feature sets from 2-5 frames, second-level from 2-3 first-level sets) with different temporal scales. This hierarchical segmentation allows the system to capture motion patterns at multiple time granularities without processing all frames uniformly, reducing computational complexity while maintaining recognition accuracy.
Solution Approach 2:
The patent introduces a temporal dimension to the feature pooling process by organizing frames into hierarchical levels based on temporal scales. First-level feature sets pool frames at one temporal scale, while second-level feature sets pool at coarser temporal scales. This dimensional organization enables efficient computation by processing features at appropriate temporal granularities rather than treating all frames equally.
2Loss of information
If frame-level features are extracted from every frame, then temporal detail is preserved, but processing time increases
Solution Approach 1:
Instead of uniformly processing all frames at the same level, the patent applies partial processing by creating multiple hierarchical levels where only certain frames are pooled at each level. First-level feature sets are generated from subsets of frames (2-5 frames per set), and second-level sets are generated from subsets of first-level sets (2-3 sets per second-level set). This partial processing approach preserves essential temporal details while significantly reducing the total number of processing operations required.
3Adaptability or versatility
If multiple hierarchical levels of pooling are applied, then temporal-adaptive representation is achieved, but system complexity increases
Solution Approach 1:
The patent implements dynamic temporal adaptation through its hierarchical pooling structure. The system can adaptively select which frames to pool at each level based on the temporal characteristics of the action being recognized. First-level feature sets handle fine-grained temporal details (2-5 frames), while second-level sets capture coarse-grained patterns (2-3 first-level sets). This dynamic multi-scale approach allows the system to adapt to varying action durations and speeds without requiring a fixed processing structure.
Data Source
AI summary
Devices, systems, and methods for computer recognition of action in video obtain frame-level feature sets of visual features that were extracted from respective frames of a video, wherein the respective frame-level feature set of a frame includes the respective visual features that were extracted from the frame; generate first-level feature sets, wherein each first-level feature set is generated by pooling the visual features from two or more frame-level feature sets, and wherein each first-level feature set includes pooled features; and generate second-level feature sets, wherein each second-level feature set is generated by pooling the pooled features in two or more first-level feature sets, wherein each second-level feature set includes pooled features.