Action Recognition via Spatio-Temporal Object Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Digital video systems and computer vision systems face difficulties in action recognition due to incorrect or spurious object detection in videos caused by motion blur and camera defocus, as object detector systems are typically trained on still images.
Innovation Solution
The implementation of a high-order interaction approach using spatio-temporal object tracking, where each frame is detected by an object detector to locate visual objects, and these objects are linked through time to form object tracks, which are then organized and combined with embeddings as input to a model trained for efficient matrix operations to generate representative embeddings and discriminative video features.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If object detector systems trained on still images are applied to video, then object detection can be performed, but detection accuracy deteriorates due to motion blur and camera defocus
Solution Approach 1:
The system performs preliminary object detection on individual frames to obtain candidate object locations, then uses spatio-temporal tracking to refine these detections across multiple frames. This preliminary detection followed by tracking refinement resolves the contradiction by allowing easy application of standard detectors while achieving high accuracy through temporal context.
Solution Approach 2:
The patent introduces spatio-temporal object tracks as an intermediary between individual frame detections and final action recognition. These tracks aggregate detection evidence across multiple frames, serving as a mediator that compensates for poor individual frame detections caused by motion blur and defocus.
2Measurement precision
If spatio-temporal object tracking is implemented, then action recognition accuracy is improved, but system complexity increases
Solution Approach 1:
The system segments the video analysis task into distinct components: object detection per frame, track formation by linking detections across frames, and action recognition from track features. This segmentation allows each component to be optimized independently while maintaining overall accuracy, reducing unnecessary complexity.
Solution Approach 2:
The spatio-temporal tracking module serves multiple functions simultaneously: it performs object detection refinement, provides temporal context for action recognition, and generates structured representations for downstream tasks. This multi-functionality reduces overall system complexity by consolidating multiple operations into a single versatile component.
3Measurement precision
If high-order interaction models with iterative processing are used, then embedding quality improves, but processing speed decreases
Solution Approach 1:
The patent replaces iterative mechanical processing with a single-pass matrix operation for high-order interaction. By formulating the embedding computation as a direct matrix multiplication rather than an iterative optimization process, the system achieves both high embedding quality and real-time processing speed.
Data Source
AI summary
Aspects of the present disclosure describe systems, methods, and structures that provide action recognition with high-order interaction with spatio-temporal object tracking. Image and object features are organized into into tracks, which advantageously facilitates many possible learnable embeddings and intra/inter-track interaction(s). Operationally, our systems, method, and structures according to the present disclosure employ an efficient high-order interaction model to learn embeddings and intra/inter object track interaction across the space and time for AR. Each frame is detected by an object detector to locate visual objects. Those objects are linked through time to form object tracks. The object tracks are then organized and combined with the embeddings as the input to our model. The model is trained to generate representative embeddings and discriminative video features through high-order interaction which is formulated as an efficient matrix operation without iterative processing delay.