Foveated Detection and Contextual Filtering for Visual Activity Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video-based activity recognition systems are limited in their ability to detect and classify activities in complex scenarios with multiple objects and clutter, particularly in intelligence, surveillance, and reconnaissance applications, where the activity of interest may occupy a small region of the video in both space and time, and are not applicable to static or moving platforms.
Innovation Solution
A system that uses foveated detection and contextual filtering, comprising convolutional neural networks and recurrent neural networks to detect objects of interest, track their activities, and classify them, allowing for real-time recognition in streaming or recorded videos from various platforms, including aerial and ground vehicles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If deep learning methods are used for activity classification, then classification accuracy is improved, but the system becomes inapplicable to ISR applications with multiple objects and clutter
Solution Approach 1:
The system segments the activity recognition process into distinct stages: object detection, object tracking, and activity classification. This segmentation allows each component to be optimized independently, with object detection handling the complex scene understanding and activity classification focusing on specific actions, thereby resolving the contradiction between accuracy and applicability to ISR applications
Solution Approach 2:
The system performs preliminary object detection and tracking before activity classification. By pre-identifying and tracking objects of interest in cluttered scenes, the system prepares structured input for the activity classifier, enabling deep learning methods to achieve high accuracy while remaining applicable to ISR applications with multiple objects
2Measurement precision
If the system processes the entire video frame at full resolution, then detection accuracy is improved, but computational complexity and processing time increase
Solution Approach 1:
The system applies different processing quality to different regions of the video frame. Full-resolution processing is applied only to regions containing detected objects of interest, while other regions receive reduced processing. This local quality approach maintains detection accuracy for critical objects while reducing overall computational complexity
3Reliability
If the system tracks and classifies all objects in the video, then comprehensive activity recognition is achieved, but processing speed decreases
Solution Approach 1:
The system performs partial tracking and classification by focusing computational resources on a subset of tracked objects that are most relevant to the activity of interest. Rather than uniformly processing all tracked objects, the system selectively classifies activities based on object importance, maintaining reliable activity recognition while improving processing speed
Data Source
AI summary
Described is a system for visual activity recognition. In operation, the system detects a set of objects of interest (OI) in video data and determines an object classification for each object in the set of OI, the set including at least one OI. A corresponding activity track is formed for each object in the set of OI by tracking each object across frames. Using a feature extractor, the system determines a corresponding feature in the video data for each OI, which is then used to determine a corresponding initial activity classification for each OI. One or more OI are then detected in each activity track via foveation, with the initial object detection and foveated object detection thereafter being appended into a new detected-objects list. Finally, a final classification is provided for each activity track using the new detected-objects list and filtering the initial activity classification results using contextual logic.