Action Detection Using Spatiotemporal Saliency Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current action detection technologies, such as bag-of-features action detectors, are not reliable for detecting complex actions like exchanging items, burying, or hauling due to short duration and interference from irrelevant actions or background clutter, leading to low detection accuracy.
Innovation Solution
The introduction of a spatiotemporal saliency map that weights features by their relevance for the specific action, improving the selectivity of the feature representation and enhancing the detection accuracy by emphasizing relevant features while reducing the impact of irrelevant ones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bag-of-features action detectors are used for detecting complex actions, then the detection process is simple and computationally efficient, but the detection accuracy is low due to interference from irrelevant actions and background clutter
Solution Approach 1:
The patent segments the feature histogram into multiple bins, each representing different spatiotemporal characteristics of actions. By dividing the feature space into discrete bins and applying specific weighting to each bin, the detector can selectively emphasize features relevant to complex actions while suppressing irrelevant background features, thereby improving detection accuracy without requiring a completely new detector architecture
Solution Approach 2:
The patent applies local quality by assigning different weights to different bins in the feature histogram based on their relevance to the target action. This allows the detector to treat different feature regions differently - enhancing important local features while suppressing less important ones, which improves detection reliability for complex actions without increasing overall system complexity
2Reliability
If standard bag-of-features detection is used, then the implementation is straightforward, but the detection reliability is low for actions with short duration and subtle motion
Solution Approach 1:
The patent changes parameters by introducing bin-specific weighting factors that modify the contribution of each feature bin to the overall detection score. This parameter adjustment allows the detector to become more sensitive to subtle motion patterns and short-duration actions by emphasizing bins that capture these characteristics, while maintaining the same underlying bag-of-features detection framework
Solution Approach 2:
The patent performs preliminary action by pre-computing and storing optimal bin weights based on training data before actual detection. This preliminary weighting preparation allows the detector to automatically emphasize relevant features during detection without requiring complex real-time computations, thus improving reliability for subtle actions while keeping implementation straightforward
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A video action detection system uses feature a data extractor to extract feature data from video data at detected spatiotemporal interest points. A feature data quantizer assigns the extracted feature values to bins of a feature vector. Bin values are computed from a sum of contributions of spatiotemporal points of interest that have been assigned to the bin, with a bin dependent adjustment of a size of the sum and/or the contributions. The video action detection system computes a sum of match scores between the feature vector and reference vectors for the predetermined type of action. The bin dependent adjustment of the size of the sum and/or the contributions is adapted in a training step, based on partial match scores for individual bins obtained using an initial action detector.