Video Action Localization via Memory Queue Context
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing online temporal action localization technologies are limited by their inability to utilize future information and correct past results, making them inefficient for real-time video analysis.
Innovation Solution
A real-time video action localization method and device that selectively uses past information by employing a memory queue to store past segments, allowing for the prediction of action regions even when start and end timestamps are not included in current input frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If online temporal action localization is performed without using future information, then real-time processing is enabled, but localization accuracy deteriorates
Solution Approach 1:
The system performs preliminary action by pre-storing video segments in a buffer before they are fully processed. This allows the model to access future context information that has already been captured but not yet fully analyzed, thereby improving localization accuracy while maintaining real-time processing capabilities.
Solution Approach 2:
The patent introduces a temporal dimension by utilizing a buffer that stores past segments. This transforms the problem from purely online (current frame only) to a hybrid approach where historical context is incorporated, effectively adding a time-based dimension to the analysis without sacrificing real-time performance.
2Measurement precision
If a memory queue storing multiple past segments is used, then action localization accuracy is improved, but computational complexity increases
Solution Approach 1:
The system extracts and stores only the necessary features from past video segments in the memory queue, rather than processing entire segments. This selective extraction reduces computational complexity while maintaining the ability to improve localization accuracy through historical context.
Solution Approach 2:
The patent implements a buffer with a fixed capacity that stores a limited number of past segments. This partial action approach provides sufficient historical context to improve accuracy without the excessive computational burden of storing and processing all past segments indefinitely.
3Adaptability or versatility
If information from past segments is utilized, then robustness to varying action durations is improved, but information processing time increases
Solution Approach 1:
Video segments are pre-buffered and pre-processed into feature representations before being needed for localization. This preliminary action allows the system to quickly access and utilize historical information without incurring processing delays during real-time operation, thus improving adaptability without significant time loss.
Data Source
AI summary
A video action localization method may comprise: receiving a first segment included in a video at a current timestamp; extracting features of the first segment; and acquiring a predicted start timestamp, end timestamp, and action class of an action region in the video by inputting the extracted features of the first segment and features of segments stored in a memory queue at previous timestamps to a neural network, wherein each of the segments stored in the memory queue at the previous timestamps satisfies a certain condition.