RGB-Only Streaming Action Detection with Chunk-Level Features
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing action detection technologies in streaming videos require large amounts of memory and time due to the use of optical flow, and fail to effectively utilize features from untrimmed videos with mixed backgrounds, leading to poor performance.
Innovation Solution
The method extracts chunk-level features from streaming videos using only RGB frames, dividing the video into segments and frame chunks, generating elevated feature information through an information elevation process, and using an action detection model trained with cross entropy loss to predict actions at the current time point.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If optical flow is used to extract features from video, then feature extraction accuracy is improved, but memory consumption and processing time increase significantly
Solution Approach 1:
The patent extracts and uses only the necessary visual information from RGB frames, eliminating the need for optical flow computation. By taking out the essential features directly from color frames without computing motion between frames, the system achieves accurate action detection while avoiding the time and memory overhead of optical flow extraction.
Solution Approach 2:
The patent employs lightweight feature extraction from RGB frames that requires minimal memory and computation, replacing the expensive and time-consuming optical flow method. This approach uses readily available RGB data without the burden of additional computational layers, enabling fast processing suitable for real-time applications.
2Measurement precision
If optical flow is used to extract features from video, then feature extraction accuracy is improved, but memory consumption increases
Solution Approach 1:
The patent extracts and uses only the necessary visual information from RGB frames, eliminating the need for optical flow computation. By taking out the essential features directly from color frames without computing motion between frames, the system achieves accurate action detection while avoiding the memory overhead of optical flow extraction.
3Measurement precision
If visual information from the entire video is used for action detection, then detection accuracy is improved, but processing efficiency decreases
Solution Approach 1:
The patent segments the video into fixed-length video segments and further divides each segment into video frame chunks. By processing these segmented chunks independently and using a sliding window approach, the system maintains detection accuracy through comprehensive feature extraction while improving processing efficiency through parallelizable, modular computation on smaller units.
Solution Approach 2:
The patent performs preliminary processing by dividing the video into segments and chunks before action detection. This pre-segmentation allows for efficient memory management and enables the system to process only relevant portions of the video at each step, maintaining accuracy while improving overall processing throughput.
4Productivity
If fixed-length video segments are used, then processing speed is improved, but adaptability to variable action durations decreases
Solution Approach 1:
The patent uses a sliding window approach that dynamically moves through the video sequence with a fixed window size. This dynamic traversal allows the system to adapt to actions of varying durations by continuously updating the detection window, maintaining processing speed through fixed-length segments while achieving versatility in detecting actions regardless of their temporal extent.
Data Source
AI summary
Disclosed herein are a method and apparatus for action detection. The method for action detection includes extracting chunk-level features for respective video frame chunks from a streaming video ranging from a past time point to a current time point, based on RGB frames, generating elevated feature information based on a chunk-level feature corresponding to the current time point for each of the video frame chunks, and detecting an action corresponding to the current time point based on the elevated feature information.