Temporal Aggregation Module for Video Understanding Feature Maps
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video understanding processing, which involves action/activity recognition and spatiotemporal analysis, is resource-intensive due to the need to process large amounts of sequential image data, particularly when dealing with long video segments, leading to high computational and memory requirements.
Innovation Solution
The method combines high-resolution and low-resolution feature maps through a Temporal Aggregation Module, applying channel-wise multiplication and summation to create temporally aggregated feature maps, reducing processing and memory needs by leveraging multiple branch Convolutional Neural Networks (CNNs) and avoiding large-scale 3D convolutions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If video understanding processing processes long video segments with many frames using traditional machine learning techniques, then prediction accuracy is improved, but computational resources and memory requirements increase significantly
Solution Approach 1:
The patent divides the video processing task into two separate streams: a high-resolution stream processing fewer frames at higher detail, and a low-resolution stream processing more frames at lower detail. This segmentation allows the system to maintain prediction accuracy through high-resolution features while reducing overall computational burden by distributing processing across multiple streams with different resolution requirements.
2Measurement precision
If video understanding processing uses high resolution feature maps for all frames, then prediction accuracy is improved, but memory requirements increase significantly
Solution Approach 1:
The patent applies different quality levels (resolutions) to different parts of the video data by maintaining high-resolution feature maps only for selected frames in the high-resolution stream, while using low-resolution feature maps for other frames. This local quality differentiation reduces overall memory requirements while preserving prediction accuracy where high resolution is most beneficial.
3Measurement precision
If video understanding processing processes more frames, then prediction accuracy is improved, but processing time increases
Solution Approach 1:
The patent segments the frame processing task by distributing frames across two parallel streams with different processing characteristics. The low-resolution stream can process more frames faster due to reduced computational complexity per frame, while the high-resolution stream processes fewer frames with higher detail. This segmentation enables the system to process more total frames without proportionally increasing processing time.
4Measurement precision
If video understanding processing uses deeper models to process more frames, then prediction accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the computational workload by creating two separate processing streams with different model depths and resolution requirements. This segmentation allows the system to achieve the benefits of deeper models and more frame processing without proportionally increasing overall computational complexity, as each stream can be optimized independently for its specific processing requirements.
Data Source
AI summary
Systems and methods for performing video understanding and analysis. Sets of feature maps for high resolution images and low resolution images in a time sequence of images are combined into combined sets of feature maps each having N feature maps. A time sequence of temporally aggregated sets of feature maps is created for each combined set of feature maps by: selecting a selected combined set of feature maps corresponding to an image at time âtâ in the time sequence of images; applying, by channel-wise multiplication, a feature map weighting vector to a number of combined sets of feature maps that are temporally adjacent to the selected combined set of feature maps; and summing elements of the number of combined set of feature maps into a temporally aggregated set of feature maps. The time sequence of temporally aggregated sets of feature maps is processed to perform video understanding processing.