Two-Dimensional Temporal Maps for Efficient Moment Localization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for moment localization in video understanding using natural language queries are inefficient and computationally costly, particularly in processing untrimmed media streams.
Innovation Solution
A neural network-based approach that extracts a two-dimensional temporal feature map from media streams, utilizing sparse sampling and a temporal adjacent network to determine correlations between moments and actions, reducing computational overhead through adaptive sampling and feature fusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional methods are used for moment localization in untrimmed media streams, then comprehensive analysis can be performed, but computational cost and processing time increase significantly
Solution Approach 1:
The patent segments the untrimmed media stream into multiple candidate clips based on detected actions, and further segments each candidate clip into multiple candidate moments. This hierarchical segmentation reduces the computational burden by breaking down the large-scale analysis into smaller, manageable units while maintaining comprehensive coverage of potential moments.
Solution Approach 2:
The patent applies different processing strategies to different regions of the temporal feature map based on their characteristics. High-confidence regions are processed with lower resolution sampling, while low-confidence regions receive more intensive analysis. This local quality approach optimizes computational resources by adapting processing intensity to the specific needs of each region.
2Measurement precision
If dense sampling is used to ensure comprehensive moment coverage, then localization accuracy improves, but computational overhead increases
Solution Approach 1:
The patent implements adaptive sampling where the sampling density is adjusted based on the confidence level and characteristics of different regions in the temporal feature map. Instead of uniformly dense sampling across the entire media stream, the system applies partial dense sampling only where necessary, achieving sufficient detection accuracy with reduced computational energy consumption.
3Reliability
If the entire media stream is processed in detail, then all potential moments are identified, but processing time becomes prohibitive
Solution Approach 1:
The patent performs preliminary processing by first detecting actions and generating candidate clips, then uses these to guide subsequent moment candidate generation. This preliminary action creates a filtered set of relevant regions that require detailed analysis, while other regions can be processed more coarsely or skipped, thereby reducing overall processing time while maintaining identification completeness for relevant moments.
Solution Approach 2:
The patent transforms the one-dimensional temporal processing into a two-dimensional temporal feature map analysis, enabling parallel processing of multiple candidate moments across different time regions. This dimensional transformation allows the system to process multiple hypotheses simultaneously, reducing the sequential processing time while maintaining comprehensive coverage.
Data Source
AI summary
Various implementations of the subject matter relate to moment localization in media stream. In some implementations, a two-dimensional temporal feature map representing a plurality of moments within a media stream is extracted from the media stream, wherein the two-dimensional temporal feature map comprises a first dimension representing a start of a respective one of the plurality of moments and a second dimension representing an end of a respective one of the plurality of moments. A correlation between the plurality of moments and an action in the media stream is determined based on the two-dimensional temporal feature map.