Video Semantic Interval Detection via Boundary Refinement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video semantic interval detection technologies face challenges in accurately identifying variable boundary semantic intervals within long videos, due to the high memory requirements of two-dimensional maps used for fine-grained interval proposals.
Innovation Solution
A system and method for detecting video semantic intervals using a deep learning-based approach that refines semantic interval proposals to produce intervals with variable boundaries, overcoming the limitations of fixed boundary proposals and improving accuracy with limited GPU memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the size N of the two-dimensional map is increased to generate fine-grained semantic interval proposals, then the measurement precision of semantic interval detection is improved, but the use of energy by stationary object (GPU memory) deteriorates
Solution Approach 1:
The patent segments the semantic interval detection process into two stages: first generating coarse-grained proposals using a lower-resolution map, then refining these proposals into fine-grained intervals through boundary refinement. This segmentation allows the system to achieve high measurement precision without requiring a large two-dimensional map throughout the entire process, thus reducing GPU memory consumption.
Solution Approach 2:
The patent performs preliminary action by generating coarse-grained semantic interval proposals first, which serve as a foundation for subsequent refinement. This preliminary step allows the system to establish initial boundaries without requiring fine-grained map resolution, thereby reducing memory requirements while maintaining the ability to produce precise final results through refinement operations.
2Manufacturing precision
If the unit length τ is reduced to generate fine-grained proposals, then the manufacturing precision of interval proposals is improved, but the quantity of substance (total number of proposals) increases
Solution Approach 1:
The patent segments the proposal generation process into coarse-grained initial proposals and fine-grained refined proposals. The coarse-grained stage uses larger unit length to generate fewer initial proposals, which are then refined to achieve the desired precision. This segmentation reduces the total quantity of proposals compared to generating all fine-grained proposals directly.
Solution Approach 2:
The patent applies partial action by generating only the necessary refinement for each proposal rather than exhaustively generating all possible fine-grained proposals. This selective refinement approach maintains manufacturing precision where needed while reducing the overall quantity of proposals generated.
3Measurement precision
If boundary refinement is applied to semantic interval proposals, then the measurement precision of variable boundary intervals is improved, but the device complexity increases
Solution Approach 1:
The patent segments the detection system into a proposal generation module and a boundary refinement module. This segmentation allows the refinement functionality to be added as a separate, manageable component that operates on existing proposals, thereby improving measurement precision for variable boundary intervals without proportionally increasing overall device complexity.
Solution Approach 2:
The patent introduces boundary refinement maps as intermediary structures that store refinement information for each proposal. These intermediary maps act as mediators between the coarse-grained proposals and the final refined intervals, enabling precise variable boundary detection while keeping the system architecture organized and manageable.
Data Source
AI summary
Provided is a system for detecting a video semantic interval. The system includes a communication module configured to receive a video and a query sentence, memory in which a program for outputting a semantic interval proposal from the video and the query sentence is stored, and a processor configured to execute the program stored in the memory. By executing the program, the processor outputs a semantic interval proposal having start timing and end timing, which is matched with the query sentence within the video, over a pre-trained video semantic interval detection network based on boundary refinements as the results of the detection of the semantic interval proposal, and outputs a semantic interval proposal having a variable boundary through the refinements of a predetermined semantic interval proposal.