Video Segment Retrieval via Scaled-Dot Product Attention

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Conventional video segment retrieval methods require significant computing resources and time, making them unsuitable for real-time applications with high execution speed, despite showing improvements in video semantics retrieval performance.

Innovation Solution

A method and system that utilize pre-trained deep neural networks to extract video and query features, integrate them using scaled-dot product attention layers, and estimate video segments corresponding to query sentences, enabling fast and accurate retrieval with reduced computational load.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If conventional fine-grained video segment retrieval methods are used, then retrieval performance is improved, but computing resources and time required increase significantly

Engineering Contradiction:
Improvevideo semantics retrieval performanceVSAvoidtime required
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The patent applies preliminary action by pre-training deep neural network algorithms before actual video retrieval operations. The system extracts video features and query features using pre-trained models, and pre-computes attention mechanisms to integrate video-query relations. This preparation work is done in advance, allowing the actual retrieval process to execute quickly without performing heavy computations during real-time queries, thus resolving the contradiction between high retrieval performance and low time consumption.

Inventive Principle:
Principle #10Preliminary action

2Measurement precision

If conventional fine-grained video segment retrieval methods are used, then retrieval performance is improved, but computing resources required increase significantly

Engineering Contradiction:
Improvevideo semantics retrieval performanceVSAvoidcomputing resources
Core Design Contradiction:
Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent uses copying by creating feature representations (video features and query features) that capture the essential semantics of the video and query sentences. Instead of processing raw video data repeatedly, the system extracts and stores compressed feature vectors that can be efficiently compared and matched. This copying of essential information into a compact form reduces the computational resources needed for subsequent retrieval operations while maintaining high retrieval performance.

Inventive Principle:
Principle #26Copying

3Productivity

If coarse-grained video segment retrieval method is used, then computing resources and time are reduced, but retrieval performance decreases

Engineering Contradiction:
Improveexecution speedVSAvoidretrieval performance
Core Design Contradiction:
ProductivityVSMeasurement precision

Solution Approach 1:

The patent applies parameter changes by transforming the retrieval problem from operating on raw video frames to operating on extracted feature vectors. The system changes the parameter space from pixel-level or frame-level data to semantic feature representations. This parameter transformation allows the system to achieve both high execution speed (by working with compact feature vectors rather than full video data) and high retrieval performance (by using semantically meaningful features captured by deep neural networks).

Inventive Principle:
Principle #35Parameter changes

Data Source

PatentUS12019678B2Method and system for retrieving video segment by a semantic query
Publication Date: 2024.06.25 ELECTRONICS & TELECOMM RES INST
  • US12019678B2 patent drawing
  • US12019678B2 patent drawing
  • US12019678B2 patent drawing

AI summary

Provided is a method of detecting a semantics section in a video. The method includes extracting all video features by inputting an inputted video to a pre-trained first deep neural network algorithm, extracting a query sentence feature by inputting an inputted query sentence to a pre-trained second deep neural network algorithm, generating video-query relation integration feature information in which all of the video features and the query sentence feature have been integrated by inputting all of the video features and the query sentence feature to a plurality of scaled-dot product attention layers, and estimating a video segment corresponding to the query sentence in the video based on the video-query relation integration feature information.