Spatio-Temporal Video Fingerprints Using Maximally Stable Volumes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video fingerprinting techniques fail to accurately and efficiently match video clips to stored videos due to sensitivity to distortions such as blurring, resizing, and frame cropping, which affects their performance in identifying videos across different formats and conditions.
Innovation Solution
A computer-implemented method using Maximally Stable Volumes (MSVs) to derive spatio-temporal fingerprints from key frames, which are then encoded and matched using affine invariant representations, allowing for robust identification of videos even under common distortions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If video fingerprints use complete frame information features (mean-luminance, centroid of gradient, etc.), then the fingerprint captures comprehensive video content, but it fails to identify videos when presented with partially cropped or scaled query data
Solution Approach 1:
The patent divides the video frame into multiple local regions and extracts spatio-temporal fingerprints from each region independently. This segmentation allows the system to handle cropped or scaled queries by matching only the relevant local regions, rather than requiring complete frame information. Each local fingerprint captures essential features of its region while remaining robust to variations in size and position.
Solution Approach 2:
The patent employs local spatio-temporal fingerprints that capture perceptually important features at specific locations within the video frame. These local fingerprints are designed to be invariant to affine transformations and can identify content even when the query is partially cropped or scaled. The local quality approach ensures that each region contributes its unique characteristics to the overall identification while maintaining robustness to distortions.
2Adaptability or versatility
If video fingerprinting uses local features to handle cropped/scaled data, then robustness to distortions improves, but discrimination between similar scenes (e.g., different seasons of same TV program) deteriorates
Solution Approach 1:
The patent extends the traditional 2D local feature extraction to a 3D spatio-temporal fingerprint by incorporating temporal information. Instead of just capturing spatial features at a single moment, the system creates fingerprints that span multiple time frames, adding a temporal dimension to the feature representation. This dimensional expansion provides additional discriminative power that helps distinguish between similar scenes across different time periods while maintaining robustness to spatial distortions.
Solution Approach 2:
The patent combines multiple types of local features into a composite spatio-temporal fingerprint representation. By integrating various perceptually important features (edges, textures, colors, motions) across multiple time frames into a unified fingerprint structure, the system achieves both robustness to distortions and precise discrimination between similar content. The composite approach allows different feature types to complement each other's strengths.
3Loss of information
If traditional video fingerprinting processes all video frames, then comprehensive content analysis is achieved, but processing time and computational complexity increase
Solution Approach 1:
The patent extracts only the essential spatio-temporal fingerprints from the video frames, discarding redundant information. By identifying and extracting only the locally important features that capture perceptually significant content, the system reduces the amount of data that needs to be processed and stored. This extraction approach maintains comprehensive content analysis capability while significantly reducing processing time and computational requirements.
Solution Approach 2:
The patent applies partial action by processing only a subset of the most informative local regions rather than all frames and regions equally. The system identifies key spatio-temporal fingerprints that represent the most discriminative content and focuses computational resources on extracting and matching these critical features. This partial processing strategy achieves sufficient content analysis for accurate identification without the full computational burden of processing every frame in detail.
Data Source
AI summary
A computer implemented method computer implemented method for deriving a fingerprint from video data is disclosed, comprising the steps of receiving a plurality of frames from the video data; selecting at least one key frame from the plurality of frames, the at least one key frame being selected from two consecutive frames of the plurality of frames that exhibiting a maximal cumulative difference in at least one spatial feature of the two consecutive frames; detecting at least one 3D spatio-temporal feature within the at least one key frame; and encoding a spatio-temporal fingerprint based on mean luminance of the at least one 3D spatio-temporal feature. The least one spatial feature can be intensity. The at least one 3D spatio-temporal feature can be at least one Maximally Stable Volume (MSV). Also disclosed is a method for matching video data to a database containing a plurality of video fingerprints of the type described above, comprising the steps of calculating at least one fingerprint representing at least one query frame from the video data; indexing into the database using the at least one calculated fingerprint to find a set of candidate fingerprints; applying a score to each of the candidate fingerprints; selecting a subset of candidate fingerprints as proposed frames by rank ordering the candidate fingerprints; and attempting to match at least one fingerprint of at least one proposed frame based on a comparison of gradient-based descriptors associated with the at least one query frame and the at least one proposed frame.