Hash Lookup Key Selection for Video Indexing Clumping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video indexing systems face inefficiencies in lookup and retrieval due to 'clumping' issues, where a large number of reference videos are associated with a particular lookup key, overwhelming processing power and memory resources.
Innovation Solution
A system that generates a reference index using training videos, partitions fingerprints into bands, selects a subset of lookup keys based on a reward function to minimize clumping, and splits lookup keys with high clumping likelihood to distribute reference identifiers across multiple keys, preventing excessive data flow during searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all lookup keys are indexed in the reference index, then complete video matching capability is achieved, but system memory and processing power are overwhelmed due to large data flow
Solution Approach 1:
The patent segments the reference index into multiple partitions, each containing a subset of lookup keys. Instead of loading all lookup keys into memory simultaneously, the system divides them into manageable segments that can be processed in batches, reducing peak memory usage and processing load while maintaining complete matching capability across all partitions.
Solution Approach 2:
The patent performs preliminary indexing of videos into multiple partitions with different lookup key subsets before the actual search operation. This preliminary organization allows the system to prepare data structures in advance, so that during search operations, only relevant partitions need to be accessed, reducing real-time processing requirements and memory demands.
2Productivity
If a small subset of lookup keys is selected for indexing, then system resources are conserved, but video matching accuracy deteriorates
Solution Approach 1:
The patent implements dynamic lookup key selection where the system adaptively chooses which lookup keys to index based on the specific search query and available system resources. Different search scenarios can trigger different subsets of lookup keys to be used, allowing the system to optimize between speed and accuracy dynamically rather than being fixed to a static subset.
Solution Approach 2:
The patent changes the parameter of lookup key selection by using multiple partitions with different key subsets. The system can adjust which partition (and thus which subset of lookup keys) is used based on query characteristics and resource availability, effectively changing the selection parameter to balance between search efficiency and matching accuracy for different scenarios.
3Productivity
If lookup keys with high clumping are used, then more reference videos are retrieved per lookup, but the number of false positives increases and processing overhead grows
Solution Approach 1:
The patent segments the lookup keys into multiple partitions, distributing the clumped reference videos across different partitions. By dividing the index structure, the system prevents any single lookup operation from returning an excessively large number of results, as the same clumped videos will be distributed across multiple smaller partitions rather than concentrated in one large list.
Solution Approach 2:
The patent introduces a new dimension to the index structure by creating multiple partitions along an additional organizational axis. Instead of a single flat index structure, the system adds a partition dimension that allows lookup operations to be distributed across multiple smaller index structures, effectively reducing the dimension of individual result sets while maintaining comprehensive coverage through the combined partitions.
Data Source
AI summary
Videos are received and indexed based on fingerprints generated for the videos, using lookup keys which correspond to sub portions of the reference fingerprints. Specifically, a set of lookup keys is selected such that the number of reference fingerprints such that the clumping associated with the associated each of the lookup keys is minimized. Clumping occurs when a number of reference identifiers associated with a lookup key exceeds a maximum value.