Vertical Cuckoo Filter for Insertion-Intensive Streaming Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Cuckoo filters and their variants face inefficiencies in insertion-intensive scenarios due to time-consuming reallocation processes and high calculation overhead, making them unsuitable for online applications where items are frequently inserted and deleted.
Innovation Solution
The Vertical Cuckoo Filter (VCF) design extends the standard Cuckoo filter by providing more candidate buckets for each item, utilizing vertical hashing to index these buckets without additional hash computation, thereby improving the reallocation process and reducing the number of hash computations required during item insertion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the standard Cuckoo filter uses reallocation to accommodate more items, then space utilization is improved, but insertion time consumption increases
Solution Approach 1:
The patent transitions from horizontal expansion (more buckets) to vertical expansion (more candidate buckets per item) by introducing d-ary cuckoo hashing with d≥4. Each item now has multiple candidate buckets instead of just two, allowing the filter to find empty slots more easily without extensive reallocation, thus improving space utilization while controlling insertion time.
Solution Approach 2:
The patent segments the bucket array into multiple candidate buckets for each item, where each candidate bucket can independently store item fingerprints. This segmentation allows parallel insertion attempts into multiple buckets simultaneously, reducing the reallocation chain length and improving insertion performance while maintaining high space utilization.
2Quantity of substance
If d-ary cuckoo hashing is used to provide more candidate buckets, then space utilization is improved, but calculation overhead increases
Solution Approach 1:
The patent pre-computes multiple hash values for each item during the insertion process, storing them in an array. These pre-computed hash values are then used to directly index into the candidate buckets, eliminating the need for iterative hashing during reallocation operations. This preliminary action reduces the calculation overhead during subsequent insertion and lookup operations.
Solution Approach 2:
The patent replaces the traditional sequential reallocation mechanism with a direct indexing mechanism using pre-computed hash values. Instead of performing multiple hash computations and iterative reallocation, the system uses the pre-computed hash array to directly determine candidate bucket positions, substituting the mechanical reallocation process with a more efficient direct access approach.
3Productivity
If more candidate buckets are provided for each item, then insertion throughput is improved, but lookup speed decreases
Solution Approach 1:
The patent uses d-ary cuckoo hashing with d≥4 to provide more candidate buckets than traditionally needed. While this excessive number of candidates increases the theoretical lookup space, the patent optimizes the actual search process by using efficient hash indexing and early termination strategies, achieving high insertion throughput while maintaining acceptable lookup performance through partial exploration of candidate buckets.
Data Source
AI summary
A system for streaming data items of a streaming application via a Vertical Cuckoo Filter (VCF), including: one or more computers communicating with a provider via a network; wherein the one or more computers includes: a processor; and a storage device coupled to the processor, wherein the storage device is configured to store the VCF, an insert module, a lookup module, and a delete module, the VCF includes buckets with one or more slots and includes at least two bitmasks, the insert module is configured to perform an insert operation of a first fingerprint of a first item from the provider on the VCF, the lookup module is configured to perform a lookup operation of a second fingerprint of a second item from the provider on the VCF, the delete module is configured to perform a delete operation of a third fingerprint of a third item from the VCF.