Compressed Cuckoo Filter with Biasing and Decoupled Sparsity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Approximate set membership data structures (ASMDS) like Bloom filters face challenges with deletion operations, storage efficiency, and insertion throughput, especially at high load factors, and lack cache-friendliness and efficient use of memory.
Innovation Solution
A compressed cuckoo filter is introduced, which uses a matrix structure with buckets and slots, employing hash functions to remap fingerprints and support deletions and counting without significant memory overhead, and optimizes storage by encoding empty slots, allowing for faster operations and reduced storage usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a Bloom filter is used for approximate set membership, then storage efficiency is improved, but deletion operations are not supported and false positive rate increases at high load factors
Solution Approach 1:
The filter is divided into multiple independent buckets, each capable of storing multiple fingerprints. This segmentation allows individual buckets to be managed separately, enabling deletion operations while maintaining overall storage efficiency. Each bucket acts as an independent container that can be modified without affecting the entire filter structure.
Solution Approach 2:
The patent introduces a new dimension by using multiple hash functions to map keys to different buckets, rather than a single hash function. This dimensional approach allows elements to have multiple possible locations, enabling deletion by clearing one bucket while maintaining presence information through other buckets, thus supporting deletions without sacrificing storage efficiency.
2Adaptability or versatility
If a cuckoo filter is used to support deletions, then deletion capability is improved, but insertion throughput decreases substantially as the filter is filled
Solution Approach 1:
The patent applies local quality by allowing different buckets to have different capacities and characteristics. Some buckets can be optimized for fast insertion while others handle deletion operations more efficiently. This local differentiation maintains overall deletion capability while preserving high insertion throughput in the majority of buckets that don't require complex deletion logic.
Solution Approach 2:
The filter pre-allocates multiple buckets for each key element, so when an element needs to be deleted, the system can simply clear one of the pre-designated buckets without requiring complex rehashing or relocation operations. This preliminary setup enables O(1) deletion operations regardless of filter fullness, maintaining insertion throughput while supporting deletions.
3Speed
If a blocked Bloom filter variant is used for cache-friendliness, then cache locality is improved, but space efficiency decreases
Solution Approach 1:
The filter structure is segmented into fixed-size blocks that can be efficiently cached. Each block contains multiple buckets, and the blocked organization allows entire blocks to be loaded into cache together, improving cache locality. The segmentation maintains space efficiency by tightly packing multiple logical buckets into each physical block without excessive overhead.
Solution Approach 2:
Multiple logical buckets are merged into single physical cache blocks, allowing the system to utilize cache memory more effectively. By combining several buckets that are frequently accessed together into one cache block, the patent improves cache hit rates while maintaining the logical independence of each bucket for deletion operations. This merging reduces the total number of cache lines needed compared to a naive blocked approach.
Data Source
AI summary
A method includes, for each key of a plurality of keys, identifying from a set of buckets a first bucket for the key based on a first hash function, and identifying from the set of buckets a second bucket for the key based on a second hash function. An entry for the key is stored in a bucket selected from one of the first bucket and the second bucket. The entry is inserted in a sequence of entries in a memory block. A position of the entry in the sequence of entries corresponds to the selected bucket. For each bucket in the set of buckets, an indication of a number of entries in the bucket is recorded.