Bloom Filter Bucket-Digest Scheme for False Positive Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Bloom filter implementations are inefficient in time-constrained systems due to high computational resources and memory access requirements, particularly when the number of hash functions is sub-optimal, leading to increased false positive probabilities and memory access frequencies.
Innovation Solution
The proposed method partitions memory into two layers, using a regular Bloom filter layer for bucket representation and a second layer of ternary digest symbols to reduce false positive probabilities, allowing for reduced hash function usage and memory accesses by employing a bucket-digests scheme that optimizes the distribution of symbols and buckets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the number of hash functions is increased to reduce false positive probability, then the reliability of the Bloom filter improves, but the time complexity and computational resources increase
Solution Approach 1:
The patent segments the Bloom filter structure into multiple layers, where each layer uses a different number of hash functions. Elements are distributed across layers, allowing the system to achieve lower overall false positive probability without requiring every query to compute the maximum number of hash functions, thus reducing average time complexity while maintaining reliability
Solution Approach 2:
The patent applies partial action by using a variable number of hash functions depending on the layer and element characteristics. Not all elements require the same computational effort, and the system can achieve the desired false positive rate by strategically distributing elements across layers with different hash function counts, avoiding the need to always use the maximum number of hash functions
2Reliability
If the number of hash functions is increased to reduce false positive probability, then the reliability improves, but the memory access frequency increases
Solution Approach 1:
By segmenting the Bloom filter into layers with different hash function requirements, the patent reduces the average memory access frequency. Elements in higher layers (with fewer hash functions) require fewer memory accesses, and the segmented structure allows the system to achieve the desired reliability without uniformly increasing memory access frequency across all elements
Solution Approach 2:
The patent applies local quality by assigning different numbers of hash functions to different layers or regions of the Bloom filter based on local requirements. This allows the system to optimize for both reliability and memory access efficiency locally, rather than applying a uniform approach that would increase memory access frequency system-wide
3Device complexity
If bucket contents are stored in slower memory to reduce cost, then the device complexity and cost decrease, but the productivity decreases due to reduced performance rate
Solution Approach 1:
The patent segments the memory structure into multiple layers with different access characteristics. By distributing bucket contents across these segmented layers, the system can store more data in slower, cheaper memory while keeping frequently accessed elements in faster memory, thus reducing overall device complexity and cost without significantly degrading performance rate
Solution Approach 2:
The patent applies partial action by storing bucket contents partially in slower memory based on access patterns and importance. Not all bucket contents require the same level of access speed, so the system can strategically place less frequently accessed elements in slower memory, reducing cost and complexity while maintaining acceptable performance for critical operations
Data Source
Figure 1~2
Figure 3~4
AI summary
A method for representing sets of elements in a randomized data structure, adapted to reduce the false-positive probability in a time-constraint system, according to which a memory space is partitioned to two layers. A first layer functions as a regular Bloom filter bits, representing buckets and a second layer that comprises digest symbols. The first layer of buckets and the second layer of digest symbols are partitioned into sets, where sets of symbols are assigned to sets of buckets. The properties of elements that belong to the buckets sets are represented as values in their corresponding symbols.