Hierarchical Bloom Filter Bit Array Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing data processing apparatuses using Bloom filters face an increase in false positive rates when the number of bits is reduced due to a decrease in the number of data elements, leading to inefficient data management and processing loads.
Innovation Solution
A data processing apparatus is designed with multiple Bloom filters associated with different data subsets, allowing for the deletion of Bloom filters when all data elements in a subset are deleted, and using a bit array with all '1' values to maintain accurate search results without increasing the false positive rate.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the number of bits in the Bloom filter is reduced when data elements are deleted, then storage space is saved, but the false positive rate increases
Solution Approach 1:
The patent divides the Bloom filter into multiple levels (hierarchical structure) where each level manages a subset of data elements. When data elements are deleted, only the corresponding level's bit array is reduced, while other levels maintain their original size. This segmentation allows selective reduction of bits without compromising the overall false positive rate, as each level independently manages its own data subset.
2Productivity
If the Bloom filter bit array is reduced to match the number of remaining data elements, then processing efficiency improves, but search accuracy deteriorates
Solution Approach 1:
The patent implements a dynamic hierarchical Bloom filter structure where the bit array size at each level adapts to the number of data elements in that specific subset. When elements are deleted, the corresponding level's bit array is dynamically reduced. This dynamic adjustment maintains processing efficiency by matching bit array size to actual data volume while preserving search accuracy through the multi-level structure that prevents false positives from propagating across all levels.
Data Source
AI summary
A data processing apparatus includes a memory configured to store a bit array including a first Bloom filter associating with a first subset containing a data element matching a first classification condition and a second Bloom filter associating with a second subset containing a data element matching a second classification condition, and a processor configured to, when a first data element to be a search target is inputted, determine whether the same data element as the first data element is present in the first subset by using the first Bloom filter, determine whether the same data element as the first data element is present in the second data subset by using the second Bloom filter, and when all the data elements contained in the first subset are deleted, delete the first Bloom filter from the bit array.