Distributable Hash Filter for Deterministic Set Inclusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for determining set membership, such as Bloom and Cuckoo filters, are probabilistic and resource-intensive, making them inefficient for large datasets and difficult to distribute in real-time environments.
Innovation Solution
A non-probabilistic method using sparse bit arrays and lossless compression to create a compact data structure that deterministically tests set membership, allowing efficient distribution and real-time access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If probabilistic techniques (Bloom/Cuckoo filters) are used for set membership testing, then resource consumption is reduced, but deterministic accuracy is lost
Solution Approach 1:
The patent segments the set membership testing problem into two distinct paths: a fast probabilistic filter for quick rejection of non-members, and a slower deterministic verification path for final confirmation. This segmentation allows the system to achieve deterministic accuracy while maintaining low average resource consumption by avoiding full deterministic verification for all queries.
Solution Approach 2:
The patent introduces an intermediary data structure (the bit array or filter structure) that sits between the query and the full deterministic verification. This intermediary provides probabilistic pre-screening, allowing the system to reject non-members quickly without invoking expensive deterministic checks, thus reducing overall resource consumption while preserving deterministic accuracy for final answers.
2Quantity of substance
If large datasets are used for set membership testing, then completeness is improved, but resource consumption increases
Solution Approach 1:
The patent extracts only the essential information needed for set membership testing into a compact data structure (bit array or filter). Instead of storing or processing the entire large dataset, the system extracts minimal sufficient information that enables membership queries to be answered with deterministic accuracy while consuming far fewer resources than processing the complete dataset.
Solution Approach 2:
The patent changes the parameter representation from storing complete dataset elements to using compact bit arrays or filter structures with specific parameters (bit positions, hash values). This parameter transformation allows the system to maintain completeness for membership testing purposes while dramatically reducing the resource footprint from O(n) to O(m) where m << n.
3Measurement precision
If deterministic set membership testing is performed on large datasets, then accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary action by pre-computing and storing the filter structure or bit array before queries are made. This preliminary processing organizes the data in a way that enables fast deterministic verification during query time, avoiding the need to process large datasets on-demand for each membership test.
Solution Approach 2:
The patent implements a dynamic two-phase verification process where the system adapts its processing depth based on the query. The probabilistic filter provides a fast first phase that dynamically determines whether to proceed to the slower deterministic verification phase, optimizing processing time while maintaining deterministic accuracy for all queries.
Data Source
AI summary
In certain embodiments, a method includes recursively performing a procedure that includes using an allowed set of object identifiers and a hash function to update a bit array, using a disallowed set of object identifiers and the hash function to further update the bit array where collisions occur, repeating the process with a new allowed set that includes object identifiers from the original allowed set that collided with the disallowed set and a new hash function, until reaching a round where no collisions occurred, generating a data structure that includes the bit arrays created during each recursive round, and compressing the data structure.