Recursive Hash Filter for Deterministic Set Inclusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing techniques for determining set membership, such as Bloom filters and Cuckoo filters, are probabilistic and cannot provide definitive answers, making them inefficient for large sets of device identifiers, which can be resource-intensive and difficult to share or access in real-time.
Innovation Solution
A compact, distributable data structure using sparse bit arrays is created by iteratively applying hash functions to both allowed and disallowed sets, resulting in a compressed data structure that can deterministically test membership, allowing it to fit within HTTP headers and be efficiently shared.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If probabilistic data structures (Bloom filters, Cuckoo filters) are used for set membership testing, then resource usage is reduced, but definitive membership determination is lost
Solution Approach 1:
The patent segments the set membership problem into two independent components: a probabilistic filter for quick rejection of non-members and a separate verification mechanism for confirming membership. This segmentation allows the system to use resource-efficient probabilistic filtering while maintaining the ability to provide definitive answers when needed, resolving the contradiction between resource usage and reliability.
Solution Approach 2:
The patent introduces an intermediary verification mechanism that bridges the gap between probabilistic filtering and definitive membership determination. This intermediary layer processes the probabilistic filter's output and provides conclusive membership status, allowing the system to benefit from both the resource efficiency of probabilistic structures and the reliability of definitive determination.
2Reliability
If large data structures are used to store complete sets of identifiers, then definitive membership testing is enabled, but distribution and real-time access become difficult
Solution Approach 1:
The patent extracts the essential membership verification capability from the complete data structure and implements it as a separate, lightweight verification mechanism. This extraction allows the system to maintain definitive membership testing without requiring distribution of the entire large data structure, thereby improving ease of operation for distribution and real-time access while preserving reliability.
Solution Approach 2:
The patent performs preliminary filtering using a compact probabilistic data structure before requiring access to the complete data set. This preliminary action eliminates many non-member queries early, reducing the need for real-time access to large distributed structures and making the system more operationally efficient while maintaining definitive testing capability.
3Quantity of substance
If probabilistic filtering is used to reduce data structure size, then distribution efficiency improves, but false positives increase
Solution Approach 1:
The patent introduces an intermediary verification step between the probabilistic filter and the final membership determination. This intermediary mechanism checks the results of the probabilistic filter and eliminates false positives by providing definitive verification, allowing the system to use compact data structures while maintaining high measurement precision.
Solution Approach 2:
The patent implements a feedback mechanism where the results of probabilistic filtering are validated against additional criteria or verification processes. This feedback loop identifies and corrects false positives, allowing the system to maintain small data structure sizes for efficient distribution while keeping the false positive rate low through iterative verification.
Data Source
AI summary
In certain embodiments, a method includes recursively performing a procedure that includes using an allowed set of object identifiers and a hash function to update a bit array, using a disallowed set of object identifiers and the hash function to further update the bit array where collisions occur, repeating the process with a new allowed set that includes object identifiers from the original allowed set that collided with the disallowed set and a new hash function, until reaching a round where no collisions occurred, generating a data structure that includes the bit arrays created during each recursive round, and compressing the data structure.