Probabilistic Key Management via Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face inefficiencies in managing and communicating large numbers of keys, leading to increased bandwidth and computing resource consumption, and delayed or inefficient recovery of storage memory due to unawareness of deleted keys across different parts of the system.
Innovation Solution
The implementation of probabilistic data structures, such as Bloom filters, to determine key membership and identify candidates for deletion, allowing for efficient communication and resource recovery by generating a compact representation of key sets and using key selectors to filter keys before generation and testing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If traditional key management methods are used to track and communicate key status across storage systems, then key tracking accuracy is maintained, but communication bandwidth and computing resources are excessively consumed
Solution Approach 1:
The patent creates a probabilistic copy (Bloom filter) of the key set instead of managing and communicating the actual keys. This filter structure allows other storage nodes to check key membership without receiving or processing the full key list, dramatically reducing communication bandwidth while enabling efficient key tracking and deletion decisions
Solution Approach 2:
The patent changes the representation parameter of the key set from exact key values to a probabilistic data structure with configurable false positive rates. By adjusting the Bloom filter parameters (number of hash functions, filter size), the system can balance between memory usage and key tracking accuracy, resolving the contradiction between resource consumption and tracking precision
2Reliability
If complete key lists are communicated across the storage system to ensure all nodes are aware of key deletions, then system consistency is maintained, but communication overhead and resource consumption increase significantly
Solution Approach 1:
The patent extracts the essential information needed for key tracking (key membership) from the complete key list and embeds it in a compact Bloom filter. This allows storage nodes to determine whether keys should be deleted without receiving or processing the entire key list, reducing communication overhead while maintaining system consistency through probabilistic membership testing
3Productivity
If probabilistic data structures are used to reduce communication overhead, then resource efficiency improves, but there is a risk of false positives affecting key deletion accuracy
Solution Approach 1:
The patent applies partial action by using probabilistic matching instead of exact matching for key deletion decisions. The Bloom filter may produce false positives (indicating a key might exist when it doesn't), but never false negatives. This allows the system to efficiently identify candidate keys for deletion with acceptable accuracy, balancing resource efficiency and deletion precision
Solution Approach 2:
The system uses feedback from Bloom filter queries to guide key deletion decisions. When a key is not found in the Bloom filter (definitive negative), the system confidently deletes the key. When a false positive occurs, the system can verify the key's actual presence before deletion, correcting the probabilistic error and maintaining overall deletion accuracy
Data Source
AI summary
A method for deleting a set of keys from a storage server is provided. The method includes generating a probabilistic data structure for a first set of keys and for each key in a second set of keys, determining whether a key of the second set of keys is found in the probabilistic data structure. The method includes identifying the key as a candidate for deletion if the key is not found in the probabilistic data structure. A system is also provided.