Perfect Hash Index for Data Storage Resource Reclamation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face challenges in efficiently managing and reclaiming resources from unused data objects due to large index structures and the introduction of false positives when using probabilistic index structures like bloom filters, especially in deduplicating storage systems with millions of files and billions of data chunks.
Innovation Solution
A computer-implemented method and apparatus that utilize a perfect hash function to identify alive physical data objects by generating identifiers, maintaining a mapping between logical and physical objects, and iteratively performing hash operations to reclaim resources from dead objects, ensuring accurate resource reclamation without false positives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Volume of stationary object
If probabilistic index structures like bloom filters are used to reduce index size, then the index structure size is reduced, but false positives are introduced causing dead objects to be incorrectly identified as alive
Solution Approach 1:
The patent introduces a two-level index structure where a bloom filter serves as a first-level intermediary to quickly filter out clearly dead objects, and a perfect hash table serves as a second-level intermediary to accurately verify objects marked as potentially alive by the bloom filter. This intermediary approach allows the system to maintain small index structures while eliminating false positives through the secondary verification layer.
Solution Approach 2:
The patent segments the index structure into two distinct parts: a bloom filter component for initial filtering and a perfect hash table component for accurate verification. This segmentation allows each component to perform its specialized function efficiently - the bloom filter handles the majority of dead objects with minimal space, while the perfect hash table handles only the potentially alive objects with high accuracy.
2Measurement precision
If traditional index structures are used to accurately track all data objects, then measurement precision is maintained, but the index structure becomes very large and expensive to manage
Solution Approach 1:
The patent segments the index structure into two distinct parts: a bloom filter component for initial filtering and a perfect hash table component for accurate verification. This segmentation allows each component to perform its specialized function efficiently - the bloom filter handles the majority of dead objects with minimal space, while the perfect hash table handles only the potentially alive objects with high accuracy.
Solution Approach 2:
Instead of maintaining complete information about all objects in the index, the patent uses partial action by having the bloom filter only indicate potential liveness without guaranteeing it. The perfect hash table then provides exact information only for the subset of objects that the bloom filter identifies as potentially alive, avoiding the need to store complete information for all objects.
3Productivity
If resource reclamation is performed without accurate identification of dead objects, then productivity is improved through faster reclamation, but loss of information occurs when alive objects are incorrectly reclaimed
Solution Approach 1:
The patent performs preliminary action by using the bloom filter to pre-identify and filter out dead objects before the actual resource reclamation process. This preliminary filtering ensures that only objects confirmed to be dead by both the bloom filter and the perfect hash table are reclaimed, preventing accidental reclamation of alive objects while maintaining fast reclamation speed for confirmed dead objects.
Solution Approach 2:
The patent implements feedback through the two-level verification process where the perfect hash table provides feedback on the accuracy of the bloom filter's identification. When the bloom filter marks an object as potentially alive, the perfect hash table verifies this marking, and this feedback mechanism ensures that no alive objects are incorrectly reclaimed while maintaining high productivity for confirmed dead objects.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for managing data objects of a data storage system are described herein. According to one embodiment, a perfect hash function is generated for data objects stored in a data storage system. For each of the data objects, a hash operation is performed using the perfect hash function to indicate whether the respective data object is alive. Resources associated with the respective data object is reclaimed if it is determined that the respective data object is not alive based on a result of the hash operation using the perfect hash function, where the reclaimed resources are released back to the data storage system as free resources. Other methods and apparatuses are also described.