Dynamic Garbage Collection for Distributed Storage Bins
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage architectures, garbage collection and bin synchronization are complex and resource-intensive due to the need to identify and free unused blocks across multiple worker nodes, which can lead to data loss and interfere with client I/O processing, especially as the scale of the storage increases.
Innovation Solution
Implementing a dynamic garbage collection process that selectively processes subsets of bins based on heuristics, such as fullness and staleness of probabilistic structures, and disjoining the generation of probabilistic structures from the garbage collection process to ensure up-to-date information, thereby reducing the false positive rate and improving scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If garbage collection processes all bins to free unused blocks, then storage efficiency is improved, but client I/O processing is interfered with and latency spikes occur
Solution Approach 1:
The patent divides the storage system into multiple worker nodes, each managing a subset of bins. Garbage collection is performed independently on each worker node rather than system-wide, allowing parallel processing and reducing interference with client I/O operations. This segmentation enables the system to maintain storage efficiency while avoiding system-wide latency spikes.
Solution Approach 2:
The patent implements probabilistic structures (such as Bloom filters) that provide approximate rather than exact information about block usage. This partial action approach allows garbage collection to make decisions based on probabilistic data, reducing the need to examine every block thoroughly and thereby decreasing processing time and interference with client I/O operations.
2Reliability
If bin synchronization copies all blocks to ensure data redundancy, then data reliability is improved, but network bandwidth and resources are wasted on copying unused blocks
Solution Approach 1:
The patent performs garbage collection before bin synchronization operations. By proactively identifying and freeing unused blocks prior to synchronization, the system ensures that only currently valid and needed blocks are copied to destination bins. This preliminary action prevents wasting network bandwidth on duplicating obsolete or unused data, thereby improving resource efficiency while maintaining data redundancy.
Solution Approach 2:
The patent uses probabilistic structures that are continuously updated with feedback about block usage patterns. This feedback mechanism allows the system to identify which blocks are actually in use versus unused, enabling intelligent decisions about which blocks should be synchronized. The feedback loop ensures that synchronization operations target only relevant blocks, reducing unnecessary network traffic.
3Measurement precision
If probabilistic structures are generated continuously to track block usage, then garbage collection accuracy is improved, but resource consumption increases
Solution Approach 1:
The patent implements probabilistic structures with configurable parameters such as false positive rates and data structures like Bloom filters with adjustable hash functions and bit array sizes. By changing these parameters, the system can balance between accuracy of block usage identification and resource consumption. This allows optimization based on specific workload requirements, maintaining sufficient accuracy while controlling computational overhead.
Data Source
AI summary
Techniques are provided for implementing garbage collection and bin synchronization for a distributed storage architecture of worker nodes managing distributed storage composed of bins of blocks. As the distributed storage architecture scales out to accommodate more storage and worker nodes, garbage collection used to free unused blocks becomes unmanageable and slow. Accordingly garbage collection is improved by utilizing heuristics to dynamically speed up or down garbage collection and set sizes for subsets of a bin to process instead of the entire bin. This ensures that garbage collection does not use stale information about what blocks are in-use, and ensures garbage collection does not unduly impact client I/O processing or conversely falls behind on garbage collection. Garbage collection can be incorporated into a bin sync process to improve the efficiency of the bin sync process so that unused blocks are not needlessly copied by the bin sync process.