Distributed Deduplication Garbage Collection Using Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated storage systems, determining when data chunks are no longer referenced is complex and resource-intensive, particularly for large data sets, leading to storage fragmentation and wasted space due to dead or unreferenced data segments.
Innovation Solution
A distributed deduplication garbage collection process using strong hash functions and Bloom filters to identify live data segments, allowing for parallel processing and efficient identification of unreferenced data segments, which are then copied forward before deleting the original storage containers, thereby reclaiming storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional garbage collection methods are used to determine unreferenced data chunks, then storage space can be reclaimed, but the process becomes complex and resource-intensive for large data sets
Solution Approach 1:
The patent introduces Bloom filters as an intermediary data structure to simplify the garbage collection process. Instead of directly tracking all references to every data chunk, the system uses Bloom filters to probabilistically determine which chunks are unreferenced. This intermediary layer reduces the complexity of reference tracking while enabling effective storage space reclamation through parallel processing across multiple storage devices.
2Loss of substance
If traditional garbage collection methods are used to determine unreferenced data chunks, then storage space can be reclaimed, but the process consumes excessive computational resources for large data sets
Solution Approach 1:
The patent divides the garbage collection process into independent segments that can be executed in parallel across multiple storage devices. Each storage device independently processes its own data chunks using local Bloom filters, eliminating the need for centralized processing of entire data sets. This segmentation dramatically reduces computational resource consumption while maintaining effective storage space reclamation through distributed parallel execution.
3Quantity of substance
If all storage devices are dedicated to primary data storage, then storage capacity is maximized, but no resources are available for garbage collection operations
Solution Approach 1:
The patent enables storage devices to serve dual purposes: primary data storage and garbage collection processing. Each storage device is equipped with Bloom filter data structures that allow it to independently perform garbage collection operations on its own data while simultaneously maintaining primary storage functions. This multi-functionality eliminates the need for dedicated garbage collection hardware, maximizing storage capacity utilization while maintaining productive garbage collection operations.
Data Source
AI summary
Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a data storage system comprising an array of redundant storage devices including a first set of storage devices to be configured as live storage devices and a second set of storage devices to be configured as spare storage devices, a spare storage device to be enabled in event of a failure of a live storage device; and a set of processing devices coupled to the array of redundant storage devices, the set of processing devices to execute logic to enable data replication and deduplication for the array of redundant storage devices and perform distributed deduplication garbage collection on the first set of storage devices using one or more devices in the second set of storage devices as temporary storage.