Distributed Garbage Collection for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated storage systems, determining when data chunks are no longer referenced is complex and resource-intensive, particularly for large data sets, leading to storage fragmentation and wasted space due to dead or unreferenced data segments.
Innovation Solution
A distributed deduplication garbage collection process using strong hash functions and Bloom filters to identify live data segments, allowing for parallel processing and efficient identification of unreferenced data segments, which are then copied forward before deleting the original storage containers, thereby reclaiming storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If traditional garbage collection methods are used to determine unreferenced data chunks, then storage space can be reclaimed, but the process becomes complex and resource-intensive for large datasets
Solution Approach 1:
The patent divides the large dataset into multiple data containers, each independently processed by different compute nodes. This segmentation allows parallel garbage collection operations across multiple containers simultaneously, reducing overall process complexity and resource requirements while maintaining effective storage space reclamation
Solution Approach 2:
The patent introduces a distributed architecture dimension by deploying multiple compute nodes across different physical or virtual machines. This dimensional expansion enables parallel processing of garbage collection tasks that would be infeasible on a single system, transforming a resource-intensive sequential process into an efficient parallel operation
2Loss of substance
If traditional garbage collection methods are used to determine unreferenced data chunks, then storage space can be reclaimed, but the process consumes excessive computational resources
Solution Approach 1:
By segmenting the garbage collection task across multiple compute nodes, each node processes only a portion of the total data containers. This division of labor significantly reduces the computational resource consumption per node while collectively achieving comprehensive storage space reclamation across the entire distributed system
Solution Approach 2:
Each compute node independently manages its assigned data containers, performing local garbage collection operations without requiring centralized coordination for every decision. This self-service approach reduces overall computational overhead and resource consumption by enabling autonomous, distributed processing
3Measurement precision
If data containers are processed sequentially to identify live data segments, then accuracy is maintained, but processing speed is slow
Solution Approach 1:
The patent segments the data container processing task across multiple compute nodes, allowing parallel execution of accuracy-critical operations. Each node maintains precise live data identification for its assigned containers through independent processing, while the aggregate system achieves high throughput through concurrent operations
Solution Approach 2:
The patent merges the results from multiple parallel compute nodes to achieve both high speed and accuracy. By combining the individually processed data containers while maintaining reference counting integrity, the system achieves fast parallel processing without sacrificing the precision needed to correctly identify live versus unreferenced data segments
Data Source
AI summary
Embodiments relating to garbage collection for a deduplicated and compressed storage device are described. One embodiment provides for a method comprising creating a first set of temporary files associated with a range of fingerprints for data within data files associated with a directory tree structure; creating a second set of temporary files associated with a range of fingerprints of storage segments stored on one or more deduplicated storage containers; sorting the fingerprints in each temporary file using distributed out of core sorting across each node in the set of multiple computing device nodes to generate a first set of sorted files and a second set of sorted files; determining an intersection of the fingerprints in the first set of sorted files and the second set of sorted files; and generating a garbage collection recipe for each of the one or more deduplicated storage containers.