Distributed Garbage Collection for Dedupe Storage Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In a dedupe storage network, conflicts arise when garbage collection operations overlap with data uploads and downloads, leading to corruption of backup images due to the transparency of the garbage collection state machine to these processes, resulting in orphan and garbage chunks being incorrectly deleted or not recognized for repurposing.
Innovation Solution
A method is implemented where the garbage collection operation acquires a write lock for data chunks with a hardlink count of one, moves them to a temporary trash directory, and then recovers data chunks from backup images created after the garbage collection initiation, ensuring accurate deletion of garbage chunks and maintaining data integrity during replication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If garbage collection operation runs concurrently with upload and download operations, then system productivity is improved, but data reliability deteriorates due to backup image corruption
Solution Approach 1:
The system performs preliminary actions by recording the garbage collection initiation time before the GC operation begins. This timestamp is used later to identify and recover backup images created during the GC operation, preventing their corruption while allowing concurrent uploads and downloads to proceed
Solution Approach 2:
A temporary trash directory serves as an intermediary between the GC operation and the actual deletion process. Garbage chunks are moved to this temporary location first, allowing the system to verify whether they are truly garbage before permanent deletion, thus preventing accidental loss of chunks needed by concurrent operations
2Loss of substance
If garbage collection deletes all chunks in Eraser DB, then loss of substance is reduced, but loss of information increases due to deletion of valid chunks
Solution Approach 1:
The system records the GC initiation time as a preliminary action, which enables subsequent identification of backup images created during the GC operation. These images are recovered from the trash directory, ensuring that valid data chunks are not permanently deleted
Solution Approach 2:
The temporary trash directory acts as an intermediary buffer between identifying garbage chunks and permanently deleting them. This allows the system to hold chunks in a suspended state, verify their status, and recover them if needed, preventing irreversible loss of valid data
3Ease of operation
If garbage collection state machine is transparent to upload and download processes, then ease of operation is improved, but reliability deteriorates due to conflicts between GC and replication operations
Solution Approach 1:
The system records the GC initiation time before the operation begins, creating a reference point that allows post-operation identification of affected backup images. This preliminary action enables the system to maintain transparency while providing a mechanism to recover from conflicts
Solution Approach 2:
The conflict between GC and concurrent replication operations, which initially causes corruption, is converted into a benefit by using the GC initiation time to identify and recover affected backup images. The harm of potential corruption becomes the basis for a recovery mechanism that ensures data integrity
Data Source
AI summary
In the dedupe storage network priority to data replication needs to be given over the garbage collection (GC) activity. For this purpose GC running on any site should not impose any locking on its peer sites for replication. For replication activity GC activity must be completely transparent. In the dedupe storage network, the retention policy of a replicated image is controlled by the site where the image was originated. Distributed GC running on the originating site can only inform the replication sites the list of expired replicated images and cleanup of replicated garbage chunks from its remote FS database for corresponding replication sites. But it cannot cleanup garbage chunks from replication sites. Garbage chunks can only be cleaned by the local GC running on that site.