Reference Tracking Garbage Collection for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Geographically distributed storage systems face challenges in garbage collection, particularly during temporary zone outages and ensuring data integrity, as BLOBs can become orphans due to concurrent CDF deletes and writes, leading to potential data loss and severe storage performance impacts.
Innovation Solution
A two-stage deletion process for orphan BLOBs, where a BLOB is first demoted to a garbage collection candidate state and then deleted after ensuring it has not obtained new references, ensuring data integrity and efficient CDF creation without foreground updates across multiple zones.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If BLOBs are deleted immediately when references are removed, then storage space is reclaimed quickly, but data loss may occur during temporary zone outages or concurrent operations
Solution Approach 1:
The system performs preliminary actions by marking BLOBs as garbage collection candidates before actual deletion. This preliminary marking allows the system to prepare for deletion while maintaining a safety mechanism to prevent actual deletion until it is certain no new references will be created during zone outages or concurrent operations.
Solution Approach 2:
The system provides beforehand cushioning by implementing a protective mechanism that prevents premature deletion of BLOBs. The garbage collection candidate state acts as a cushion that protects against data loss while still enabling eventual reclamation of storage space, balancing the need for space reclamation with data protection during uncertain conditions.
2Reliability
If the system waits to confirm no new references before deleting BLOBs, then data loss is prevented, but storage performance deteriorates due to delayed deletion
Solution Approach 1:
The system performs the reference verification action preliminarily by checking for new references before final deletion. This preliminary check ensures data integrity while allowing the deletion process to proceed efficiently once verification is complete, rather than delaying the entire process.
Solution Approach 2:
The deletion process is segmented into distinct phases: marking as garbage collection candidate, verifying no new references, and final deletion. This segmentation allows each phase to be optimized independently, with the verification phase ensuring reliability and the final deletion phase maximizing productivity.
3Reliability
If BLOBs are marked as garbage collection candidates across all zones, then data integrity is ensured during zone outages, but system complexity increases due to replication overhead
Solution Approach 1:
The garbage collection candidate state serves multiple functions: it marks BLOBs for potential deletion, propagates deletion intent across all zones during replication, and provides a verification mechanism before final deletion. This multi-functionality reduces the need for separate mechanisms in each zone, thereby managing complexity while ensuring data integrity.
Solution Approach 2:
The system implements feedback by monitoring whether BLOBs in garbage collection candidate state obtain new references before deletion. This feedback mechanism ensures that deletion only occurs when safe, providing reliability while the centralized management of the candidate state helps control system complexity through coordinated replication.
Data Source
AI summary
The described technology is generally directed towards reference tracking garbage collection, including for operating in a geographically distributed storage system. Content Addressed Storage (CAS) data objects comprising BLOBs (Binary Large Objects) are referenced by data objects comprising C-Clip Descriptor Files (CDFs). BLOBs having no remaining reference, referred to as orphan BLOBs, are deleted in a two-stage deletion operation that ensures that no new reference has been created. An alive BLOB that became an orphan is demoted to a garbage collection candidate with limited access to the BLOB. The BLOB is only deleted if the garbage collection candidate status of the BLOB is distributed across geographically distributed zones, and there is no new reference to the BLOB.