Snapshot Garbage Collection for Cloud Storage BLOBs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cloud-based data storage systems face performance issues due to the need for 'stop-the-world' disruptions to prevent data loss during garbage collection, especially in geographically distributed setups, where orphaned BLOBs can become referenced again, leading to prolonged write operations and impaired storage performance.
Innovation Solution
Implementing a snapshot-based garbage collection process that traverses object reference tables to identify and delete orphaned BLOBs, with background processing to restore BLOBs referenced during the traversal and reclaim capacity without halting incoming data traffic, thereby eliminating write barriers and synchronizing garbage collection with data writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a write barrier is implemented to update all referenced BLOBs before client acknowledgement, then data loss is prevented, but storage performance severely degrades due to prolonged write operations
Solution Approach 1:
The system performs preliminary actions by creating a snapshot of the object reference table before garbage collection, and pre-updating the reference count in system metadata. This allows the garbage collection process to proceed without requiring time-consuming updates to all referenced BLOBs, thus maintaining both data integrity and performance
Solution Approach 2:
The patent introduces an intermediary mechanism using snapshots and reference counting. Instead of directly updating all BLOB references (which causes performance degradation), the system uses a snapshot as an intermediary to track changes and a reference count as a mediator to determine when updates are actually necessary, thereby resolving the contradiction between reliability and performance
2Reliability
If a stop-the-world mode is implemented to halt incoming data traffic, then garbage collection can proceed without synchronization complexity, but storage performance is severely impacted due to disrupted data writes
Solution Approach 1:
The system enables continuous data writes during garbage collection by using snapshots to capture the state at the beginning of GC and comparing it with the current state. This allows the useful action of data writing to continue uninterrupted while garbage collection proceeds in the background, resolving the contradiction between safety and throughput
Solution Approach 2:
The patent implements a dynamic approach where the garbage collection process adapts to ongoing data writes by using snapshots and reference counting. Instead of statically halting all writes, the system dynamically determines which BLOBs can be safely collected based on reference count changes, allowing flexible continuation of data operations while maintaining GC safety
3Reliability
If background processing is used to restore BLOBs referenced during garbage collection traversal, then data integrity is maintained, but capacity reclamation experience short delays
Solution Approach 1:
The system performs preliminary actions by creating snapshots before garbage collection and pre-calculating reference counts. This allows background processing to quickly determine whether BLOBs can be safely collected without extensive verification, reducing the time delay while maintaining data integrity through the snapshot verification mechanism
Data Source
AI summary
The technology describes performing garbage collection while data writes are occurring, which can lead to a conflict in that a new reference to an otherwise non-referenced candidate object for garbage collection is written after the non-referenced candidate object is detected. In one example implementation, orphaned binary large objects (BLOBs) that are not referenced by a descriptor file and are beyond a certain age are detected and deleted via an object references table traversal as part of garbage collection. Before reclaiming a deleted BLOB's capacity, a background process operates to restore the deleted BLOB if a new descriptor file reference to the BLOB was written during the object references table traversal. Capacity is only reclaimed after the object references table traversal and the background processing completes, for those BLOBs that were deleted and had not been restored.