Weakly Synchronized Garbage Collection for Object Store Replicas
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face challenges in managing consistency and efficiency of object replicas across geographically remote locations due to the high cost and complexity of compaction processes, especially when dealing with aggregated stores and deleted objects, which leads to bitwise non-identical replicas and increased integrity check complexity.
Innovation Solution
The solution involves splitting objects into chunks, storing them in journals, and implementing a garbage collection and compaction scheme that maintains bitwise identicality of replicas by using metadata to track chunk locations and sizes, performing regional garbage collection, and scheduling compactions based on profitability and threshold criteria, with synchronization compactions ensuring replica consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If compaction is performed periodically to reclaim space occupied by deleted objects, then storage efficiency is improved, but operational cost and complexity increase due to rewriting all objects in the aggregated store
Solution Approach 1:
The patent divides objects into smaller chunks (e.g., 2MB, 4MB, or 8MB) that can be independently stored and managed in the aggregated store. This segmentation allows the system to perform compaction at the chunk level rather than requiring complete rewriting of entire objects, significantly reducing the operational cost and complexity of compaction operations while maintaining storage efficiency.
2Productivity
If each storage cluster performs independent scheduling of compaction, then resource usage is optimized, but replica consistency and integrity check complexity increase
Solution Approach 1:
The patent introduces a coordination mechanism that merges compaction scheduling decisions across multiple storage clusters. By using a coordinator to manage compaction operations and track compaction identifiers across replicas, the system maintains replica consistency while allowing each cluster to perform compaction operations. This coordination approach ensures that replicas remain in sync without requiring complex synchronization protocols.
Solution Approach 2:
The patent implements a feedback mechanism where storage clusters report their compaction status and resource usage to a coordinator. The coordinator uses this feedback information to make informed scheduling decisions, balancing the workload across clusters and ensuring replica consistency. This feedback loop allows the system to optimize resource usage while maintaining simple integrity checks through compaction identifiers.
3Measurement precision
If garbage collection joins the list of known live objects with directories of all data stores to find deleted objects, then accuracy of deleted object identification is improved, but operational cost increases due to opening all stores
Solution Approach 1:
The patent extracts the garbage collection process from a centralized approach that requires opening all data stores. Instead, it uses a distributed approach where each storage cluster independently identifies and removes deleted chunks locally. The coordinator maintains a list of live objects and distributes this information to clusters, which then perform garbage collection without needing to scan all stores, significantly reducing operational cost while maintaining identification accuracy.
Data Source
AI summary
Managing consistency of object replicas is performed at a first instance of a distributed storage system. The first instance performs garbage collection on a shard that includes a first plurality of object chunks, thereby removing a second plurality of object chunks from the shard. This leaves a third plurality of object chunks in the shard, where the first plurality of object chunks is the union of the second and third pluralities of object chunks. The first instance sends a first list of identifiers to a second instance of the distributed storage system. The second instance has a replica of the shard. The first list of identifiers specifies the object chunks in the third plurality of object chunks. The second instance removes all object chunks from the replica of the shard that are not included in the first list.