In-Place Garbage Collection for Distributed Shared Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
State machine replication (SMR) systems face storage space exhaustion due to the ever-growing shared log without effective garbage collection, leading to high write amplification and performance impacts from inefficient checkpointing and trimming processes.
Innovation Solution
Implementing a generic in-place garbage collection mechanism that identifies obsolete log entries at the client level and compacts them within the shared log servers, reducing write amplification and improving system performance by leveraging fine-grained garbage information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If checkpointing and trimming are used to reclaim storage space, then storage space is recovered, but write amplification increases and system performance deteriorates
Solution Approach 1:
The system performs preliminary identification of obsolete log entries at the client level before compaction occurs. Clients track and send garbage information to servers in advance, allowing servers to prepare for efficient in-place compaction without triggering performance-degrading checkpointing operations
Solution Approach 2:
The patent extracts the garbage collection function from the traditional checkpointing process. By separating obsolete entry identification (done at client level) from compaction (done at server level), the system avoids the write amplification associated with full checkpointing while still reclaiming storage space
2Reliability
If the shared log grows without garbage collection, then all update operations are preserved for fault tolerance, but storage space is exhausted
Solution Approach 1:
The system applies different quality requirements to different parts of the log. Recent log entries are preserved with full redundancy for fault tolerance, while older entries that have been superseded by newer versions are identified as obsolete and eligible for removal. This local differentiation allows selective garbage collection that maintains reliability where needed while freeing space where possible
Solution Approach 2:
The patent implements a mechanism to discard obsolete log entries (those superseded by newer versions of the same operation) while recovering storage space. The system tracks which entries are still needed for fault tolerance and safely removes only those that are redundant, balancing space reclamation with reliability requirements
3Quantity of substance
If inefficient checkpointing is used for garbage collection, then storage space is reclaimed, but CPU cycles and I/O bandwidth are excessively consumed
Solution Approach 1:
The garbage collection process is segmented into distinct phases performed by different system components. Clients perform segmentation by identifying and sending garbage information for specific obsolete entries, while servers perform compaction in-place. This division eliminates the need for full-system checkpointing operations that consume excessive CPU and I/O resources
Solution Approach 2:
The patent replaces the mechanical checkpointing process (which requires rewriting large portions of the log to storage) with a more efficient in-place compaction mechanism. By substituting the traditional checkpointing mechanism with fine-grained garbage information processing, the system achieves space reclamation with minimal CPU and I/O overhead
Data Source
AI summary
Garbage collection processing in a distributed shared log system includes a client identifying obsoleted log entries for a shared data object. The client sends information associated with the identified obsoleted log entries to a shared log server. The shared log server receives information associated with obsoleted log entries from all clients in the distributed shared log system and uses the information to delete the obsoleted log entries. The shared log server can update a snapshot mark to indicate the earliest time that a snapshot of the shared log can be taken. The snapshot mark can be updated based on the information associated with obsoleted log entries.