Garbage Collection Using Logical Clocks for Immutable Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing garbage collection techniques in file systems are inefficient for managing immutable data stores, particularly in commercial products, as they fail to effectively reclaim storage space and handle complex object graphs with extensive use of snapshots and deduplication.
Innovation Solution
A method involving a logical clock that monotonically increases, associating objects with numbers, and updating these numbers based on the logical clock and subtree relationships, along with a consensus store for reservation numbers, to efficiently delete unreferenced objects and manage storage resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional mark and collection garbage collection techniques are used, then unreferenced objects can be removed from the object store, but the system cannot effectively handle complex object graphs with extensive snapshots and deduplication, and storage space reclamation is inefficient
Solution Approach 1:
The patent segments the object graph into multiple independent trees, each with its own root and set of objects. This segmentation allows the garbage collector to process each tree separately using depth-first search, avoiding the complexity of analyzing the entire object graph at once. Each tree can be independently traversed and cleaned, making the garbage collection process scalable and efficient even for large datasets with extensive snapshots and deduplication.
Solution Approach 2:
The patent performs preliminary actions by maintaining version information and parent-child relationships for all objects before garbage collection begins. Each object stores its version number and references to parent objects, allowing the system to pre-compute which objects are reachable from which roots. This preliminary structuring of data enables the depth-first search algorithm to efficiently determine garbage objects without needing to perform complex analysis during the actual collection phase.
2Reliability
If snapshots and deduplication are extensively used to manage file system versions, then data integrity and version control are improved, but determining whether objects are still in use requires analysis of the full object graph, increasing processing complexity and time
Solution Approach 1:
By dividing the object graph into separate trees, the patent enables parallel or incremental analysis of each tree independently. This segmentation reduces the time required to analyze the full object graph, as the system can process multiple trees concurrently or focus on only the affected tree when a snapshot is created or modified. The version information stored with each object allows quick determination of which trees need analysis.
Solution Approach 2:
The patent applies partial action by analyzing only the specific tree or subset of trees that are affected by snapshot operations, rather than always analyzing the entire object graph. When a snapshot is created or an object is modified, the system identifies the affected tree and performs garbage collection analysis only on that tree, significantly reducing processing time while maintaining data integrity through version tracking.
3Ease of manufacture
If simple storage addition is used instead of garbage collection, then system simplicity is maintained, but storage space cannot be reclaimed for obsolete data, leading to inefficient resource utilization in commercial products
Solution Approach 1:
The patent implements self-service by enabling the storage system to automatically identify and remove obsolete objects through the garbage collection process. The system uses the version information and parent-child relationships stored with each object to autonomously determine which objects are no longer referenced by any snapshot or active data. This automatic self-management allows the system to reclaim storage space without requiring manual intervention or complex external management tools, maintaining ease of operation while improving storage utilization.
Data Source
AI summary
Provided are systems and methods for garbage collection of objects in storage. An example method may include providing a monotonically increasing logical clock. Each object is associated with a first number and a second number. The second number is a minimum of the first numbers of objects in a subtree to which the object refers. When the logical clock increases, objects with the first number less than the logical clock from the storage are deleted. When a new object is added to the storage, the first number of the new object is set to a new first number. The new first number is equal to or greater than the logical clock. The first number of each object in a subtree to which the new object is referring is updated. The updated first number is a function of a previous first number and a previous logical clock.