Deduplicated Storage Garbage Collection via Generation Numbers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated storage systems, particularly in cloud-based environments, deleting dead objects is complex due to the need to avoid impacting live objects, and conventional reference count methods are cumbersome and resource-intensive.
Innovation Solution
A system and method for garbage collection that uses a microservice or container-based application to identify and remove unreferenced segments without impacting live segments, employing a deletion bucket and worker nodes to manage the process, allowing for concurrent reads/writes and scalable performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reference count methods are used to track and delete dead objects in deduplicated storage systems, then object deletion can be achieved, but the system requires maintaining billions of counts which consumes significant storage and is very difficult to manage
Solution Approach 1:
The patent extracts the reference counting mechanism from the system and replaces it with a generation-based approach. Instead of tracking individual reference counts for billions of objects, the system uses generation numbers that are much simpler to maintain. When an object's generation number doesn't match the current generation, it is identified as dead and can be deleted without complex reference counting.
Solution Approach 2:
The patent changes the parameter used for tracking object liveness from individual reference counts to generation numbers. This parameter change transforms the problem from managing billions of individual counters to managing a single generation counter and associated metadata, dramatically reducing storage requirements and management complexity while maintaining deletion accuracy.
2Quantity of substance
If segments of dead objects are deleted immediately in deduplicated storage systems, then storage space is freed, but live objects may be impacted because the same segments may correspond to live objects
Solution Approach 1:
The patent performs preliminary identification of dead objects using generation number comparison before deletion. By checking whether an object's generation number matches the current generation, the system preliminarily determines which objects are dead and safe to delete, ensuring that live objects are not impacted while freeing storage space efficiently.
Solution Approach 2:
The patent introduces generation numbers as an intermediary mechanism between object creation and deletion. This intermediary allows the system to safely determine which segments can be deleted without directly tracking individual object references, thus protecting live objects while enabling storage reclamation.
3Productivity
If conventional garbage collection approaches are used in distributed and cloud based systems, then dead objects can be removed, but the process is cumbersome and resource-intensive due to the need to maintain and manage billions of reference counts
Solution Approach 1:
The patent changes the fundamental parameter for garbage collection from individual reference counts to generation numbers. This transformation reduces the computational resources needed for garbage collection from managing billions of counters to processing simple generation number comparisons, dramatically improving efficiency and reducing energy consumption in distributed and cloud-based systems.
Solution Approach 2:
The patent uses inexpensive generation number metadata instead of expensive reference count structures. Generation numbers are simple, disposable values that require minimal storage and computation, making garbage collection much more efficient and resource-friendly compared to maintaining complex reference counting systems.
Data Source
AI summary
Systems and methods for cleaning a storage system. A deduplicated storage system is cleaned by identifying structures that include dead or unreferenced segments. This includes processing recipes to identify the segments that are no longer part of a live object recipe. Then, the dead segments are removed. This is accomplished by copying forward the live segments and then deleting, as a whole, the structure that included the dead segments.