Breadth-First Garbage Collection for Deduplicated Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In deduplicated file systems, reclaiming space is inefficient due to shared data segments, as conventional garbage collection methods, such as mark-and-sweep, are sensitive to metadata fragmentation and require repeated traversal of shared segments, leading to prolonged garbage collection times, especially with large numbers of small files.
Innovation Solution
Implementing a breadth-first garbage collection approach that traverses physical segments of the file system namespace level-by-level, using live and walk vectors to mark and copy alive segments, thereby avoiding repeated processing and reducing the impact of metadata fragmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a depth-first garbage collection approach is used to traverse file trees, then each file can be processed systematically, but the same shared segments are repeatedly traversed for multiple files, leading to prolonged garbage collection time
Solution Approach 1:
The patent segments the garbage collection process by separating the traversal of file metadata from the actual processing of data segments. It introduces a two-phase approach: first collecting all segment references from multiple files, then processing each unique segment only once. This segmentation eliminates redundant traversal of shared segments while maintaining systematic processing of all files.
Solution Approach 2:
The patent performs preliminary action by first gathering all segment references from the file system before actual garbage collection processing. It uses a reference counting mechanism that pre-identifies all segments referenced by any file, allowing the system to determine which segments are truly garbage (zero references) before initiating the costly sweep operation. This preliminary reference gathering prevents repeated traversal during the collection phase.
2Loss of substance
If conventional mark-and-sweep garbage collection is used, then space can be reclaimed for deleted files, but the process is highly sensitive to metadata fragmentation and the number of small files, causing garbage collection time to increase significantly
Solution Approach 1:
The patent uses copying to create a reference count for each segment by traversing all file metadata structures. Instead of immediately sweeping segments during the traversal, it copies reference information into a segment reference table, allowing multiple files to contribute their segment references without repeated processing. This copying mechanism decouples the metadata traversal from the actual segment evaluation, making the process insensitive to metadata fragmentation.
Solution Approach 2:
The patent changes the fundamental parameter of garbage collection from a file-centric approach to a segment-centric approach. Instead of processing files sequentially and sweeping segments immediately, it aggregates all segment references first, then processes segments based on their reference counts. This parameter change transforms the complexity from being proportional to the size of the file system namespace to being proportional only to the number of unique segments, eliminating sensitivity to the number of files and metadata fragmentation.
3Stability of the object's composition
If a file-by-file depth-first traversal is performed, then the garbage collection can follow the file system structure, but garbage collection time becomes proportional to the size of the logical space rather than the amount of metadata
Solution Approach 1:
The patent introduces a new dimension to the garbage collection process by adding a segment reference aggregation phase that operates independently of the file system hierarchy. Instead of moving strictly through the file tree structure, it collects segment references from all files simultaneously in a parallel aggregation step, then processes segments in a separate phase. This dimensional change allows the system to maintain awareness of file structure while achieving efficiency proportional to metadata size rather than logical space size.
Data Source
AI summary
A garbage collector of a storage system traverses a namespace of a file system of the storage system to identify segments that are alive in a breadth-first manner. The namespace includes information identifying files that are represented by segments arranged in a plurality of levels in a hierarchy, where an upper level segment includes one or more references to one or more lower level segments, and at least one segment is referenced by multiple files. All live segments of an upper level are identified before any of live segments of a lower level are identified. Upon all live segments of all levels have been identified, the live segments are copied from their original storage locations to a new storage location, and a storage space associated with the original storage locations is reclaimed.