Distributed Storage Space Reclamation via Parallel Copy-Forward
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing mark-and-sweep approach for identifying and copying forward live clumps in file systems is inefficient, particularly in large storage systems, as it consumes significant compute and memory resources and becomes increasingly time-consuming.
Innovation Solution
A distributed storage system is implemented with a job controller that breaks down the space reclamation process into phases, each with independent tasks that can be performed in parallel by multiple task workers, using clump fingerprints and container identifiers to efficiently identify and copy forward live clumps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the mark-and-sweep approach is used to identify and copy forward live clumps, then the space reclamation process is performed, but the process consumes significant compute and memory resources and becomes increasingly time-consuming in large storage systems
Solution Approach 1:
The patent divides the storage space into multiple containers and the reclamation process into multiple phases (identification phase and copy-forward phase). Task workers are assigned to process specific containers in parallel, breaking down the monolithic mark-and-sweep operation into smaller, manageable segments that can be executed concurrently, thereby improving productivity while reducing overall reclamation time.
2Productivity
If the mark-and-sweep approach is used, then live clumps are identified and copied forward, but the computational burden increases significantly in large storage systems
Solution Approach 1:
The patent segments the identification and copy-forward operations into separate phases that can be performed by different task workers on different containers simultaneously. This segmentation allows the system to process large storage spaces without proportionally increasing computational burden on any single worker, improving overall reclamation efficiency while distributing compute resources more effectively.
Solution Approach 2:
The patent performs preliminary identification of live clumps and their container assignments before the actual copy-forward operation. By pre-computing which clumps are live and which containers they belong to, the system reduces the computational burden during the execution phase, as task workers simply need to copy data rather than perform identification calculations, thereby improving reclamation efficiency while reducing peak compute resource usage.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Techniques for copying forward live data within storage regions used by a file system to new storage regions are provided. Storage regions refer to ranges of persistent storage within a storage medium. One or more live-clump-identifying workers identify a set of live-clump-references. The live-clump references are references to live clumps that belong to live objects. Live clumps represent a unit of data. After identifying the set of live-clump-references, one or more container-identifying workers identify a set of storage regions that contain clumps that correspond to the set of live-clump references. After identifying the set of storage regions corresponding to the set of live-clump-references, a plurality of copy-forward task workers perform copy-forward operations on the set of storage regions to copy live data to new storage regions.