Cloud Storage Shard Compaction by Predicted Lifespan
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems, such as cloud storage, face inefficiencies due to the mixing of data objects with varying lifespans within a single shard file, leading to increased storage and computational overheads as live data objects are repeatedly rewritten during garbage collection, which compacts short-lived data while affecting long-lived data.
Innovation Solution
The method involves predicting the lifespan of each data object and grouping it into shard files with associated predicted lifespan ranges, allowing for independent compaction of each shard file based on the predicted lifespans, thereby reducing unnecessary rewriting of live data and optimizing storage and computational resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of substance
If garbage collection is performed frequently to minimize storage overhead, then storage overhead is reduced, but computational overhead increases due to wasteful rewriting of live data objects
Solution Approach 1:
The patent segments data objects into different lifespan categories (short-lived, medium-lived, long-lived) and creates separate shard files for each category. This segmentation allows garbage collection to be performed selectively on short-lived object shards without affecting long-lived objects, thereby reducing unnecessary computational work while maintaining storage efficiency.
Solution Approach 2:
The patent applies different quality characteristics to different parts of the storage system by creating lifespan-specific shard files with tailored compaction policies. Short-lived object shards undergo frequent compaction while long-lived object shards maintain stability, optimizing the balance between storage overhead and computational overhead for each segment.
2Quantity of substance
If data objects of varying lifespans are grouped into the same shard file, then storage capacity utilization is improved, but long-lived data objects are repeatedly rewritten during compaction
Solution Approach 1:
The patent divides the storage system into multiple lifespan-specific shard files rather than using a single mixed shard file. This segmentation ensures that long-lived objects are isolated in dedicated shards that are not subject to frequent compaction, eliminating unnecessary rewrites while maintaining high storage capacity utilization through efficient packing of appropriately-sized objects into lifespan-matched shards.
Data Source
AI summary
A method includes receiving data objects, determining a predicted lifespan of each data object, and instantiating multiple shard files. Each shard file has an associated predicted lifespan range. The method also includes writing each data object into a corresponding shard file having the associated predicted lifespan range that includes the predicted lifespan of the respective data object and storing the shard files in a distributed system. The method also includes determining whether any stored shard files satisfy a compaction criteria based on a number of deleted data objects in each corresponding stored shard file. For each stored shard file satisfying the compaction criteria, the method also includes compacting the stored shard file by rewriting the remaining data objects of the stored shard file into a new shard file.