Log-Structured Storage Consolidation for Read Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Archival storage of versioned data objects, such as virtual machine disks, in cloud-based storage systems faces significant read amplification issues due to data fragmentation, leading to increased recovery times as new snapshots are added, which can significantly prolong the time required to read and restore the latest snapshot.
Innovation Solution
A distributed storage system employing a tiered structure with a log-structured format and content-addressable storage manages snapshots by breaking down storage objects into chunks, using a segment usage table to identify and consolidate live blocks, and periodically consolidating blocks for the latest snapshot into new log segments to reduce read amplification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If new snapshots are continuously added to cloud-based storage service, then storage capacity is increased, but read amplification increases and recovery time is significantly increased
Solution Approach 1:
The storage system segments snapshots into different tiers: hot snapshots (latest) are kept in local storage for fast access, while cold snapshots (older) are archived in cloud-based storage. This segmentation allows the system to maintain fast recovery times for recent snapshots while preserving unlimited storage capacity in the cloud for historical snapshots.
Solution Approach 2:
The patent extracts the latest snapshot data from the cloud-based storage service and places it in local storage infrastructure. This extraction eliminates read amplification issues for the most recently accessed snapshot while maintaining the ability to store unlimited snapshots in the cloud.
2Quantity of substance
If all snapshots are stored in cloud-based storage service, then storage costs are reduced, but read amplification occurs and access time increases
Solution Approach 1:
The system applies different storage qualities to different snapshots based on their age and access patterns. The latest snapshot receives high-quality local storage for fast access, while older snapshots use cost-effective cloud storage. This local quality differentiation resolves the contradiction between storage cost and access speed.
Solution Approach 2:
The patent introduces a temporal dimension to storage architecture by creating a two-tier storage hierarchy (local and cloud) that operates across different time scales. Recent snapshots are accessed from local storage while historical snapshots reside in cloud storage, optimizing both cost and performance across the time dimension.
3Adaptability or versatility
If latest snapshot is highly fragmented across multiple storage locations, then storage flexibility is improved, but more data needs to be read and recovery time increases
Solution Approach 1:
The system performs preliminary consolidation of the latest snapshot blocks into a single log segment before recovery operations. This preliminary action eliminates fragmentation for the most recently accessed snapshot, ensuring fast recovery while maintaining storage flexibility for older snapshots through the tiered architecture.
Data Source
AI summary
System and method for managing snapshots of storage objects in a storage system use a consolidation operation to reduce read amplification for stored snapshots of a storage object that are stored in log segments in the storage system according to a log-structured file system as storage service objects. The consolidation operation involves identifying target log segments among the log segments that include live blocks that are associated with the latest snapshot of the storage object and determining the number of the live blocks included in each of the target log segments. Based on the number of the live blocks in each of the target log segments, candidate consolidation log segments are determined from the target log segments. The live blocks in the candidate consolidation log segments are then consolidated to new log segments, which are uploaded to the storage system as new storage service objects.