LSM-Tree Block Allocation by Predicted Data Expiration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face high Write Amplification due to the migration of a large number of valid pages during Garbage Collection (GC) in block storage devices like SSDs, which degrades GC performance.
Innovation Solution
A method and device that predict the expected expiration time of data files based on their attribute information and allocate them to storage blocks with similar expiration times, reducing the migration of valid pages during GC.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If data is stored into blocks without considering expected expiration time, then storage simplicity is maintained, but Write Amplification increases during GC
Solution Approach 1:
The patent predicts the expected expiration time of data files before they are actually expired, and pre-allocates storage blocks based on this prediction. This preliminary action allows the system to proactively organize data by expiration time, so that when GC occurs, data with similar expiration times are already grouped together, reducing the need to migrate valid pages and thereby reducing Write Amplification.
2Ease of manufacture
If data is stored into blocks without considering expected expiration time, then storage allocation simplicity is maintained, but GC performance deteriorates
Solution Approach 1:
The system performs preliminary prediction of expiration times and pre-organizes data into blocks based on predicted expiration times before GC is triggered. This preliminary organization ensures that when GC executes, the data is already optimally arranged, improving GC performance without adding complex real-time decision-making during the GC process itself.
Solution Approach 2:
The patent introduces dynamic block allocation based on predicted expiration times. Instead of static block assignment, the system dynamically selects blocks that match the predicted expiration characteristics of the data, allowing the storage system to adapt to different data lifecycles and optimize GC performance for varying workloads.
3Reliability
If more valid pages are migrated during GC, then garbage data is effectively collected, but Write Amplification increases
Solution Approach 1:
By predicting expiration times in advance and organizing data into blocks based on these predictions, the system ensures that when GC is triggered, the data in each block is already grouped by similar expiration times. This preliminary organization allows GC to efficiently identify and collect garbage data with minimal migration of valid pages, thereby maintaining GC effectiveness while reducing Write Amplification.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method for data storage includes determining an expected expiration time of a new data file based on attribute information of the new data file and recorded attribute information of data files, in response to creation of the new data file in a level of a Log-Structured Merge Tree (LSM-Tree). Additionally, a block set corresponding to the expected expiration time is allocated for the new data file in a storage device. The recorded attribute information of the data files indicates a level where each of the data files is located, key information of each of the data files, and lifetime information of each of the data files. The data files that have been created comprise unexpired data files or both unexpired data files, and expired data files that expired within a preset period before creation time of the new data file.