LSM Tree File Segmentation for Storage Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In LSM tree-based data storage systems, the merging of upper-layer and lower-layer files often results in unnecessary data being merged, leading to severe read/write amplification and decreased storage performance.
Innovation Solution
The method involves dividing files stored in the LSM tree into sub-files based on data types, allowing for targeted merging of only relevant data types, thereby avoiding the passive participation of unnecessary data in the merging process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the lower-layer file is searched for files that include data of the same key as the upper-layer file for merging, then the merge operation can reduce duplicate data, but a large amount of unnecessary data is passively merged, causing severe read/write amplification
Solution Approach 1:
The patent segments the lower-layer file into multiple sub-files based on data types. Each sub-file contains data of a specific type, allowing the merge operation to selectively process only the relevant sub-files that contain matching keys, rather than scanning and merging all data in the entire lower-layer file. This segmentation eliminates unnecessary data processing and reduces read/write amplification.
2Reliability
If all data in the lower-layer file is merged with the upper-layer file, then complete data coverage is achieved, but read/write amplification increases significantly due to processing unnecessary data
Solution Approach 1:
The patent extracts only the necessary sub-files from the lower-layer file based on type matching with the upper-layer file. By extracting and processing only the relevant sub-files that contain data of the same type as the upper-layer file, the system achieves complete data coverage for merging while avoiding the energy waste of processing unrelated data, thus reducing read/write amplification.
Data Source
AI summary
This specification provides an LSM tree-based data storage method and a related device, applied to an LSM tree-based data storage system. The method includes: determining whether the first storage layer meets a merge condition for merging with the second storage layer, and if yes, selecting a to-be-merged target file from the at least one first file stored at the first storage layer, where the target file includes data corresponding to a target type; and searching the plurality of second sub-files for a target sub-file that includes data corresponding to the target type, and merging the target file and the target sub-file.