LSM-Tree Storage Segmentation for KV SSD Space Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The log-structured merge-tree (LSM-Tree) database storage architecture experiences severe space amplification due to append-only writing, leading to shortened SSD life and increased storage costs, while leveraging its fast writing performance and SSTable file storage.
Innovation Solution
A data access method that generates an index for key-value pairs, storing key-index pairs in a block storage device using the LSM-Tree structure and index-value pairs on a key-value solid state drive (KV SSD), allowing for sequential writing and reducing space amplification by avoiding in-place updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If append-only writing mechanism is used in LSM-Tree for block storage device, then writing performance is improved, but space amplification increases severely
Solution Approach 1:
The patent segments the storage system into two distinct components: a block storage device (HDD or block SSD) for storing key-index pairs using LSM-Tree structure, and a key-value SSD for storing index-value pairs. This segmentation allows each storage device to handle specific data types optimally, enabling the LSM-Tree to maintain its sequential writing advantage while the key-value SSD handles updates efficiently, thereby reducing space amplification.
Solution Approach 2:
The patent extracts the value storage function from the traditional LSM-Tree structure and places it in a separate key-value SSD. Only key-index pairs are stored in the block storage device using append-only writing, while index-value pairs are stored separately. This extraction eliminates the need for in-place updates in the block storage device, maintaining writing performance while reducing space amplification caused by invalid data retention.
2Productivity
If append-only writing is used instead of in-place update, then writing performance is improved, but service life of SSD is shortened
Solution Approach 1:
The patent divides the storage system into block storage device for key-index pairs and key-value SSD for index-value pairs. The block storage device uses append-only writing which is efficient but has limited write endurance, while the key-value SSD handles the update-intensive value storage. This segmentation protects the block storage device from excessive write operations, extending its service life while maintaining high writing performance.
Solution Approach 2:
The patent extracts the update operations from the block storage device by storing value data in a separate key-value SSD. Since key-value SSDs are optimized for handling updates and have better write endurance, this extraction allows the block storage device to use efficient append-only writing without suffering from shortened service life, as the high-write operations are offloaded to the key-value SSD.
3Productivity
If all writing operations are sequential append operations, then writing performance is improved, but data access complexity increases
Solution Approach 1:
The patent segments data access into two distinct phases: first, searching for the key in the LSM-Tree structure in the block storage device to obtain the index, and second, using the index to retrieve the value from the key-value SSD. This segmentation of access operations simplifies the writing process (maintaining sequential append efficiency) while providing a clear, two-step access mechanism that balances complexity with performance.
Data Source
AI summary
Provided is a data access method including generating an index corresponding to a key in a first key-value pair in response to receiving the first key-value pair, writing the key-index pair corresponding to the key and the index into a block storage device, the key-index pair being stored in the block storage device in a log-structured merge-tree (LSM-Tree) structure, writing an index-value pair corresponding to the index and the value in the first key-value pair into a key-value solid state drive (KV SSD).