Storage Index File for Faster Reads in LSM-Tree Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Log-Structured Merge-Tree (LSM-tree) approach in storage systems sacrifices read performance for write efficiency, leading to time-consuming read operations due to numerous disk reads as data is written and compacted.
Innovation Solution
A data access method that constructs a quick read path using a second index structure, such as a hash structure, to maintain metadata correlations, allowing direct access to memory or disk based on the index file, reducing the need for level-by-level SSTable searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If LSM-tree is used to maximize write capability, then write performance is improved, but read performance deteriorates due to requiring large quantity of disk read actions
Solution Approach 1:
The patent segments the read path into two distinct paths: a quick read path for frequently accessed data and a traditional LSM-tree path for other data. This segmentation allows the system to optimize read performance for hot data without compromising the write-optimized LSM-tree structure, thereby resolving the contradiction between write capability and read speed.
Solution Approach 2:
The patent introduces an intermediary component (the quick read path with its own index structure) that mediates between the LSM-tree and the read access requests. This intermediary captures frequently accessed data and provides fast access to it, eliminating the need to traverse the entire LSM-tree for hot data reads, thus improving read speed while maintaining write performance.
2Productivity
If data is written into storage system using LSM-tree, then write capability is maximized, but read operation becomes time-consuming due to multiple disk reads
Solution Approach 1:
The patent performs preliminary action by pre-loading frequently accessed data into the quick read path's memory-based index structure. This preliminary action ensures that when read requests arrive for hot data, the system can immediately serve them from memory without performing time-consuming disk reads, thus reducing read operation time while maintaining the write-optimized LSM-tree structure.
Solution Approach 2:
The patent applies local quality by creating a specialized quick read path with different characteristics (memory-based, fast access) specifically for frequently accessed data, while the rest of the system continues to use the LSM-tree structure. This localized optimization reduces read operation time for hot data without affecting the overall write capability of the system.
3Device complexity
If conventional LSM-tree read path is used, then system structure remains simple, but read access overhead and delay increase
Solution Approach 1:
The patent introduces dynamics by making the read path adaptive and configurable. The system can dynamically select between the quick read path and the traditional LSM-tree path based on data access patterns and system state. This dynamic approach allows the system to optimize read access delay for hot data while maintaining a relatively simple overall structure that can adapt to different workloads.
Data Source
Figure 1~2
Figure 3
Figure 4A
AI summary
A data access method and a storage device are provided. In the method, the storage device receives a write access request, where the write access request includes a first key and to-be-written first data; writes the first key and the first data into a memory of the storage device based on a first index structure; and records a correspondence between the first key and first metadata in an index file based on a second index structure, where the first metadata indicates a storage address of the first data in the memory or a disk of the storage device, the index file is stored in the memory of the storage device, and the index file is used for reading data stored in the memory or the disk. This method helps to provide an index solution with excellent read-write performance, so as to reduce overheads and a delay of data read access.