LSM Tree SST File Compression via SSD Intermediary
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log structured merge (LSM) trees face inefficiencies in data compression and merging due to the gap between sequential and random read/write performance of storage devices, which affects the overall compression efficiency.
Innovation Solution
The method involves selecting SST files for compression and merging, sequentially reading and writing them from a slow storage device to a fast storage device, and then randomly reading them into memory based on key sequence numbers, utilizing the higher random read/write performance of the fast storage device, while updating a mapping table for quick data location.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If SST files are randomly read from a slow storage device (e.g., magnetic disk) during compression, then data can be accessed in any order, but the read speed is slow due to the sequential nature of magnetic disk access
Solution Approach 1:
The patent introduces a fast storage device (SSD) as an intermediary between the slow magnetic disk and the memory. SST files are first sequentially read from the magnetic disk to the SSD, then randomly read from the SSD to memory during compression. The SSD acts as a buffer that converts slow random access into fast sequential access, resolving the contradiction between read speed and random access capability.
Solution Approach 2:
The patent performs preliminary action by sequentially reading SST files from the magnetic disk to the SSD before the actual compression process. This pre-positioning of data in the fast storage device enables subsequent random reads during compression to be performed at high speed, eliminating the need for slow random access directly from the magnetic disk.
2Speed
If SST files are sequentially read from slow storage device during compression, then read speed is improved, but random access capability is lost
Solution Approach 1:
The patent segments the data access process into two distinct phases: a preliminary phase where SST files are sequentially read from magnetic disk to SSD, and a compression phase where data is randomly read from SSD to memory. This segmentation allows each phase to utilize the most appropriate access pattern for its specific requirements, maintaining both sequential read speed and random access flexibility.
3Productivity
If compression is performed directly on SST files from magnetic disk, then storage space is utilized efficiently, but compression efficiency is low due to slow random read performance
Solution Approach 1:
The SSD serves as an intermediary storage device that enables efficient compression operations. By pre-loading SST files into the SSD from the magnetic disk, the system can perform random reads during compression at high speed from the SSD, dramatically improving compression efficiency while maintaining efficient use of the magnetic disk's large storage capacity.
Solution Approach 2:
The patent performs preliminary action by copying SST files from the magnetic disk to the SSD before compression begins. This pre-positioning of data in fast storage enables the compression process to access data randomly at high speed, resolving the contradiction between compression efficiency and random read speed.
4Speed
If fast storage device (SSD) is used for all operations, then read/write speed is improved, but storage cost increases
Solution Approach 1:
The patent applies local quality by using different storage devices for different purposes: the magnetic disk is used for storing large volumes of SST files where capacity is more important than speed, while the SSD is used for the smaller subset of data being actively compressed where speed is critical. This localized optimization resolves the contradiction between speed and storage capacity by matching each storage device's strengths to its specific use case.
Data Source
AI summary
A data processing method for a log structured merge (LSM) tree includes selecting SST files to be compressed and merged in a current layer and a next layer, sequentially reading the SST files to be compressed and merged in the current layer and the next layer from a first storage device and sequentially writing the SST files in a second storage device, randomly reading the SST files to be compressed and merged from the second storage device into a memory according to key sequence numbers of data blocks included in the SST files to be compressed and merged, and performing compression and merge processing on the SST files to be compressed and merged. Sequential and random read and write speed of the second storage device is higher than that of the first storage device.