In-Storage SST File Search to Reduce LSM-Tree Read Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing LSM-Tree based data storage systems suffer from high read amplification, leading to inefficient read operations, increased CPU occupancy, and power consumption due to the need to load large amounts of data from storage to the host for search operations.
Innovation Solution
Implementing a storage apparatus with a computing unit, such as an FPGA, to perform binary search and bloom filter operations directly on stored SST files, reducing the need to move data to the host by offloading the search process within the storage apparatus.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is loaded from storage to host memory for search operations in LSM-Tree based databases, then search functionality is achieved, but read amplification increases and search efficiency decreases
Solution Approach 1:
The patent combines the storage apparatus and computing unit into an integrated system where the computing unit resides within the storage apparatus. This merging allows search operations to be performed on data while it remains in storage, eliminating the need to load entire files to host memory and thereby reducing read amplification while maintaining search functionality.
Solution Approach 2:
The computing unit acts as an intermediary between the storage apparatus and the host. Instead of the host directly loading and searching data files, the computing unit receives search requests from the host, loads only necessary data blocks into its internal memory, performs searches, and returns results. This intermediary approach significantly reduces the data transfer burden on the host.
2Reliability
If large amounts of data are moved from storage to host for search operations, then complete search capability is provided, but CPU occupancy rate increases
Solution Approach 1:
The patent extracts the search computation function from the host CPU and relocates it to a dedicated computing unit within the storage apparatus. This extraction allows the host CPU to avoid the intensive computational workload of searching large data files, thereby reducing CPU occupancy rate while preserving complete search capability through the computing unit's binary search and bloom filter implementations.
3Reliability
If data files are loaded to host memory for searching, then search operations can be performed, but power consumption increases
Solution Approach 1:
Instead of loading entire data files to host memory for searching, the computing unit loads only the necessary data blocks into its internal memory based on the search request. This partial action approach loads minimal data (only what is needed for the specific search), significantly reducing the energy required for data transfer and processing while maintaining full search operation capability.
4Reliability
If search operations are performed on SST files stored in hierarchical form on disk, then data can be searched, but read amplification increases due to large data movement
Solution Approach 1:
The patent replaces the mechanical data movement process (loading files from disk to host memory) with a computational approach. The computing unit within the storage apparatus performs searches directly on data blocks loaded into its internal memory using binary search and bloom filters, substituting the physical data transfer mechanism with an in-storage computational mechanism that eliminates unnecessary data movement.
Data Source
AI summary
A method of data reading for a storage device, including: receiving, by the storage device, a search request from a host, wherein the search request includes an address of a file to be searched; loading the file into a computing unit included in the storage device based on the address; obtaining a search result by searching, by the storage device, for data to be read in the file; and returning the search result to the host.