LSM Tree MacroBlock Storage for Disk IO Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log Structured Merge Tree (LSM Tree) based data storage systems face inefficiencies in query response times and disk IO due to the need for rewriting all files during inter-layer data merger, leading to high overhead and space wastage.
Innovation Solution
Implementing an N-layer data structure where each layer has decreasing storage space, using MacroBlocks and MicroBlocks to organize data, allowing for efficient merging without rewriting entire layers and optimizing storage space usage by compressing data into variable-length MicroBlocks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all files on a lower layer are rewritten during inter-layer data merger, then data consistency across layers is maintained, but disk IO overhead increases and storage efficiency decreases
Solution Approach 1:
The patent extracts only the necessary data blocks that need to be merged from the lower layer, rather than rewriting all files. The merge operation identifies and transfers only the minimal set of blocks required to maintain data consistency, significantly reducing disk IO overhead while preserving reliability.
Solution Approach 2:
The patent segments the data storage into hierarchical layers with different block sizes (MacroBlocks in lower layers, MicroBlocks in upper layers). This segmentation allows selective merging of specific blocks rather than entire files, reducing the scope of rewriting operations during inter-layer data merger.
2Ease of manufacture
If fixed-size data blocks are used in LSM Tree layers, then storage allocation is simplified, but storage space wastage increases
Solution Approach 1:
The patent applies local quality by using different block sizes for different layers and positions within the LSM Tree. Lower layers use larger MacroBlocks for efficient sequential writes, while upper layers use smaller MicroBlocks for precise space utilization. This variable block size strategy reduces storage wastage while maintaining allocation simplicity through hierarchical organization.
Solution Approach 2:
The patent changes the block size parameter dynamically based on the layer position in the LSM Tree. Each layer is configured with optimized block sizes that balance allocation simplicity and space efficiency, transforming the fixed-parameter approach into a variable-parameter system that adapts to different storage scenarios.
3Productivity
If multi-layer data structure is used in LSM Tree, then write performance is improved, but query response time deteriorates in some cases
Solution Approach 1:
The patent performs preliminary actions by pre-sorting and pre-organizing data blocks during the write phase in lower layers. Data is written in sorted order and organized into MacroBlocks with predictable structures, which enables faster query operations by reducing the need for random access and minimizing the search space in upper layers.
Solution Approach 2:
The patent implements a nested block structure where MicroBlocks are contained within MacroBlocks, and layers are nested hierarchically. This nesting allows queries to first search smaller MicroBlocks in upper layers for quick hits, and only descend to larger MacroBlocks in lower layers when necessary, improving query response time while maintaining the multi-layer write performance benefits.
Data Source
AI summary
Embodiments provide a data block storage method applied to a database employing an LSM tree. According to the method, data in the 0th layer of the LSM tree is organized in MacroBlocks. When the data in the LSM tree is transferred from an upper layer to a lower layer, the data is transferred to the lower layer in the form of MacroBlocks, so that all layers of the LSM tree are organized in the form of MacroBlocks. When inter-layer data merger is performed, the merger may be performed based on the MacroBlocks, which can avoid rewriting of all data on an entire layer and can effectively reduce disk IO and temporary space consumption. Moreover, as each MacroBlock comprises a plurality of variable-length MicroBlocks, the data may be stored in a storage space in a more compact manner.