LSM Tree Batch Merge for Write I/O Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log-structured merge trees (LSM trees) face significant performance issues due to the time-consuming sequential scanning and writing required during merge operations, which disrupt user activities and incur substantial costs.
Innovation Solution
Implementing a batch-wise merge process where intermediate level trees are collectively merged into a base level tree, eliminating the need to create a new large tree and reducing the number of write I/O operations, thereby optimizing the merge process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sequential scanning and writing is performed to create a new larger tree during merge operations, then data integrity is maintained, but the operation time and write I/O operations increase significantly
Solution Approach 1:
The patent segments the merge operation into two distinct phases: an incremental update phase that modifies the base tree in-place with minimal I/O, and a final reconciliation phase that ensures data integrity. This segmentation allows the majority of the merge to proceed efficiently without full sequential scanning, resolving the contradiction between speed and reliability.
Solution Approach 2:
The patent performs preliminary actions by pre-processing and organizing data from intermediate trees before the final merge. Records are sorted, deduplicated, and prepared in advance, allowing the base tree to be updated incrementally rather than requiring a complete sequential scan and rewrite, thus reducing merge operation time while maintaining integrity.
2Reliability
If sequential scanning and writing is performed to create a new larger tree during merge operations, then complete data merging is achieved, but the number of write I/O operations increases significantly
Solution Approach 1:
The patent extracts and updates only the necessary records from intermediate trees into the base tree, rather than performing a complete sequential scan and rewrite of the entire tree structure. By extracting only the changed or new records and applying them incrementally to the base tree, the system achieves complete data merging with significantly reduced write I/O operations.
Solution Approach 2:
Instead of creating a completely new large tree through sequential writing, the patent copies and applies only the necessary updates to the existing base tree. This selective copying approach maintains complete data merging while avoiding the energy-intensive process of writing out an entire new tree structure.
3Reliability
If write locks are acquired during merge operations to ensure data consistency, then data consistency is maintained, but user activities are disrupted
Solution Approach 1:
The patent implements dynamic locking strategies where write locks are acquired only on specific portions of the base tree that are being updated, rather than locking the entire tree structure. This dynamic, fine-grained locking approach maintains data consistency for updated records while allowing concurrent read operations and user activities to continue on other parts of the tree, thus reducing disruption.
4Reliability
If a new large tree is created during merge operations to ensure data completeness, then data completeness is achieved, but the device complexity and resource requirements increase
Solution Approach 1:
The patent employs temporary data structures and intermediate representations that are created only when needed for the merge operation and then discarded once the incremental update is complete. These temporary objects ensure data completeness during the merge process but do not require the complexity of maintaining a permanent new large tree structure, thus reducing overall device complexity.
Data Source
AI summary
Disclosed herein is a technique for merging a batch of intermediate trees into a base tree stored in a storage device. The batch of intermediate trees is stored in a storage device and each intermediate tree in the batch represents a tree that was previously moved from memory to the storage device. When the number of intermediate trees in the batch satisfies a threshold, the batch of intermediate trees is collectively merged into the base tree within the storage device.