Log-Structured Merge Tree Compaction for High Density Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern distributed databases face challenges in managing large data volumes efficiently, leading to high storage costs and increased communication costs as additional nodes are added for horizontal scaling.
Innovation Solution
A write-optimized high data density key-value storage engine is implemented, combining log-structured merge trees (LSM Trees) and segmented log-structured object storage, which minimizes write amplification through separate storage of sequence numbers and document values, and performs scalable and incremental compaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If horizontal scaling is performed by adding more nodes to increase storage capacity, then storage capacity is improved, but storage cost and communication cost increase
Solution Approach 1:
The patent segments the storage system into multiple levels: hot data in memory, warm data in log-structured merge trees, and cold data in object storage. This segmentation allows efficient access to frequently accessed data while archiving less frequently accessed data, improving storage capacity utilization without requiring proportional increases in node count
Solution Approach 2:
The patent introduces a time-based dimension to storage by implementing log-structured merge trees that maintain historical versions of data over time. This allows the system to store temporal sequences of data efficiently, increasing storage capacity in the time dimension without adding spatial dimensions (nodes)
2Productivity
If log-structured merge tree is used for write-optimized storage, then write throughput is improved, but write amplification increases
Solution Approach 1:
The patent extracts the compaction operation from the write path by implementing background compaction threads that separately manage space reclamation. This allows write operations to proceed without waiting for compaction, maintaining high write throughput while reducing write amplification through asynchronous space management
Solution Approach 2:
The patent performs preliminary filtering of duplicate data before writing to disk by maintaining in-memory buffers that detect and eliminate duplicates. This preliminary action reduces the amount of data written to disk, thereby reducing write amplification while preserving write throughput
3Speed
If data is stored in log-structured object store with sequence numbers, then data retrieval efficiency is improved, but storage space is wasted due to stale document versions
Solution Approach 1:
The patent implements incremental compaction that identifies and discards stale document versions while recovering storage space. The system maintains a delete list that tracks stale sequence numbers, and during compaction, rewrites only the necessary data portions, efficiently reclaiming space without affecting retrieval efficiency of active documents
4Speed
If main memory size is increased to improve data access speed, then data access speed is improved, but system cost increases
Solution Approach 1:
The patent applies local quality by optimizing memory usage for specific access patterns. The system uses bloom filters in memory to quickly determine document existence without loading full documents, and uses sequence number indexing to efficiently locate documents. This selective optimization of memory structures improves data access speed without requiring proportional increases in main memory size
Data Source
AI summary
A system uses a hybrid key-value storage engine that uses log-structured merge tree and a segmented log-structured object store. The system performs garbage collection of stale document versions avoiding index lookup during log segment compaction. The system separates index and document data to minimize write amplification. The system maintains a delete list using a log-structured merge-tree to store stale document sequence numbers and corresponding sizes per log segment. For each log segment from the plurality of log segments, the system determines a measure of fragmentation of the log segment based on sizes of deleted documents of the log segment from the second log-structured merge-tree. If the fragmentation exceeds a threshold, the system initiates a compaction operation for the log segment.