High-Density Document Storage Compaction with Split LSM Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern distributed databases face challenges in storage efficiency and cost-effectiveness as they scale out to handle large data volumes, leading to increased communication costs and write amplification due to frequent writes.
Innovation Solution
A high-density data storage system using a hybrid key-value storage engine with log-structured merge trees and segmented log-structured object storage, which separates index and document data to minimize write amplification and performs efficient garbage collection and scalable compaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If distributed database scales out by adding more nodes to handle large data volumes, then storage capacity and throughput are improved, but cost of ownership and communication costs increase
Solution Approach 1:
The patent segments the database system into multiple autonomous nodes that can independently store and process data. Each node maintains a local copy of the database with its own storage engine, allowing the system to scale horizontally while keeping individual node complexity manageable. The segmentation enables distributed storage capacity without requiring centralized management complexity.
2Quantity of substance
If distributed database scales out by adding more nodes to handle large data volumes, then storage capacity is improved, but communication costs between nodes increase
Solution Approach 1:
By segmenting the database across autonomous nodes, the patent enables local data processing and reduces the need for inter-node communication. Each node can independently serve local queries, minimizing communication overhead while maintaining distributed storage capacity.
3Device complexity
If log-structured merge tree stores both keys and document values together, then storage structure is simplified, but write amplification increases
Solution Approach 1:
The patent segments the storage structure into two separate components: an LSM tree that stores only keys and sequence numbers, and a separate document value storage. This segmentation prevents write amplification because updates to document values do not require rewriting the entire key-value entry in the LSM tree, only updating the separate value storage.
Solution Approach 2:
The patent extracts document values from the traditional LSM tree structure and stores them separately. The LSM tree retains only the indexing functionality (keys and sequence numbers), while document values are stored in a dedicated storage area. This extraction eliminates the write amplification problem caused by redundant value storage in the LSM tree.
4Quantity of substance
If database performs frequent compaction operations to maintain storage efficiency, then storage density is improved, but write throughput and performance deteriorate
Solution Approach 1:
The patent segments compaction operations to occur independently in the background for each node's local storage, rather than requiring coordinated compaction across the distributed system. This allows compaction to proceed without blocking write operations, maintaining both storage density and write throughput.
Solution Approach 2:
The patent performs preliminary organization of data during the write phase, where sequence numbers and keys are sorted and structured as they are inserted into the LSM tree. This preliminary organization reduces the amount of work required during compaction, allowing compaction to maintain storage density without significantly impacting write throughput.
Data Source
AI summary
A system stores multiple documents in a log-structured object store, each document associated with a sequence number and a document value. The log-structured object store includes multiple log segments, each storing documents within a defined sequence number range. A first log-structured merge-tree is maintained to map document keys to corresponding sequence numbers, while a second log-structured merge-tree maps sizes of stale documents to their sequence numbers. A ratio is determined between the size of stale documents and the total size of documents in at least one log segment. In response to the ratio exceeding a predetermined threshold, a compaction operation is initiated to delete stale documents and consolidate active documents into a reduced number of log segments using the reclaimed space. The first and second log-structured merge-trees are updated based on the compaction results to reflect the current state of the data.