Log-Structured Merge Bush for Fast Writes and Range Lookups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional key-value stores face a performance compromise between log-structured hash tables (LSH-tables) and log-structured merge-trees (LSM-trees), with LSH-tables offering fast writes but impractical range lookups and high memory usage, while LSM-trees provide fast range lookups but expensive writes and modest memory requirements, necessitating a need for a data structure that bridges these extremes.
Innovation Solution
The introduction of a new multi-level data structure, log-structured merge bush (LSM-bush), which allows newer data to be merged more lazily, incorporating a cap level and staging levels with varying capacities, and allocates Bloom filters to minimize false positive rates, enabling faster writes and scalable range lookups while optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If log-structured hash table (LSH-table) is used, then write speed is improved, but range lookup performance deteriorates and memory usage increases
Solution Approach 1:
The patent segments the data structure into multiple levels (L levels total, where L ≥ 2), with Level 1 containing the hash table for fast writes and Point Lookups, and Level 2 containing sorted runs for efficient Range Lookups. This segmentation allows each level to specialize in specific operations, resolving the contradiction between write speed and range lookup performance.
Solution Approach 2:
The patent adds a vertical dimension to the traditional single-level hash table by creating a multi-level hierarchy. Level 1 handles insertions and point lookups, while Level 2 handles range lookups. This dimensional expansion allows the system to optimize for different operation types simultaneously, improving both write speed and range lookup performance without compromise.
2Speed
If log-structured hash table (LSH-table) is used, then write speed is improved, but memory consumption increases
Solution Approach 1:
The patent extracts the range lookup functionality from the hash table structure and places it in a separate Level 2 with sorted runs. This extraction allows Level 1 to maintain a smaller, more memory-efficient hash table while Level 2 handles range queries using disk-based sorted data, reducing overall memory consumption while preserving fast write performance.
Solution Approach 2:
The patent introduces Level 2 as an intermediary structure between the hash table and disk storage. This intermediary layer handles range lookups using sorted runs stored on disk, preventing the need to load all data into memory and thus reducing memory consumption while maintaining fast write capabilities in Level 1.
3Productivity
If log-structured merge-tree (LSM-tree) is used, then range lookup performance is improved, but write cost increases
Solution Approach 1:
The patent creates a simplified copy of the LSM-tree approach at Level 2, using sorted runs for range lookups. However, unlike traditional LSM-trees that require complex multi-level merging, this implementation uses a single Level 2 that periodically merges runs in the background, allowing fast range lookups without the write amplification penalty of full LSM-tree merging.
Solution Approach 2:
The patent changes the merging parameter from aggressive continuous merging (traditional LSM-tree) to periodic background merging. This parameter change allows Level 1 to accept writes rapidly without immediate merging, improving write speed, while Level 2 periodically performs merges to maintain range lookup efficiency, thus resolving the contradiction between write speed and range lookup performance.
Data Source
AI summary
Embodiments of the present invention provide a new multi-level data structure, log-structured merge bush (LSM-bush), to alleviate the performance compromise between LSH-table and LSM-tree data structures. Similar to LSM-tree, LSM-bush may buffer writes in memory, merge the writes as sorted runs across multiple levels in storage, and use in-memory fence pointers and Bloom filters to facilitate lookups. LSM-bush differs from LSM-tree in that it allows newer data to be merged more “lazily” than LSM-tree. This can be achieved by allowing larger numbers of runs to be collected at the smaller levels before merging them.