Dynamic Shard Splitting and Merging in Key-Value Stores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing key-value stores face inefficiencies in transaction commits due to the need to write entire tree blocks to disk, even when only a few key-value pairs have changed, leading to excessive write amplification and disk usage.
Innovation Solution
Implementing a key-value store architecture with multiple sub-KV stores, each responsible for a range of keys, allowing for independent transaction commits and dynamic sharding/splitting/merging based on key thresholds to reduce unnecessary disk writes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire tree block is written to disk during transaction commit, then data consistency is ensured, but write amplification increases and disk usage becomes excessive
Solution Approach 1:
The patent segments the key-value store into multiple sub-KV stores, each responsible for a specific key range. Each sub-KV store has its own B-tree structure and can be committed independently. This segmentation allows only the modified sub-KV stores to be written to disk during commit, rather than the entire tree block, thereby reducing write amplification while maintaining data consistency through selective persistence of changed segments.
2Reliability
If the entire tree block is written to disk during transaction commit, then data consistency is ensured, but disk usage becomes excessive
Solution Approach 1:
By dividing the key-value store into sub-KV stores with separate B-trees, the patent enables selective writing of only those sub-KV stores that have accumulated changes. This reduces the amount of data written to disk during commit operations, optimizing disk usage while ensuring that all modified data is properly persisted for data consistency.
3Loss of energy
If multiple sub-KV stores are maintained with independent transactions, then write amplification is reduced, but system complexity increases
Solution Approach 1:
The patent organizes sub-KV stores with independent B-trees and transactions, allowing parallel and independent commit operations. Each sub-KV store manages its own transaction log and commit process, which reduces write amplification by avoiding redundant writes of unchanged segments. The modular structure manages complexity through clear separation of concerns, where each sub-KV store is self-contained and can be managed independently.
Solution Approach 2:
The patent implements lazy commit semantics where sub-KV stores accumulate changes and commit only when a threshold is reached or explicitly triggered. This partial action approach allows the system to defer writes until necessary, reducing the frequency and volume of disk I/O operations, thereby reducing write amplification while maintaining acceptable performance through on-demand commits.
4Adaptability or versatility
If sub-KV stores are split and merged dynamically, then scalability is improved, but operational complexity increases
Solution Approach 1:
The patent implements dynamic splitting and merging of sub-KV stores based on key range thresholds and workload characteristics. When a sub-KV store grows beyond a threshold, it is automatically split into smaller units; when sub-KV stores become too small, they are merged back together. This dynamic adaptation allows the system to scale efficiently with varying data volumes and access patterns, optimizing performance while managing operational complexity through automated threshold-based decisions.
Data Source
AI summary
A method is provided for a key-value (KV) store database management system (DBMS) to provide a KV store to an application. The method includes providing a first low-level KV store and a top-level KV store. The first low-level KV store includes a root node and leaf nodes. The top-level KV store includes a leaf node pointing to the first low-level KV store. The leaf node in the top-level KV store includes a minimum key in the first low-level KV store and a pointer to the root node of the first low-level KV store. The method further includes determining a key-value insertion to the KV store would increase a number of keys in the first low-level KV store over an upper threshold and, in response to said determining, creating a second low-level KV store. The method additionally includes determining if a total number of keys in the first low-level store and the second low-level store is less than a lower threshold from key-value deletions and, when the total number of keys in the first low-level store and the second low-level store is less than the lower threshold, merging the first and the second low-level stores.