Fingerprint Filter for LSM Tree Key-Value Store I/O Bottleneck
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern key-value stores face memory I/O bottlenecks due to the increasing number of Bloom Filters (BFs) as data grows, leading to higher latency and storage I/O costs, while existing Fingerprint Filters (FFs) face challenges in scaling storage I/Os and maintaining up-to-date run IDs efficiently.
Innovation Solution
The Chucky system employs a fingerprint filter with compressed run IDs that are opportunistically updated during merge operations, using Huffman coding to minimize run ID size and align fingerprints and run IDs within FF buckets, allowing for efficient scaling of both memory and storage I/Os without additional storage reads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If Bloom Filters are used for each run in LSM-tree, then application point reads are optimized by allowing ruling out runs that do not contain target entry, but memory I/O becomes a bottleneck as data size grows and number of runs increases
Solution Approach 1:
The patent combines multiple Bloom Filters into a single Fingerprint Filter that handles all runs. Instead of maintaining separate BFs for each run (which increases memory I/O as runs increase), the invention uses one unified FF that stores fingerprints and run IDs together, reducing the number of memory I/O operations from O(number of runs) to O(1).
Solution Approach 2:
The patent changes the fundamental parameters of the filtering structure by transitioning from Bloom Filters (which use hash functions and bit arrays) to Fingerprint Filters (which use direct fingerprint storage with hash-based bucket indexing). This parameter change allows the system to maintain constant memory access time regardless of the number of runs, as the FF uses O(1) hash-based lookup rather than probing multiple BFs.
2Productivity
If Fingerprint Filters are used to replace multiple Bloom Filters, then memory I/O is reduced to constant access time, but run IDs must grow with data size to identify more runs uniquely, increasing false positive rate
Solution Approach 1:
The patent adds a new dimension to the filtering structure by storing both fingerprints and run IDs together in the same Fingerprint Filter buckets. Instead of using run IDs as a separate indexing dimension that grows independently, the invention integrates run IDs into the fingerprint storage structure, allowing them to share the same hash-based bucket organization and benefiting from the same O(1) access performance.
Solution Approach 2:
The patent changes how run IDs are managed by storing them alongside fingerprints in the FF rather than separately. The run IDs are hashed to the same buckets as their corresponding fingerprints, and the FF's collision resolution mechanisms (cuckoo hashing, linear probing) automatically handle any conflicts. This parameter change allows the system to maintain constant access time while supporting an unbounded number of runs.
3Productivity
If merge operations are performed more greedily to reduce number of runs and Bloom Filters, then fewer BFs need to be probed, but storage write I/Os increase and BF construction cost increases
Solution Approach 1:
The patent extracts the Bloom Filter probing overhead from the read path by replacing it with a Fingerprint Filter. This extraction eliminates the need to probe multiple BFs during reads, as the FF provides O(1) lookup that directly identifies the relevant run. By removing this bottleneck, the system can afford to perform fewer merges while maintaining high read throughput, as the FF construction cost is amortized over all future reads.
Solution Approach 2:
The patent performs preliminary action by constructing the Fingerprint Filter during merge operations rather than during reads. The FF is built incrementally as runs are merged, storing fingerprints and run IDs in advance. This preliminary construction eliminates the need for expensive runtime probing of multiple BFs, and the amortized construction cost is much lower than the cumulative cost of probing many BFs during reads.
Data Source
AI summary
A method for managing a log structured merged (LSM) tree of key value (KV) pairs, the LSM tree is stored in a non-volatile memory, the method may include merging runs of the LSM tree to provide merged runs; writing merged runs to the non-volatile memory; adding new runs to the LSM tree, wherein the adding comprises writing runs to the non-volatile memory; and updating at least one management data structure (MDS) to reflect the merging and the adding; wherein an MDS of the at least one MDS stores a mapping between keys of the KV pairs of the LSM tree, fingerprints associated with the KV pairs of the LSM tree, and compressed run identifiers that identify runs of the LSM tree.