Counter-Based Compaction for Key-Value Store Tree Data Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Key-value store (KVS) tree data structures face efficiency issues due to accumulation of garbage data, leading to decreased search performance and increased write operations on storage media, particularly in systems using lower endurance memory devices like QLC NAND cells.
Innovation Solution
A counter-based merge operation is implemented to merge sub-sequences of sub-indexes into a merged sub-index, reducing the length of the index sequence, eliminating garbage data, and minimizing write operations by assigning merge count values and replacing sub-sequences with merged indices, thereby improving query performance and reducing data written to storage media.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If key-value store tree data structure is used for data storage, then data organization and retrieval capability is improved, but garbage data accumulation occurs leading to decreased search performance
Solution Approach 1:
The patent implements a compaction operation that identifies and removes obsolete key-value pairs (garbage data) from the tree structure. When a key is updated or deleted, the old version is marked as obsolete but retained in the structure. The compaction process systematically identifies these obsolete entries and removes them, recovering storage space and maintaining search performance by preventing garbage data accumulation.
2Productivity
If frequent merge operations are performed to eliminate garbage data, then search performance is improved, but write operations on storage media increase
Solution Approach 1:
The patent implements incremental compaction that processes only a portion of the tree structure at each operation rather than performing complete merges. The compaction operation traverses the tree and compacts subsets of nodes, progressively eliminating garbage data through multiple partial operations. This approach maintains search performance by continuously removing obsolete data without requiring full-tree rewrites, thereby reducing total write operations on storage media.
3Loss of substance
If counter-based merge operation is implemented to reduce write operations, then write amplification is reduced, but system complexity increases
Solution Approach 1:
The patent implements a counter-based compaction mechanism where each node in the tree maintains a counter value that automatically tracks the number of obsolete key-value pairs. When the counter reaches a threshold, the node is automatically scheduled for compaction without requiring external intervention or complex decision-making logic. This self-service approach reduces write amplification by triggering compaction only when necessary, while keeping system complexity manageable through simple counter-based decision rules.
Data Source
AI summary
Aspects of the present disclosure provide for operations of a key-value tree data structure that merges key-value pair data of a node, in a key-value tree data structure using counter values.


