Targeted Sweep Queue for Key-Value Database Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing sweep methods for key-value databases are slow and performance degrades over time due to the accumulation of historical cell versions, leading to significant performance degradation as the database size increases.
Innovation Solution
A targeted sweep method is implemented, where entries for write transactions are written to a queue with metadata, and a single ranged tombstone is used to delete prior versions of cells based on a current sweep timestamp, optimizing the deletion process and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If historical cell versions are maintained to ensure database consistency, then database consistency is improved, but storage space consumption increases and read operation performance degrades
Solution Approach 1:
The patent introduces a sweep queue that pre-collects and organizes deletion information before actual deletion occurs. When a cell is deleted, its version history is recorded in the sweep queue with metadata about when and why the deletion should occur, allowing systematic cleanup later without impacting current read operations.
Solution Approach 2:
The patent segments the deletion process into multiple phases: (1) recording deletion information in the sweep queue, (2) organizing versions by timestamp ranges, and (3) executing deletions in batches based on sweep timestamps. This segmentation allows the system to maintain consistency while planning deletions separately from execution.
2Quantity of substance
If traditional sweep methods are used to remove historical versions, then storage space is reduced, but sweep performance degrades over time as database size increases
Solution Approach 1:
The sweep queue stores deletion information in advance, organized by timestamp ranges. When sweeping, the system processes pre-organized data structures rather than scanning the entire database, significantly improving sweep performance as the database grows.
Solution Approach 2:
The patent implements dynamic sweep strategies where the sweep timestamp progresses forward in time, and the sweep queue dynamically organizes deletion candidates based on current timestamp ranges. This dynamic approach adapts to growing databases and maintains consistent sweep performance.
3Reliability
If read operations scan all historical cell versions, then database consistency is maintained, but read operation performance degrades significantly
Solution Approach 1:
The patent segments version history into active versions (currently readable) and deleted versions (marked for removal). By separating these categories and using tombstones to mark deletions, read operations only scan active versions rather than all historical versions, maintaining consistency while improving performance.
Solution Approach 2:
The patent introduces tombstone entries as intermediaries between deleted cell versions and read operations. Tombstones act as markers that tell readers which versions should be ignored, allowing consistent reads without scanning through all historical versions, thus improving read performance.
Data Source
AI summary
A computer-implemented method for targeted sweep of a key-value data storage is provided. The method comprises before a write transaction to a database having a key value store commits, and before each of one or more write commands of the write transaction are persisted to the key value store, writing an entry for each of the one or more write commands to an end of a targeted sweep queue, the entry comprising metadata including: data identifying a cell to which the write command relates, a start timestamp of the write transaction, and information identifying a type of the write transaction.