Compaction Policies for Append-Only Store Streams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing append-only memory stores face high write amplification during compaction, leading to increased CPU overhead, I/O operations, and storage device wear, as they write and copy data records repeatedly, especially when mixing records with different update frequencies in the same stream.
Innovation Solution
Divide the append-only store into multiple streams based on estimated update frequencies, using exponentially weighted moving averages to determine the appropriate stream for new or updated records, and optimize memory allocation among streams to minimize write amplification by separating frequently and infrequently updated records.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If records with different update frequencies are mixed in the same stream, then the store structure is simple, but write amplification increases during compaction
Solution Approach 1:
The patent divides the append-only store into multiple streams, where each stream is dedicated to storing records with similar update frequencies. This segmentation separates frequently updated records from infrequently updated records, allowing compaction to proceed more efficiently by copying only the necessary records, thereby reducing write amplification while maintaining a manageable store structure.
2Quantity of substance
If frequent compaction is performed to manage memory growth, then memory usage is controlled, but CPU overhead and I/O operations increase
Solution Approach 1:
The patent applies different compaction strategies to different streams based on their local characteristics. Each stream is managed independently with compaction triggered when its memory usage exceeds a threshold, rather than performing global compaction. This local quality approach reduces unnecessary CPU overhead and I/O operations by avoiding compaction of streams that do not require it.
3Reliability
If all records are copied during compaction, then data consistency is maintained, but storage device wear increases
Solution Approach 1:
The patent extracts only the necessary records for copying during compaction by identifying which records need to be transferred between streams based on their update frequencies and memory thresholds. This selective extraction approach maintains data consistency for affected records while minimizing the total amount of data copied, thereby reducing storage device wear.
Data Source
AI summary
A method of reducing write amplification in an append-only memory store of data records, by which the store is subdivided into streams, each of which for storing records having an update frequency within a variable range of update frequencies. By defining an update frequency that does not rely on time, statistical methods can be used to select the streams in which data records can be written. The range of update frequencies of each stream can be fixed or variable and based on the stored records. The memory allocated to each stream can be determined based on numerically solving an optimization problem that determines the write amplification resulting from different memory allocations in the streams.