Data Storage Deduplication With Pre-Flush Index Promotion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face performance degradation due to the need for slower disk I/O operations during the flush process, particularly when accessing on-drive deduplication indexes, which leads to CPU utilization drops and reduced system throughput.
Innovation Solution
A pre-flush stage is introduced to promote selected fingerprints from the on-drive deduplication index to an in-memory deduplication index, allowing deduplication lookups to be confined solely to the in-memory index during the flush process, thereby reducing reliance on slower disk I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the flush process accesses the on-drive deduplication index for each data unit, then deduplication completeness is maintained, but disk I/O operations increase and system throughput decreases
Solution Approach 1:
The patent performs preliminary deduplication lookups during a pre-flush stage before the actual flush process. The system scans the write log and queries the on-drive deduplication index in advance to identify deduplication opportunities, copying matching entries to the in-memory deduplication index. This preliminary action ensures that when the flush process executes, it can rely on pre-fetched deduplication information, reducing the need for slow disk I/O operations during the critical flush path and thereby improving system throughput while maintaining deduplication completeness.
Solution Approach 2:
The patent copies deduplication entries from the on-drive deduplication index to the in-memory deduplication index during the pre-flush stage. By copying the necessary deduplication information into memory beforehand, the system eliminates the need to repeatedly access the slow on-drive index during the flush process. This copying mechanism allows the flush operation to proceed at high speed using only memory accesses, significantly improving throughput while preserving the completeness of deduplication checks.
2Reliability
If the flush process performs deduplication lookups during the actual flush, then data consistency is ensured, but CPU utilization drops due to waiting for disk I/O
Solution Approach 1:
The patent performs all necessary deduplication lookups and index queries during the pre-flush stage before the actual flush process begins. By completing these I/O-bound operations in advance, the CPU is freed from waiting during the critical flush path. The pre-flush stage scans the write log, queries the on-drive deduplication index, and copies matching entries to memory, ensuring that when the flush executes, the CPU can process data without being blocked by slow disk operations, thereby improving CPU utilization while maintaining data consistency through the structured two-stage approach.
3Measurement precision
If the system maintains a full on-drive deduplication index for complete deduplication coverage, then deduplication accuracy is maximized, but memory resources are consumed and access speed is reduced
Solution Approach 1:
The patent segments the deduplication index into two distinct components: a small in-memory deduplication index for fast local lookups during flush operations, and a comprehensive on-drive deduplication index for complete deduplication coverage. The in-memory index stores only the entries needed for immediate flush processing, enabling fast memory-based access. The on-drive index maintains the full historical record for accurate deduplication. This segmentation allows the system to achieve both high access speed (via the small in-memory index) and high accuracy (via the complete on-drive index) without requiring the entire index to reside in memory.
Solution Approach 2:
The patent copies relevant deduplication entries from the on-drive index to the in-memory index during the pre-flush stage. This copying operation transfers only the necessary information into memory, allowing fast local lookups during flush without requiring the full index to be in memory. The copied entries represent the subset of deduplication data needed for current flush operations, enabling the system to achieve fast access speeds for the active workload while maintaining the ability to perform accurate deduplication by referencing the complete on-drive index when necessary.
Data Source
AI summary
A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.