On-Disk Dedupe Index Eviction Rules for Higher Deduplication Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing storage systems face inefficiencies in data deduplication due to random eviction policies at the on-disk dedupe index level, leading to the eviction of highly deduplicable datasets and reduced efficiency.
Innovation Solution
Implementing specialized eviction rules for the on-disk dedupe index, including avoiding eviction of datasets with infinite reference counts, preferentially evicting those with maximum reference counts, non-deduplicable datasets, and avoiding eviction of clean entries previously destaged from the in-memory index, to manage space effectively.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If random eviction policies are applied at the on-disk dedupe index level, then space management is simplified, but highly deduplicable datasets are evicted and deduplication efficiency is reduced
Solution Approach 1:
The patent changes the eviction policy parameters from random selection to rule-based selection, where eviction decisions are made based on dataset characteristics such as deduplication potential and reference counts. This allows the system to prioritize keeping highly deduplicable datasets in the index while still managing space effectively.
2Quantity of substance
If the on-disk dedupe index is kept full to maximize deduplication coverage, then more datasets can be deduplicated, but space for new entries is insufficient
Solution Approach 1:
The patent implements dynamic space management where the dedupe index adapts its contents based on incoming data patterns. The eviction rules allow the index to dynamically adjust which entries are retained or removed, ensuring that space is always available for new entries while maximizing the retention of highly deduplicable datasets.
3Use of energy by moving object
If in-memory dedupe index is frequently destaged to on-disk level, then memory usage is reduced, but on-disk index size increases and may require evictions
Solution Approach 1:
The patent implements feedback mechanisms where eviction rules monitor the state of the on-disk dedupe index and adjust eviction decisions based on current conditions. This feedback loop ensures that destaging operations do not cause the index to grow uncontrollably, as the eviction policy responds to index size and entry characteristics to maintain optimal performance.
Data Source
AI summary
Techniques for keeping on-disk, physical deduplication (“dedupe”) index levels up to date to increase the efficiency of data deduplication in storage systems. The techniques include providing a plurality of dedupe indexes across a memory level and a disk level of a storage system, in which the plurality of dedupe indexes include an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level. The techniques include applying a plurality of eviction rules regarding evicting entries of the on-disk dedupe index, in which the plurality of eviction rules avoid random evictions of entries from the on-disk dedupe index. The techniques include, in response to applying the plurality of eviction rules, identifying a plurality of entries of the on-disk dedupe index as eviction candidates. The techniques include evicting the identified eviction candidates from the on-disk dedupe index.