Hash Prefix Indexing for Stale Entry Removal in On-Drive Dedupe Tables

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Maintaining stale index entries in on-drive dedupe indexes is detrimental to the effectiveness of deduplication processes, leading to inefficient use of storage space and potential missed deduplication opportunities due to evicted valid index entries.

Innovation Solution

Implement hash prefix indexing to assign index entries to buckets based on their hash prefixes, constructing address bags on-demand to identify and remove stale entries by comparing virtual addresses, reducing the need for full hash value computation and IO operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If full hash values are computed and stored for each invalid data page to identify stale entries, then stale index entry removal accuracy is improved, but processing resource consumption and memory usage increase significantly

Engineering Contradiction:

Improvestale index entry removal accuracyVSAvoidprocessing resource consumption

Core Design Contradiction:

Measurement precisionVSUse of energy by moving object

Solution Approach 1:

The patent extracts only the necessary portion of the hash value (the prefix) rather than computing and storing the full hash value. This allows identification of stale entries through prefix matching while avoiding the computational overhead of generating complete hash values for all invalid data pages.

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The patent performs partial hash computation by calculating only the prefix portion of the hash value rather than the complete hash. This partial action is sufficient for identifying potential stale entries through prefix matching, reducing processing resources while maintaining effective stale entry detection.

Inventive Principle:

Principle #16Partial or excessive action

2Measurement precision

If full hash values are computed for invalid data pages to identify stale entries, then stale index entry removal accuracy is improved, but memory resource consumption increases

Engineering Contradiction:

Improvestale index entry removal accuracyVSAvoidmemory resource consumption

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent extracts and stores only the hash prefix along with the virtual address of invalid data pages in the dedupe log, rather than storing complete hash values. This extraction approach maintains the ability to identify stale entries while significantly reducing the memory footprint of the dedupe log structure.

Inventive Principle:

Principle #2Taking out (Extraction)

3Measurement precision

If data pages are read from storage and decompressed to compute hash values for stale entry identification, then stale index entry removal accuracy is improved, but IO operations and processing time increase

Engineering Contradiction:

Improvestale index entry removal accuracyVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs preliminary action by capturing and logging the hash prefix and virtual address at the moment a data page is invalidated (when reference count reaches zero), rather than performing hash computation later when stale entry removal is needed. This eliminates the need to read and decompress data pages during stale entry identification.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The system serves itself by automatically logging the hash prefix and virtual address information when a data page becomes invalid, without requiring external intervention or subsequent processing. The dedupe log is self-populated with the necessary identification data at the time of invalidation.

Inventive Principle:

Principle #25Self-service

4Quantity of substance

If on-drive dedupe index capacity is filled with stale index entries, then storage space utilization appears improved, but deduplication performance deteriorates due to evicted valid index entries

Engineering Contradiction:

Improvestorage space utilizationVSAvoiddeduplication performance

Core Design Contradiction:

Quantity of substanceVSProductivity

Solution Approach 1:

The patent implements feedback by continuously monitoring the reference count of data pages and automatically logging hash prefix information when pages are invalidated. This feedback mechanism ensures that stale index entries are identified and removed based on actual data validity status, preventing performance degradation while maintaining storage efficiency.

Inventive Principle:

Principle #23Feedback

Data Source

PatentUS12411810B1Efficient removal of stale entries from on-drive deduplication hash tables using hash prefix indexing

Publication Date: 2025.09.09 DELL PROD LP

AI summary

Techniques for achieving efficient removal of stale index entries from on-drive deduplication indexes using hash prefix indexing. The techniques include providing an on-drive dedupe index that includes a plurality of index entries. Each index entry includes a hash value of a data page, and an address associated with a storage location of the data page. Each index entry is assigned to a bucket defined by a respective hash prefix. The techniques include, for each data page associated with a reference count decremented to zero, logging a hash prefix and a storage address of the data page. The techniques include, for each bucket, constructing an address bag, storing, in the address bag, logged addresses whose corresponding hash prefix is the same as the respective hash prefix defining the bucket, and removing, from the bucket, each stale index entry that includes an address matching one of the addresses in the address bag.