Deduplication Hash Collision Resolution via Metadata Pre-Filtering

Solution Overview

Problem

Non-cryptographic hash functions used in deduplication systems are inefficient due to the need for additional data validation steps, especially when dealing with compressed data blocks, which require decompression and byte-for-byte comparisons to confirm matches, consuming significant time and resources.

Innovation Solution

The technique involves obtaining metadata from candidate and target blocks with hash-based matches and comparing this metadata to quickly disqualify mismatches, such as checksums and compression details, thereby abandoning deduplication attempts for non-matching blocks, thus avoiding unnecessary decompression and byte comparisons.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If non-cryptographic hash functions are used for deduplication, then compute time and storage space are reduced, but hash collisions occur requiring additional validation

Engineering Contradiction:

Improvededuplication processing speedVSAvoidmatch accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The validation process is segmented into multiple stages: first comparing metadata (compression method, compressed size) to quickly eliminate obvious mismatches, then performing byte-for-byte comparison only on candidates that pass metadata validation. This segmentation reduces the number of full validations needed while maintaining accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Metadata comparison is performed as a preliminary action before full byte-for-byte validation. By checking compression methods and compressed sizes first, the system pre-filters candidate blocks, ensuring that only those with matching metadata proceed to expensive byte comparisons, thereby improving overall efficiency.

Inventive Principle:

Principle #10Preliminary action

2Reliability

If byte-for-byte comparisons are performed to validate hash matches, then match accuracy is ensured, but processing time and resource consumption increase significantly

Engineering Contradiction:

Improvematch accuracyVSAvoidvalidation time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The validation process is segmented into multiple stages: first comparing metadata (compression method, compressed size) to quickly eliminate obvious mismatches, then performing byte-for-byte comparison only on candidates that pass metadata validation. This segmentation reduces the number of full validations needed while maintaining accuracy.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of performing complete byte-for-byte comparisons on all hash-matching blocks, the system performs partial validation by first checking metadata fields. Only blocks that pass this partial validation proceed to full byte comparison, reducing the total validation workload while maintaining reliability.

Inventive Principle:

Principle #16Partial or excessive action

3Reliability

If compressed data blocks are validated through decompression and byte comparison, then accurate match detection is achieved, but compute resources and time are consumed

Engineering Contradiction:

Improvematch detection accuracyVSAvoidcompute resource consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

Metadata comparison is performed as a preliminary action before full byte-for-byte validation. By checking compression methods and compressed sizes first, the system pre-filters candidate blocks, ensuring that only those with matching metadata proceed to expensive byte comparisons, thereby improving overall efficiency.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of performing complete byte-for-byte comparisons on all hash-matching blocks, the system performs partial validation by first checking metadata fields. Only blocks that pass this partial validation proceed to full byte comparison, reducing the total validation workload while maintaining reliability.

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20240231622A9Resolving deduplication matches when using non-cryptographic hash

Publication Date: 2024.07.11 DELL PROD LP

AI summary

A technique for managing deduplication using a non-cryptographic hash function includes obtaining metadata associated with both a candidate block presented for deduplication and a target block having a hash-based match to the candidate block. The technique further includes checking for a mismatch between the candidate block and the target block based on the obtained metadata. In response to the checking determining a mismatch, the technique further includes abandoning deduplication of the candidate block, such that the candidate block is stored independently of the target block.