Locality-Sensitive Hashing for Distributed Deduplication

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Distributed deduplication storage systems face inefficiencies in optimizing deduplication efficiency, particularly when identical or similar data is stored across multiple nodes, leading to unnecessary storage consumption and inefficient data retrieval.

Innovation Solution

The implementation of locality-sensitive hashing to assign backup data to deduplication nodes based on metadata, including characteristics derived from the data itself, optimizes deduplication by grouping similar data together, thereby minimizing storage consumption and improving retrieval efficiency.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If distributed deduplication storage systems store identical or similar data across multiple nodes, then data redundancy and storage consumption increase, but data availability and system reliability improve

Engineering Contradiction:

Improvedata availabilityVSAvoidstorage consumption

Core Design Contradiction:

ReliabilityVSLoss of substance

Solution Approach 1:

The system segments data into fixed-size chunks and computes independent hash values for each chunk. This segmentation allows the system to identify and deduplicate individual data blocks across distributed nodes, storing only unique chunks while maintaining data availability through distributed replication of deduplicated blocks.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The system creates hash-based copies (fingerprints) of data chunks rather than storing redundant actual data copies. By computing and comparing hash values across nodes, the system can verify data uniqueness without duplicating storage, achieving both space efficiency and data availability through hash replication.

Inventive Principle:

Principle #26Copying

2Productivity

If distributed deduplication storage systems use traditional hashing methods, then data distribution across nodes is uniform, but deduplication efficiency decreases due to scattered similar data

Engineering Contradiction:

Improvededuplication efficiencyVSAvoiddata distribution

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The system applies locality-sensitive hashing that preserves local data characteristics and similarities. By designing hash functions that map similar input data to nearby output values, the system ensures that similar data chunks are distributed to the same or adjacent nodes, improving deduplication efficiency while maintaining appropriate data distribution across the cluster.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The system changes the hashing parameter from traditional uniform distribution to locality-sensitive hashing with configurable dimensions and thresholds. This parameter change allows the hash function to adapt to data characteristics, grouping similar data together while still distributing data across nodes based on hash value ranges, thereby optimizing both deduplication efficiency and data distribution.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS9268784B1Content-aware distributed deduplicating storage system based on locality-sensitive hashing

Publication Date: 2016.02.23 EMC IP HLDG CO LLC

AI summary

Backup data is processed by obtaining a set of metadata associated with backup data. A locality-sensitive hash key is generated for the backup data based at least in part on the set of metadata. The backup data is assigned to one of a plurality of deduplication nodes based at least in part on the locality-sensitive hash key.