Distributed Deduplication via Locality-Sensitive Hashing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data deduplication techniques in distributed storage systems result in excessive metadata generation and network congestion due to individual indexing of data chunks, leading to increased storage consumption and network traffic.
Innovation Solution
Implementing a deduplication manager that groups data chunks into extents, using Locality-Sensitive Hashing (LSH) to identify similar extents and associate metadata with these groups, reducing the need for individual chunk metadata and minimizing network queries.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If individual data chunks are indexed separately for deduplication, then deduplication accuracy is improved, but metadata storage consumption increases excessively
Solution Approach 1:
The patent merges multiple individual data chunk indexes into a single extent-level index. Instead of creating separate metadata entries for each data chunk, the system groups related chunks into extents and creates one index entry per extent, dramatically reducing metadata storage while maintaining deduplication capability through the extent-level abstraction.
Solution Approach 2:
The patent introduces a new hierarchical dimension by organizing data chunks into extents as an intermediate level between individual chunks and full files. This extent-level organization adds a grouping dimension that reduces the number of indexable units from individual chunks to extent groups, thereby reducing metadata overhead.
2Measurement precision
If individual data chunks are indexed separately, then deduplication completeness is improved, but network traffic increases due to excessive metadata exchange
Solution Approach 1:
The patent combines multiple data chunk indexing operations into a single extent-level indexing operation. By merging the metadata exchange requirements of individual chunks into one extent-level metadata update, the system reduces network traffic while maintaining the ability to identify and deduplicate duplicate data across the distributed system.
3Productivity
If data chunks are distributed across multiple servers, then storage scalability is improved, but network congestion increases due to numerous node queries
Solution Approach 1:
The patent merges the location and access information of multiple data chunks into a single extent metadata structure. When accessing data, the system queries for the extent once and retrieves all associated chunks from that location, rather than querying multiple servers individually for each chunk, thereby reducing network congestion while maintaining distributed storage scalability.
4Manufacturing precision
If small data chunk sizes are used, then deduplication granularity is improved, but the number of metadata entries increases excessively
Solution Approach 1:
The patent merges the metadata representation of multiple small data chunks into a single extent metadata entry. This allows the system to maintain fine-grained deduplication at the chunk level while representing all chunks in an extent through one consolidated metadata structure, thereby preserving deduplication granularity without proportionally increasing metadata complexity.
Solution Approach 2:
The patent introduces the extent as an intermediate organizational dimension between individual chunks and files. This additional hierarchical level allows small chunk sizes for fine-grained deduplication while grouping them into extents that reduce the overall metadata footprint through aggregation.
Data Source
AI summary
Deduplication in a distributed storage system is described. A deduplication manager defines a first extent on a first node in a distributed storage system. The deduplication manager selects a second extent from the one or more groups of extents. The second extent matches the first extent. The deduplication manager removes, from the first extent, one or more data chunks that are included in the second extent. The deduplication manager associates, with the first extent, a pointer to the second extent for the removed one or more data chunks.