Scalable Data Deduplication via Hash-Based Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deduplication techniques are not scalable and become excessively time-consuming and resource-intensive for large data sets, impacting storage system performance due to their reliance on sorting algorithms that require disproportionate time and resources.
Innovation Solution
The method involves partitioning a data set into logical partitions, where each data block is assigned to a specific partition based on its fingerprint, allowing for independent deduplication within each partition, using a hash algorithm to generate fingerprints for efficient duplicate detection and elimination, and scheduling deduplication processes to optimize resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sorting algorithms are used for deduplication, then duplicate data blocks can be identified and eliminated, but the time required increases disproportionately with data set size
Solution Approach 1:
The patent divides the data set into multiple partitions, each processed independently for deduplication. This segmentation allows parallel processing of partitions, reducing overall deduplication time from geometric growth to additive growth as data set size increases, while maintaining complete deduplication coverage across the entire data set
2Reliability
If sorting algorithms are used for deduplication, then duplicate data blocks can be eliminated, but processor and memory resources are consumed excessively
Solution Approach 1:
By partitioning the data set into smaller subsets that can be processed independently, the patent reduces the memory and processor resources required for each individual deduplication operation. Each partition requires only a fraction of the total resources compared to processing the entire data set as one unit, enabling resource-constrained systems to perform effective deduplication
3Quantity of substance
If deduplication is performed on large data sets, then storage space is optimized, but storage system performance for servicing user requests is adversely impacted
Solution Approach 1:
The patent enables deduplication of large data sets by processing partitions in parallel or sequentially with reduced resource requirements, making deduplication feasible without causing excessive performance degradation. The segmented approach allows the system to maintain responsiveness to user requests while performing deduplication operations on enterprise-scale data sets
Solution Approach 2:
The patent performs deduplication operations during periods when the storage system is less busy servicing user requests, or schedules deduplication to occur in background threads. This preliminary or background action ensures that deduplication does not significantly impact user-facing performance while still achieving storage space optimization
Data Source
AI summary
In a method and apparatus for scalable deduplication, a data set is partitioned into multiple logical partitions, where each partition can be deduplicated independently. Each data block of the data set is assigned to exactly one partition, so that any two or more data blocks that are duplicates of each are always be assigned to the same logical partition. A hash algorithm generates a fingerprint of each data block in the volume, and the fingerprints are subsequently used to detect possible duplicate data blocks as part of deduplication. In addition, the fingerprints are used to ensure that duplicate data blocks are sent to the same logical partition, prior to deduplication. A portion of the fingerprint of each data block is used as a partition identifier to determine the partition to which the data block should be assigned. Once blocks are assigned to partitions, deduplication can be done on partitions independently.