Autonomous Deduplication Pool Partitioning for Scalable Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional deduplication systems face scalability issues and prolonged recovery times due to the growth of deduplication pools, which can exceed acceptable service levels and lead to data loss or corruption from hardware or filesystem failures.
Innovation Solution
The method involves partitioning a deduplication pool by grouping data containers based on data sharing thresholds, creating autonomous partitions with minimal inter-partition data sharing, allowing for efficient recovery and deduplication processes by comparing fingerprints within each group rather than across all containers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a traditional deduplication approach is used with a single global fingerprint index and reference database, then deduplication can be applied across all data containers, but the recovery process takes an unacceptable length of time and scalability is compromised
Solution Approach 1:
The patent divides the single global deduplication pool into multiple independent partitions, each with its own fingerprint index and reference database. This segmentation allows recovery operations to be performed on individual partitions rather than the entire pool, dramatically reducing recovery time while maintaining deduplication effectiveness within each partition.
Solution Approach 2:
The patent introduces a new organizational dimension by creating hierarchical grouping of data containers into partitions based on data sharing thresholds. This transforms the flat single-pool structure into a multi-level hierarchy, enabling parallel processing and localized recovery operations that improve overall system productivity.
2Quantity of substance
If the deduplication pool grows to store terabyte and petabyte ranges of data, then data capacity increases, but managing the deduplication pool becomes unwieldy and service level agreements are breached
Solution Approach 1:
By segmenting the large deduplication pool into smaller manageable partitions, the system can store terabyte and petabyte ranges of data while keeping each partition's management complexity manageable. Each partition operates independently with its own index and database, preventing the entire system from becoming unwieldy.
Solution Approach 2:
The patent applies local quality by allowing each partition to be managed independently with localized deduplication policies and parameters. This enables optimized management of each partition's data characteristics without affecting the entire pool, reducing overall system complexity while maintaining high data capacity.
3Productivity
If data containers are grouped with high data sharing thresholds, then deduplication efficiency improves within groups, but inter-group data sharing is minimized which may impact overall deduplication
Solution Approach 1:
The patent dynamically adjusts the data sharing threshold parameter to determine partition membership. By optimizing this threshold, the system achieves high deduplication efficiency within partitions while maintaining appropriate data integrity across the entire system through controlled inter-partition data relationships.
Data Source
AI summary
A method to partition a deduplication pool is provided. The method includes determining that an amount of data in a plurality of data containers of the deduplication pool has reached a data capacity threshold and comparing each data container of the plurality of data containers with at least one other of the plurality of data containers as to amount of shared data. The method includes grouping, based on results of the comparing, the plurality of data containers into a plurality of groups of data containers, with data sharing from each of the plurality of groups of data containers to each other of the plurality of groups of data containers less than a data sharing threshold and data sharing inside each of the plurality of groups of data containers greater than the data sharing threshold.