Balanced Fingerprint Distribution for Storage Controller Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In data storage systems with multiple active controllers, the redistribution of deduplication processing to accommodate new active controllers consumes significant system resources and degrades performance, as existing solutions require reshuffling of fingerprint ownership and movement of billions of fingerprints, leading to substantial resource utilization and potential system degradation.
Innovation Solution
A method that groups fingerprints into balanced ownership groups across active controllers, allowing new controllers to absorb fingerprint ownership without reassigning existing ownership, minimizing resource usage by moving only a percentage of fingerprints proportional to the number of active controllers, thus maintaining efficient deduplication processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fingerprints are redistributed across controllers when a new active controller is added, then deduplication processing can be balanced across all active controllers, but system resources are significantly consumed and performance degrades due to moving billions of fingerprints
Solution Approach 1:
The patent segments the fingerprint table into multiple partitions, with each partition maintained by a specific controller. When a new controller is added, only the metadata structure is updated to include the new controller's partition, while existing data partitions remain unchanged. This segmentation allows the system to scale horizontally without requiring redistribution of existing fingerprint data, thus maintaining deduplication balance while avoiding excessive resource consumption.
Solution Approach 2:
The patent pre-allocates fingerprint table partitions to controllers based on their identity or configuration, establishing a stable mapping relationship before any data is stored. This preliminary assignment ensures that when controllers are added or removed, the system can dynamically adjust the partition distribution without requiring movement of existing fingerprints, thereby maintaining adaptability while minimizing resource usage during scaling operations.
2Adaptability or versatility
If fingerprints are redistributed across controllers when a new active controller is added, then deduplication processing can be balanced across all active controllers, but system performance degrades during the redistribution process
Solution Approach 1:
By dividing the fingerprint table into independent partitions that can be owned by different controllers, the system avoids the need to stop or slow down deduplication operations during controller addition. Each controller continues to manage its assigned partitions independently, allowing the system to maintain full productivity while adapting to changes in the number of active controllers.
Solution Approach 2:
The patent implements a dynamic partition assignment mechanism where the mapping between controllers and fingerprint partitions is adjusted based on the current set of active controllers. This dynamic structure allows the system to seamlessly incorporate new controllers or remove failed ones without disrupting ongoing deduplication operations, thus maintaining high productivity while providing adaptability.
3Loss of substance
If finer granularity deduplication is implemented, then match likelihood increases and data reduction is greater, but the number of fingerprints to be stored and processed increases significantly
Solution Approach 1:
The patent segments the large set of fingerprints into multiple smaller partitions distributed across controllers. Each controller manages a subset of fingerprints locally, which reduces the memory footprint and processing burden on each individual controller. This segmentation enables the system to handle fine-grained deduplication at scale by parallelizing fingerprint storage and comparison operations across multiple controllers, thus achieving greater data reduction without being overwhelmed by the total number of fingerprints.
Solution Approach 2:
The patent introduces a new dimension of distribution by organizing fingerprints not just by content but also by controller ownership and partition location. This multi-dimensional organization allows the system to efficiently manage fine-grained deduplication by distributing the workload across both the fingerprint space and the controller space, enabling scalable processing of billions of fingerprints through parallel operations on distributed partitions.
Data Source
AI summary
Fingerprints of data portions are distributed in a balanced manner across active controllers of a data storage system, and may be done so in such a manner that, when a new active controller is added to the system, fingerprint ownership and movement between pre-existing active controllers, and active controllers overall, is minimized When a new active controller is added to the system and fingerprints are redistributed, no fingerprint ownership may be re-assigned between pre-existing active controllers and no fingerprints may be moved between pre-existing active controllers, for example, between local memories of the active controller.