Distributed Fingerprint Partitioning for Storage De-duplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Scalable storage systems face performance and scalability issues due to the limitations of traditional monolithic dictionaries in data de-duplication, which are not suitable for clustered scale-out storage systems.
Innovation Solution
A storage system with a de-duplication module that generates fingerprints for data items, identifies partitions, associates them with hardware instances, and queries these instances to determine fingerprint existence, allowing for efficient de-duplication across multiple information handling resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a monolithic dictionary is used for data de-duplication, then data consistency is maintained, but scalability and performance deteriorate in clustered scale-out storage systems
Solution Approach 1:
The patent divides the monolithic dictionary into multiple distributed dictionary instances across different nodes in the clustered storage system. Each node maintains its own dictionary partition, allowing the system to scale horizontally while maintaining de-duplication functionality. This segmentation resolves the contradiction by enabling scalability without sacrificing data consistency through the use of distributed consensus protocols.
Solution Approach 2:
The patent transitions from a single-dimensional monolithic dictionary structure to a multi-dimensional distributed dictionary architecture across multiple nodes and partitions. By adding the dimension of distribution across multiple hardware instances, the system achieves both scalability and data consistency simultaneously.
2Device complexity
If a monolithic dictionary is used for data de-duplication, then simple data structure is maintained, but performance deteriorates in scalable storage systems
Solution Approach 1:
The dictionary is segmented into multiple distributed instances, allowing parallel processing of de-duplication operations across different nodes. This improves performance by eliminating the single-point bottleneck of a monolithic dictionary while maintaining manageable complexity through standardized partitioning protocols.
Solution Approach 2:
Multiple distributed dictionary instances are merged through coordination protocols to provide unified de-duplication functionality. The system combines the processing power of multiple nodes while presenting a consistent interface to applications, thereby improving performance without significantly increasing operational complexity.
3Device complexity
If de-duplication operations are centralized in a single dictionary, then system complexity is reduced, but scalability is limited
Solution Approach 1:
The centralized dictionary is segmented and distributed across multiple independent nodes, allowing the system to scale horizontally. Each node manages its own dictionary partition independently, reducing inter-node dependencies and enabling linear scalability while keeping individual node complexity manageable.
Solution Approach 2:
Coordination intermediaries and standardized protocols are introduced to manage communication between distributed dictionary instances. These intermediaries handle consistency protocols and partition management, allowing the system to scale without proportionally increasing overall system complexity.
Data Source
AI summary
In accordance with embodiments of the present disclosure, a storage system may include a storage array comprising one or more storage resources, a processor communicatively coupled to the storage array, and a de-duplication module comprising instructions embodied on a computer-readable medium communicatively coupled to the processor. The de-duplication module may be configured to, when read and executed by the processor: generate a fingerprint for an item of data stored on the storage array; identify a partition for the fingerprint; associate the partition with a hardware instance selected from a plurality of hardware instances, wherein each particular hardware instance comprises one or more information handling resources; and query the selected hardware instance to determine if the fingerprint exists on the hardware instance.