Cross-Node Data Deduplication in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed storage systems, data deduplication across multiple nodes is inadequate, leading to redundant data copies and complexity due to shared snapshots and clones, which existing methods fail to manage effectively.
Innovation Solution
A centralized database maintains sampled data units and lineage information across nodes, using an entity array and signature array to determine candidate data sets for migration and select destination nodes based on deduplicable data, reducing redundant copies by managing data movement across nodes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is performed at individual nodes in a distributed storage system, then storage space utilization is improved at each node, but redundant data copies persist across multiple nodes and deduplication effectiveness deteriorates
Solution Approach 1:
A coordination service acts as an intermediary between storage nodes to enable cross-node deduplication. The coordination service receives deduplication requests from storage nodes, compares data across the distributed system using fingerprint matching, and coordinates the elimination of redundant copies across nodes, thereby achieving system-wide deduplication effectiveness while maintaining individual node storage efficiency
Solution Approach 2:
The patent merges deduplication operations across multiple independent storage nodes into a unified cross-node deduplication process. By combining data from multiple nodes and applying fingerprint-based comparison, the system achieves comprehensive deduplication that eliminates redundant copies across the entire distributed storage system, not just at individual nodes
2Reliability
If cross-node data deduplication is implemented in distributed storage systems, then redundant data copies are reduced, but system complexity increases due to coordination requirements
Solution Approach 1:
The system segments deduplication operations into node-level fingerprint generation and coordination-service-level comparison. Each storage node independently generates data fingerprints locally, while the coordination service handles cross-node fingerprint matching and coordination. This segmentation reduces system complexity by distributing computational tasks and avoiding centralized processing of all data
Solution Approach 2:
The patent uses fingerprint copies instead of actual data copies for cross-node comparison. Storage nodes send only fingerprint identifiers (not the actual data) to the coordination service for matching. This copying approach maintains deduplication effectiveness while dramatically reducing network bandwidth requirements and coordination complexity compared to transferring actual data blocks
3Quantity of substance
If data is migrated between storage nodes to eliminate redundancy, then storage space utilization is optimized, but migration overhead and time consumption increase
Solution Approach 1:
The system performs preliminary fingerprint generation and comparison before data migration decisions are made. By pre-computing fingerprints and identifying redundant copies in advance, the system can make informed migration decisions and execute data movement only when necessary, reducing unnecessary migration operations and associated time overhead
Solution Approach 2:
The patent implements local deduplication at each storage node before cross-node deduplication. Each node first eliminates local redundant copies, then participates in cross-node deduplication only for unique data. This local quality approach reduces the volume of data requiring migration and optimizes storage space utilization by applying appropriate deduplication strategies at each level
Data Source
AI summary
Example implementation relates to a method for managing movement of set of entities associated with sampled data units in a cluster of nodes of a distributed storage system. A database is maintained for sampled data units received from the cluster of nodes. The method includes maintaining lineage information for the set of entities associated with the sampled data units received from each node. Based on the lineage information of a source node, a data set for migration is determined.