Cluster Data Replication via Global Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods between storage clusters are inefficient as they replicate duplicate data, leading to increased network traffic and resource utilization, which can be improved by leveraging global deduplication to identify and avoid copying duplicate data.
Innovation Solution
The technique involves creating a snapshot of data as a baseline at the destination cluster, transmitting only new data that differs from the baseline, and using a hash function to generate extent keys stored in an extent store hash table, ensuring that the same hash function is used across clusters to facilitate efficient replication by avoiding duplicate data transmission.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication transmits all data including duplicates to ensure completeness, then data consistency is maintained, but network traffic and resource utilization increase
Solution Approach 1:
The system performs preliminary actions by establishing baseline snapshots at the destination cluster before replication begins. These snapshots serve as reference points that enable the system to identify and exclude duplicate data during subsequent replication operations, thereby reducing network traffic while maintaining data consistency.
Solution Approach 2:
The system implements feedback mechanisms through hash function comparisons between source and destination data. By calculating hashes of data blocks and comparing them against existing data at the destination, the system receives feedback about which data blocks are duplicates and adjusts replication behavior accordingly, transmitting only necessary data while ensuring consistency.
2Reliability
If duplicate data is replicated to ensure data completeness, then data consistency is maintained, but replication efficiency decreases
Solution Approach 1:
The system extracts and removes duplicate data from the replication process by using hash comparisons to identify identical data blocks at the destination before transmission. This extraction of duplicate identification logic enables efficient replication by filtering out redundant data while preserving the completeness and consistency of the replicated dataset.
Solution Approach 2:
The system creates copies of data at the destination cluster and uses these copies as reference for identifying duplicates during replication. By maintaining baseline snapshots and using hash functions to compare data blocks, the system efficiently determines which data needs to be copied and which are duplicates, thereby improving replication efficiency while ensuring data consistency.
3Productivity
If global deduplication is implemented to avoid duplicate data transmission, then replication efficiency improves, but system complexity increases
Solution Approach 1:
The system achieves universality by implementing a single hash function that serves multiple purposes: data integrity verification, duplicate detection, and replication efficiency optimization. This universal approach to data handling simplifies the overall system architecture compared to multiple specialized mechanisms, while still achieving global deduplication benefits across the storage cluster.
Data Source
AI summary
A technique preserves efficiency for replication of data between a source node of a source cluster (“source”) and a destination node of a destination cluster (“destination”) of a clustered network. Replication in the clustered network may be effected by leveraging global in-line deduplication at the source to identify and avoid copying duplicate data from the source to the destination. To ensure that the copy of the data on the destination is synchronized with the data received at the source, the source creates a snapshot of the data for use as a baseline copy at the destination. Thereafter, new data received at the source that differs from the baseline snapshot are transmitted and copied to the destination. In addition, the source and destination nodes negotiate to establish a mapping of name-to-data when transferring data (i.e., an extent) between the clusters. Illustratively, the name is an extent key for the extent, such that the negotiated mapping established by the source and destination is based on the extent key associated with the extent.