Cross-Site Deduplication Bandwidth-Aware Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional replication systems do not provide an optimized mechanism to utilize processing resources in cross-site deduplication environments, leading to suboptimal performance due to varying network bandwidths among source systems, which can negatively impact the efficiency of data replication across sites.
Innovation Solution
Implement a mechanism that allows source servers to share information about their current data processing capacities, including network bandwidth, and use fingerprints of data objects to load balance replication tasks across source servers with higher available bandwidths, dynamically selecting source servers to maximize bandwidth usage and reduce replication time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data replication is performed using traditional mechanisms without considering network bandwidth variations, then replication can be implemented with simple mechanisms, but replication performance is degraded due to suboptimal utilization of available bandwidth
Solution Approach 1:
The system dynamically selects source servers based on real-time network bandwidth availability. The destination server receives bandwidth information from multiple source servers and adaptively chooses the optimal source for replication, making the replication mechanism flexible and responsive to changing network conditions rather than static
Solution Approach 2:
Source servers provide feedback about their current network bandwidth availability to the destination server. This feedback mechanism enables the destination server to make informed decisions about which source server to use for replication, optimizing bandwidth utilization and replication performance
2Adaptability or versatility
If common data segments are transferred through source systems with lower network bandwidth, then all source systems can participate in cross-site deduplication, but overall cross-site deduplication performance is negatively impacted
Solution Approach 1:
The system changes the selection parameter from arbitrary or static source selection to dynamic selection based on network bandwidth parameter. By evaluating and comparing bandwidth parameters of different source servers, the system identifies and selects the source with optimal bandwidth for transferring common data segments
Solution Approach 2:
The source server selection is made dynamic rather than static. The destination server continuously evaluates the bandwidth availability of participating source servers and adaptively selects the most appropriate source server for each replication operation, allowing the system to fully utilize the capabilities of high-bandwidth sources
Data Source
AI summary
Methods and apparatuses are described to provide a mechanism to improve replication performance by understanding available bandwidths of multiple source servers and load balancing the replication among the source servers based on fingerprints of data units to be replicated. A separate source server can be identified for one source server to replicate the data units to a target server. A replication request is sent from the one source server to the separate source server to identify the data units. A reply may be received from the separate source server to indicate that one or more of the data units are replicated from the separate source server. Subsequently, the remaining ones of the data units not indicated in the reply may be sent directly from the one source server to the target server without sending the one or more data units indicated in the reply.