Distributed Storage Data Replication via Delta Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Replicating large quantities of data across a distributed storage system is inefficient due to limited bandwidth in long-haul network paths and the lack of a robust solution for handling network outages, leading to slow data transmission and manual intervention requirements.
Innovation Solution
Data changes are tracked as deltas and transmitted instead of the entire data piece, utilizing a network tree structure to reduce long-haul data transmission, and implementing a method where data is dynamically replicated from one storage sub-system to another, prioritizing real-time client requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large quantities of data are replicated across distributed storage systems using traditional methods, then data consistency is maintained, but network bandwidth consumption increases and replication speed decreases
Solution Approach 1:
The patent segments data replication into two distinct phases: a push phase where data is proactively replicated to remote sites, and a pull phase where data is retrieved on-demand. This segmentation allows the system to optimize network bandwidth by performing replication only when necessary, rather than continuously maintaining full synchrony across all nodes.
Solution Approach 2:
The system dynamically adjusts replication behavior based on real-time conditions. The push-pull model enables the system to switch between proactive replication (push) and reactive retrieval (pull) depending on data access patterns, network conditions, and consistency requirements, thereby optimizing bandwidth consumption while maintaining data integrity.
2Reliability
If data is replicated to multiple distributed instances, then availability and fault tolerance improve, but network transmission time increases
Solution Approach 1:
The push phase performs preliminary data replication to remote storage sites in advance, proactively establishing data copies before they are needed. This preliminary action reduces the time required for data retrieval during the pull phase, as data is already positioned at remote locations rather than needing to be transmitted from the origin during access operations.
3Ease of manufacture
If traditional replication strategies are used with static network topology, then implementation is simple, but efficiency decreases when network links become unavailable
Solution Approach 1:
The system dynamically adapts to changing network conditions by switching between push and pull phases based on real-time network availability and data access patterns. When network links are unavailable, the system can continue serving data from local caches or alternative replicas, and resume replication when connectivity is restored, thereby maintaining productivity despite network fluctuations.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
A method for asynchronously replicating data onto a tape medium is implemented at one or more server computers associated with a distributed storage system and connected to a tape storage system. Upon receiving a first request from a client for storing an object within the tape storage system, a server computer stores the object within a staging sub-system of the distributed storage system and provides a first response to the requesting client. If a predefined condition is met, the server computer transfers objects from the staging sub¬ system to the tape storage system. For each transferred object, the server computer adds a reference to the object to a tape management sub-system, identifies a corresponding parent object associated with the object and its metadata within a parent object management sub¬ system of the distributed storage system, and updates the parent object's metadata to include the object's location within the tape storage system.