Distributed Storage Data Replication via Delta Transmission
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Replicating large quantities of data in distributed storage systems is inefficient due to limited network bandwidth and the lack of a robust solution for handling network outages, leading to slow data updates and manual intervention requirements.
Innovation Solution
The system tracks changes as deltas and transmits these deltas rather than the entire data, utilizing a network tree structure to reduce long-haul data transmission, and dynamically replicates data by identifying a second storage subsystem as the source and a chunk store as the destination, with priority given to real-time client requests.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data piece is transmitted to each database instance for replication, then data consistency is maintained, but network bandwidth consumption increases significantly
Solution Approach 1:
The patent extracts only the changed portions (deltas) of the data rather than transmitting the entire data piece. The system identifies what has changed since the last replication and transmits only those differences, significantly reducing network bandwidth consumption while maintaining data consistency across all database instances.
Solution Approach 2:
The patent segments the data transmission process into identifying changes (deltas) and transmitting only those segments. By dividing the replication process into change detection and selective transmission, the system reduces the amount of data moving across the network while ensuring all instances receive the necessary updates.
2Reliability
If data is replicated to multiple database instances globally, then data availability improves, but replication speed decreases due to long-haul network paths
Solution Approach 1:
The patent implements preliminary action by establishing a hierarchical replication structure where data is first replicated to regional hubs before being distributed to final destinations. This preliminary replication at intermediate nodes enables faster subsequent distributions and allows the system to handle global replication efficiently without waiting for direct long-haul transmissions to complete.
Solution Approach 2:
The patent introduces intermediary nodes (regional hubs or replication servers) that facilitate data transmission between source and destination instances. These intermediaries receive data from the source and forward it to multiple destinations simultaneously, reducing the impact of long-haul network paths by breaking up the transmission into shorter segments through the intermediary infrastructure.
3Device complexity
If static network topology is used for replication, then system simplicity is maintained, but adaptability to network outages and changes is reduced
Solution Approach 1:
The patent implements a dynamic replication topology that automatically adapts to network conditions. Instead of using a fixed static topology, the system dynamically determines optimal replication paths based on current network availability, successfully handling outages and changes by rerouting replication through alternative nodes without manual reconfiguration.
Solution Approach 2:
The patent incorporates feedback mechanisms that monitor network availability and replication success. When outages or network changes are detected, the system uses this feedback to automatically adjust the replication topology, selecting alternative paths or destinations to maintain data consistency and availability despite changing network conditions.
Data Source
AI summary
A server computer at a first storage sub-system of a distributed storage system receives from a client a first client request for an object. If the object is not present in the first storage sub-system, the server computer identifies a second storage sub-system of the distributed storage system as having a replica of the requested object, the requested object including content and metadata. The server computer submits an object replication request for the requested object to the second storage sub-system and independently receives the content and metadata of the requested object from the second storage sub-system. The server computer generates a new replica of the object at the first storage sub-system using the received metadata and content and returns the metadata of the new replica of the object to the client.