Asynchronous Tape Replication via Delta Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Replicating large quantities of data across a distributed database system is inefficient due to limited bandwidth in long-haul network paths and the challenges of integrating tape storage, which supports serial access but not random access, and handling network outages gracefully.
Innovation Solution
The method involves tracking changes as deltas and transmitting these deltas rather than the entire data, utilizing a network tree structure to reduce data transmission across long-haul links, and implementing a system for asynchronous replication of data onto a tape medium within a distributed storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire large piece of data is transmitted to each database instance for replication, then data consistency is ensured, but network bandwidth consumption increases significantly and replication speed decreases
Solution Approach 1:
The patent segments the data transmission process by transmitting only the changed portions (deltas) rather than the entire large piece of data to each database instance. This maintains data consistency while significantly reducing bandwidth consumption and improving replication speed.
Solution Approach 2:
The patent extracts only the necessary changed information (deltas) from the complete data and transmits these extracted portions to other database instances. This allows the system to maintain consistency for changed data while avoiding unnecessary transmission of unchanged data, thus improving replication efficiency.
2Reliability
If the same large piece of data is transmitted to each database instance, then all instances are updated, but bandwidth usage is multiplied by the number of database instances
Solution Approach 1:
The patent divides the data transmission into segments by sending only the delta (changed portions) rather than the complete data to each instance. This segmentation approach ensures all instances receive necessary updates while dramatically reducing total bandwidth consumption across the network.
Solution Approach 2:
The patent uses a copying approach by creating and transmitting only the delta representation of changes rather than copying the entire large piece of data to each instance. This reduces the amount of data copied and transmitted across the network while maintaining replication completeness.
3Reliability
If tape storage is integrated into the distributed storage system, then cost-effectiveness and reliability are improved, but random access capability is lost and throughput is reduced
Solution Approach 1:
The patent introduces an intermediary layer (the distributed storage system with object storage) between the tape storage and the distributed database. This intermediary provides random access capability and high throughput for frequently accessed data, while tape storage handles cost-effective long-term storage, thus resolving the contradiction between reliability/cost and speed.
4Device complexity
If static network topology is used for replication, then replication strategy is simple, but system adaptability to network outages and changes is reduced
Solution Approach 1:
The patent implements a dynamic replication strategy that automatically adapts to network changes and outages by monitoring network conditions and adjusting replication paths in real-time. This dynamic approach maintains system adaptability while managing complexity through automated decision-making rather than manual configuration.
Data Source
AI summary
A method for asynchronously replicating data onto a tape medium is implemented at one or more server computers associated with a distributed storage system and connected to a tape storage system. Upon receiving a first request from a client for storing an object within the tape storage system, a server computer stores the object within a staging sub-system of the distributed storage system and provides a first response to the requesting client. If a predefined condition is met, the server computer transfers objects from the staging sub-system to the tape storage system. For each transferred object, the server computer adds a reference to the object to a tape management sub-system, identifies a corresponding parent object associated with the object and its metadata within a parent object management sub-system of the distributed storage system, and updates the parent object's metadata to include the object's location within the tape storage system.