Hybrid Data Replication in Distributed Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed storage systems face performance issues due to delays in updating secondary databases during data replication, which can hinder timely access to data in case of primary database failures or maintenance.
Innovation Solution
Implementing a hybrid combination of log shipping and data pulling for data replication, where small data objects use log shipping and larger objects are striped across multiple partitions, allowing for parallel data pulling from a primary location to a secondary location, reducing replication time and minimizing input/output transactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated from primary database to secondary database, then failover capability is provided, but replication time increases causing performance delays
Solution Approach 1:
The patent segments data into small data objects and large data objects, applying different replication strategies to each. Small objects use log shipping while large objects use parallel data pulling, optimizing replication efficiency for each category and reducing overall replication time while maintaining reliability
Solution Approach 2:
The system performs preliminary actions by maintaining a blocks table that pre-stores information about data objects, their locations, and metadata. This preliminary organization enables faster replication operations when failover is needed, reducing the actual replication time while ensuring data consistency
2Loss of time
If data objects are striped across multiple partitions, then parallel data pulling is enabled reducing replication time, but device complexity increases
Solution Approach 1:
The blocks table serves multiple functions: it stores metadata about data objects, tracks their locations across partitions, and enables both parallel data pulling and log shipping operations. This multi-functionality reduces the need for separate complex systems while achieving fast replication
Solution Approach 2:
The patent introduces an intermediary blocks table that mediates between the distributed data partitions and the replication process. This intermediary structure simplifies the complexity by providing a centralized coordination point that enables parallel operations without requiring complex direct peer-to-peer coordination between all partitions
3Productivity
If log shipping is used for small data objects, then replication efficiency is improved, but data consistency delays occur
Solution Approach 1:
The patent applies different quality levels of replication to different data objects based on their size and characteristics. Small objects use log shipping for efficiency, while large objects use parallel pulling for speed. The blocks table maintains local metadata that ensures consistency is achieved appropriately for each object type, balancing efficiency and consistency locally rather than applying a single global approach
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Data is replicated between primary and secondary storage systems using a data pulling process or a log shipping process. If data is to be replicated through the log shipping process, a transaction and its associated data get stored in a transaction log in the secondary storage system. Subsequently, when the transaction gets executed, actual data gets persisted from the transaction log to a data log in the secondary storage system. If the data is to be replicated through the data pulling process, the transaction is stored in the transaction log. However, the associated data for the transaction is retrieved from the first primary storage system and stored directly in the data log in the secondary system.