Replicating Big Data Using Temporary Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face challenges in efficiently replicating large volumes of data across multiple data centers, leading to data inconsistencies and prolonged replication times, especially during system maintenance or failures, which affects data availability and performance.
Innovation Solution
The method involves generating a new replica by copying a sorted string table file, metadata, and replication state, and applying mutations from the replication log, while using a temporary table to pin recent mutations and prevent garbage collection, thereby reducing replication time and resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional data replication methods are used in distributed storage systems, then data can be replicated across multiple data centers, but the replication time is prolonged and data inconsistencies occur
Solution Approach 1:
The system performs preliminary actions by creating a temporary table and pinning replication logs before actually generating the replica. This ensures that all mutations are captured and preserved in advance, preventing data inconsistencies during the replication process. The temporary table is prepared with the correct structure and associations before the actual data copying occurs.
Solution Approach 2:
A temporary table serves as an intermediary between the source table and the final replica. This intermediary structure allows the system to pin replication logs and manage the replication process without affecting the source table or the final replica. The temporary table acts as a buffer that ensures data consistency while enabling efficient replication.
2Reliability
If replication logs are pinned using a temporary table, then data consistency is enhanced, but additional storage resources are consumed
Solution Approach 1:
The system discards the temporary table and its associated replication log associations after the replica generation is complete. The temporary table is deleted and its associations are removed from the replication configuration, allowing the pinned replication logs to be garbage collected. This recover process releases the additional storage resources consumed during the replication process.
3Ease of operation
If multiple associations are added to replication configuration files, then replication control is improved, but device complexity increases
Solution Approach 1:
The system adds temporary table associations to the replication configuration file as a preliminary step before generating the actual replica. These temporary associations enable precise control over which replication logs are pinned during the replication process. After the replica is generated, these temporary associations are removed, leaving only the necessary permanent associations in the configuration file.
Data Source
AI summary
A method includes identifying a first table including data. The first table has associated metadata, an associated replication state, an associated replication log file including replication logs logging mutations of the first table, and an associated replication configuration file including a first association that associates the first table with a replication family. The method includes inserting a second association in the replication configuration file that associates a second table having a non-loadable state with the replication family. The association of the second table with the replication family causes persistence of any replication logs in the replication log file that correspond to any mutations of the first table during the existence of the second table. The method further includes generating a third table from the first table, the metadata associated with the first table, and the associated replication state of the first table.