Database Replication Coordination for Multi-Partition Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems, database replication methods fail to ensure data consistency across all partitions of a secondary cluster due to inconsistent execution of multi-partition transactions, leading to discrepancies in data across different partitions.
Innovation Solution
A database replication method involving a coordination server that determines a target timestamp for each partition based on the timestamps of multi-partition transactions across the secondary cluster, ensuring that each partition executes transactions consistently by using a coordinated timestamp to execute or skip specific log records, thereby maintaining data consistency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If each partition of the primary cluster sends replication logs to corresponding partitions of the secondary cluster independently, then the replication process is simple and fast, but data consistency across all partitions of the secondary cluster cannot be ensured
Solution Approach 1:
The patent introduces a coordination server as an intermediary between primary and secondary clusters. This server receives timestamps from each partition of the secondary cluster, determines a target timestamp that ensures consistency, and sends it back to partitions. This mediator resolves the contradiction by enabling independent fast replication while guaranteeing data consistency through centralized coordination.
Solution Approach 2:
The patent implements a feedback mechanism where each partition of the secondary cluster sends its timestamp to the coordination server, receives a target timestamp in response, and uses this feedback to determine whether to execute the replication log. This feedback loop ensures that partitions only execute transactions when all necessary data is available, maintaining consistency without sacrificing replication speed.
2Loss of information
If a replication log of a multi-partition transaction is saved in each partition of the primary cluster, then complete transaction information is available in each partition, but buffers of partitions being in different statuses or periods causes inconsistent execution at the secondary cluster
Solution Approach 1:
The patent replaces the mechanical synchronization approach (where partitions wait for each other or use complex locking mechanisms) with a timestamp-based coordination system. The coordination server uses timestamps to determine execution order and consistency, substituting complex mechanical synchronization with a simpler, more reliable temporal ordering mechanism.
3Reliability
If the primary cluster sends replication logs frequently to ensure data freshness, then data consistency is improved, but network bandwidth and system resources are consumed
Solution Approach 1:
The patent implements periodic action by having each partition of the secondary cluster send its timestamp to the coordination server at regular intervals or when specific conditions are met (such as buffer full or period elapsed). This periodic coordination ensures data freshness and consistency without requiring continuous communication, thereby reducing network bandwidth consumption while maintaining reliability.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
A database replication method and apparatus for a distributed system are provided and relate to the database field. The method includes: sending, by a first partition of a secondary cluster to a coordination server, a timestamp of a multi-partition transaction newly added to the first partition; determining, by the coordination server, a target timestamp for the first partition based on the timestamp of the newly added multi-partition transaction and a multi-partition transaction timestamp, of each partition of the secondary cluster, stored on the coordination server; sending, by the coordination server, the target timestamp to the first partition; and executing, by the first partition, a replication log in the first partition based on the target timestamp. By maintaining a timestamp of a multi-partition transaction included in each partition of the secondary cluster, the coordination server determines which multi-partition transactions are present in all the partitions, and informs a corresponding partition which log records can be executed. In this way, the corresponding partition can execute, without waiting, a multi-partition transaction that is present in all the partitions but has not been executed, thereby avoiding data inconsistency and increasing replication efficiency.