Global Sequence Number Data Replication Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Ensuring data consistency and accuracy across multiple computing devices is challenging, especially when data is mutating, as existing technologies struggle to maintain synchronized views without causing performance delays or resulting in incorrect operations.
Innovation Solution
The implementation of a global sequence number (GSN) system that increments or modifies with each data change, allowing computing devices to replicate data in chunks and compare initial and current GSN values to identify mutations, ensuring that local copies remain consistent with primary data stores by applying changes from the transaction log.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is replicated across multiple computing devices, then data availability and access speed are improved, but data consistency and accuracy deteriorate when data is mutating
Solution Approach 1:
The system captures data mutations in a transaction log before they are fully committed, and replicates these mutations to secondary computing devices in advance. This preliminary capture and replication of mutations ensures that when data is accessed from secondary devices, the mutations are already present, maintaining data consistency without requiring real-time synchronization during data access operations.
Solution Approach 2:
The patent introduces a transaction log as an intermediary mechanism between the primary data store and secondary computing devices. This transaction log buffers and manages data mutations, allowing them to be replicated asynchronously to secondary devices. The transaction log acts as a mediator that ensures mutation integrity while allowing independent access at secondary devices, thus maintaining both speed and consistency.
2Reliability
If data replication is performed in real-time to maintain consistency, then data accuracy is improved, but system performance and processing speed deteriorate
Solution Approach 1:
Instead of continuous real-time replication, the system uses periodic action by batching data mutations and replicating them in discrete transactions. The transaction log accumulates mutations and replicates them periodically to secondary computing devices, reducing the overhead of constant synchronization while maintaining data accuracy through structured transaction processing.
Solution Approach 2:
The system performs preliminary capture of mutations in the transaction log before replication occurs. This allows mutations to be prepared and staged for replication without immediately impacting primary system performance. The actual replication to secondary devices can then occur with reduced performance impact, as the mutation capture phase is separated from the replication execution phase.
3Productivity
If data is copied to multiple devices simultaneously, then data availability is improved, but the risk of obtaining outdated or inconsistent data increases
Solution Approach 1:
The system implements feedback mechanisms where secondary computing devices can query the transaction log to determine if newer mutations are available. This feedback loop allows devices to check for updates before accessing data, ensuring they obtain the most current version without requiring continuous real-time synchronization. The feedback mechanism maintains data freshness awareness while preserving data availability.
Solution Approach 2:
Mutations are preliminarily captured in the transaction log before being replicated to secondary devices. This preliminary capture creates a record of the latest data state that secondary devices can reference. When devices need data, they can check the transaction log for the latest mutation sequence numbers, ensuring they retrieve the most current data without waiting for real-time updates.
Data Source
AI summary
Routing data is copied from a primary data store to a local data store of a computing device. As the routing data is being copied by the computing device, changes to the data continue to occur. After copying all or a portion of the data, the computing device uses a value of a global sequence number (GSN) to determine if mutations have been made to the primary data. The routing data in the local data store is not consistent with the routing data stored in the primary data store when the value of the GSN is larger after the copying of the routing data as compared to the initial value of the GSN when the copying started. When changes have been made to the data, the computing device integrates the changes into the local data store before beginning routing operations.