Data Replication Sequence ID Verification for Network Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High write frequency applications in long-distance data synchronization face data inconsistency due to network packet order issues, especially in high write frequency conditions, where hash values alone cannot ensure correct order, leading to inconsistencies during failover.
Innovation Solution
Transmitting identifiers and sequence IDs independently from values, allowing the remote system to maintain data consistency by checking sequence IDs in a queue for correct replication status and issuing error messages when inconsistencies are detected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hash values are used to check data consistency in high write frequency applications, then data verification is simplified, but network packet order issues cause data inconsistency
Solution Approach 1:
The patent segments the data transmission into two independent parts: metadata (including sequence ID and hash value) transmitted first, and actual data values transmitted separately. This segmentation allows the receiving system to verify data consistency using sequence IDs without being affected by network packet reordering, resolving the contradiction between verification simplicity and data consistency.
Solution Approach 2:
The patent performs preliminary transmission of metadata containing sequence IDs and hash values before transmitting the actual data values. This preliminary action enables the receiving system to prepare for verification and maintain correct ordering, ensuring data consistency even when data packets arrive out of order due to network conditions.
2Reliability
If metadata is replicated synchronously to remote system, then data consistency is improved, but network latency increases
Solution Approach 1:
The patent applies partial synchronous replication by transmitting metadata (sequence ID and hash value) synchronously to ensure consistency, while allowing data values to be transmitted asynchronously. This partial action approach maintains data consistency for verification purposes without requiring full synchronous replication of all data, thereby reducing network latency.
3Measurement precision
If sequence ID checking is implemented for every data entry, then data ordering accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent uses sequence IDs as simplified copies of the actual data ordering information. Instead of implementing complex ordering logic, the system transmits sequence IDs with metadata and uses these IDs for verification. This copying approach maintains high ordering accuracy while keeping processing complexity low, as the sequence ID provides a simple numerical representation of data entry order.
Data Source
AI summary
A first system receives values with identifiers of the values from one or more clients. The first system enters the values sequentially into a first data store. The first system associates each of the values with a sequence ID indicating a position in entry sequence of the values into the first data store. The first system transmits a first identifier of a first value and a first sequence ID associated with the first value to a second system. The first system transmits the first sequence ID and the first value to the second system after transmitting the first identifier and the first sequence ID. The second system holds the first identifier and the first sequence ID transmitted from the first system in a first queue. The second system enters the first value received after the first identifier from the first system into a second data store.