Incremental Data Synchronization via Sequence Number Logging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data synchronization methods using periodic updates suffer from poor real-time performance and inefficient network bandwidth usage, as they often synchronize large amounts of data at once, leading to bottlenecks and reduced efficiency in service processing.
Innovation Solution
Implement an incremental synchronization method where a source end records write operations and sequence numbers, and only pushes real-time write operation information to a destination end if the source end's sequence number is greater, allowing for real-time updates and reducing network bandwidth usage by synchronizing only incremental data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If periodic synchronization is used to synchronize data from source end to destination end, then data consistency is maintained, but real-time performance is poor and network bandwidth is wasted due to synchronizing large amounts of data at once
Solution Approach 1:
The patent segments the data synchronization process into incremental updates based on sequence numbers. Instead of synchronizing all data periodically, only the changed portions (incremental data) are synchronized. The source end divides data into multiple segments identified by sequence numbers, and the destination end requests or receives only the segments that have changed, thereby improving real-time performance while maintaining data consistency.
Solution Approach 2:
The patent implements periodic checking of sequence numbers to determine whether incremental synchronization is needed. The source end periodically checks if new data changes have occurred by comparing sequence numbers, and only triggers synchronization when changes are detected. This periodic action mechanism ensures real-time performance by avoiding unnecessary synchronous operations while maintaining data consistency.
2Loss of time
If periodic synchronization with short duration is used to improve real-time performance, then real-time performance improves, but network bandwidth is wasted due to synchronizing huge quantity of data again in each process
Solution Approach 1:
The patent extracts only the necessary incremental data for synchronization by comparing sequence numbers between source and destination ends. Instead of transferring all data, the system identifies and transfers only the specific portions that have changed (data with sequence numbers greater than the last synchronized sequence number). This extraction approach improves real-time performance while minimizing network bandwidth consumption.
Solution Approach 2:
The patent uses sequence numbers as a parameter to control the synchronization process. By changing the synchronization criterion from periodic time-based updates to sequence number-based incremental updates, the system achieves real-time performance without wasting network bandwidth. The sequence number parameter enables the system to determine precisely what data needs to be synchronized, avoiding redundant data transmission.
3Productivity
If agent query is used to acquire basic data locally, then processing bottleneck at basic data query server is solved, but service processing time is occupied by UDP packet queries
Solution Approach 1:
The patent implements preliminary action by pre-synchronizing basic data from the basic data entry end to service servers before it is needed. The agent on each service server proactively pulls or receives incremental data updates in advance, storing them locally. When service processing requires basic data, it can immediately access the pre-synchronized local copy without occupying service processing time with UDP queries, thus resolving the time occupation issue while maintaining the relief of query server bottleneck.
Data Source
AI summary
The disclosure provides a data synchronization system, apparatus and method. A primary memory area is created at a destination end apparatus, and an incremental synchronization process is performed. A source end apparatus records a log for a write operation, including operation information and a sequence number for the operation. A latest destination-end write operation sequence number is recorded. The source end pushes in real-time write operation information corresponding to a source-end write operation sequence number greater than the latest destination-end sequence number to the destination end if a latest source-end write operation sequence number is greater than the latest destination-end sequence number. The destination end apparatus performs a write operation on the primary memory area according to the write operation information, records a write operation sequence number of the primary memory area and sends a latest write operation sequence number of the primary memory area to the source end apparatus.