Backup Server Data Replication via Serialized Stream Sorting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Periodic replication of changed data can be time-consuming and resource-intensive, leading to inconsistencies between replica and master servers, especially during updates.
Innovation Solution
A method and system for creating backups or replicas involve receiving a serialized stream of data changes, sorting and storing them, sending acknowledgments, writing to a single file, and merging with existing backup data on the replica server, optimizing data transfer and maintaining coherence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If periodic replication is used to backup changed data, then data consistency between master and replica servers is maintained, but the replication process becomes time-consuming and resource-intensive
Solution Approach 1:
The system performs preliminary actions by buffering incoming data changes in memory before writing to disk, and by pre-establishing the replication infrastructure. This allows the actual data transfer to be completed more quickly when replication is triggered, reducing overall replication time while maintaining consistency through the buffered data queue.
Solution Approach 2:
The replication process is segmented into distinct phases: data buffering in memory, acknowledgment sending to the master server, and disk writing. This segmentation allows parallel processing of multiple operations and enables the system to handle replication more efficiently by not blocking on any single operation.
2Reliability
If periodic replication is used to backup changed data, then data consistency between master and replica servers is maintained, but system resources are consumed excessively
Solution Approach 1:
The system extracts only the changed data portions from the master server and buffers them in memory on the replica server, rather than replicating entire files or databases periodically. This extraction approach significantly reduces the amount of data that needs to be processed and transferred, lowering system resource consumption while maintaining data consistency for the changed portions.
Solution Approach 2:
The system performs partial replication by only buffering and replicating the specific data changes that occur, rather than performing complete periodic backups. This partial action approach reduces resource consumption by avoiding redundant replication of unchanged data while still ensuring consistency for the modified portions.
3Productivity
If data is written to multiple files during replication, then data can be processed in parallel, but data coherence and consistency become difficult to maintain
Solution Approach 1:
The system merges multiple incoming data change streams into a single buffered queue in memory before writing to disk. This merging approach maintains data coherence by ensuring all changes are collected and ordered properly before persistence, while still allowing parallel processing of the buffering and acknowledgment operations that lead up to the unified write operation.
4Loss of energy
If acknowledgment is sent immediately upon receiving data stream, then network bandwidth is utilized efficiently, but data integrity cannot be ensured if replication is interrupted
Solution Approach 1:
The system performs preliminary buffering of data in memory and sends acknowledgments to the master server before actually writing to disk. This preliminary action allows the network transaction to complete efficiently while the data is still in the buffer, ensuring that if an interruption occurs, the buffered data can be recovered and written without requiring retransmission, thus maintaining both bandwidth efficiency and data integrity.
Data Source
AI summary
Systems, methods, and software program products discussed herein can create a backup or replica of a master. A method can include receiving, at a backup server, a serialized stream of data representative of changes to a first file of a computer and sorting the received stream of data and storing the sorted data in a memory. The method can include sending an acknowledgment to the computer indicating that the stream of data was received, writing all the sorted data to a single second file, and merging, on the backup server, the written data with data representative of a backup of data on the computer in response to writing all the sorted data to the single second file.