Journal Event Consolidation for Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication systems face challenges in preserving recorded operations during system malfunctions, leading to difficulties in accurately determining and replicating data changes, and often result in slow performance due to file-by-file or directory-by-directory comparison.
Innovation Solution
A system and method for data replication that records operations as journal events, consolidates them, and transfers them to auxiliary storage during system malfunctions, allowing for efficient replication to a backup server without impacting performance, using a master application with modules for snapshot creation, synchronization, and event management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If file-by-file or directory-by-directory comparison is used to determine data changes between snapshots, then data change accuracy is improved, but replication speed deteriorates
Solution Approach 1:
The patent segments the data change detection process into two parts: (1) using file system journal events to quickly identify potential changes at the operation level, and (2) performing detailed comparison only on affected files or directories. This segmentation allows the system to avoid full file-by-file comparison while maintaining accuracy for changed data.
Solution Approach 2:
The patent implements preliminary action by maintaining a file system journal that pre-records all data operations (create, delete, modify, rename) as they occur. This preliminary recording of operations allows the replication system to directly identify changes without performing time-consuming comparisons during the replication process itself.
2Measurement precision
If journal events are continuously recorded in memory during system operation, then data change tracking accuracy is improved, but system reliability deteriorates due to memory overflow risk
Solution Approach 1:
The patent introduces an intermediary mechanism - a persistent storage journal or log - that acts as a buffer between the volatile memory and the replication process. Journal events are written to this intermediary storage structure, which can grow dynamically or be managed with circular buffering, thereby preventing memory overflow while maintaining complete operation records.
Solution Approach 2:
The patent applies parameter changes by dynamically adjusting the retention policy and storage capacity allocation for journal events based on system conditions. The system can prioritize storing critical operation events while allowing less critical events to be overwritten or aggregated, thereby adapting the journal management parameters to balance accuracy and reliability.
3Loss of information
If all journal events are sent for replication without consolidation, then data completeness is improved, but CPU usage and network overhead increase
Solution Approach 1:
The patent merges multiple journal events that represent the same logical data change into single consolidated replication units. For example, multiple write operations on the same file are combined into one replication event containing the final file state, and sequential operations on the same directory are batched together. This consolidation maintains data completeness while significantly reducing the number of individual replication transmissions and CPU processing overhead.
Solution Approach 2:
The patent implements continuous consolidation of journal events in the background without interrupting the replication workflow. The system continuously processes and merges journal events as they arrive, maintaining an up-to-date consolidated view of data changes that can be efficiently transmitted to the backup system, thereby keeping the useful action of data protection continuous and efficient.
Data Source
AI summary
Systems and methods for replicating data from a production server to a backup server include recording at least one operation on one or more data items stored in a volume of a production server. The operation may be recorded as at least one journal event in a memory. A determination may then be made regarding whether a system malfunction incident has occurred in the production server and if so, a first set of journal events may be transferred from the memory to an auxiliary storage at a first time instant. At a second time instant, a second set of journal events recorded in the memory between the first and second time instants may be transferred to the auxiliary storage. At one journal event stored in the auxiliary storage unit may then be sent for replication to a backup server.