Incremental Checkpoints for Data Replication Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data replication techniques are inefficient and time-consuming, particularly when dealing with large datasets, as they often require locking the file system, blocking other accesses, and incurring significant overhead due to the need for full system scans or checkpointing, which impacts server performance and availability.
Innovation Solution
The implementation of incremental checkpoints, where changes to the file system are mirrored and stored on a remote node in real-time, allowing for iterative updates and reducing the need for full system scans, thereby improving replication efficiency and minimizing downtime.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional checkpoint replication is used to ensure data consistency, then data accuracy is improved, but system performance deteriorates due to file system locking and blocking other accesses
Solution Approach 1:
The patent segments the replication process into two distinct phases: a full copy phase that establishes initial data consistency, and an incremental update phase that captures only changes using copy-on-write mechanisms. This segmentation allows the system to achieve data consistency without continuously locking the file system, thereby maintaining system performance during normal operations.
Solution Approach 2:
The patent performs a preliminary full copy of the file system to the remote node before establishing incremental replication. This preliminary action ensures that a complete baseline exists, and subsequent incremental changes can be applied without requiring full file system locks, thus resolving the contradiction between data consistency and system performance.
2Loss of information
If full system scans are performed to track changes for replication, then data completeness is improved, but time consumption increases significantly
Solution Approach 1:
The patent extracts only the changed data blocks from the file system using copy-on-write mechanisms, rather than performing full system scans. By taking out only the necessary incremental changes and storing them in a change log, the system maintains data completeness while dramatically reducing the time required for replication operations.
Solution Approach 2:
The patent uses copy-on-write technology to create incremental copies of only the modified data blocks. This copying mechanism captures changes efficiently without scanning the entire file system, thereby maintaining data completeness while minimizing time consumption for replication.
3Measurement precision
If read-copy-modify cycle is used for writes to ensure data tracking, then data accuracy is improved, but server performance deteriorates due to doubled write operations
Solution Approach 1:
The patent introduces a change log as an intermediary structure that captures write operations incrementally. Instead of performing read-copy-modify cycles for every write operation, the system uses the change log to record modifications, thereby maintaining data tracking accuracy while avoiding the performance penalty of repeated write operations to the primary file system.
Data Source
AI summary
Incremental checkpoint, for use in data replication, track the changes made to a file system after a point in time at which the incremental checkpoint is created. Data replication techniques using the incremental checkpoints may include taking a regular checkpoint of the file system and creating the first time full copy on remote node using the regular checkpoint. Changes made to the file system are then tracked in an incremental checkpoint that are stored on the remote node. The processes of taking the incremental checkpoint and storing the incremental checkpoint are iteratively performed. The first time fully copy and the incremental checkpoints may then be used for data replication, backup, continuous data protection (CDP), or the like.