Asynchronous Journaling Policy for Continuous Data Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional data protection systems face challenges in maintaining continuous data replication under dynamically changing I/O rates, leading to potential production site shutdowns due to backlog of un-logged data transactions and inefficiencies in data recovery processes.
Innovation Solution
Implementing an optimized asynchronous journaling policy that dynamically adapts by switching between nominal and accelerated journaling policies based on transaction backlogs, and using near real-time journaling when transaction rates allow, to manage I/O operations efficiently and prevent site shutdowns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If synchronous journaling is used to ensure data consistency, then data reliability is improved, but system performance deteriorates due to inability to keep pace with high I/O rates
Solution Approach 1:
The system dynamically switches between synchronous and asynchronous journaling modes based on real-time backlog monitoring. When the journal backlog exceeds a threshold, the system transitions to asynchronous mode to prevent shutdown; when the backlog is cleared, it returns to synchronous mode to ensure consistency. This dynamic adaptation resolves the contradiction by allowing the system to optimize performance when reliability constraints are met.
Solution Approach 2:
The system changes the journaling parameter (synchronous vs. asynchronous) based on backlog conditions. By modifying this critical parameter dynamically, the system can tolerate temporary reductions in data consistency guarantees in exchange for maintaining high I/O processing speed, thereby resolving the performance-reliability tradeoff.
2Productivity
If asynchronous journaling is used to improve performance, then I/O processing speed is improved, but data reliability deteriorates due to potential loss of recent transactions
Solution Approach 1:
The system continuously monitors the journal backlog size and uses this feedback to control the journaling mode. When the backlog grows beyond a safe threshold, the system receives feedback that reliability is compromised and switches to synchronous mode. This feedback mechanism ensures that asynchronous mode is only used when data consistency is maintained, resolving the reliability-performance contradiction.
3Reliability
If the backup site processes transactions synchronously, then data consistency is improved, but recovery time increases due to processing delays
Solution Approach 1:
The system performs preliminary journaling actions asynchronously to prepare transaction records in advance, reducing the time required during actual recovery operations. By pre-processing journal entries before they are strictly required, the system maintains consistency while accelerating recovery, thereby resolving the time-consistency contradiction.
Data Source
AI summary
Methods and systems for continuous data replication, including receiving a write transaction from a production site, the write transaction including raw data (RWNEW), a size indicator (SZ1) for the size of the raw data, and a start address (ADD1) for writing the raw data, copying the raw data (RWNEW) into a first journal data stream at a current journal address (JADD1), and copying the size indicator (SZ1), the start address (ADD1) and the journal address (JADD1) into a first journal meta-data stream.