ML-Predicted Data Flush Replication Mode Switch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Continuous replication systems face challenges in handling large data spikes from applications with low and periodic data traffic patterns, leading to resource strain and disruption, as existing 'fast forward' modes require post-peak identification and result in redundant work due to journal loss.
Innovation Solution
Implementing machine learning to predict large data flushes and proactively switch to fast forward mode, using time-series models and analytics to forecast data flush sizes, thereby reducing resource waste and minimizing disruptions to other applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If continuous replication is used to maintain journal integrity, then data protection is improved, but resource strain and disruption to other workloads occurs during large data flushes
Solution Approach 1:
The system performs preliminary actions by using machine learning models to predict large data flushes before they occur. When a flush is predicted, the replication mode is proactively switched from journal-based continuous replication to fast-forward replication in advance, allowing the system to prepare for the incoming data surge and prevent resource exhaustion while maintaining the ability to protect data.
2Productivity
If fast forward mode is used to reduce resource requirements, then system performance is improved, but journal integrity is lost
Solution Approach 1:
The system dynamically switches between two replication modes based on predicted workload conditions. It transitions from journal-based continuous replication (high reliability, high resource usage) to fast-forward replication (lower resource usage, reduced journal protection) and back again. This dynamic adaptation allows the system to optimize performance during predicted large flushes while maintaining journal integrity during normal operations.
3Reliability
If resources are allocated to handle peak data flushes, then data replication reliability is improved, but disruption to other workloads occurs
Solution Approach 1:
By predicting large data flushes using machine learning analysis of historical patterns, the system takes preliminary action to switch replication modes before the peak occurs. This allows resources to be gradually allocated and prepared in advance rather than abruptly reallocated during the flush, preventing sudden resource contention that would disrupt other workloads while still ensuring replication reliability is maintained.
Data Source
AI summary
Embodiments for predicting large data flushes in a data replication system collecting usage data for assets in the system; analyzing the data using machine learning processes on the basis of each asset and the system as a whole to determine usage trends with respect to the data flush operations; predicting a next large data flush using a time-series model; obtaining a capacity of a journal storage space used for write operations to a storage device in the system; and determining if a size of the predicted next flush size is too large relative to this capacity, and if so, invoking a fast forward mode to not retain I/O history information for undo operations during a replication in order to save resources in the system.