Replication Completion Time Estimation via Machine Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Estimating the completion time of data replication operations in data deduplication systems is challenging, especially with limited information available, and conventional offline techniques often produce inaccurate results due to lack of access to entire datasets.
Innovation Solution
The use of machine learning techniques, specifically curve fitting functions generated from historical replication operation data, to estimate replication completion time and perform automated actions based on these estimates, employing an aggregated Mondrian forest algorithm for online learning and dynamic curve fitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional offline techniques are used for replication completion estimation, then the system can operate with simple processing, but the estimation accuracy deteriorates due to lack of access to entire datasets
Solution Approach 1:
The system performs preliminary actions by collecting and storing replication operation data during the replication process before completion. Historical data including replication speed, data size, and progress information are accumulated in advance, enabling accurate completion time estimation without waiting for the entire dataset to be processed by complex offline techniques.
Solution Approach 2:
A machine learning model serves as an intermediary between the replication system and the estimation requirement. The model processes partial replication data and transforms it into accurate completion time predictions, bridging the gap between limited available information and the need for precise estimation without requiring access to entire datasets.
2Measurement precision
If limited information is available for estimation, then the data processing burden is reduced, but the estimation accuracy deteriorates
Solution Approach 1:
The system implements feedback by continuously monitoring replication progress and using historical replication data to refine completion time estimates. The machine learning model learns from past replication operations and adjusts predictions based on actual performance patterns, maintaining high accuracy even when complete dataset information is not available.
Solution Approach 2:
The system performs preliminary data collection and analysis during the replication process, gathering relevant information about replication speed, data characteristics, and system performance before completion is needed. This preliminary processing enables accurate estimation without requiring access to the entire dataset at once.
3Measurement precision
If dynamic curve fitting functions are generated using machine learning techniques, then the estimation accuracy is improved, but the computational complexity increases
Solution Approach 1:
The system applies partial action by using machine learning techniques only for the specific task of generating curve fitting functions from historical data, rather than applying complex computational methods to the entire replication process. This targeted approach achieves high estimation accuracy while minimizing overall computational resource consumption.
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for estimating replication completion time using machine learning techniques are provided herein. An example computer-implemented method includes obtaining, from one or more data deduplication storage systems, data related to one or more historical replication operations; generating at least one curve fitting function by processing at least a portion of the obtained data using one or more machine learning techniques; generating an estimate for completion time of at least one unexecuted replication operation associated with at least one of the one or more data deduplication storage systems by processing input data from the at least one data deduplication storage system using the at least one curve fitting function; and performing one or more automated actions based at least in part on the generated estimate.