Storage Data Transfer Scheduling via Performance Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data transfer systems lack an efficient method to determine the optimal time for conducting data transfers between storage devices, often leading to overloading or underutilization of resources, which can result in poor performance and increased costs due to inefficient scheduling.
Innovation Solution
A system and method that schedules data transfers by estimating the performance of source and target storage devices based on recent actual data transfers and proposed transfer metrics, comparing these estimates to performance thresholds to decide whether to execute the transfer immediately or postpone it, thereby optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data transfer is executed immediately without scheduling, then data transfer speed is improved, but storage device performance deteriorates due to resource overload
Solution Approach 1:
The system performs preliminary estimation of storage device performance before executing data transfer. The performance estimation module evaluates whether the storage device can handle the proposed data transfer by comparing estimated performance against thresholds, and only initiates transfer when conditions are favorable, preventing resource overload while maintaining transfer efficiency
2Reliability
If data transfer is scheduled based on performance estimation, then storage device performance is improved, but data transfer time increases due to scheduling delays
Solution Approach 1:
The system dynamically adjusts data transfer scheduling based on real-time performance estimation. Rather than using fixed scheduling intervals, the system continuously monitors storage device performance metrics and adapts the timing of data transfers to match current system conditions, optimizing both performance maintenance and transfer timing
Solution Approach 2:
The system implements feedback mechanisms where performance monitoring continuously tracks storage device metrics, feeds this information back to the performance estimation module, which then adjusts scheduling decisions. This closed-loop control ensures transfers are scheduled at optimal moments based on actual system state rather than predetermined schedules
3Reliability
If data transfer is postponed when performance is below threshold, then storage device performance is maintained, but productivity decreases due to delayed data transfer
Solution Approach 1:
The system applies partial action by selectively postponing only those data transfers that would degrade performance below thresholds, while allowing other transfers to proceed immediately. This selective approach maintains performance for critical operations while preserving productivity for non-critical transfers where immediate execution is safe
Data Source
AI summary
A method and system. In response to a request for a proposed additional data transfer from a source storage system to a target storage system, an estimate is made of utilization of one or more data transfer components of the source storage system and the target storage system that would occur during the proposed additional data transfer combined with other scheduled data transfers involving the source storage system and the target storage system, based in part on (i) an I/O request rate for recent actual data transfers involving the source storage system and the target storage system combined with (ii) an estimated I/O request rate for the proposed additional data transfer. It is ascertained whether the estimated utilization exceeds a corresponding utilization threshold. If so, execution of the proposed additional data transfer is postponed. If not, execution of the proposed additional data transfer is initiated.