Elastic Cloud Data Transfer Queue Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current bandwidth provisioning methods often result in over-provisioning, leading to excess costs due to peak demand occurring infrequently, and attempts to lower costs by provisioning less bandwidth are ineffective as they lead to expensive overcharges during peak usage in large data center applications like data mirroring or backup.
Innovation Solution
A computer-implemented method that analyzes data properties for transfer across an elastic cloud computing network, assigns transfer queue weightings based on parameters, and controls bandwidth allocation and schedule to optimize data transfer costs, allowing for staged transfers with intermediate storage to reduce latency and optimize costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bandwidth is provisioned to meet projected peak data demand, then data transfer reliability is improved, but cost increases due to over-provisioning for unused bandwidth
Solution Approach 1:
The system dynamically adjusts bandwidth allocation based on actual data transfer needs rather than static peak demand projections. The elastic network bandwidth is provisioned on-demand during staged transfers, allowing the bandwidth capacity to flexibly expand or contract according to real-time transfer requirements, thus eliminating over-provisioning while maintaining transfer reliability.
Solution Approach 2:
The data transfer is divided into multiple staged transfers rather than a single bulk transfer. Each stage transfers a portion of the data, allowing bandwidth to be allocated and released in manageable increments. This segmentation enables cost-effective bandwidth utilization by matching capacity to actual needs at each stage while ensuring continuous reliable transfer progress.
2Loss of energy
If bandwidth is provisioned lower to reduce costs, then bandwidth cost decreases, but overcharges occur when peak demand exceeds provisioned bandwidth
Solution Approach 1:
The elastic network provides dynamic bandwidth provisioning that can scale up during peak transfer periods within each stage. Rather than committing to a fixed low bandwidth allocation, the system leverages on-demand elastic capacity to handle peak demands as they arise, preventing overcharges while maintaining transfer reliability through adaptive capacity adjustment.
Solution Approach 2:
The system provisions bandwidth partially for each staged transfer rather than allocating full peak capacity upfront. Each stage uses only the bandwidth necessary for that specific transfer portion, avoiding payment for excessive unused capacity while ensuring sufficient bandwidth is available to complete each stage without interruption or overcharge penalties.
3Speed
If data is transferred in a single bulk operation, then transfer speed is improved, but transfer cost increases and latency increases for large data sets
Solution Approach 1:
The large data set is divided into multiple smaller data subsets transferred in staged operations. Each stage transfers a manageable portion of the total data, enabling optimized bandwidth allocation for each subset. This segmentation maintains effective transfer speed by avoiding resource exhaustion while reducing overall cost through efficient elastic bandwidth utilization across multiple stages.
Solution Approach 2:
The data transfer occurs in periodic staged intervals rather than as a single continuous bulk operation. Each stage represents a periodic transfer cycle that can be optimized independently for cost and speed. This periodic approach allows the system to release and re-allocate bandwidth efficiently between stages, maintaining high transfer speeds while minimizing total cost through elastic resource management.
4Loss of time
If all data is transferred simultaneously, then completion time is reduced, but cost optimization is lost due to inability to prioritize fast-changing data
Solution Approach 1:
Different data subsets receive different transfer priorities and bandwidth allocations based on their specific characteristics. Fast-changing data is identified and prioritized for earlier transfer with higher bandwidth allocation, while less critical data is transferred with lower priority. This local quality differentiation optimizes both completion time for critical data and overall cost efficiency through targeted resource allocation.
Solution Approach 2:
The system performs preliminary analysis of data properties before transfer to identify fast-changing data and establish transfer priorities. This preliminary characterization enables the staged transfer process to allocate bandwidth and schedule transfers optimally from the outset, ensuring that time-sensitive data is transferred first with appropriate speed while maintaining cost optimization through efficient resource utilization across all stages.
Data Source
AI summary
Controlled transfer of data over an elastic network is provided and includes analyzing properties of a set of data for transfer from a source storage location to a target storage location across an elastic cloud computing network as part of a staged transfer of the set of data, assigning transfer queue weightings to data subsets of the set of data based on weighting parameters, the transfer queue weightings dictating order of transfer of the data subsets from the source storage location to the target storage location, and controlling transfer of the data subsets from the source storage location to the target storage location consistent with the transfer queue weightings assigned to the data subsets, the controlling transfer including controlling, based on costs to transfer the data subsets, bandwidth allocation of the elastic cloud computing network and a transfer schedule for the data subsets.