Dynamic Bandwidth Allocation for Distributed Data Copy Jobs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge in cloud computing is the unpredictable nature of data movement between geographically distributed data centers, which can lead to delayed high-priority data transfers due to the congestion caused by large amounts of low-priority data, resulting in an inability to maintain the agreed-upon level of service for customers.
Innovation Solution
A bandwidth sharing system that allocates network bandwidth to multiple copy jobs across distributed computing systems using a global traffic manager node, which coordinates with a data copier to prioritize copy jobs based on service level agreements, actual bandwidth usage, and registration information, ensuring that high-priority data transfers are not delayed by low-priority data movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If bandwidth is dedicated to different types of traffic to ensure service level agreements, then high-priority data transfers are protected from delays, but the system cannot adapt to unpredictable customer data movement patterns and still fails to maintain agreed-upon service levels
Solution Approach 1:
The patent implements dynamic bandwidth allocation where the system continuously monitors actual bandwidth usage by different copy jobs and adjusts allocations in real-time. The global traffic manager receives bandwidth usage information from source nodes and dynamically reallocates bandwidth to ensure service level agreements are met, transitioning from static dedicated bandwidth to adaptive dynamic allocation that responds to changing traffic patterns
Solution Approach 2:
The system establishes a feedback loop where source nodes report actual bandwidth usage to the global traffic manager, which then uses this information to adjust bandwidth allocations. This closed-loop control enables the system to adapt to unpredictable customer data movement by continuously monitoring usage patterns and reallocating bandwidth to maintain service level agreements
2Productivity
If large amounts of low-priority data are transferred between data centers, then storage capacity is optimized and data is backed up, but high-priority data transfers are delayed due to network congestion
Solution Approach 1:
The patent segments bandwidth into different priority allocations based on copy job types. The global traffic manager divides available bandwidth among different categories of data transfers (e.g., high-priority customer data vs. low-priority backup data), ensuring that high-priority transfers receive sufficient bandwidth to meet service level agreements while still allowing low-priority transfers to occur within allocated portions
Solution Approach 2:
The system applies different quality of service characteristics to different data transfer types by allocating specific bandwidth portions to different copy job types. High-priority customer data receives guaranteed bandwidth allocations with stricter service level requirements, while low-priority backup data receives remaining bandwidth with more flexible timing, allowing each type to be optimized for its specific requirements
3Reliability
If dedicated bandwidth is allocated to different traffic types, then service level agreements can be maintained, but the system lacks flexibility to handle unpredictable customer data movement patterns
Solution Approach 1:
The global traffic manager serves multiple functions: it allocates bandwidth to different copy job types, monitors actual bandwidth usage across the network, collects registration information about ongoing transfers, and dynamically adjusts allocations based on aggregated usage data. This multi-functional approach consolidates complexity into a single coordinating entity rather than requiring complex distributed control at each node
4Productivity
If multiple copy jobs execute simultaneously across distributed data centers, then data movement throughput is increased, but bandwidth congestion occurs and service level agreements cannot be guaranteed
Solution Approach 1:
The global traffic manager acts as an intermediary between multiple source nodes executing copy jobs and the shared network bandwidth. It coordinates bandwidth allocation across all source nodes, aggregates usage information from multiple locations, and distributes allocation decisions back to individual nodes, enabling simultaneous execution of multiple copy jobs while preventing any single node from causing network congestion that would violate service level agreements
Data Source
AI summary
A bandwidth sharing system is provided that has worker nodes with executor threads for copying data from source nodes to destination nodes based on assigned copy jobs. To execute its assigned copy job, a worker thread of a worker node registers its copy job with the source node. The source node allocates a portion of its bandwidth to the copy job. The source node sends the data of the copy job to the worker thread, and the worker thread forwards the data to the destination node. Upon completion of the copy job, the worker thread deregisters the copy job. The deregistration allows the source node to reallocate the portion of its bandwidth that was allocated to the copy job to another copy job.