Task Scheduling Reducing Cross-Cluster Bandwidth Usage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems experience high bandwidth usage across clusters due to tasks reading and writing across different service units, leading to inefficient network resource utilization.
Innovation Solution
A method that analyzes network resources needed for cross-cluster reading and writing, calculating usage information to schedule tasks based on input-output ratios, optimizing task placement to minimize network resource occupancy by scheduling tasks to clusters where dependent or result data is located.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If tasks are scheduled to clusters where dependent data is located to reduce cross-cluster reads, then network bandwidth usage is reduced, but task scheduling complexity increases due to need to analyze input-output ratios
Solution Approach 1:
The system pre-calculates and stores the input-output ratios of tasks in advance. When a task needs to be scheduled, the scheduler directly queries these pre-computed ratios rather than calculating them in real-time, significantly reducing the computational complexity of the scheduling decision while achieving optimal task placement that minimizes cross-cluster data access
Solution Approach 2:
The patent introduces an intermediary component (the task management device) that acts as a mediator between the task scheduler and the cluster nodes. This intermediary pre-analyzes task characteristics and maintains a repository of input-output ratios, allowing the scheduler to make informed decisions without directly computing complex metrics for each scheduling event, thus reducing overall system complexity
2Productivity
If tasks are scheduled to minimize cross-cluster data access, then network resource utilization improves, but data processing time may increase due to task migration overhead
Solution Approach 1:
The system changes the scheduling parameter from simple service-type matching to a composite metric that includes input-output ratios. By adjusting this parameter to reflect actual data access patterns, the scheduler can make more accurate decisions about task placement, minimizing cross-cluster data access and improving network resource utilization while accounting for migration overhead through the pre-computed ratios
3Ease of operation
If current scheduling methods are used where tasks are assigned by service type only, then scheduling simplicity is maintained, but bandwidth usage across clusters remains high
Solution Approach 1:
The system performs preliminary analysis of task input-output ratios before scheduling decisions are made. By pre-computing and storing these ratios, the system maintains operational simplicity during actual scheduling while incorporating sophisticated optimization criteria that significantly reduce cross-cluster bandwidth usage
Solution Approach 2:
An intermediary task management device is introduced that handles the complex analysis of input-output ratios, allowing the actual scheduling process to remain simple. This mediator pre-processes task characteristics and provides the scheduler with optimized task-to-cluster mappings that reduce bandwidth usage without complicating the scheduling operation itself
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The disclosed embodiments provide a task scheduling method and apparatus. Network resources needed for a task to perform cross-cluster reading and writing are analyzed to obtain usage information of the occupied network resources for reading and writing; and the task is scheduled according to the usage information of the network resources needed for reading and writing. Because the usage information of the network resources occupied for reading and writing respectively represent network resources that can be saved by the cluster where access data is located when the task is scheduled for reading and writing, it can be determined that the cluster to which the task is scheduled can enable the task to occupy less network resources, thereby solving the problem of high bandwidth usage across clusters in current systems.