Deep-learning job scheduling method considering multi-resource interleaving
Patent Information
- Application Number
- PCT/CN2025/132963
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-04-14
- Filing Date
- 2025-11-06
- Publication Date
- 2026-10-22
Smart Images
Figure CN2025132963_22102026_PF_FP_ABST
Abstract
Description
Considering deep learning job scheduling methods with multiple resource interleaving Technical Field
[0001] This invention belongs to the field of computing resource management and task scheduling technology, specifically relating to a deep learning job scheduling method that considers the interleaving of multiple resources. Background Technology
[0002] With the improvement of computing power and the rapid growth of data volume, deep learning has become an important tool for solving complex problems and is widely used in fields such as image recognition, natural language processing, and recommender systems. However, the training process of DL models has extremely high requirements for computing resources, especially when processing massive amounts of data, where the computational load increases significantly. Therefore, enterprises and institutions typically build GPU clusters to better support intelligent business processing. The core task of the cluster scheduler is to manage these high-cost infrastructures and allocate various DL training tasks. Efficient scheduling strategies are particularly important when managing a large number of training jobs in the cluster, not only improving resource utilization but also accelerating the overall training process.
[0003] However, most existing scheduling strategies treat GPUs as the sole bottleneck in deep learning training, neglecting the allocation of storage I / O, CPU, and network I / O. However, with the rapid development of deep learning, models of varying sizes and types have emerged, and the training of these models may encounter performance bottlenecks on other resources. For example, in the training of small deep learning models, storage I / O often becomes a resource bottleneck; the limited number of model parameters allows GPUs to complete training quickly, and the GPU's computational power far exceeds the data retrieval speed. In reinforcement learning, agents need to frequently interact with the environment, and the simulation of complex environments often requires significant CPU resources, potentially exceeding the GPU computation time required for neural network inference. Furthermore, in the distributed training of large deep learning models, the frequent synchronization of model weights and gradient information among multiple nodes makes network I / O a key factor limiting efficiency. Therefore, GPU-based scheduling methods alone are insufficient to meet the diverse resource demands of modern deep learning training jobs.
[0004] Given the phased, iterative computational nature of deep learning training jobs, fine-grained, interleaved management of multiple resources can be achieved over time. Specifically, each training iteration typically includes steps such as data loading, preprocessing, forward propagation, back propagation, and gradient synchronization, each with varying degrees of resource dependency. For example, the data loading phase primarily relies on storage I / O, while forward and back propagation place greater demands on GPUs. Technical issues
[0005] To address the shortcomings and deficiencies of existing technologies, this invention provides a deep learning job scheduling method that considers multiple resource interleaving. By using phased resource binding, hybrid optimization algorithms, and dynamic scheduling strategies, it significantly shortens the global completion time of deep learning jobs in a cluster environment. Technical solutions
[0006] The core innovations of this solution include:
[0007] Multi-resource phased binding model: Divide deep learning jobs into four phases: data loading (storage I / O), preprocessing (CPU), forward / backward propagation (GPU), and gradient synchronization (network I / O), clarify the resource requirements of each phase, and provide a theoretical basis for multi-resource staggered optimization;
[0008] DPGB Hybrid Algorithm Architecture: Combining Discrete Particle Swarm Optimization and Genetic Algorithm (DDPSO-GA) with multi-round Blossom algorithm, it implements "job-node mapping" and "multi-resource interleaved grouping" step by step, and reduces the complexity of high-dimensional search through discrete encoding, adaptive parameter adjustment and hypergraph matching strategies;
[0009] Dynamic resource scheduling mechanism: Based on the time sequence arrangement of jobs within the group and the maximum time consumption accumulation rule of each stage, the time of a single iteration is quantified (Formula 10). Combined with the exhaustive sorting optimization of four job groups, it realizes the parallel and efficient utilization of non-GPU resources such as storage IO and CPU.
[0010] Hybrid initialization and differentiated fitness design: Elite particles are generated based on job submission time, combined with a random population initialization strategy to accelerate convergence and ensure global search capability. At the same time, invalid solutions are gradually eliminated through a feasible / infeasible particle hierarchical evaluation mechanism.
[0011] In experiments on the Microsoft Philly dataset, this invention significantly shortens the completion time and improves resource utilization compared to traditional scheduling algorithms (SRTF, AntMan, etc.), making it particularly suitable for cluster scheduling scenarios with high load and multiple types of deep learning jobs.
[0012] The specific technical solution adopted by this invention to solve its technical problem is as follows:
[0013] A deep learning job scheduling method considering multi-resource interleaving includes the following steps:
[0014] Deep learning tasks are divided into four stages: data loading, data preprocessing, forward / backward propagation, and gradient synchronization, which are respectively bound to four types of resource requirements: storage I / O, CPU, GPU, and network I / O.
[0015] Based on the resource demand vectors and node resource capacity vectors at each stage of the task, establish the mapping constraints from task to node.
[0016] Based on the phase division and constraints, the DPGB algorithm is used to determine the mapping scheme from job to node, and the job is grouped into multiple resources within the same node to generate a scheduling scheme that minimizes the global completion time. The DPGB algorithm includes a hybrid strategy of discrete particle swarm optimization and genetic algorithm, DDPSO-GA, and a multi-round Blossom algorithm.
[0017] Based on the grouping scheme, resources are dynamically allocated on the nodes in stages to achieve parallel utilization of multiple resources.
[0018] Furthermore, in the DPGB algorithm:
[0019] The DDPSO-GA is used to map jobs to nodes, including discrete particle encoding, hierarchical design of fitness functions, and dynamic parameter tuning;
[0020] The multi-round Blossom algorithm is used to perform hypergraph matching and grouping of jobs on the same node, and optimizes resource interleaving efficiency through multi-round minimum weight matching.
[0021] Furthermore, in the DDPSO-GA, the position of each particle is determined by a mapping relationship from job to node:
[0022] Each particle's position contains multiple elements, the number of which is equal to the total number of tasks n, and each element corresponds to one task.
[0023] The value of each element is the target node number to which the job corresponding to that element is assigned, and the target node number is an integer number of the cluster node;
[0024] The target node number must ensure that the resource requirements of each stage of the operation do not exceed the resource capacity of the target node.
[0025] Furthermore, the fitness function is designed hierarchically as follows:
[0026] For feasible particles, the fitness value is the global completion time T. makespan ;
[0027] For infeasible particles, the fitness value is the number of jobs that violate resource constraints |j inf (X i )|.
[0028] Furthermore, the dynamic parameter adjustment includes:
[0029] Dynamic adjustment of inertia weight: The inertia weight w is dynamically adjusted based on the time difference between the current particle and the global optimal solution. The calculation formula is as follows:
[0030] Among them, wmin and w max Here, represents the lower and upper bounds of the inertia weight, respectively, used to control the global search and local convergence capabilities of the particle; represents the particle's weight in the t-th iteration. The global completion time of the corresponding scheduling scheme; the historical best particle g in the t-th iteration. t Global completion time;
[0031] Acceleration factor linear adjustment: Individual cognitive coefficient c1 decreases linearly from the initial value to the termination value, while social cognitive coefficient c2 increases linearly from the initial value to the termination value.
[0032] Furthermore, the multi-round Blossom algorithm includes the following steps:
[0033] The tasks on the same node are modeled as a four-dimensional hypergraph, and the weight of the hyperedge is the single iteration time of the task grouping;
[0034] The jobs are gradually merged through log24 rounds of matching to generate the minimum weight grouping scheme.
[0035] Furthermore, the optimization objective of the multi-resource staggered grouping is: each group contains 4 jobs, and the grouping scheme with the shortest single iteration time is selected by exhaustively searching all sorts.
[0036] Furthermore, the formula for calculating the time of a single iteration is as follows:
[0037] in, Indicates job grouping G i Different tasks In the The time taken for each stage; 'a' indicates the number of resource types considered; The value is the sum of the execution times of all stages; Q i This is the sorted set of tasks within the group; is the sorting position of the job within the group; z is the stage index in the iteration, corresponding to the four resource types.
[0038] Furthermore, the initial population generation strategy of the DDPSO-GA includes:
[0039] Elite particles are generated based on the job submission time;
[0040] The remaining particles are generated randomly to maintain population diversity.
[0041] And an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method described above.
[0042] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above. Beneficial effects
[0043] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:
[0044] Breakthrough in multi-resource collaborative scheduling capabilities: By binding resource requirements in stages and using a dynamic staggered scheduling mechanism, it achieves for the first time the fine-grained collaborative allocation of four types of resources: storage I / O, CPU, GPU, and network I / O, solving the problem of resource idleness and competition under the single resource perspective of traditional scheduling strategies;
[0045] Balancing algorithm efficiency and solution quality: Based on discrete coding and hybrid optimization strategies (DDPSO-GA+Blossom algorithm), it significantly reduces the search complexity of high-dimensional scheduling problems, effectively approximating the global optimal solution while ensuring the efficiency of generating feasible solutions;
[0046] Improved utilization of non-GPU resources: By optimizing the timing arrangement within job groups and using the maximum time accumulation rule for each stage, the parallel potential of non-GPU resources such as CPU and storage I / O is explored, alleviating the resource bottleneck caused by GPU-centric scheduling;
[0047] Enhanced convergence speed and robustness: The hybrid initialization strategy combined with the dynamic parameter adjustment mechanism accelerates algorithm convergence while avoiding premature convergence and getting trapped in local optima, making it particularly suitable for complex scenarios with high load and mixed submission of heterogeneous jobs.
[0048] Feasibility-oriented hierarchical optimization: By designing differentiated fitness for feasible / infeasible particles, high-quality solutions that meet resource constraints are prioritized, invalid search paths are reduced, and the overall iteration efficiency of the algorithm is improved. Attached Figure Description
[0049] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0050] Figure 1 illustrates the effect of job order on the time of a single iteration: (a) the optimal order of staggered jobs within a group, and (b) the poor order of staggered jobs within a group.
[0051] Figure 2 is an overview diagram of the DPGB algorithm according to an embodiment of the present invention;
[0052] Figure 3 is an example diagram of encoded particles according to an embodiment of the present invention;
[0053] Figure 4 is an example diagram of the variation process in an embodiment of the present invention;
[0054] Figure 5 is an example diagram of the cross process in an embodiment of the present invention;
[0055] Figure 6 is a comparison chart of completion time under normal load in the embodiments of the present invention;
[0056] Figure 7 is a comparison chart of completion time under high load in the embodiments of the present invention;
[0057] Figure 8 shows the impact of the number of tasks within a group on the completion time in an embodiment of the present invention. Embodiments of the present invention
[0058] To make the features and advantages of the present invention more apparent and understandable, specific embodiments are described below in detail:
[0059] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0060] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.
[0061] This invention proposes a multi-resource interleaving-based DL training job scheduling strategy. By interleaving the various stages of the job at different time periods, it achieves efficient resource sharing. A scheduling algorithm, DPGB, is designed to search for the optimal job scheduling scheme and minimize the job completion time in a cluster environment. A scheduling method, DPGB, combining DDPSO-GA and the multi-round Blossom algorithm, is also designed. This method first determines the node to which the job is mapped, and then considers grouping jobs on the same node for multi-resource interleaving, thereby optimizing the job completion time in a cluster environment.
[0062] 1. Problem Definition
[0063] The purpose of this invention is to minimize the completion time of the DL training job set while satisfying the resource requirement constraints of each DL training job. Table 1 defines the main symbols used in this invention.
[0064] 1.1 System Environment
[0065] The system environment consists of m homogeneous nodes (i.e., servers or virtual machines), denoted as S = {s1, s2, ..., sm}. m Each node has four types of resources required for DL training jobs, including storage I / O, CPU, GPU, and network I / O.
[0066] This invention uses Represents each node s i Its resource capacity, of which each element Represents node s i The total amount of resource type y in the node. Besides the resources mentioned above, it is assumed that other resources in the node are sufficient and will not become a bottleneck for the DL training job.
[0067] 1.2 Operation Model
[0068] The set of assignments J = {j1,j2,…,j} n This folder stores deep learning (DL) training jobs submitted by different users. Considering the iterative process and stage division characteristics of DL training jobs, job j is... i This can be represented as shown in formula (1): j i =<α i ,k i ,s i ,R i > (1)
[0069] Where, α i Indicates DL training job j i Submission time, k i Indicates the number of iterations, s i This indicates the node for training the task. DL model training consists of four stages: data loading, data preprocessing, forward / backward propagation, and gradient synchronization. Each stage primarily uses one type of resource, as shown in Table 2.
[0070] Table 1 Symbol Definitions
[0071] Table 2 Bottlenecks in each stage of DL training
[0072] R i For j i The main resource quantities required at each stage are shown in formula (2):
[0073] Each pair Includes the duration of the i-th job in the y-th running phase. and the demand for resource type z at this stage.
[0074] 1.3 Job Scheduling Scheme
[0075] In this embodiment, the job scheduling scheme Γ includes the job-to-node mapping M, the job grouping scheme G, and the completion time T corresponding to the job scheduling scheme. makespan The specific definition is shown in formula (3): Γ=(S,J,M,G,T) makespan (3)
[0076] The mapping M from job to node is shown in formula (4), and each tuple (j i ,s y ) indicates assignment j i At node s y During training, use Indicate j i Mapped to node s y M = {(j i ,s y )|j i ∈J,s y ∈S} (4)
[0077] A feasible scheduling scheme that ensures each job is mapped to a node that meets the resource requirements must satisfy the constraints shown in formula (5):
[0078] Group the jobs waiting to be trained on the same node into a grouping scheme G = {G1, G2, ..., G...} u For each job group G i This is expressed as shown in formula (6):
[0079] Q i The group contains the jobs and their sorting order, as shown in formula (7):
[0080] Each pair Indicates job grouping G i It includes assignment j y And j y Sort by position within the group Use To represent the task grouping G i Internal sort The task is to achieve high resource utilization through multi-resource interleaving within each group. There are multiple sorting methods to alternate tasks within a group, and different sorting methods have different resource interleaving efficiencies, as shown in Figure 1. (a) shows the optimal sorting for interleaving four tasks within a group. In contrast, (b) shows a worse sorting where the four tasks cannot completely overlap. Since the maximum number of tasks in a group is set to 4, all sorting methods can be exhaustively enumerated and their corresponding iteration times calculated. Finally, the shortest iteration time and the corresponding sorting method are selected, and group G is updated. i The order of the tasks in Q i .
[0081] s i It is assignment group G i The trained node has the same value as the node to which all jobs in the group are mapped. Indicates job grouping G i It includes the maximum number of iterations for the job, calculated as shown in formula (8):
[0082] Indicates job grouping G i The earliest time that can be executed is calculated as shown in formula (9):
[0083] Group G of assignments i The time required for one iteration, and Q i It is related to the order of the assignments. The calculation is shown in formula (10):
[0084] Indicates job grouping G i Different tasks In the The time taken for each stage. 'a' represents the number of resource types considered, equal to the number of stages in each iteration within the group. This example considers four types of resources. The execution time of each stage depends on the maximum execution time of all jobs within that stage. The value is the sum of the execution times of all stages.
[0085] T makespan The following variables represent the job completion time corresponding to the scheduling scheme:
[0086] i)t start (G i ,s y ) represents the task grouping G i At node s y The start execution time is determined by node s.y Available time t end (s y The earliest time that the group can be executed. Jointly decided, t start (G i ,s y The definition is shown in formula (11). For the first execution of a node, it is assumed that the node is always available, so its start time depends only on the earliest execution time of the group.
[0087] ii) Assignment Grouping G i Training time t train (G i ) by G i Number of iterations With iteration time The decision is defined as shown in formula (12):
[0088] iii)G i At node s y End time t end (G i ,s y ), equal to G i start time t start (G i ,s y ) and training time t train (G i The sum of t is defined as shown in formula (13): end (G i ,s y )=t start (G i ,s y )+t train (G i (13)
[0089] iv)t end (s y _ represents node s y The end time of the last executed task group is defined as shown in formula (14):
[0090] Based on the variables defined above, the completion time of scheduling scheme Γ is defined as the time of the latest completed job group, as shown in formula (15):
[0091] 1.4 Problem Formulation
[0092] In the context described in this invention, different job scheduling schemes directly affect the completion time of the job set. This invention aims to find a scheduling scheme that is as efficient as possible to minimize the completion time of the job set. Therefore, the problem is formulated as shown in Equation (16):
[0093] 2 Job scheduling strategy based on DPGB algorithm
[0094] Grouping jobs based on various resources, and considering the earliest executable time and end time of each group in different grouping schemes, makes the scheduling order of each group interdependent and impossible to optimize independently. This type of job scheduling problem with dependencies has been proven to be NP-hard. Swarm intelligence algorithms such as PSO and GA are feasible solutions, but uniformly encoding the training position and grouping scheme of each job leads to an excessively large search space and requires a long convergence time. Therefore, this invention proposes a DPGB algorithm to solve the above problem. This algorithm is a deep learning training job scheduling strategy that combines DDPSO-GA and a multi-round Blossom algorithm. DDPSO-GA combines the crossover and mutation operators of the GA algorithm with the PSO algorithm to map jobs to the most suitable nodes. The Blossom algorithm can solve the minimum weight matching problem, and by adjusting it to multiple rounds, it can solve the job grouping problem considering various resources.
[0095] DDPSO-GA combines the crossover and mutation operators of the GA algorithm with the PSO algorithm to map jobs to the most suitable nodes. The Blossom algorithm can solve the minimum weight matching problem, and by extending it to multiple rounds, it can address job grouping problems that consider various resources.
[0096] 2.1 Overview of the DPGB Algorithm
[0097] Figure 2 shows an overview of the DPGB algorithm. First, an initialization algorithm generates a mapping scheme from jobs to nodes, and a multi-round Blossom algorithm determines the job grouping scheme. Second, the completion time of the job set is calculated based on the job mapping scheme and the grouping scheme. Finally, the DDPSO-GA algorithm is used to iteratively update and find the optimal job scheduling scheme.
[0098] 2.2 Job Mapping Strategy Based on DDPSO-GA Algorithm
[0099] In the traditional PSO algorithm, the population contains n p There are *n* particles, and at the *t*th iteration, each particle can be represented as... For the position information of the particles, This provides information about the particle's velocity and position. This represents a candidate solution, speed It helps each particle search for a better position. Representing the individual's historical best particle, using g t Let represent the best particle in the history of the population. The update process of the position and velocity of each particle is described by formulas (17) and (18):
[0100] Where w is the inertia weight, representing the tendency of the particle to maintain its current direction and velocity. c1 and c2 represent the individual cognitive coefficient and the social cognitive coefficient, respectively, controlling the... and g t The attractive force on the particle. ξ1 and ξ2 are random numbers in the interval [0,1), which adjust the particle's direction. and g t The degree of proximity.
[0101] Traditional PSO algorithms are typically used to solve continuous optimization problems. However, DL training job scheduling involves exploring the optimal mapping of jobs to different nodes, which is a discrete optimization problem. Furthermore, traditional PSO algorithms suffer from slow convergence speed and low efficiency in early iterations. Therefore, this invention proposes the DDPSO-GA algorithm to solve the mapping problem from DL training jobs to nodes.
[0102] 2.2.1 Problem Coding
[0103] To improve algorithm performance and search efficiency, a good coding strategy should meet the following three principles:
[0104] i) Completeness: Each candidate solution in the problem space can be encoded as a particle.
[0105] ii) Non-redundancy: A candidate solution in the problem space has only one corresponding encoded particle.
[0106] iii) Feasibility: Each encoded particle corresponds to a candidate solution in the problem space.
[0107] The encoding of constrained optimization problems often fails to satisfy all the above constraints. Inspired by the literature, a discrete strategy is adopted to generate n-dimensional candidate solution particles. A particle represents a mapping scheme from a DL training job to a node. Figure 3 shows an example of the encoding of the problem in this invention. In the t-th iteration, the i-th particle is represented as shown in formula (19):
[0108] Where n is the number of tasks, and each particle is an integer vector of dimension n. This represents the node mapped to by the y-th task within the i-th particle in the t-th iteration, and its value is the node number, i.e.
[0109] Each job can be executed on different nodes, and the value of the corresponding dimension in each particle is the number of a different node. Therefore, each job scheduling scheme has a corresponding coded particle, which conforms to the completeness principle. Furthermore, the scheduling scheme for each job is represented in the particle by a unique node number, preventing different particle codes from corresponding to the same job scheduling scheme, thus conforming to the non-redundancy principle. However, some particles may be infeasible. In these particles, some jobs may be assigned to nodes that cannot meet their resource requirements, and the coding scheme in this embodiment does not meet the feasibility principle.
[0110] 2.2.2 Fitness Function
[0111] The fitness function is used to evaluate the performance of particles; particles with smaller fitness function values have better performance. The goal of this embodiment is to minimize the completion time of the job set, setting the fitness function to the completion time of the corresponding job scheduling scheme for each particle. However, the discrete encoding strategy proposed in this invention does not satisfy the feasibility principle, as there are two types of particles in the problem space:
[0112] i) Feasible particles: The encoded particles satisfy the job resource requirement constraints, and all DL training jobs are scheduled to be trained on nodes that meet their resource requirements.
[0113] ii) Infeasible particles: Encoded particles do not meet the job resource requirement constraints. At least one DL training job is scheduled for training on a node that does not meet its resource requirements. inf (X i To represent particle X i The set of jobs that do not meet resource requirements.
[0114] In the DPGB scheduling strategy, a differentiated fitness function is designed for infeasible particles, and infeasible solutions are gradually eliminated in conjunction with a population evolution mechanism. Specifically, the fitness function is defined according to different situations, which can be divided into the following three cases:
[0115] i) Both types of particles being compared are feasible particles: the particle with the shorter completion time of the task set is selected, and the fitness function is defined as shown in formula (20): f(X i ) = T makespan (X i (20)
[0116] ii) One of the particles being compared is feasible, while the other is not: the feasible particle will be selected, and the fitness function is defined as shown in Equation (21):
[0117] iii) Both particles being compared are infeasible: choose j inf (X i Shorter particles are more likely to become feasible particles in subsequent iterations. The fitness function is defined as shown in equation (22): f(X) i )=|j inf (X i (22)
[0118] 2.2.3 Initialization Strategy
[0119] Traditional PSO typically initializes the swarm by randomly generating particles. However, this random initialization can lead to a prolonged convergence time to the global optimum. DDPSO-GA employs a hybrid initialization method, generating a high-performance elite particle based on the job submission time, while the remaining particles are generated randomly.
[0120] Algorithm 1 generates initial particles based on job submission time.
[0121] Input: J, S
[0122] Output: X e
[0123] Algorithm 1 demonstrates the elite particle X e The generation first initializes the elite particle X. e and counter n cur First, sort the job set J (lines 1-3) according to the job submission time. Then, retrieve the sorted jobs in order, and evenly distribute them across the nodes based on the remainder of the current counter divided by the total number of nodes. Finally, record the scheduling scheme generated by the algorithm in X. e The algorithm generates particles that prevent nodes from becoming idle due to a lack of work, thus improving the overall efficiency of the system. Using a hybrid initialization strategy helps the algorithm quickly find a better solution in the early stages, while avoiding getting trapped in local optima and preserving the algorithm's ability to explore globally.
[0124] 2.2.4 Particle Update Strategy
[0125] As shown in formulas (17) and (18), the traditional PSO algorithm consists of three main parts: inertia, individual cognition, and social cognition. To improve the algorithm's search capability, this embodiment uses crossover and mutation operators from the genetic algorithm to update the particles, thereby exploring a wider solution space. The particle update strategy is shown in formula (23).
[0126] In this context, F1() and F2() represent crossover operations, and F3() represents mutation operations.
[0127] In this embodiment, the mutation operator of GA is used to replace the inertial part, as shown in formula (24):
[0128] Among them, M u () denotes the mutation operator, where ξ3 is a random number in the interval [0,1). Mutation occurs if ξ3 is less than the inertia weight w. The mutation process uses multi-point mutation, where the number of mutation points is a random integer in the interval [1,|J|]. Different mutation points p are randomly selected for each mutation. i Set it to a new random integer between [1, |S|]. Figure 4 shows a mutation process with 2 mutation points:
[0129] For the individual cognition and social cognition components, the crossover operator of GA is used for replacement, as shown in formulas (25) and (26):
[0130] Where C() represents the crossover operator, and acceleration factors c1 and c2 are used to represent the interaction with the individual's historical best, respectively. The best historical population g t The crossover probability. The crossover process randomly selects two indices p1 and p2 from the old particles and replaces the fragment between them with... (or Fragments of particles, Figure 5 illustrates the crossover process:
[0131] 2.2.5 Parameter Update
[0132] The inertia weight w significantly impacts the search capability and convergence of the PSO algorithm. A larger inertia weight w results in a stronger global search capability but makes convergence more difficult; conversely, a smaller inertia weight w results in a stronger local search capability and easier convergence. The classic adjustment mechanism linearly changes the inertia weight w based on the number of iterations; however, this does not satisfy the non-linear characteristics of DL training job scheduling. A novel adjustment mechanism is used in DDPSO-GA, which adjusts the inertia weight w based on the current particle... Compared with the best historical population g t The search capability is dynamically adjusted based on the difference in completion time, as shown in formula (27):
[0133] When the completion time of a particle differs significantly from the population's historical best, inertia weights must be increased to enhance global search capabilities. Conversely, local search capabilities must be enhanced to accelerate convergence and find the optimal solution.
[0134] Furthermore, the acceleration factors c1 and c2 are adjusted using a linear strategy. and This represents the initial value of the acceleration factor, while and This indicates the end value of the acceleration factor.
[0135] 2.3 Job Grouping Strategy Based on Multi-Round Blossom Algorithm
[0136] After multiple jobs are assigned to the same node, it is necessary to group the jobs using various resources to reduce the completion time of each group, thereby indirectly optimizing the total completion time of the job set. Therefore, this can be viewed as matching i jobs on an i-dimensional hypergraph, where i represents the resource type; this invention considers four different resource types. Thus, the grouping process can be abstracted as a rank-4 hypergraph minimum weight matching problem. In the hypergraph, vertices represent jobs, and hyperedges connect four vertices. The weight of each hyperedge corresponds to the completion time of the four job combinations. The goal is to select a set of non-overlapping hyperedges that minimizes the total weight (i.e., the total completion time of all groups). In graph theory, this is called "minimum weight k-consistent hypergraph matching," which is NP-hard.
[0137] Blossom's algorithm is a graph algorithm used to solve the minimum weight matching problem. However, it can only handle the minimum weight matching problem in simple graphs with two resource types, and cannot handle the minimum weight matching problem in hypergraphs with multiple resource types. This embodiment modifies the Blossom algorithm to a multi-round approach, dividing the matching process into multiple rounds. In each round, the Blossom algorithm is used to group nodes. At the end of each round, each matched pair of nodes is merged, and other nodes are matched in the next round.
[0138] Algorithm 2 illustrates the main process of the multi-round Blossom algorithm. First, the job scheduling scheme corresponding to particle X is extracted, and the set of jobs to be trained at each node is recorded in dictionary D (lines 1-3). The job grouping scheme is calculated for each node, and all jobs at that node are placed in graph H, starting a multi-round matching process of log2 a times (lines 4-8). An edge is added to each pair of nodes in graph H, with its weight being the grouping iteration time after the tasks corresponding to the nodes are grouped. The Blossom algorithm is used to find the minimum weight matching in the graph (lines 9-12). For each pair of jobs in the matching process, their corresponding nodes are merged for matching in the next iteration (lines 13-17). After the multi-round matching is completed, each merged node corresponds to a job grouping scheme, which is recorded in the grouping scheme set G. The time complexity of the Blossom algorithm is O(n log n). 3 ), where n is the number of nodes. The multi-round Blossom algorithm used in this embodiment employs log2 a Blossom algorithms, resulting in a time complexity of O(n^2). 3 ).
[0139] Algorithm 2: Multi-round Blossom Algorithm
[0140] Input: X, a = 4
[0141] Output: G
[0142] 2.4 Mapping of Encoded Particles to Scheduling Schemes
[0143] Algorithm 3 describes the mapping from particles to scheduling schemes. The inputs include a set of nodes S, a set of jobs J, encoded particles X, and a job grouping scheme G. The output is the scheduling scheme Γ = (S, J, M, G, T) corresponding to each particle. makespan Specifically, the process first determines whether the encoded particle X is feasible. This requires checking if each job node has been scheduled to a node that meets all its resource requirements. If a job's resource exceeds the capacity of the corresponding node's resource, the particle is considered infeasible (lines 2-6). The completion time of the job set is then calculated based on the scheduling position of each job and the job grouping scheme (line 7). Finally, the scheduling scheme corresponding to the encoded particle X is obtained, where the job-to-node mapping M can be obtained through the encoding particle X (line 8).
[0144] Algorithm 3: Mapping of Encoded Particles to Scheduling Schemes
[0145] Input: S,J,X,G
[0146] Output: Γ
[0147] Based on the above design, this embodiment of the invention provides a deep learning training job scheduling method (DPGB) based on DDPSO-GA and the Blossom algorithm. The DPGB method combines PSO-GA and graph heuristic algorithms to schedule jobs based on the state of cluster nodes and the initial data of the jobs. Its main steps are as follows:
[0148] Step 1: Initialize the relevant parameters of the DDPSO-GA algorithm, and mix the elite initial particles with other randomly generated particles to form the initial population according to the mixed initialization strategy.
[0149] Step 2: Use the multi-round Blossom algorithm to analyze each particle in the population, and group the tasks on each node based on the fact that the tasks have been mapped to the corresponding nodes.
[0150] Step 3: Generate a scheduling scheme for each particle based on the mapping position and grouping scheme of each job, and calculate the completion time of the job queue as the fitness of the particle.
[0151] Step 4: Based on the fitness of each new particle, update the historical best particle and the population best particle for all individuals, and update parameters such as the acceleration factor.
[0152] Step 5: Determine if the algorithm's stopping condition is met. If not, proceed to step 6.
[0153] Step 6: Following the update strategy of formulas (23)-(26), mutate and cross over the particles, and then proceed to step 2 to recalculate the fitness of the updated particles.
[0154] 3. Methods of evaluation
[0155] Deploying the algorithm in a real cluster environment is difficult and time-consuming, therefore, a simulation experiment was conducted to simulate the cluster environment. The simulation machine was configured with an AMD Ryzen 4800H@2.8GHz processor and 16GB of RAM, and a 64-bit Windows operating system. The DDPSO-GA parameters were set as follows: population size of 50, maximum number of iterations of 1000, and w max =0.9,w min =0.4, 3.1 Experimental Setup
[0156] The dataset used for testing is from Microsoft Philly, which records the jobs executed by Microsoft in a multi-tenant cluster over two months. It was divided into four datasets with job sizes ranging from 1457 to 5754 based on the virtual cluster number for experimental simulation. Since the Microsoft Philly dataset does not provide a training model for each job, eight common deep learning models were randomly selected in this embodiment. Table 3 shows the resource bottleneck types of these models, and the number of iterations for each job is calculated based on its duration and the time per iteration.
[0157] Table 3 shows the DL models used in the dataset.
[0158] This embodiment constructs a simulator to evaluate the performance of the DPGB algorithm on large-scale datasets and under different configurations. The simulator is developed based on open-source code and its error compared to the real cluster environment is controlled within 5%, demonstrating high accuracy and reliability. The simulated cluster environment consists of 8 nodes, each equipped with 8 NVIDIA V100 GPUs, 92 CPU cores, 520GB of memory, ample storage space, and 1Gbps of network bandwidth.
[0159] 3.2 Comparison Algorithm
[0160] This embodiment compares three algorithms with the DPGB-based multi-resource interleaved deep learning job training scheduling strategy. The specific experimental setup is as follows:
[0161] 1) SRTF (Shortest Remaining Time First). This algorithm allocates jobs to nodes based on their remaining training time. The less training time a job has left, the higher its scheduling priority. This algorithm does not use multi-resource cross-running.
[0162] 2) AntMan (GPU Cluster Dynamic Scaling Algorithm). This algorithm introduces a dynamic scaling mechanism for memory and computation within the deep learning framework, allowing for fine-grained coordination between jobs and preventing job interference. This algorithm does not use multi-resource cross-running.
[0163] 3) Muri (Multi-Resource Cluster Scheduling Algorithm). This algorithm uses a graph heuristic to group jobs, improves resource utilization through multi-resource interleaving, and allocates nodes according to the number of GPUs required for each group.
[0164] 3.3 Results and Analysis
[0165] To compare the scheduling performance of DPGB and other comparative algorithms for DL training jobs in a cluster environment, 10 independent and repeated experiments were conducted under two different load scenarios. Figure 6 shows the completion time of each algorithm under normal load, in units of 100,000 seconds, where datasets 1-4 are the original datasets partitioned from Microsoft Philly. As can be observed from the figure, compared with the SRTF, AntMan, and Muri algorithms, the DPGB algorithm shows an average performance improvement of 32.24%, 23.08%, and 2.51%, respectively. This performance enhancement stems from the DPGB algorithm's effective reduction of the search space through a step-by-step optimization strategy. First, DDPSO-GA combines the advantages of PSO and GA to efficiently allocate jobs to the most suitable nodes; subsequently, the multi-round Blossom algorithm further groups the jobs on the nodes, thereby optimizing the scheduling order. This step-by-step approach not only avoids high-dimensional search of the overall problem but also significantly improves the convergence speed, making resource utilization more efficient in a multi-tasking environment. The SRTF algorithm schedules jobs solely based on their remaining time, neglecting resource requirements. This singular scheduling strategy leads to inefficient resource allocation, preventing some jobs from obtaining necessary resources under high resource demands and further extending completion times. The Antman algorithm achieves GPU sharing through dynamic scaling; however, it doesn't consider the cross-operation of other resources, potentially limiting overall resource utilization efficiency under diverse job requirements and failing to fully leverage the cluster's potential. The graph heuristic algorithm used in Muri experiences an exponential increase in possible job combinations and resource allocation schemes as the number of nodes increases. This dramatically increases the possibilities to be processed during matching, significantly raising computational complexity. Since each matching round can only handle a limited number of node combinations, it cannot comprehensively consider all resource configurations, potentially leading to degraded matching results and failing to fully utilize resource potential.
[0166] By setting the submission time of all jobs in the dataset to 0, a high-load scenario of a large number of jobs being submitted in a short period of time was simulated. Dataset 1`-4` represents this high-load scenario. Figure 7 visually reflects the shortest completion time achievable by the four algorithms under high load. It can be seen that the DPGB algorithm leads on all datasets, followed by the MURI algorithm, then the AntMan algorithm, with the SRTF algorithm performing the worst. On all datasets, the DPGB algorithm shows a greater improvement in completion time compared to other algorithms under high load. This is because when all jobs are submitted at time 0, the resource contention is high, providing more opportunities for resource interleaving, allowing the DPGB algorithm to provide significant improvements. Experimental results show that the algorithm proposed in this embodiment can maintain good performance in high-load environments.
[0167] To observe the impact of the number of jobs within a group on completion time, this embodiment adjusts the number of jobs within a group based on the settings of the basic experiment, setting the number of jobs to {2, 3, 4}. Figure 8 shows the impact of different numbers of jobs within a group on completion time. As the number of jobs within a group increases, the completion time decreases accordingly. This is because increasing the number of jobs within a group can improve the efficiency of resource interleaving. When executing in groups, the number of resources interleaved in the time dimension is consistent with the number of jobs within the group. The minimum completion time is reached when the number of jobs within a group is 4.
[0168] This embodiment also measures the average job completion time (JCT) of the DPGB algorithm under different load conditions. Tables 4-5 show the normalized average JCT results of different algorithms, with the DPGB algorithm as the benchmark. Experimental results show that the DPGB algorithm improves the average JCT by more than 1.74 times compared to SRTF and more than 6.63 times compared to Antman under normal load. There are slight differences with Muri due to different datasets, but overall, the performance of both is similar, showing a high level of optimization. The performance under high load is similar to that under normal load. This indicates that the DPGB algorithm can further optimize the overall completion time while maintaining a good average JCT.
[0169] Table 4 Comparison of Normalized Average JCT under Normal Load
[0170] Table 5 Comparison of Normalized Mean JCT under High Load
[0171] In the proposed scheme of this invention, a DL training job scheduling model based on multi-resource interleaving in a cluster environment is first proposed. Next, discrete particles are used to encode the DL training job scheduling problem, and a scheduling method DPGB combining DDPSO-GA and the multi-round Blossom algorithm is designed to optimize job completion time in a cluster environment. Finally, extensive experiments are conducted to verify the effectiveness and feasibility of the proposed method. Experimental results show that the proposed method has better performance compared to comparative algorithms.
[0172] Based on the same inventive concept, this invention also provides a computer device, comprising: one or more processors, and a memory for storing one or more computer programs; the programs include program instructions, and the processor executes the program instructions stored in the memory. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, used to implement one or more instructions, specifically for loading and executing one or more instructions stored in a computer storage medium to implement the above-described method.
[0173] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium storing a computer program, which, when executed by a processor, performs the above-described method. This storage medium can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0174] It should be noted that, unless otherwise defined, the technical or scientific terms used in this invention should have the ordinary meaning understood by one of ordinary skill in the art to which this invention pertains. The terms "first," "second," and similar terms used in this invention do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0175] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.
[0176] This invention is not limited to the preferred embodiment described above. Anyone inspired by this invention can derive other forms of deep learning job scheduling methods that consider multiple resource interleaving. All equivalent changes and modifications made within the scope of the claims of this invention shall fall within the scope of this invention.
Claims
1. A deep learning job scheduling method considering multi-resource interleaving, characterized in that, Includes the following steps: Deep learning tasks are divided into four stages: data loading, data preprocessing, forward / backward propagation, and gradient synchronization, which are respectively bound to four types of resource requirements: storage I / O, CPU, GPU, and network I / O. Based on the resource demand vectors and node resource capacity vectors at each stage of the task, establish the mapping constraints from task to node. Based on the phase division and constraints, the DPGB algorithm is used to determine the mapping scheme from job to node, and the job is grouped into multiple resources within the same node to generate a scheduling scheme that minimizes the global completion time. The DPGB algorithm includes a hybrid strategy of discrete particle swarm optimization and genetic algorithm, DDPSO-GA, and a multi-round Blossom algorithm. Based on the grouping scheme, resources are dynamically allocated on the nodes in stages to achieve parallel utilization of multiple resources.
2. The deep learning job scheduling method considering multi-resource interleaving according to claim 1, characterized in that: In the DPGB algorithm: The DDPSO-GA is used to map jobs to nodes, including discrete particle encoding, hierarchical design of fitness functions, and dynamic parameter tuning; The multi-round Blossom algorithm is used to perform hypergraph matching and grouping of jobs on the same node, and optimizes resource interleaving efficiency through multi-round minimum weight matching.
3. The deep learning job scheduling method considering multi-resource interleaving according to claim 2, characterized in that: In the DDPSO-GA, the position of each particle is determined by the mapping relationship from job to node: Each particle's position contains multiple elements, the number of which is equal to the total number of tasks n, and each element corresponds to one task. The value of each element is the target node number to which the job corresponding to that element is assigned, and the target node number is an integer number of the cluster node; The target node number must ensure that the resource requirements of each stage of the operation do not exceed the resource capacity of the target node.
4. The deep learning job scheduling method considering multi-resource interleaving according to claim 2, characterized in that: The fitness function is designed hierarchically as follows: For feasible particles, the fitness value is the global completion time T. makespan ; For infeasible particles, the fitness value is the number of jobs that violate resource constraints |j inf (X i )|.
5. The deep learning job scheduling method considering multi-resource interleaving according to claim 2, characterized in that: The dynamic parameter adjustment includes: Dynamic adjustment of inertia weight: The inertia weight w is dynamically adjusted based on the time difference between the current particle and the global optimal solution. The calculation formula is as follows: Among them, w min and w max Here, represents the lower and upper bounds of the inertia weight, respectively, used to control the global search and local convergence capabilities of the particle; represents the particle's weight in the t-th iteration. The global completion time of the corresponding scheduling scheme; the historical best particle g in the t-th iteration. t Global completion time; Acceleration factor linear adjustment: Individual cognitive coefficient c1 decreases linearly from the initial value to the termination value, while social cognitive coefficient c2 increases linearly from the initial value to the termination value.
6. The deep learning job scheduling method considering multi-resource interleaving according to claim 2, characterized in that: The multi-round Blossom algorithm includes the following steps: The tasks on the same node are modeled as a four-dimensional hypergraph, and the weight of the hyperedge is the single iteration time of the task grouping; The jobs are gradually merged through log24 rounds of matching to generate the minimum weight grouping scheme.
7. The deep learning job scheduling method considering multi-resource interleaving according to claim 1, characterized in that: The optimization objective of the multi-resource staggered grouping is: each group contains 4 jobs, and the grouping scheme with the shortest single iteration time is selected by exhaustively searching all sorts.
8. The deep learning job scheduling method considering multi-resource interleaving according to claim 1, characterized in that: The formula for calculating the time of a single iteration is: in, Indicates job grouping G i Different tasks In the The time taken for each stage; 'a' indicates the number of resource types considered; The value is the sum of the execution times of all stages; Q i This is the sorted set of tasks within the group; is the sorting position of the job within the group; z is the stage index in the iteration, corresponding to the four resource types.
9. The deep learning job scheduling method considering multi-resource interleaving according to claim 2, characterized in that: The initial population generation strategy of the DDPSO-GA includes: Elite particles are generated based on the job submission time; The remaining particles are generated randomly to maintain population diversity.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as claimed in any one of claims 1-9.