Scheduling method for high performance computing

Abstract

Scheduling method for high performance computers in which a network of servers (3) are connected to each other via connections to form respective nodes n generally having mutually different hardware structures, the method comprising the steps of : submitting ( 100 ) a JOB specifying the required computational resources, i. e. the necessary nodes n and for each node n required the desired hardware resources and the time of access to the nodes; defining the number of SUBJOBs as the number of required nodes forming a submitted JOB; defining a binary variable (block 120 ) x s,n ϵ { 0. 1 } where s indicates the subjob to be allocated and n indicates the computation node; defining a cost function H comprising the sum of several addends, in which each addend is given by the multiplication of a coefficient λ I or weight and of a term H i which is in turn a function of the binary variable x s,n . Each term H i corresponds to an objective function to be optimised or to a constraint; operating the search for the global minimum of the cost function H by quantum annealing; the binary variables x s,n of the global minimum of the cost function H identify the computation nodes n on which the subjobs s x s, n can be allocated most efficiently; assigning the identified computation nodes (block 150 ) to the JOB.

Description

[0001] SCHEDULING METHOD FOR HIGH PERFORMANCE COMPUTING

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This Patent Application claims priority from Italian Patent Application No. 102025000000504 filed on January 14, 2025, the entire disclosure of which is incorporated herein by reference.

[0004] TECHNICAL FIELD

[0005] The present invention relates to a scheduling method for High Performance Computing.

[0006] BACKGROUND OF THE INVENTION

[0007] As is known, the term High Performance Computing (HPC) refers to the ability to process data and perform complex calculations at high speed. An HPC system is a cluster of computers and is often referred to as a supercomputer.

[0008] A supercomputer contains hundreds or thousands of servers connected to each other through a network, these servers if required, are able to work together in a synchronized manner to complete one or more tasks according to a parallel processing mode.

[0009] Each server constitutes a node belonging precisely to the supercomputer. The nodes in a cluster are able to work in parallel, increasing processing speed by providing a high-performance computing environment. The cluster nodes can in turn be heterogeneous and make different type of hardware and software resources available.

[0010] The HPC systems are usually used by several users who send requests for access to the hardware resources of nodes for a certain time interval.A request is known by the term JOB. Requests of different users can be mutually very different.

[0011] In the HPC systems, the management of the JOBS of the different users is carried out by a software component called Scheduler, also known as process planner or manager.

[0012] The scheduler implements a scheduling or planning algorithm which, given a set of JOBS coming from different users, establishes a temporal order for the execution of these JOBS, privileging - for example -requests that comply with certain parameters according to a certain scheduling policy, in order to optimise access to the nodes. In more detail, the scheduler determines when a JOB must be processed (JOB scheduling) and which nodes (JOB allocation) must be made available for the execution of that job.

[0013] The most common job schedulers are the following: Simple Linux Utility for Resource Management, better known as SLURM; Portable Batch System (PBS); Torque (resource manager) often used in combination with Moab (workload manager); IBM Spectrum Load Sharing Facility (LSF).

[0014] Many of the commercial schedulers perform a first-in-first-out (FIFO) scheduling attempt of the various JOBS received.

[0015] Generally, the JOBS are considered in order of priority; defining priority rules is one of the most difficult tasks for system administrators, because the rules have a significant impact on workload distribution. This is why this class of JOB schedulers is also referred to as Priority-Based Rules (PRB) scheduler in the literature.

[0016] Typically, a scheduler examines the list of pending JOBS to see whether there are any "holes" in the scheduling where a JOB can be executedwithout delaying pending JOBS with a higher priority. A "hole" is a time interval during which the resources required for the JOB are available. If the scheduler identifies a hole in the scheduling, it can "fill" the hole with the identified JOB, even if there are JOBs with a higher priority in the queue. The scheduler performs this filling or padding operation since it does not contribute to modifying and therefore delaying the start time of any higher priority JOB.

[0017] Although schedulers which carry out a scheduling are known, the Applicant is not aware of commercial software that realises job allocation process through a real optimisation algorithm.

[0018] Optimization would in fact play an important role as it has been demonstrated that in some cases the use of a more sophisticated scheduler brings advantages, especially in the cloud environment. Here is a list of published cases that have highlighted the potential benefits:

[0019] Situations in which the jobs have complicated dependencies on each other, e.g., workflows in which some jobs read output files written by other jobs as in the infrastructures designed for data analytics (Andrew Chung, 2020);

[0020] Cases in which the jobs can be shaped according to available resources (Litzinger, 2022 );

[0021] Situations in which multiple objective functions need to be optimised at the same time e.g. on cloud environment (Prodan, 2014 ) (Deng, 2018 );

[0022] Cases in which the cluster has heterogeneous nodes both in terms of hardware (CPU, GPU, memory,... ) and in terms of Software;Cases in which there are specific policies for the requests of the users, for example in cloud computing (Rao, 2017 ); and

[0023] Cases in which power and energy consumption are particularly relevant (Jingbo Li, 2021 ).

[0024] OBJECT OF THE PATENT

[0025] The scope of the present invention is to realise a scheduling method that uses an efficient optimisation algorithm.

[0026] In accordance with the present invention, a scheduling method of the type described by claim 1 is realised.

[0027] BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Further characteristics and advantages of the present invention will become clear from the following description of a non-limiting embodiment, with reference to the attached figures, wherein:

[0029] Figure 1 schematically shows a plurality of servers connected by means of a network between them;

[0030] Figure 2 schematically shows the main operations realised by the scheduler of the present invention; and

[0031] Figures 3 are graphs showing the advantages obtained by the scheduler of the present invention.

[0032] DETAILED DESCRIPTION OF THE PRESENT INVENTION

[0033] With reference to Figure 1, number 1 denotes as a whole a network of servers 3 connected to each other by connections (of any type) and forming respective nodes n (of known type and therefore represented schematically) generally having a different hardware structure from each other and different computing capabilities. A scheduler 9 is provided which is realised according to the present invention and adapted to receive a plurality of requests, knownby the term JOBs, from different users 12 (indicated schematically).

[0034] The users 12 submit to the scheduler 9 (block 100 ), realised according to the present invention, a JOB specifying the required computational resources and that is the necessary nodes n and for each node n required the desired hardware resources (number of processors, memory needed, etc. ). The time of access to the nodes is also specified.

[0035] As will be clarified by the following description, the number of nodes required represents the number of SUBJOBs that forms that specific JOB.

[0036] For example, according to a PBS programming mode (Portable Batch System or simply PBS is the name of the software that performs the job scheduling) this request is made through the "-1 select" command with which chunks are requested which represent a description of the requested resources. For example, a user may request n all-equal chunks or heterogeneous chunks. For example, using the PBS formalism we have: "-2

[0037] select=1:ncpus=1:mpiprocs=4:ompthreads=12:ngpus=0:host_mem=131503mb:gpu_mem=20302mb:disk=768626mb" which indicates the request for a chunk ( -1 select=l), and for each chunk 1 cpus, 4 mpiprocs, 12 ompthreads and so on are required. Starting from the PBS settings, the number of chunks is then translated into the required number of nodes. In one embodiment, the number of chunks coincides with the number of nodes and thus with the number of sub obs. In other cases a chunk may correspond to a fraction of a node. It depends on the PBS settings that are defined. In PBS formalism, the time of access to the nodes is required by setting the Walltime variable: "-Walltime=190h"

[0038] As mentioned above, the number of nodes required indicates the number of subjobs forming a job. For example, the line: -1select=2:ncpus=1:mpiprocs=4:ompthreads=12:ngpus=0:host_mem=131503mb:gpu_mem=20302mb:disk=768626mb same as the one above but this time two chunks are required so in this case the job consists of two subj obs. After sending a JOB to the scheduler 9 and defining the number of subj obs, the server 9 is configured to verify (block 110 ) whether it is possible for the node n to host the subjobs, i. e. whether the node n has sufficient available hardware resources.

[0039] In case of positive verification, a binary variable is defined (block 120 ):

[0040] Xs, n e { 0, 1 }

[0041] where:

[0042] s indicates the subjob to be allocated; and

[0043] n indicates the computation node (is the node where the executable program known as PBS MOM runs ).

[0044] The binary variable xs,n is worth one if for the node n the subjob s is assigned (xs,n =1 - the subjob s has been assigned on node n) and is worth zero if for the node n the subjob s is not assigned. Subsequently, a cost function H is defined (block 130) comprising the sum of several addends, in which each addend is given by the multiplication of a coefficient

[0045]

[0046] or weight and of a term Hi which is in turn a function of the binary variable xs,n. An example of a cost function H is the following, which is given by the sum of four addends, but the number of the addends could be different, i. e. less or more than the number represented:

[0047] H=λ_A H_A+λ_B H_B+λ_C H_C+λ_D H_DEach term Hi corresponds to an objective function to be optimised or to a constraint.

[0048] For example, the term HA is an objective function used to optimise the use of the resources and is computed using the formula:

[0049] H_A= − ∑_n(∑_s x_sn RES_S)

[0050] Where RESsare the hardware resources that have been required by the sub job s.

[0051] The term HB is a constraint that ensures that a subjob s is not assigned to more than one node n:

[0052] H_B = ∑_s(∑_{n,n':n≠n'} x_sn x_sn')

[0053]

[0054] s n.n'.n^ni

[0055] In the variablexsn' s represents the sub job index and n' represents a further node index.

[0056] is a constraint that ensures that subjobs referring to the same job are executed simultaneously:

[0057] He= xs n ~ I J Ixsj( / [)

[0058] jnn>

[0059]

[0060] s:j(s)=j

[0061] in this equation, j (s) indicates the job to which the subjob s belongs, \ j \ is the number of subjobs forming the job j, j (l) is one of these subjobs, used as representations of the job.HD is a constraint that ensures that the sum of the resources required by all subjobs assigned to the same node must not exceed the available resources of the node:

[0062] ∑_s x_sn RES_S + ∑_k w_k s_{n,k} − R_n)²

[0063]

[0064] k

[0065] RESsare the resources required by the sub ob s, Rnare the available resources of the node at the time of optimisation.

[0066] The variables s_{n,k}

[0067]

[0068] represent slack variables and are needed in optimisation to transform inequalities into equalities. Note that these variables have no physical meaning. They are only entered to ensure that the resources of the subjobs do not exceed the residual resources of the node and then are not considered.

[0069] Another usable objective function HE expresses the physical distance between the computers that form the nodes on which the subjobs of a same job are assigned: this function contains a term that penalizes the solutions that put subjobs of the same job on nodes that are not on the same rack. As is known, the rack indicates the mechanical support structure ( frame cabinet), on which the actual hardware parts, of arbitrary value, are then mounted. In this case, in fact, we only have two possible distance values dn mbetween nodes:

[0070] f dn m= 0 for nodes belonging to the same rack

[0071] (dnm = 1 fornodes belonging to different racks

[0072]

[0073] H_E: minimizes the distance between nodes on which subjobs of a same job are assigned:H_E = ∑_{s,s':j(s)=j(s'), n≠m} (d_{nm} − 1)x_{sn} x_{s'm}

[0074] s,s':j(s)=j(s')

[0075]

[0076] n=tm

[0077] It is also possible to use an objective function that represents job priorities, if present in one' s production workflow. In this case the objective function HAm has the following expression:

[0078] H_{Am}= − ∑_n(∑_s x_{sn} RES_S (1 + λ_G Pr_S)

[0079] where RESsare the hardware resources that have been required by the sub job s;

[0080] the coefficient λ_G expresses the importance given to the priority of the jobs with respect to the occupation of nodes.

[0081] PR_s represents a user-settable priority value.

[0082] Subsequently (block 140) the search for the global minimum of the cost function H is operated by known techniques of quantum annealing (this operation is also known with the term quantum annealing, acronym QA) which, as is known, is a general method to find the global minimum of a given function on a set of candidate solutions (the search space), by means of a process analogous to quantum fluctuations.

[0083] The binary variables x_{s,n} of the global minimum of the function H identify the computation nodes n on which the subjobs s can be allocated most efficiently. Conveniently the operation of calculating the global minimum is performed by a quantum calculator which, as is known, is a computer that uses the quantum properties of matter, such as the superposition of states, entanglement, and tunnelling effect, in order to find the minimum of a functionexpressed in QUBO form (acronym for Quadratic unconstrained binary optimization).

[0084] The nodes thus identified are (block 150) assigned to the JOB and used for the subsequent processing required by the user.

[0085] Based on the tests realised by the Applicant, the cluster employment and therefore the total throughput are optimised. The tests were carried out in an environment consisting of 100 nodes with 4 gpus per node and 48 cpus. A simulation of about 4 hours was filmed. In the representation of Figure 3, the solid line represents the result obtained according to the present invention compared with a solution operating according to the known art (i. e. the PBS programming indicated in the known art indicated with the hatching). The upper left pane shows the number of running jobs, the upper right pane shows the number of jobs in the queue, the lower left pane shows the percentage of CPUs used, and the lower right pane shows the percentage of GPUs used. We can see that the solution of the present invention ensures a better employment of the clusters compared to what is obtained from the prior art.

Claims

CLAIMS1. Scheduling method for high performance computers in which a network of servers (3) are connected to each other via connections to form respective nodes n having different hardware structures from each other and different computational capacities, the method comprising the steps of:submitting ( 100) a JOB specifying the required computational resources, namely the necessary nodes n and for each node n required the desired hardware resources and the time of access to the nodes;defining the number of SUBJOBs as the number of the required nodes that forms a submitted JOB;defining a binary variable (block 120)xs,n∈ {0.1}where s indicates the subjob to be allocated and n indicates the computation node;defining a cost function H comprising the sum of several addends, in which each addend is given by the multiplication of a coefficient λior weight and of a term Hiwhich is in turn a function of the binary variable xs,n; each term Hicorresponds to an objective function to be optimised or to a constraint; operating the search for the global minimum of the cost function H by quantum annealing;the binary variables xs,nof the global minimum of the cost function H identify the computation nodes n on which the subjobs s can be allocated with greater efficiency;assigning (block 150) the identified computation nodes to the JOB and using them for a subsequent processing.

2. - Method according to claim 1, wherein the objective function is one of the following:- an objective function HA computed through the formula:H_A= − ∑_n(∑_s x_sn RES_S)where RESsare the hardware resources that have been required by the subjob s;an objective function HAm computed according to the following expression:HAm= − Σn(ΣsxsnRESs(1 + λGPrs))where RESsare the hardware resources that have been required by the sub job s;the coefficient λGexpresses the importance given to the priority of the jobs over nodes employment.PR_s represents a settable priority value;An objective function HEthat expresses the physical distance between the computers that form the nodes on which the sub jobs of a same job are assigned: H^ minimizes the distance between nodes on which subjobs of a same job are assigned:HE= Σ (dnm− 1)xsnxs'ms,s’:j(s')=j(sr)n=tm- where the distance values dn mbetween nodes:dn,m= 0 for nodes belonging to the same rackdn,m= 1 for nodes belonging to different racks3. - Method according to claim 1 or 2, where the constraint is one of the following:a constraint HB which ensures that a subjob s is not assigned to more than one node n:HB= Σs(Σn,n':n≠n'xsnxsn')s n.n'.n^niin the variable xss represents the subjob index and n' represents a further node index;a constraint HQ that ensures that sub jobs referring to a same job are executed simultaneously, the function HQ is defined as:HC= Σj(Σn, s:j(s)=jxs,n− |j| Σn'xsj(1),n')2Jnn>s:j(s)=jin this equation, j (s) indicates the job to which the subjob s belongs, \ j \ is the number of subjobs forming the job j, j (l) is one of these subjobs, used as representations of the job.a constraintHD that ensures that the sum of the resources required by all subjobs assigned to the same node must not exceed the available resources of the node:HD= Σn(ΣsxsnRESs+ Σkwksn,k− Rn)2n kRESsare the resources required by the sub job s, Rnare the available resources of the node at the time of optimisation, the variables sn,krepresent slack variables and are needed in optimisation to transform inequalities into equalities.

4. - Method according to any one of the preceding claims wherein the step of defining a binary variable is preceded by the step of verifying (block 110) whether it is possible for the required node n to host the subjob s i.e. whether the node n has sufficient available hardware resources.

5. - Method according to any one of the preceding claims wherein, according to a Portable Batch System programming mode, chunks representing a description of the required resources are required; the method comprising the step of translating the number of chunks into the required number of requested nodes.

Images