Cluster scheduling method and apparatus, and computer device, readable storage medium and program product

Abstract

The present application relates to a cluster scheduling method and apparatus, and a computer device, a computer-readable storage medium and a computer program product. The method comprises: acquiring a target job; on the basis of node states and node snapshot data of nodes, searching, in order of partition priority, a plurality of partitions for a target node that meets an operating condition of the target job; scheduling the target job to the target node; and on the basis of a scheduling result of the target node, updating node snapshot information of the target node in the node snapshot data, and updating real-time node data of the target node among real-time node data.

Description

Cluster scheduling method and device, computer equipment, readable storage medium and program product

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 2024116571317, filed on November 19, 2024, and entitled "Cluster scheduling method and device, computer equipment, readable storage medium and program product", the contents of which are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] The present application relates to the field of cloud native technology, and in particular, to a cluster scheduling method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0004] Kubernetes (K8s, which is an abbreviation of "ubernete" with 8 replacing the 8 characters in the middle of the name) has become a de facto standard for resource management and container orchestration, which can easily manage massive containers and node resources. More and more enterprises use it as a key technology for building cloud native infrastructure, and deploy and run big data and artificial intelligence jobs.

[0005] As the business scale becomes larger, the number of nodes and jobs in the cluster also becomes larger. The number of nodes can reach tens of thousands or even hundreds of thousands, and the number of jobs can even reach millions. In this case, in addition to the stability of the entire system, the job scheduling system also faces great pressure.

[0006] To this end, in some traditional technologies, multiple clusters are combined into a large logical cluster, and a unified distribution system distributes jobs to each cluster, and then the scheduler in the cluster is responsible for scheduling the jobs, which needs to go through two scheduling to run. In some other traditional technologies, multiple schedulers in the cluster schedule in parallel, and a special component is needed to load coordinate the scheduling results of multiple schedulers. This technology requires each scheduler to save snapshot information of the entire cluster, and does not effectively reduce the storage pressure of the scheduler. Therefore, the current traditional technology has the technical problem that it is difficult to effectively cope with the influence of the expansion of the cluster scale on the cluster scheduling performance. SUMMARY

[0007] According to various embodiments of the present application, a cluster scheduling method, device, computer equipment, readable storage medium and program product are provided.

[0008] In a first aspect, the present application provides a cluster scheduling method, comprising:

[0009] obtaining a target job;

[0010] finding, in a plurality of partitions in order of partition priorities, a target node satisfying a running condition of the target job according to node states and node snapshot data of the nodes; each of the partitions contains a plurality of the nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to a scheduling success rate of the partition; the node snapshot data stores node snapshot information of each of the nodes; the node snapshot information is updated according to node real-time information of the node in node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores node real-time information of each of the nodes; the node state indicates whether the node snapshot information of the node is in a to-be-updated state; the to-be-updated state is used to indicate that the node snapshot information is updated according to the node real-time information before it is judged whether the node satisfies the running condition;

[0011] scheduling the target job to the target node;

[0012] updating, according to a scheduling result of the target node, the node snapshot information of the target node in the node snapshot data and the node real-time information of the target node in the node real-time data.

[0013] In a second aspect, the present application further provides a cluster scheduling apparatus, comprising:

[0014] a job obtaining unit configured to obtain a target job;

[0015] a node finding unit configured to find, in a plurality of partitions in order of partition priorities, a target node satisfying a running condition of the target job according to node states and node snapshot data of the nodes; each of the partitions contains a plurality of the nodes in the cluster; each of the partitions has the partition priority; the partition priority is related to a scheduling success rate of the partition; the node snapshot data stores node snapshot information of each of the nodes; the node snapshot information is updated according to node real-time information of the node in node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores node real-time information of each of the nodes; the node state indicates whether the node snapshot information of the node is in a to-be-updated state; the to-be-updated state is used to indicate that the node snapshot information is updated according to the node real-time information before it is judged whether the node satisfies the running condition;

[0016] a job scheduling unit configured to schedule the target job to the target node;

[0017] The first updating unit is configured to update node snapshot information of the target node in the node snapshot data and update node real-time information of the target node in the node real-time data according to a scheduling result of the target node.

[0018] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the operations of the cluster scheduling method when executing the computer program.

[0019] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the operations of the cluster scheduling method when executed by a processor.

[0020] In a fifth aspect, the present application provides a computer program product, which comprises a computer program, and the computer program implements the operations of the cluster scheduling method when executed by a processor.

[0021] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the present application will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or in the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description only illustrate the embodiments of the present application, and for those skilled in the art, other drawings can be obtained from the disclosed drawings without any creative effort, and additional details or examples used to describe the drawings should not be considered as limiting the scope of any one of the disclosed inventions, the presently described embodiments and / or examples, and the presently understood best mode of these inventions.

[0023] FIG. 1 is an application environment diagram of a cluster scheduling method according to one or more embodiments.

[0024] FIG. 2 is a flow diagram of a cluster scheduling method according to one or more embodiments.

[0025] FIG. 3 is a schematic diagram of updating node information according to one or more embodiments.

[0026] FIG. 4 is a schematic diagram of updating node information according to another or more embodiments.

[0027] FIG. 5 is a structural block diagram of a cluster scheduling apparatus according to one or more embodiments.

[0028] FIG. 6 is an internal structure diagram of a computer device according to one or more embodiments. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0030] In order to make the purposes, technical solutions and advantages of the present application more clear, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application.

[0031] Some terms involved in the present application are explained as follows:

[0032] Partition: a logical set containing a group of nodes, the nodes in the cluster are divided into several partitions.

[0033] Node real-time data: a data structure for storing node information in the scheduler, which can always be consistent with the node information in the interface service module. The node real-time data is organized by partition.

[0034] Node snapshot data: a data structure for storing node information in the scheduler, which can be a copy of the node real-time data and may differ from the node real-time data. The node snapshot data is organized by partition.

[0035] As shown in FIG. 1, the application environment of the cluster scheduling method of the present application can be referred to as a scheduling system, which can include an interface service module and a scheduler, wherein the scheduler can include a scheduling module and a synchronization module.

[0036] The interface service module can serve as a system access entrance of the scheduling system and can provide interface operations such as job submission, update, search and deletion, and node creation, update, search and deletion. The interface service module can also store data based on an ETCD (a distributed, reliable and fast key-value storage system that can be used for shared configuration, service discovery and consistency guarantee), and can be responsible for authentication, authorization and verification of various resources, and can provide external RESTful APIs (Representational State Transfer (REST) application program interface API), and can use a native kubernetes api-server (one of the core components of Kubernetes, responsible for exposing the Kubernetes API to users and clients, receiving and processing requests from clients, and storing them in ETCD).

[0037] The node real-time data and the node snapshot data can be stored in the scheduler.

[0038] The scheduling module in the scheduler can be responsible for job scheduling. In this application, the scheduling module can perform job scheduling based on the node snapshot data, and can also be responsible for synchronization of part of the node information between the node real-time data and the node snapshot data. The scheduling module can also dynamically maintain partitions and their partition priorities, and perform job scheduling according to the partition priorities.

[0039] The synchronization module in the scheduler can be responsible for synchronization of node information, which can include synchronization between the interface service module and the node real-time data, and synchronization of part of the node information between the node real-time data and the node snapshot data.

[0040] Based on the scheduling system shown in FIG. 1, the cluster scheduling method of the present application is described in combination with various embodiments and corresponding drawings.

[0041] In an exemplary embodiment, as shown in FIG. 2, a cluster scheduling method is provided, which can be applied to the scheduler in FIG. 1 and can be executed by the scheduling module and the synchronization module in the scheduler. The cluster scheduling method can include the following operations:

[0042] Operation S201, obtaining a target job.

[0043] The target job can be a job to be scheduled. In this operation, the scheduling module in the scheduler can obtain a job from a plurality of jobs to be scheduled as a target job.

[0044] Operation S202, according to the order of the partition priority, in the plurality of partitions according to the node state and the node snapshot data of the node to find out the target node meeting the running condition of the target job.

[0045] Wherein, the order of the partition priority can be from higher to lower. In this operation, the partition priority can be from higher to lower, and each time a partition is taken out to find the target node.

[0046] Wherein, each partition contains a plurality of nodes in the cluster, and the nodes can be used to process jobs.

[0047] Wherein, each partition has a respective partition priority, and the partition priority is related to the scheduling success rate of the partition. Wherein, the partition with a higher scheduling success rate can have a higher partition priority.

[0048] In this operation, the scheduling module can take out one partition at a time according to the order of the partition priority from high to low. For this partition, the scheduling module can find the target node meeting the running condition of the target job in the partition according to the node state and the node snapshot data of the node.

[0049] Wherein, the node snapshot data stores the node snapshot information of each node. As shown in FIG. 1, the node snapshot data can manage the node snapshot information of each node by partition, for example, for 0 to k partitions, each partition can contain, for example, N nodes, and then the node snapshot data can manage the node snapshot information of different nodes according to 0 to k partitions. Wherein, the node snapshot information of a node can be updated by the synchronization module according to the node real-time information of the node in the node real-time data in the case that the node snapshot information of the node in the partition where the node is located is in the unoccupied state, thereby ensuring the timeliness of the node snapshot information update. Wherein, the node real-time data stores the node real-time information of each node, and correspondingly, as shown in FIG. 1, the node real-time data also manages the node real-time information of each node by partition, for example, for 0 to k partitions, each partition can contain, for example, N nodes, and then the node real-time data can manage the node real-time information of different nodes according to 0 to k partitions.

[0050] Wherein, the node state of the node indicates whether the node snapshot information of the node is in the to-be-updated state, which can be used to indicate that the scheduling module updates the node snapshot information of the node according to the node real-time information of the node before judging whether the node meets the running condition, thereby ensuring the timeliness of the node snapshot information update and the accuracy of judging whether the node is the target node.

[0051] Wherein, the synchronization module and the scheduling module can update the node snapshot information in different partitions in parallel, which can improve the efficiency of the node snapshot information update.

[0052] In the operation, the scheduling module can determine whether the node needs to update the node snapshot information according to the node state of the node, and can determine whether the node is a target node satisfying the running condition of the target job according to the updated node snapshot information. As an example, the node snapshot information can be used to represent the current service performance and other information of the node, and the scheduling module can determine whether the node can satisfy the running condition of the target job according to the node snapshot information, such as whether the service performance needs to reach a certain index. Thus, the scheduling module can find the target node satisfying the running condition of the target job according to the node state and the node snapshot data in the multiple partitions in the order of the partition priority.

[0053] In operation S203, the target job is scheduled to the target node.

[0054] In operation S204, the node snapshot information of the target node in the node snapshot data and the node real-time information of the target node in the node real-time data are updated according to the scheduling result of the target node.

[0055] For operations S203 and S204, the target job can be successfully scheduled to the target node for processing, and the scheduling result of the target node can indicate that the target job is included in the current jobs to be processed. Thus, according to the scheduling result, the node snapshot information and the node real-time information of the target node need to be updated. In combination with FIG. 3, the target node can be nodes 1-3 in partition 1, and according to the scheduling result, the scheduling module can update the node snapshot information of nodes 1-3 in the node snapshot data and update the node real-time information of nodes 1-3 in the node real-time data. The node snapshot information of nodes 1-3 in the node snapshot data can be updated first, and then the node real-time information of nodes 1-3 in the node real-time data can be updated according to the updated node snapshot information of nodes 1-3. This process can be referred to as post-scheduling update. Unlike pre-scheduling update, the scheduling module can determine whether the node needs to update the node snapshot information according to the node state of the node in operation S202. For example, the scheduling module can determine that node 1-2 in partition 1 needs to update the node snapshot information according to the node state of node 1-2. At this time, the scheduling module can update the node snapshot information of node 1-2 in the node snapshot data according to the node real-time information of node 1-2 in the node real-time data, and then determine whether node 1-2 belongs to the target node according to the updated node snapshot information of node 1-2.

[0056] After the scheduling of the target job is completed, the scheduling module can continue to schedule the next job.

[0057] The method of the above embodiment obtains a target job, finds out a target node satisfying a running condition of the target job according to node states and node snapshot data of nodes in multiple partitions in an order of partition priorities, schedules the target job to the target node, and updates node snapshot information of the target node in the node snapshot data and node real-time data of the target node according to a scheduling result of the target node, wherein each partition contains multiple nodes in the cluster, the partition priority is related to a scheduling success rate of the partition, the node snapshot data stores node snapshot information of each node, the node snapshot information is updated according to node real-time information of the node in the node real-time data when a partition where the node is located is in an unoccupied state, and the node state indicates whether the node snapshot information of the node is in a to-be-updated state or not, and the to-be-updated state is used to indicate that the node snapshot information is updated according to the node real-time information before it is judged whether the node satisfies the running condition or not.

[0058] Therefore, the method of the above embodiment can avoid copying all node information every time scheduling is performed based on independent node snapshot data, reduce time consumption, avoid locking the entire node snapshot data during scheduling based on partition-based node management, update node information of nodes in different partitions in parallel based on a partition-based snapshot updating manner, ensure the updating efficiency and timeliness of the node information, and ensure that each time scheduling is performed, a partition with a higher scheduling success rate is selected for scheduling based on the partition priority related to the scheduling success rate, thereby effectively dealing with the influence of the expansion of the cluster size on the scheduling performance of the cluster without changing the structure of the cluster and the scheduling system thereof, reducing the time consumption of copying node snapshot information, and ensuring that the scheduling result meets the global optimal in terms of the scheduling success rate.

[0059] In an exemplary embodiment, the cluster scheduling method of the present application can further create partitions and divide nodes by the following operations before obtaining a target job:

[0060] obtaining a total number of nodes in the cluster and a number of nodes contained in each partition in a configuration file, creating multiple partitions according to the total number of nodes and the number of nodes, and sequentially dividing the nodes in the cluster into the multiple partitions.

[0061] In the embodiment, the scheduling module can obtain the total number M of nodes in the cluster, and the scheduling module can obtain the number N of nodes contained in each partition in the configuration file. The configuration file can be a related file for dividing the partitions, and the configuration file can contain the number N of nodes contained in each partition. Therefore, the scheduling module can determine the number K of partitions = M / N according to the total number M of nodes and the number N of nodes, and create partitions such as P1, P2, …, Pk (as an example of partition identification), and the like. The scheduling module can traverse all nodes in the cluster, and sequentially divide the nodes into the partitions P1, P2, …, Pk until all nodes in the cluster are processed.

[0062] In an example embodiment, the cluster scheduling method of the present application can further update the relevant node information by the following operations:

[0063] obtaining to-be-updated node information; if there is node real-time information of a first node to which the to-be-updated node information belongs in the node real-time data, updating the node real-time information to the to-be-updated node information in the case that the to-be-updated node information is different from the node real-time information; and updating node snapshot information of the first node in the node snapshot data according to the updated node real-time information in the case that a partition where the first node is located is in an unoccupied state; and setting a node state of the first node to a to-be-updated state in the case that the partition where the first node is located is in an occupied state.

[0064] The scheme of the present embodiment can be implemented by a synchronization module in the scheduler to update the relevant node information of the node real-time data and the node snapshot data according to the to-be-updated node information.

[0065] In combination with FIG. 1 and FIG. 4, the synchronization module in the scheduler can obtain the to-be-updated node information through the interface service module. After obtaining the to-be-updated node information, the synchronization module can first search the node real-time data to find whether there is node real-time information of a first node to which the to-be-updated node information belongs.

[0066] If there is node real-time information of the first node to which the to-be-updated node information belongs in the node real-time data, the synchronization module can further judge whether the to-be-updated node information is the same as (whether there is a change in) the node real-time information.

[0067] In the case that the to-be-updated node information is the same as (no change in) the node real-time information, the synchronization module can process the next to-be-updated node information.

[0068] In the case that the to-be-updated node information is different from (there is a change in) the node real-time information, the synchronization module can update the node real-time information to the to-be-updated node information. At this time, the synchronization module also needs to judge whether the partition where the first node is located is in a state occupied (or used) by a scheduling module in the scheduler (in an unoccupied state or in an occupied state).

[0069] In the case that the partition where the first node is located is in an unoccupied state (such as partition 0, partition k, and the like shown in FIG. 4), the synchronization module can update the node snapshot information of the first node in the node snapshot data according to the updated node real-time information, and then the synchronization module can process the next to-be-updated node information.

[0070] In the case that the partition where the first node is located is in the occupied state (the partition 1 shown in FIG. 4), the synchronization module can set the node state of the first node as the to-be-updated state, thereby indicating the scheduling module that the node snapshot information of the first node needs to be updated before judging whether the first node is the target node. Then the synchronization module can process the next to-be-updated node information.

[0071] In an exemplary embodiment, further, after obtaining the to-be-updated node information in the above embodiment, the method can further include:

[0072] If the node real-time information of the first node to which the to-be-updated node information belongs does not exist in the node real-time data, a second node is added in the partition that meets the node quantity condition among the multiple partitions; and according to the to-be-updated node information, the node real-time information of the second node is added in the node real-time data, and the node snapshot information of the second node is added in the node snapshot data.

[0073] In the embodiment, for the case that the node real-time information of the first node to which the to-be-updated node information belongs does not exist in the node real-time data, the synchronization module can request the scheduling module to add a second node (i.e. to create a new node in the partition) in the partition that meets the node quantity condition among the multiple partitions, and the synchronization module can also add the node real-time information of the second node in the node real-time data, and add the node snapshot information of the second node in the node snapshot data. Then the synchronization module can process the next to-be-updated node information. The node quantity condition can be the least number of nodes, i.e. the synchronization module can request the scheduling module to add the second node in the partition that has the least number of nodes among the multiple partitions, thereby balancing the scheduling performance of each partition. In some other embodiments, if one or more nodes need to be deleted, the scheduling module can delete the nodes in the partitions to which the nodes belong and delete the node real-time information and the node snapshot information of the nodes in the node real-time data and the node snapshot data according to the nodes to be deleted.

[0074] In an exemplary embodiment, the operation S202 of searching out the target node that meets the running condition of the target job according to the node state and the node snapshot data in the multiple partitions in the order of the partition priority can include:

[0075] polling the plurality of partitions in order of partition priority; in the polling of the plurality of partitions, setting a current partition as occupied and non-current partitions as unoccupied; and traversing nodes contained in the current partition according to node status and node snapshot data in the current partition; in the node traversal of the current partition, if the node status of a current node is to-be-updated, updating node snapshot information of the current node according to node real-time information of the current node, and judging whether a running condition is met according to the updated node snapshot information; if yes, determining that the current node is a target node; if no, entering target node judgment for a next node of the current partition; if no target node is found in the node traversal of the current partition, entering node traversal for a next partition of the plurality of partitions.

[0076] In the embodiment, the scheduling module can poll the plurality of partitions in order from high to low partition priority. The scheduling module can sort the plurality of partitions in order from high to low partition priority, with the highest priority partition at the front and the lowest priority partition at the back. In the polling of the plurality of partitions, the scheduling module can set a current partition (being polled) as occupied and other partitions, i.e. non-current partitions, as unoccupied (which can be indicated by not being marked as in use).

[0077] Then, the scheduling module traverses nodes contained in the current partition according to node status and node snapshot data. In the node traversal of the current partition, the scheduling module can judge whether the node status of a current node (being traversed) is to-be-updated. If the node status of the current node is to-be-updated, the scheduling module updates node snapshot information of the current node according to node real-time information of the current node in node real-time data, and deletes the to-be-updated status after the update. The scheduling module can judge whether the current node meets a running condition according to the updated node snapshot information, i.e. whether a target job can run on the current node. If yes, the scheduling module can determine that the current node is a target node. If no, the scheduling module can enter target node judgment for a next node of the current partition, i.e. taking the next node of the current partition as a new current node, and continuing to judge whether the node status of the new current node is to-be-updated, whether the new current node meets the running condition, and the like, until a target node is found or the current partition is traversed. If the scheduling module finds a target node in the node traversal of the current partition, the scheduling module enters node traversal for a next partition of the plurality of partitions, and takes the next partition of the plurality of partitions as a new current partition, and continues to traverse nodes contained in the new current partition according to node status and node snapshot data, until a target node is found or all partitions are traversed. In some embodiments, if no target node is found after all partitions are traversed, the scheduling module can determine that scheduling of the target job fails.

[0078] In an example embodiment, further, after finding out the target node satisfying the running condition of the target job according to the node state and the node snapshot data of the node in the multiple partitions in the order of the partition priority in operation S202, the method of the present application can further include the following operations:

[0079] For the first partition in which the target node is found, the partition priority of the first partition is raised; for the second partition in which the target node is not found after polling, the partition priority of the second partition is lowered.

[0080] The scheme of the present embodiment can adjust the partition priority of the partition according to the scheduling success rate, ensuring that the partition with a higher scheduling success rate can be preferentially selected for scheduling each time, improving the efficiency and effectiveness of cluster scheduling.

[0081] In the present embodiment, for the first partition in which the target node is found, the scheduling module can raise the partition priority thereof; for the second partition in which the target node is not found after polling, the scheduling module can lower the partition priority thereof. For the raising and lowering of the partition priority, as an implementation, the following methods can be used:

[0082] An upper limit H and a lower limit L of the partition priority can be set.

[0083] A first threshold T1 and a second threshold T2 for adjusting the partition priority can be set, which can be used to control the raising and lowering of the partition priority, respectively. As an example, T1=T2=(H+L) / 2.

[0084] For a partition with a partition priority P, when raising the partition priority thereof:

[0085] If the partition priority P is less than the first threshold T1, the raised partition priority P is P*2, and if the raised partition priority P is greater than the upper limit H, the raised partition priority P is set to the upper limit H.

[0086] If the partition priority P is greater than or equal to the first threshold T1, the raised partition priority P is P+1, and if the raised partition priority P is greater than the upper limit H, the raised partition priority P is set to the upper limit H.

[0087] For a partition with a partition priority P, when lowering the partition priority thereof:

[0088] If the partition priority P is less than the second threshold T2, the lowered partition priority P is P-2, and if the lowered partition priority P is less than the lower limit L, the lowered partition priority P is set to the lower limit L.

[0089] If the partition priority P is greater than or equal to the second threshold T2, the reduced partition priority P is P / 2, and if the reduced partition priority P is less than the lower limit L, the reduced partition priority P is set to the lower limit L.

[0090] In this way, the partition priority can be adjusted hierarchically according to the relationship between the partition priority and the corresponding threshold.

[0091] It should be understood that, although each operation in the flowchart involved in each embodiment as described above is shown in sequence according to the arrow, these operations are not necessarily executed in sequence according to the arrow. Unless explicitly stated herein, the execution of these operations is not strictly limited in sequence, and these operations can be executed in other sequences. Moreover, at least part of the operations in the flowchart involved in each embodiment as described above can include multiple operations or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these operations or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other operations or stages in other operations.

[0092] Based on the same inventive concept, the embodiments of the present application also provide a cluster scheduling device for implementing the cluster scheduling method as described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more cluster scheduling device embodiments provided below can refer to the limitations of the cluster scheduling method in the above text, and will not be repeated here.

[0093] In an exemplary embodiment, as shown in FIG. 5, a cluster scheduling device is provided, which can include:

[0094] The job acquisition unit 501 is configured to acquire a target job.

[0095] The node searching unit 502 is configured to search for a target node satisfying a running condition of a target job in a plurality of partitions according to node states and node snapshot data of nodes in an order of partition priorities, wherein each of the partitions contains a plurality of the nodes in a cluster, each of the partitions has the partition priority, the partition priority is related to a scheduling success rate of the partition, the node snapshot data stores node snapshot information of each of the nodes, the node snapshot information is updated according to node real-time information of the node in node real-time data when a partition where the node is located is in an unoccupied state, the node real-time data stores node real-time information of each of the nodes, the node state indicates whether the node snapshot information of the node is in a to-be-updated state, and the to-be-updated state is used to indicate that the node snapshot information is updated according to the node real-time information before it is judged whether the node satisfies the running condition.

[0096] The job scheduling unit 503 is configured to schedule the target job to the target node.

[0097] The first updating unit 504 is configured to update node snapshot information of the target node in the node snapshot data and update node real-time information of the target node in the node real-time data according to a scheduling result of the target node.

[0098] In an exemplary embodiment, the apparatus 500 can further include a second updating unit configured to acquire to-be-updated node information, update node real-time information of a first node to which the to-be-updated node information belongs as the to-be-updated node information when the node real-time information of the first node exists in the node real-time data and is different from the to-be-updated node information, and update node snapshot information of the first node in the node snapshot data according to the updated node real-time information when a partition where the first node is located is in an unoccupied state, or set a node state of the first node to the to-be-updated state when the partition where the first node is located is in an occupied state.

[0099] In an exemplary embodiment, the second updating unit is further configured to add a second node in a partition satisfying a node quantity condition in the plurality of partitions when the node real-time information of the first node to which the to-be-updated node information belongs does not exist in the node real-time data, and add node real-time information of the second node in the node real-time data and add node snapshot information of the second node in the node snapshot data according to the to-be-updated node information.

[0100] In an example embodiment, the node searching unit 502 is configured to poll the plurality of partitions in the order of the partition priorities; in the polling of the plurality of partitions, set a current partition as an occupied state and set a non-current partition as an unoccupied state; and in the current partition, traverse nodes included in the current partition according to the node state and the node snapshot data; in the node traversal of the current partition, if the node state of a current node is the to-be-updated state, update the node snapshot information according to the node real-time information of the current node, and determine whether the running condition is satisfied according to the updated node snapshot information; if yes, determine that the current node is the target node; if no, proceed to the target node determination of a next node of the current partition; if the target node is not found in the node traversal of the current partition, proceed to the node traversal of a next partition of the plurality of partitions.

[0101] In an example embodiment, the node searching unit 502 is further configured to, for a first partition in which the target node is found, raise the partition priority of the first partition; and for a second partition that has been polled and in which the target node is not found, lower the partition priority of the second partition.

[0102] In an example embodiment, the apparatus 500 can further include a partition unit configured to obtain a total number of nodes in the cluster and obtain a number of nodes included in each partition in a configuration file; create the plurality of partitions according to the total number of nodes and the number of nodes, and sequentially divide the nodes in the cluster into the plurality of partitions.

[0103] The above modules in the cluster scheduling apparatus can be realized by software, hardware, or a combination thereof, in whole or in part. The above modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the above modules.

[0104] In an example embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 6. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The database of the computer device is configured to store node snapshot data, node real-time data, and the like. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with terminals outside through a network connection. The computer program is executed by the processor to implement a cluster scheduling method.

[0105] Those skilled in the art can understand that the structure shown in FIG. 6 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. Specifically, the computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0106] In an example embodiment, a computer device is also provided, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the operations in the above method embodiments.

[0107] In an example embodiment, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the operations in the above method embodiments.

[0108] In an example embodiment, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the operations in the above method embodiments.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use, and processing of related data need to comply with relevant regulations.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0111] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above-mentioned embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0112] The above embodiments only express several implementation ways of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation to the patent scope of the application. It should be pointed out that for ordinary skilled persons in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A cluster scheduling method, the method comprising: Obtain the target task; According to the partition priority, target nodes that meet the running conditions of the target job are found in multiple partitions based on the node status and node snapshot data; wherein, each partition contains multiple nodes in the cluster; each partition has a partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each node; the node snapshot information is updated according to the node real-time information in the node real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node real-time information of each node; the node status indicates whether the node snapshot information of the node is in a pending update state; the pending update state is used to indicate that the node snapshot information is updated according to the node real-time information before determining whether the node meets the running conditions; Schedule the target job to the target node; Based on the scheduling result of the target node, update the node snapshot information of the target node in the node snapshot data and update the real-time node information of the target node in the node real-time data.

2. The method according to claim 1, further comprising: Get the node information to be updated; If the real-time node data contains real-time node information of the first node to which the node information to be updated belongs, then if the node information to be updated is different from the real-time node information, the real-time node information will be updated to the node information to be updated. as well as, When the partition where the first node is located is in an unoccupied state, the node snapshot information of the first node in the node snapshot data is updated according to the updated real-time information of the node; If the partition where the first node is located is occupied, the node status of the first node is set to the pending update status.

3. The method according to claim 2, further comprising, after obtaining the node information to be updated: If the real-time node information of the first node to which the node information to be updated belongs does not exist in the real-time node data, then a second node is added to the partition that meets the node quantity condition among the multiple partitions; and, according to the node information to be updated, the real-time node information of the second node is added to the real-time node data, and the node snapshot information of the second node is added to the node snapshot data.

4. The method according to claim 1, wherein finding the target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data in order of partition priority includes: The multiple partitions are polled in round-robin order according to their partition priority. In the polling of the multiple partitions, the current partition is set to occupied status, and the non-current partitions are set to unoccupied status; as well as, In the current partition, the nodes contained in the current partition are traversed according to the node status and node snapshot data; During the node traversal of the current partition, if the node state of the current node is the state to be updated, the node snapshot information is updated according to the real-time information of the current node, and it is determined whether the running conditions are met based on the updated node snapshot information. If the conditions are met, then the current node is determined to be the target node; If the condition is not met, proceed to determine the target node of the next node in the current partition; If the target node is not found during the node traversal of the current partition, then the node traversal of the next partition among the multiple partitions is initiated.

5. The method according to claim 4, after finding the target node that meets the running conditions of the target job in multiple partitions according to the node status and node snapshot data in the order of partition priority, the method further includes: For the first partition where the target node is found, increase the partition priority of the first partition; For the second partition that has been polled but the target node has not been found, the partition priority of the second partition is reduced.

6. The method according to any one of claims 1 to 5, further comprising: Obtain the total number of nodes in the cluster, and obtain the number of nodes contained in each partition of the configuration file; Based on the total number of nodes and the number of nodes, the multiple partitions are created, and the nodes in the cluster are sequentially assigned to the multiple partitions.

7. A cluster scheduling device, the device comprising: The job acquisition unit is used to acquire the target job. A node lookup unit is used to find target nodes that meet the running conditions of the target job in multiple partitions according to the node status and node snapshot data, based on the partition priority. Each partition contains multiple nodes in the cluster; each partition has a partition priority; the partition priority is related to the scheduling success rate of the partition; the node snapshot data stores the node snapshot information of each node; the node snapshot information is updated based on the node's real-time information in the node's real-time data when the partition where the node is located is in an unoccupied state; the node real-time data stores the node's real-time information; the node status indicates whether the node's node snapshot information is in a pending update state; the pending update state indicates that the node snapshot information should be updated based on the node's real-time information before determining whether the node meets the running conditions. A job scheduling unit is used to schedule the target job to the target node; The first update unit is used to update the node snapshot information of the target node in the node snapshot data and the node real-time information of the target node in the node real-time data according to the scheduling result of the target node.

8. The apparatus according to claim 7, further comprising: The second update unit is used to obtain information about the node to be updated. If the real-time node data contains real-time node information of the first node to which the node information to be updated belongs, then if the node information to be updated is different from the real-time node information, the real-time node information will be updated to the node information to be updated. In addition, when the partition where the first node is located is in an unoccupied state, the node snapshot information of the first node in the node snapshot data is updated according to the updated real-time information of the node; If the partition where the first node is located is occupied, the node status of the first node is set to the pending update status.

9. The apparatus according to claim 8, wherein the second updating unit is further configured to: if the real-time node information of the first node to which the node information to be updated belongs does not exist in the real-time node data, then add a second node in the partitions among the plurality of partitions that meet the node quantity condition; and, according to the node information to be updated, add the real-time node information of the second node in the real-time node data, and add the node snapshot information of the second node in the node snapshot data.

10. The apparatus according to claim 7, wherein the node lookup unit is further configured to poll the plurality of partitions in order of partition priority; in the polling of the plurality of partitions, the current partition is set to occupied state, and the non-current partitions are set to unoccupied state; And, in the current partition, traverse the nodes contained in the current partition according to the node status and node snapshot data; During the node traversal of the current partition, if the node state of the current node is the state to be updated, the node snapshot information is updated according to the real-time information of the current node, and it is determined whether the running conditions are met based on the updated node snapshot information. If the conditions are met, then the current node is determined to be the target node; If the condition is not met, proceed to determine the target node of the next node in the current partition; If the target node is not found during the node traversal of the current partition, then the node traversal of the next partition among the multiple partitions is initiated.

11. The apparatus according to claim 10, wherein the node lookup unit is further configured to, for a first partition where the target node is found, increase the partition priority of the first partition; and for a second partition that has been polled but where the target node has not been found, decrease the partition priority of the second partition.

12. The apparatus according to any one of claims 7 to 11, further comprising: A partition unit is used to obtain the total number of nodes in the cluster and the number of nodes contained in each partition in the configuration file; Based on the total number of nodes and the number of nodes, the multiple partitions are created, and the nodes in the cluster are sequentially assigned to the multiple partitions.

13. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to perform the operation of the method according to any one of claims 1 to 6.

14. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the operation of the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program that, when executed by a processor, implements the operation of the method according to any one of claims 1 to 6.

Images