Container scheduling method and apparatus, electronic device and medium

Abstract

The embodiments of the present application relate to the technical field of computers. Provided are a container scheduling method and apparatus, an electronic device and a medium. The method comprises: constructing a container utility function and a load balancing objective function and, on the basis of the container utility function and the load balancing objective function, determining a pheromone update policy for a multi-objective ant colony algorithm, the container utility function indicating the utility of allocating target containers to target compute nodes, the utility of allocating the target containers to the target compute nodes being directly proportional to the amount of data to be read in the target compute nodes, and the load balancing objective function indicating the degree of balance of allocating the target containers to the target compute nodes; and, on the basis of initial parameters of the multi-objective ant colony algorithm and the pheromone update policy, determining a container scheduling solution. The method preferentially deploys containers onto compute nodes where data to be read is stored, and ensures balanced allocation of the containers, avoiding network IO and increasing the running speed of tasks within the containers.

Description

Container scheduling methods, apparatus, electronic devices and media

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 2024117444111, filed on November 30, 2024, entitled "Container Scheduling Method, Apparatus, Electronic Device and Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of computer technology, and in particular to a container scheduling method, apparatus, electronic device, and medium. Background Technology

[0004] Containers are a lightweight and portable virtualization technology used to encapsulate applications and all their dependencies and configurations so that they can run in different computing environments. YARN (Yet Another Resource Negotiator) is a resource scheduling component in the Hadoop (distributed system infrastructure) ecosystem, used for resource management and job scheduling, providing server computing resources for computing programs. In YARN, applications perform work through containers running on the underlying operating system. For application jobs, reading data or files locally on a node is faster than reading from a remote machine. However, current container scheduling methods have a high degree of randomness in container allocation, rarely consider local read / write priority strategies, and struggle to avoid unnecessary network I / O (network I / O refers to the process of data interaction over a network), making it difficult to effectively improve the running efficiency of jobs in containers. Summary of the Invention

[0005] In view of this, embodiments of this application provide a container scheduling method, apparatus, electronic device and medium that prioritizes localization when allocating containers to ensure the efficiency of task execution within containers. At the same time, in order to prevent the problem of overly concentrated container allocation, the optimization objective of node load balancing is designed to avoid this problem.

[0006] In a first aspect, embodiments of this application provide a container scheduling method, including:

[0007] Construct a container utility function and a load balancing objective function, and determine the pheromone update strategy for the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function; wherein, the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node; the load balancing objective function is used to indicate the degree of balance in allocating the target container to the target computing node;

[0008] Initialize the initial parameters of the multi-target ant colony algorithm;

[0009] The container scheduling scheme is determined based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy.

[0010] Optionally, determining the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy includes: establishing resource constraint rules between containers and computing nodes; and determining the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm, the pheromone update strategy, and the resource constraint rules.

[0011] Optionally, the method further includes:

[0012] Based on the container runtime graph of the job to be executed, the containers are divided into MapNode type and ReduceNode type; the MapNode type container is the container before the shuffle, and the ReduceNode type container is the container after the shuffle.

[0013] The container utility function includes:

[0014] Construct a first utility function and a second utility function; the first utility function is used to indicate the utility of allocating a container of type MapNode to a target compute node, and the second utility function is used to indicate the utility of allocating a container of type ReduceNode to a target compute node;

[0015] The step of determining the pheromone update strategy for the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function includes: using the first utility function and the load balancing objective function as the first pheromone update strategy for determining the scheduling scheme of MapNode type containers, and using the second utility function and the load balancing objective function as the second pheromone update strategy for determining the scheduling scheme of ReduceNode type containers.

[0016] Optionally, determining the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy includes:

[0017] Based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, a first scheduling scheme for MapNode type containers is determined.

[0018] Based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme, a scheduling scheme for ReduceNode type containers is determined.

[0019] Optionally, determining the first scheduling scheme for the MapNode type container based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy includes:

[0020] Construct a first node candidate set, which includes a first computing node that stores the data to be read containing the instruction of the job to be executed and a second computing node that does not store the data to be read.

[0021] Based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, a first target computing node for deploying the MapNode type container is determined from the first node candidate set, so as to determine the first scheduling scheme for the MapNode type container;

[0022] The step of determining the scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes:

[0023] Construct a second node candidate set, which includes a third computing node that has the first target computing node deployed and a container of the MapNode type not deployed.

[0024] Based on the initial parameters of the multi-target ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying the ReduceNode type container is determined from the second node candidate set, so as to determine the second scheduling scheme for the ReduceNode type container.

[0025] Optionally, determining the first scheduling scheme for the MapNode type container based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy includes:

[0026] In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first computing node in the first node candidate set, and the remaining ants are randomly assigned to the second computing node in the first node candidate set.

[0027] The step of determining the scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes:

[0028] In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

[0029] Optionally, in the first iteration of the multi-target ant colony algorithm, assigning a portion of the ants to the first computing nodes in the first node candidate set includes:

[0030] In the first iteration of the multi-target ant colony algorithm, for each first computing node, it is determined whether to assign ants to the first computing node based on the proportion of the data blocks to be read included in the first computing node.

[0031] Secondly, embodiments of this application provide a container scheduling device, including:

[0032] A construction module is used to construct a container utility function and a load balancing objective function, and to determine the pheromone update strategy for a multi-objective ant colony algorithm based on the container utility function and the load balancing objective function. The container utility function indicates the utility of allocating a target container to a target computing node, and the utility of allocating a target container to a target computing node is proportional to the amount of data to be read included in the target computing node. The load balancing objective function indicates the degree of balance in allocating the target container to the target computing node.

[0033] The initialization module is used to initialize the initial parameters of the multi-target ant colony algorithm.

[0034] The determination module is used to determine the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy.

[0035] Optionally, the determining module is further configured to: establish resource constraint rules between containers and computing nodes; and determine a container scheduling scheme based on the initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy, and the resource constraint rules.

[0036] Optionally, the device further includes a container partitioning module, used to: partition containers into MapNode type and ReduceNode type according to the container operation graph of the job to be executed; the MapNode type container is the container before the shuffle occurs, and the ReduceNode type container is the container after the shuffle occurs;

[0037] The construction module is used to: construct a first utility function and construct a second utility function; the first utility function is used to indicate the utility of allocating a MapNode type container to a target compute node, and the second utility function is used to indicate the utility of allocating a ReduceNode type container to a target compute node; the first utility function and the load balancing objective function are used as the first pheromone update strategy for determining the scheduling scheme of MapNode type containers, and the second utility function and the load balancing objective function are used as the second pheromone update strategy for determining the scheduling scheme of ReduceNode type containers.

[0038] Optionally, the determining module is configured to: determine a first scheduling scheme for a MapNode type container based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy; and determine a scheduling scheme for a ReduceNode type container based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme.

[0039] Optionally, the determining module is used to:

[0040] Construct a first node candidate set, which includes a first computing node that stores the data to be read containing the instruction of the job to be executed and a second computing node that does not store the data to be read.

[0041] Based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, a first target computing node for deploying the MapNode type container is determined from the first node candidate set, so as to determine the first scheduling scheme for the MapNode type container;

[0042] Construct a second node candidate set, which includes a third computing node that has the first target computing node deployed and a container of the MapNode type not deployed.

[0043] Based on the initial parameters of the multi-target ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying the ReduceNode type container is determined from the second node candidate set, so as to determine the second scheduling scheme for the ReduceNode type container.

[0044] Optionally, the determining module is used to:

[0045] In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first computing node in the first node candidate set, and the remaining ants are randomly assigned to the second computing node in the first node candidate set.

[0046] The step of determining the scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes:

[0047] In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

[0048] Optionally, the determining module is used to: during the first iteration of the multi-target ant colony algorithm, for each first computing node, determine whether to assign ants to the first computing node based on the proportion of the data blocks to be read included in the first computing node.

[0049] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement the container scheduling method provided in any embodiment of this application when executing the program stored in the memory.

[0050] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the container scheduling method provided in any embodiment of this application.

[0051] The technical solution provided in this application has at least the following beneficial effects:

[0052] The container scheduling method provided in this application uses a container utility function (indicating the allocation of target containers to target computing nodes) and a load balancing objective function (indicating the degree of balance in the allocation of target containers to target computing nodes) as pheromone update strategies for a multi-objective ant colony algorithm. This pheromone update strategy is then used to determine the optimal container scheduling method. This method prioritizes the decomposition of application tasks onto computing nodes containing data to be read through the container utility function, facilitating rapid retrieval of locally stored data from computing nodes, avoiding network I / O, and accelerating task execution within containers. Simultaneously, by introducing a load balancing objective function, it addresses the issue of concentrated container allocation that might result from prioritizing only local reads, resulting in a more balanced container allocation and preventing unreasonable allocation from causing a few computing nodes to become abnormally busy. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0054] Figure 1 shows a schematic flowchart of a container scheduling method according to an embodiment of this application;

[0055] Figure 2 shows a flowchart of a container scheduling method according to another embodiment of this application;

[0056] Figure 3 shows a schematic diagram of the structure of the container scheduling device according to an embodiment of this application;

[0057] Figure 4 shows a schematic diagram of the structure of the container scheduling and analysis system according to an embodiment of this application;

[0058] Figure 5 shows a schematic diagram of the structure of an electronic device according to an embodiment of this application. Specific Implementation

[0059] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0060] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and are not used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more.

[0061] Figure 1 shows a flowchart of a container scheduling method according to an embodiment of this application. As shown in Figure 1, the method includes:

[0062] Step S101: Construct the container utility function and the load balancing objective function, and determine the pheromone update strategy of the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function.

[0063] Step S102: Initialize the initial parameters of the multi-target ant colony algorithm.

[0064] Step S103: Determine the container scheduling scheme based on the initial parameters and pheromone update strategy of the multi-target ant colony algorithm.

[0065] The container utility function indicates the utility of allocating a target container to a target compute node. Utility can be used to measure the effect of deploying a target container to a target compute node, such as reducing network I / O and improving container operating efficiency. Optionally, the higher the utility of deploying a container to a compute node, the better the effect of deploying the container to that compute node, such as effectively reducing network I / O and improving container operating efficiency. In this embodiment, to reduce network I / O, local reading of data files or data blocks is prioritized, avoiding pulling data files or data blocks from remote storage nodes. Therefore, the utility is greater when deploying a container to a compute node that stores data blocks to be read, and the utility of allocating a container to a compute node is proportional to the amount of data to be read included in the compute node. Optionally, the utility of deploying a container to a compute node can be measured based on the number of data files or data blocks contained in the compute node.

[0066] The load balancing objective function indicates the degree of balance in distributing target containers across target compute nodes. The container scheduling method in this embodiment deploys containers based on a strategy prioritizing local reads, which can easily lead to containers being concentrated on a few specific compute nodes. This can result in abnormally high I / O and computational loads on those specific compute nodes, causing read / write anomalies for other files allocated to those nodes and affecting the stable operation of the distributed storage system. To avoid this problem, this embodiment controls container deployment through the load balancing objective function, thereby ensuring that the load is evenly distributed across each compute node.

[0067] This embodiment of the multi-objective ant colony optimization algorithm aims to solve the optimization problem of maximizing the container utility function and minimizing the load balancing objective function. Ant Colony Optimization (ACO) is an optimization algorithm that simulates the foraging behavior of ants and is mainly used to solve combinatorial optimization problems. ACO finds the optimal solution by simulating ant behavior. Specifically, when ants move along a path, they choose the path they will take based on the pheromones left by ants that have already passed by. The probability of choosing a path is proportional to the intensity of the secretions on that path. Therefore, the collective behavior of a large group of ants actually constitutes a positive feedback phenomenon of learning information: the more ants that have traversed a certain path, the greater the probability that subsequent ants will choose that path. This mechanism allows ACO to find the optimal solution by imitating the behavior of ants. The multi-objective ant colony optimization algorithm combines ant colony optimization and multi-objective optimization algorithms, aiming to solve optimization problems with multiple objectives.

[0068] Optional initial parameters for the multi-objective ant colony algorithm include, but are not limited to: the number of ants (antNum), the pheromone matrix, the critical point matrix (criticalPointMatrix) of the ants using a random allocation strategy in one iteration, the percentage increase pheromone concentration q for each path traversed by an ant, the percentage decrease pheromone concentration p after each iteration, and the number of iterations numIter. The pheromone matrix is ​​a two-dimensional matrix used to record the pheromone concentration when assigning container i to computing node a. Initially, all elements of the pheromone matrix have a value of 1. During the iterative process of the ant colony algorithm, some ants follow the allocation strategy with the highest pheromone concentration, while others follow a random allocation strategy to discover new local optima. The criticalPointMatrix controls the numbering of the ants using the random allocation strategy. For example, if the number of ants is set to 10, then in each iteration, 10 ants will complete all task allocation. Furthermore, the allocation process is performed in ascending order of ant numbers (ants are numbered starting from 0). If criticalPointMatrix[0] = 5, when assigning the 0th task, ants numbered 0 to 5 will assign tasks based on pheromone concentration (i.e., assign the task to the node with the highest pheromone concentration in the row), while ants numbered 6 to 9 will be assigned randomly (i.e., assign the task to any node randomly).

[0069] Based on the container utility function and load balancing objective function, a pheromone update strategy for the multi-objective ant colony algorithm is determined to control the pheromone concentration update. When the ant colony algorithm completes one iteration, the pheromone matrix needs to be updated, including:

[0070] 1) Reduce the concentration of all pheromones in the pheromone matrix by p% to simulate pheromone volatilization.

[0071] 2) Determine the optimal allocation scheme in this iteration based on the container utility function and load balancing objective function, and increase the pheromone concentration of the scheme by q%. In each iteration of the multi-objective ant colony algorithm, AntNum ants will generate AntNum container scheduling schemes. Based on the container utility function and load balancing objective function, determine the path with the maximum container utility and the most balanced container allocation, and increase the pheromone concentration of the path by q.

[0072] The container scheduling scheme provided in this application uses the container utility function and the load balancing objective function as pheromone update strategies to control the pheromone update in each iteration of the ant colony algorithm, thereby determining the optimal container deployment scheme. The container utility function ensures priority for local data reading, allocating containers as much as possible to the computing nodes where the data to be read resides. This allows applications to read data from local nodes as much as possible, avoiding access to data from remote nodes, thus reducing network I / O and improving job execution efficiency. The load balancing objective function ensures that the load is evenly distributed across each computing node, preventing system instability caused by high load on specific computing nodes.

[0073] In an optional embodiment, the container scheduling method further includes: establishing resource constraint rules between containers and computing nodes, and determining a container scheduling scheme based on the initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy, and the resource constraint rules.

[0074] To ensure successful container deployment during container allocation, it's crucial that the resource specifications of a single container are less than the available resources of all compute nodes in the current node set. For example, the resource specifications of a single container (e.g., the number of CPU cores and memory) must be less than the minimum available resources of all compute nodes in the current node set. In an optional embodiment, fixed container resource specifications can be set to facilitate rapid allocation of multiple containers. On the other hand, from the perspective of a compute node, its available resources (also known as remaining allocable resources) should be greater than the container's resource specifications. For instance, the compute node's available CPU cores and memory must be greater than the number of CPU cores and memory required by the container. Only when this condition is met is container allocation allowed on that compute node permitted.

[0075] In this embodiment of the application, multiple containers are allowed to be assigned to the same computing node. When using the multi-target ant colony algorithm to determine the optimal container deployment scheme, it is necessary to ensure that the resource constraint rules between the container and the computing node are met, so as to ensure successful container deployment.

[0076] Figure 2 shows a flowchart of a container scheduling method provided in another embodiment of this application. As shown in Figure 2, the method includes:

[0077] Step S201: Based on the container runtime graph of the job to be executed, divide the containers into MapNode type and ReduceNode type; the MapNode type container is the container before the shuffle, and the ReduceNode type container is the container after the shuffle.

[0078] In optional implementation scenarios, large-scale data processing engines such as Spark can be used to process jobs to be executed. The Spark computing engine performs mutual transformations of RDDs (Resilient Distributed Datasets) based on dependencies. During the processing of the Spark computing engine, each stage's computing node only processes a portion of the task's data. If the next stage needs to depend on all the computation results of the previous stage, then all the computation results of the previous stage need to be re-integrated and classified, that is, a shuffle is required to re-integrate and classify the data on all nodes in the cluster.

[0079] In this embodiment, containers are divided into MapNode and ReduceNode types based on whether a shuffle has occurred. MapNode containers are those before the shuffle, while ReduceNode containers are those after the shuffle. MapNode containers focus on reading raw data from the storage system, processing these raw files or data blocks in parallel, and then passing the processed data to downstream nodes. The main performance overhead for this type of container is in reading the raw data files; since network I / O overhead is much higher than local I / O, this type of container prioritizes local I / O. ReduceNode containers primarily receive data read or processed by MapNode containers, and their executable tasks include repartitioning, aggregation, and join operations. The performance overhead for this type of container mainly lies in the I / O overhead during the shuffle process.

[0080] In an optional embodiment, the Spark computing engine can be used to generate a container runtime graph showing the interdependent containers within the job, and the containers can be divided according to this graph. For example, the container runtime graph can be represented as a Directed Acyclic Graph (DAG). Each node in the DAG represents a container. Analyzing the graph, containers are divided into two categories based on whether a shuffle occurs. Simultaneously, based on the DAG head, the number and timing of shuffles in the job execution process can be identified, thereby identifying the number of times and the timing of calls to the multi-objective ant colony algorithm to determine the container scheduling scheme.

[0081] When modeling the allocation and scheduling of container sets before and after shuffle, it is necessary to consider the state transition paths of containers before and after shuffle, and determine the states related to container scheduling based on the state transition paths of containers. Optionally, the state transition paths of containers before and after shuffle are as follows: (1) Data_Input intensive state; (2) CPU intensive state; (3) shuffle_Output intensive state; (4) shuffle_Input intensive state; (5) CPU intensive state; (6) Data_Output intensive state. In the above state transitions, states (1), (2), and (3) belong to the states of MapNode type containers, and states (4), (5), and (6) belong to the states of ReduceNode type containers. Among the five states, the CPU intensive state is optimized, which has low correlation with container scheduling and allocation. Its state transition occurs after data loading IO, and its performance is related to the number of CPU cores allocated to the container. Since the data writing process is not related to the allocation and scheduling of containers, the Data_Output intensive state and the shuffle_Output intensive state do not need to be adjusted and optimized. Therefore, it is advisable to optimize the Data_Input intensive state of MapNode type containers and the shuffle_Input intensive state of ReduceNode type containers.

[0082] Step S202: Construct a first utility function and a second utility function. The first utility function indicates the utility of allocating a MapNode type container to the target compute node. The second utility function indicates the utility of allocating a ReduceNode type container to the target compute node.

[0083] In step S201, the Data_Input intensive state of the MapNode type container is determined to be optimizable, which means optimizing the data reading stage of the MapNode type container. The data to be read in the data reading stage is raw data, which is large in volume, rich and complex in data type, and has a large data writing pressure. For the MapNode type container, its container allocation location is usually coupled with the nodes of the distributed storage system. Therefore, in order to avoid reading data files or data blocks from distributed remote storage nodes, reduce network reading pressure, and improve container reading efficiency, local file reading is given priority, that is, the container is deployed on the computing node that stores the data to be read. Therefore, the utility of deploying the MapNode type container on the computing node can be represented by the number of data to be read (files to be read or data blocks to be read) contained in the computing node. The more data to be read contained in the computing node, the greater the utility of deploying the container on the computing node. Optionally, the first utility function corresponding to the MapNode type container is as follows (1): Ci (k) = count k (m) (1)

[0084] C i (k) represents the utility of assigning a container i of type MapNode to compute node k, count k (m) indicates that the number of data to be read (e.g., files or data blocks to be read) contained in computing node k is m.

[0085] For n MapNode type containers to be scheduled, the scheduling scheme should maximize the utility of each container, that is, maximize the following first objective function:

[0086] Here, t1 represents the target value of the scheduling scheme for n MapNode type containers. The larger t1 is, the better the effectiveness of the container scheduling scheme and the better the performance of the scheme.

[0087] In step S201, it is determined that the shuffle_Input intensive state of the ReduceNode type container can be optimized, that is, the shuffle_Input reading stage of the ReduceNode type container can be optimized. The core needs to optimize the cross-container data transmission in shuffle. Since cross-container data transmission is divided into two categories, one is cross-container transmission within the same compute node, which is essentially local IO; the other is data transmission from the compute node to a remote container, which is essentially network IO. The local IO speed of different containers within the same compute node is much higher than network IO. Therefore, in this optimization process, it is necessary to consider coupling the ReduceNode type container and the MapNode type container to the same compute node. Therefore, the utility of deploying the ReduceNode type container on the compute node can be represented by the number of connections between the ReduceNode type container and the MapNode type container within the same compute node. The larger the number of connections between the ReduceNode type container and the MapNode type container within the same compute node, the greater the utility of deploying the ReduceNode type container on the compute node. Optionally, the second utility function corresponding to the ReduceNode type container is as shown in the following equation (3): C j (k) = count k (i,s) (3)

[0088] C i (k) represents the utility of allocating a container j of type ReduceNode to compute node k, count k(i,s) represents the number of connections s between the MapNode type container deployed in compute node k and the ReduceNode type container j.

[0089] Optionally, one or more MapNode type containers or one or more ReduceNode type containers can be deployed within the same compute node. The number of connections between ReduceNode and MapNode type containers within the same compute node is the total number of full connections between them. For example, if one MapNode type container and one ReduceNode type container are deployed within the same compute node, the number of connections between them is 1. If one MapNode type container and two ReduceNode type containers are deployed within the same compute node, the number of connections between them is 2. If two MapNode type containers and two ReduceNode type containers are deployed within the same compute node, the number of connections between them is 4.

[0090] For z ReduceNode type containers to be allocated, the scheduling scheme should maximize the utility of each container, that is, maximize the following objective function:

[0091] Here, t2 represents the target value of the scheduling scheme for z ReduceNode type containers. The larger this target value, the better the efficiency of the z ReduceNode type containers.

[0092] Step S203: Construct the load balancing objective function. This load balancing objective function is used to indicate the degree of balance in distributing target containers across target compute nodes.

[0093] In step S202, local read is preferred to optimize Data_Input and Shuffle_Input, but this leads to another serious problem. To maximize the objective function, the allocated MapNode and ReduceNode containers are concentrated on a few specific computing nodes, easily resulting in a concentrated container allocation. The IO and computational load of a specific computing node may be abnormally high, causing abnormal read / write operations on other files allocated to that node, affecting the stable operation of the system. Therefore, it is necessary to eliminate the uneven node load caused by the overly concentrated container distribution. Here, a load balancing objective function can be introduced to measure the degree of balance in container allocation. As an example, the load balancing objective function is shown in equation (5):

[0094] Among them, L p Let L represent the load of the p-th compute node, and L represent the average load across the p compute nodes. To ensure that the load is evenly distributed across each compute node, it is necessary to minimize the above objective function.

[0095] Step S204: Establish resource constraint rules for containers and compute nodes.

[0096] To ensure successful container deployment during the allocation process, it's crucial to guarantee that the resource specifications of a single container are less than the available resources of all compute nodes in the current node set. For example, the resource specifications of a single container (such as the number of CPU cores and memory) must be less than the minimum available resources of the compute nodes in the current node set. In other words, the following constraints must be met:

[0097] Core i Mem represents the number of CPU cores required for container i. i Indicates the memory requirements of container i, min t∈A {Core_R t} represents the minimum number of available CPU cores in node set A, min t∈A {Mem_R t} represents the minimum amount of available memory in the node set A.

[0098] In an optional embodiment, a fixed container resource specification can be set to facilitate the rapid allocation of multiple containers.

[0099] On the other hand, from the perspective of the compute node, the available resources (also known as remaining allocable resources) of the compute node to which the container is to be allocated should be greater than the resource specifications of the container. For example, the number of available CPU cores and memory of the compute node needs to be greater than the number of CPU cores and memory required by the container. Only when this condition is met can the container be allocated on that compute node. For example, for a compute node a, the condition for allowing the allocation of container i on that node is:

[0100] Core_Remain a Mem_Remain represents the number of available CPU cores in compute node a. a This indicates the available memory for compute node a.

[0101] In this embodiment of the application, multiple containers are allowed to be assigned to the same computing node. When using the multi-target ant colony algorithm to determine the optimal container deployment scheme, it is necessary to ensure that the resource constraint rules between the container and the computing node are met, so as to ensure successful container deployment.

[0102] Step S205: Initialize the initial parameters of the multi-target ant colony algorithm.

[0103] In this embodiment, the objective of the multi-objective ant colony algorithm is: given |I| containers and |A| nodes, where each container i∈I and node a∈A, establish an allocation or mapping relationship I→A between containers and nodes, allowing multiple containers to be assigned to a single node. The initial parameters of the multi-objective ant colony algorithm include, but are not limited to: the number of ants (antNum), the pheromone matrix (pheromoneMatrix), the critical point matrix (criticalPointMatrix) of ants using a random allocation strategy in one iteration, the percentage increase q in pheromone levels for each path traversed by an ant, the percentage decrease p in pheromone levels after each iteration, and the number of iterations numIter.

[0104] Step S206: Using the first utility function and the load balancing objective function as the pheromone first update strategy for determining the scheduling scheme of MapNode type containers, the first scheduling scheme for MapNode type containers is determined based on the initial parameters of the multi-objective ant colony algorithm, the pheromone first update strategy, and the resource constraint rules.

[0105] Optionally, considering that the cluster size of computing nodes is generally large, if all computing nodes are used as candidate nodes, the multi-target ant colony algorithm will consume a lot of time to calculate the optimal container scheduling scheme. Therefore, the complexity and time of the algorithm search can be reduced by narrowing the candidate node set, thereby improving efficiency. Since the allocation of MapNode type containers should prioritize reading data from local storage, nodes containing data to be read are selected from all computing nodes as candidate set A. At the same time, in order to ensure that container allocation is not too concentrated on specific nodes, computing nodes that do not contain data to be read are randomly selected and added to candidate set A. Therefore, this application embodiment also includes: constructing a first node candidate set, which includes a first computing node storing data to be read that is to be executed and a second computing node that does not store data to be read; and determining the first target computing node for deploying MapNode type containers from the first node candidate set according to the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, so as to determine the first scheduling scheme for MapNode type containers.

[0106] In the first iteration of the multi-objective ant colony algorithm, to approximate the optimal scheduling scheme and accelerate the iteration convergence speed, a portion of the ants (e.g., 50% of the ants) can be assigned to the first computing node in the first node candidate set, and the remaining ants can be randomly assigned to the second computing node in the first node candidate set. More preferably, when assigning some ants to the first computing node, the number of data blocks to be read contained in the first computing node can be used to determine whether to assign an ant to that computing node. If the first computing node contains a large number of data blocks to be read, then the ants are preferentially assigned to that computing node. Optionally, the proportion of data blocks to be read contained in each first computing node can be used to determine whether to assign an ant to that computing node. Optionally, ants can be assigned based on the probability that an ant is assigned to a first computing node. The probability that ant t is assigned to the first computing node a is calculated according to the following formula:

[0107] p(t,a) represents the probability that ant t is assigned to computing node a, u(a) represents the number of data blocks to be read contained in the first computing node a, ∑ a∈A μ(a) represents the total amount of data blocks to be read.

[0108] In subsequent iterations of the multi-objective ant colony algorithm, ants are allocated as follows: Based on the critical point matrix (criticalPointMatrix), ants that follow the highest pheromone allocation and randomly allocated ants are determined. For ants that need to follow the highest pheromone allocation, they are assigned to the node with the highest pheromone concentration in the current row according to the current pheromone matrix. For ants that need to follow random allocation, they are randomly assigned to any computing node in the current row.

[0109] In each iteration of the multi-objective ant colony algorithm, the pheromone matrix is ​​updated once after each iteration. The process of updating the pheromone matrix includes:

[0110] 1) Reduce the concentration of all pheromones in the pheromone matrix by p% to simulate pheromone volatilization.

[0111] 2) Determine the optimal allocation scheme in this iteration based on the container utility function and the load balancing objective function, and increase the pheromone concentration corresponding to this scheme by q%. Specifically, calculate the objective function value for each ant based on the first objective function and the load balancing objective function. Optionally, the original size and distribution of the data for the first objective function and the load balancing objective function can be balanced using regularization parameters λ1 and λ2, i.e., the objective function value for each ant is calculated according to the following formula (9): T=λ1t1+λ2t3 (9)

[0112] In each iteration, AntNum ants will generate AntNum allocation schemes. According to Equation (9), the path that satisfies the objective function maximization is found, and the pheromone concentration of the path is increased.

[0113] Step S206: Using the second utility function and the load balancing objective function as the pheromone second update strategy for determining the scheduling scheme of ReduceNode type containers, the scheduling scheme for ReduceNode type containers is determined based on the initial parameters of the multi-objective ant colony algorithm, the pheromone second update strategy, the resource constraint rules, and the first scheduling scheme.

[0114] For ReduceNode type containers, they are preferentially deployed on compute nodes that contain MapNode containers to reduce network I / O. For example, the compute nodes containing MapNode type containers can be used as a search candidate set A, and nodes that do not contain MapNode type containers can be randomly added to candidate set A. This ensures that ReduceNode type containers can also maximize the utilization of the localized computation results of MapNodes. Therefore, this application embodiment also includes: constructing a second node candidate set, which includes a first target compute node and a third compute node that does not deploy MapNode type containers; and determining a second target compute node for deploying ReduceNode type containers from the second node candidate set according to the initial parameters of the multi-target ant colony algorithm and the second pheromone update strategy, so as to determine a second scheduling scheme for ReduceNode type containers.

[0115] In the first iteration of the multi-target ant colony algorithm, in order to get closer to the optimal scheduling scheme and accelerate the iteration convergence speed, some ants (e.g., 50% of the ants) can be assigned to the first target computing node in the second node candidate set, and the remaining ants can be randomly assigned to the third computing node in the second node candidate set.

[0116] In subsequent iterations of the multi-objective ant colony algorithm, ants are allocated as follows: Based on the critical point matrix (criticalPointMatrix), ants that follow the highest pheromone allocation and randomly allocated ants are determined. For ants that need to follow the highest pheromone allocation, they are assigned to the node with the highest pheromone concentration in the current row according to the current pheromone matrix. For ants that need to follow random allocation, they are randomly assigned to any computing node in the current row.

[0117] In each iteration of the multi-objective ant colony algorithm, the pheromone matrix is ​​updated once after each iteration. The process of updating the pheromone matrix includes:

[0118] 1) Reduce the concentration of all pheromones in the pheromone matrix by p% to simulate pheromone volatilization.

[0119] 2) Determine the optimal allocation scheme in this iteration based on the container utility function and the load balancing objective function, and increase the pheromone concentration corresponding to this scheme by q%. Specifically, calculate the objective function value for each ant based on the second objective function and the load balancing objective function. Optionally, the original size and distribution of the data for the first objective function and the load balancing objective function can be balanced using regularization parameters λ3 and λ2, i.e., the objective function value for each ant is calculated according to the following formula (10): T=λ3t2+λ2t3 (10)

[0120] In each iteration, AntNum ants will generate AntNum allocation schemes. According to Equation (10), the path that satisfies the objective function maximization is found, and the pheromone concentration of the path is increased.

[0121] The container scheduling method provided in this application divides containers into two types: MapNode and ReduceNode, with the shuffle process as the boundary. For MapNode type container allocation, priority should be given to the local files to be read. However, the allocation of ReduceNode type containers depends on the distribution of existing MapNode type containers. Therefore, the allocation of MapNode type containers and ReduceNode type containers needs to be divided into two stages: first, the scheduling calculation of MapNode type containers is completed, and then the scheduling calculation of ReduceNode type containers is completed. Stage 1 is used to perform the scheduling calculation of MapNode type containers, and the algorithm optimization objectives are Equations (2) and (5). Stage 2 is used to perform the scheduling scheme calculation of ReduceNode type containers, and the nodes allocated with MapNode type containers in Stage 1 are taken as part of the candidate set, and the algorithm optimization objectives are changed to Equations (4) and (5). Both stages use the above multi-objective ant colony algorithm to iteratively search for the optimal value. This embodiment divides containers into MapNode and ReduceNode types, using the shuffle process as the boundary. It prioritizes allocating MapNode type containers first, followed by ReduceNode type containers, dynamically completing the allocation in a two-stage manner. This breaks down the problem into two stages of local search, balancing algorithm performance and search efficiency. In allocating both types of containers, this embodiment reduces the number of nodes the ant colony algorithm needs to search by narrowing the candidate set of nodes, thus lowering computational complexity. Furthermore, in the first iteration of the multi-objective ant colony algorithm, this embodiment initializes an allocation scheme close to the target, reducing iterative search time and allowing the algorithm to approach the optimal solution more quickly.

[0122] In optional embodiments, to improve container scheduling efficiency and avoid spending a lot of time waiting for scheduling schemes before task startup, a pre-training approach can be adopted. This involves pre-determining the number and specifications of one or more containers to calculate the corresponding container allocation scheme, which is then pre-stored in the system. For example, a resident memory thread pool can be started, and a thread can be submitted to the pool to extract the distribution of various files and their data blocks in the distributed storage system in real time. This thread can obtain information such as file block size and block distribution by establishing an RPC connection (Remote Procedure Call) with the NameNode (which maintains the file system's metadata, including the directory tree, file and block location information, etc.). Additionally, a thread can be submitted to the thread pool to obtain metadata about the used and remaining CPU and memory resources of the compute nodes. This thread obtains this information by establishing an RPC connection with the ResourceManager. The metadata obtained by the thread can be persistently saved to a key-value (kv) storage system. For each file read from the metadata, the optimal container scheduling scheme is calculated separately. In essence, for a given file, different container specifications (CPU, memory) and container quantities are predefined. Under the current compute node resource distribution, a multi-objective ant colony algorithm is used to calculate allocation schemes for MapNode and ReduceNode type containers, which are then persistently saved to the key-value store system. It's important to note that if compute node resources change, the container scheduling scheme for that file in the current environment must be modified. When a container needs to be started, the relevant scheduling plan can be directly queried and quickly returned to the ResourceManager (responsible for resource allocation across the entire cluster, used for quickly starting relevant containers). For example, when a client needs to initiate a container scheduling request, it sends a scheduling request specifying the container specifications and number of containers. In response to this scheduling request, the key-value store is queried. If a relevant scheduling scheme exists, it is provided to the client, facilitating a quick lookup of the pre-stored scheme and reducing waiting time before task execution. If no relevant scheduling scheme exists in the key-value store, the multi-objective ant colony algorithm is used to calculate allocation schemes for MapNode and ReduceNode type containers based on the container specifications and capacity, under the current compute node resource distribution.

[0123] The container scheduling scheme provided in this application uses the container utility function and the load balancing objective function as pheromone update strategies to control the pheromone update in each iteration of the ant colony algorithm, thereby determining the optimal container deployment scheme. The container utility function ensures priority for local data reading, allocating containers as much as possible to the computing nodes where the data to be read resides. This allows applications to read data from local nodes as much as possible, avoiding access to data from remote nodes, thus reducing network I / O and improving job execution efficiency. The load balancing objective function ensures that the load is evenly distributed across each computing node, preventing system instability caused by high load on specific computing nodes.

[0124] Figure 3 shows a schematic diagram of the container scheduling device provided in an embodiment of this application. As shown in Figure 3, the container scheduling device 300 includes:

[0125] The construction module 301 is used to construct a container utility function and a load balancing objective function, and to determine the pheromone update strategy of the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function; wherein, the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the load balancing objective function is used to indicate the degree of balance in allocating the target container to the target computing node;

[0126] Initialization module 302 is used to initialize the initial parameters of the multi-target ant colony algorithm;

[0127] The determination module 303 is used to determine the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy.

[0128] Optionally, the determination module is also used to: establish resource constraint rules for containers and computing nodes; and determine a container scheduling scheme based on the initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy, and the resource constraint rules.

[0129] Optionally, the device also includes a container partitioning module, used to: partition containers into MapNode type and ReduceNode type according to the container operation graph of the job to be executed; the MapNode type containers are containers before the shuffle occurs, and the ReduceNode type containers are containers after the shuffle occurs;

[0130] The construction module is used to: construct a first utility function and construct a second utility function; the first utility function is used to indicate the utility of allocating a MapNode type container to a target compute node, and the second utility function is used to indicate the utility of allocating a ReduceNode type container to a target compute node; the first utility function and the load balancing objective function are used as the first pheromone update strategy for determining the scheduling scheme of MapNode type containers, and the second utility function and the load balancing objective function are used as the second pheromone update strategy for determining the scheduling scheme of ReduceNode type containers.

[0131] Optionally, the determining module is used to: determine a first scheduling scheme for MapNode type containers based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy; and determine a scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme.

[0132] Optionally, the determination module is used for:

[0133] Construct a first node candidate set, which includes a first computing node that stores data to be read containing instructions for jobs to be executed and a second computing node that does not store data to be read.

[0134] Based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, the first target computing node for deploying MapNode type containers is determined from the first node candidate set, so as to determine the first scheduling scheme for MapNode type containers.

[0135] Construct a second node candidate set, which includes a third computing node that has deployed the first target computing node and a container of type MapNode that has not been deployed.

[0136] Based on the initial parameters of the multi-target ant colony algorithm and the second pheromone update strategy, the second target computing node for deploying ReduceNode type containers is determined from the second node candidate set, so as to determine the second scheduling scheme for ReduceNode type containers.

[0137] Optionally, the determination module is used for:

[0138] In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first computing node in the first node candidate set, and the remaining ants are randomly assigned to the second computing node in the first node candidate set.

[0139] Based on the initial parameters of the multi-objective ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme, the scheduling scheme for ReduceNode type containers is determined, including:

[0140] In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

[0141] Optionally, the determining module is used to: during the first iteration of the multi-target ant colony algorithm, for each first computing node, determine whether to assign ants to the first computing node based on the proportion of the data blocks to be read included in the first computing node.

[0142] The above-described apparatus can execute the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for executing the methods. Technical details not described in detail in this embodiment can be found in the container scheduling method provided in the embodiments of this application.

[0143] [Corrected according to Rule 91, 11.12.2025] Optionally, embodiments of this application also provide a container scheduling system 400, including a resource analysis module 401, a modeling and calculation module 402, and a query response module 403.

[0144] [Corrected according to Rule 91, 11.12.2025] Resource Analysis Module 401: This module extracts in real time the distribution of each file and its data blocks in the distributed storage system, as well as the used and remaining resources of each computing node, to provide basic data for scheduling modeling. The execution process of this module is as follows: A resident in-memory thread pool is started to obtain metadata information of scheduling resources. A thread is submitted to the thread pool to obtain block metadata information. This thread obtains information such as file block size and block distribution by establishing an RPC connection with the NameNode node. In addition, a thread is submitted to the thread pool to obtain metadata of the used and remaining CPU and memory resources of the computing nodes. This thread obtains the above information by establishing an RPC connection with the ResourceManager. The metadata obtained by the thread can be persistently saved to the key-value storage system.

[0145] [Corrected according to Rule 91, 11.12.2025] Modeling and Calculation Module 402: Performs modeling and calculation of container scheduling using the container scheduling method provided in the embodiments of this application. To avoid spending a lot of time waiting for the scheduling scheme before the task starts, it is recommended to adopt a pre-training approach, that is, to pre-determine the number and specifications of one or more containers to calculate the corresponding container allocation scheme and store it in the system. The execution process of this module is as follows: For the files read from the metadata, calculate the optimal container scheduling scheme for each file. That is, for a file, under different container specifications (CPU, memory) and container numbers, use the container scheduling method provided in the embodiments of this application to calculate the allocation scheme of MapNode and ReduceNode under the current computing node resource distribution, and persist it to the key-value storage system. It should be noted that if the computing node resources change, the container scheduling scheme of the file in the current environment needs to be modified. The method proposed in this paper is also used, but the previous optimal solution is still used during solution initialization, and then the new optimal solution is quickly calculated using this method.

[0146] [Corrected according to Rule 91, 11.12.2025] Query Response Module 403: Used to respond to users. When a container needs to be started, it can directly query the relevant scheduling plan and quickly return it to the ResourceManager. The execution process of this module is as follows: When the client needs to initiate a container scheduling request, it sends a scheduling request specifying the container specifications and number of containers to this module. The query response module will query the key-value store. If a relevant scheduling scheme exists, it will hand the container scheduling scheme to the client. If no relevant scheduling scheme exists in the key-value store, it will submit the container specifications and container capacity data to the modeling and calculation module to calculate the container scheduling scheme under the specification constraints and submit the calculation result to the client. Figure 5 shows a schematic diagram of the structure of the electronic device according to an embodiment of this application. As shown in Figure 5, the electronic device includes:

[0147] [Corrected according to Rule 91, 11.12.2025] Processor 501, communication interface 502, memory 503, and communication bus 504, wherein processor ... and communication bus 504 are specified. [0147.1][Correction 11.12.2025 according to detailed rules 91] 502, the memory 503 completes communication with each other through the communication bus 504.

[0148] [Correction 11.12.2025 based on Rule 91] Memory 503 is used to store computer programs;

[0149] [Correction 11.12.2025 based on Rule 91] When processor 501 executes a program stored in memory 503, it performs the following steps:

[0150] Construct a container utility function and a load balancing objective function, and determine the pheromone update strategy for the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function; wherein, the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node; the load balancing objective function is used to indicate the degree of balance in allocating the target container to the target computing node;

[0151] Initialize the initial parameters of the multi-target ant colony algorithm;

[0152] The container scheduling scheme is determined based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy.

[0153] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0154] The communication interface is used for communication between the aforementioned terminal and other devices.

[0155] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0156] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0157] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the bandwidth auditing methods described in the above embodiments.

[0158] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the bandwidth auditing methods described in the above embodiments.

[0159] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0160] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0161] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0162] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. A container scheduling method, characterized in that, include: Construct a container utility function and a load balancing objective function, and determine the pheromone update strategy for the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function; wherein, the container utility function is used to indicate the utility of allocating the target container to the target computing node, and the utility of allocating the target container to the target computing node is proportional to the amount of data to be read included in the target computing node; the load balancing objective function is used to indicate the degree of balance in allocating the target container to the target computing node; Initialize the initial parameters of the multi-target ant colony algorithm; The container scheduling scheme is determined based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy.

2. The method according to claim 1, characterized in that, The step of determining the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy includes: Establish resource constraint rules for containers and compute nodes; The container scheduling scheme is determined based on the initial parameters of the multi-objective ant colony algorithm, the pheromone update strategy, and the resource constraint rules.

3. The method according to claim 1, characterized in that, The method further includes: Based on the container runtime graph of the job to be executed, the containers are divided into MapNode type and ReduceNode type; the MapNode type container is the container before the shuffle, and the ReduceNode type container is the container after the shuffle. The container utility function includes: Construct a first utility function and a second utility function; the first utility function is used to indicate the utility of allocating a container of type MapNode to a target compute node, and the second utility function is used to indicate the utility of allocating a container of type ReduceNode to a target compute node; The step of determining the pheromone update strategy for the multi-objective ant colony algorithm based on the container utility function and the load balancing objective function includes: using the first utility function and the load balancing objective function as the first pheromone update strategy for determining the scheduling scheme of MapNode type containers, and using the second utility function and the load balancing objective function as the second pheromone update strategy for determining the scheduling scheme of ReduceNode type containers.

4. The method according to claim 1, characterized in that, The step of determining the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy includes: Based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, a first scheduling scheme for MapNode type containers is determined. Based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme, a scheduling scheme for ReduceNode type containers is determined.

5. The method according to claim 4, characterized in that, The step of determining the first scheduling scheme for MapNode type containers based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy includes: Construct a first node candidate set, which includes a first computing node that stores the data to be read containing the instruction of the job to be executed and a second computing node that does not store the data to be read. Based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy, a first target computing node for deploying the MapNode type container is determined from the first node candidate set, so as to determine the first scheduling scheme for the MapNode type container; The step of determining the scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes: Construct a second node candidate set, which includes a third computing node that has the first target computing node deployed and a container of the MapNode type not deployed. Based on the initial parameters of the multi-target ant colony algorithm and the second pheromone update strategy, a second target computing node for deploying the ReduceNode type container is determined from the second node candidate set, so as to determine the second scheduling scheme for the ReduceNode type container.

6. The method according to claim 4, characterized in that, The step of determining the first scheduling scheme for MapNode type containers based on the initial parameters of the multi-target ant colony algorithm and the first pheromone update strategy includes: In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first computing node in the first node candidate set, and the remaining ants are randomly assigned to the second computing node in the first node candidate set. The step of determining the scheduling scheme for ReduceNode type containers based on the initial parameters of the multi-target ant colony algorithm, the second pheromone update strategy, and the first scheduling scheme includes: In the first iteration of the multi-target ant colony algorithm, some ants are assigned to the first target computing node in the second node candidate set, and the remaining ants are randomly assigned to the third computing node in the second node candidate set.

7. The method according to claim 6, characterized in that, In the first iteration of the multi-target ant colony algorithm, assigning a portion of the ants to the first computing node in the first node candidate set includes: In the first iteration of the multi-target ant colony algorithm, for each first computing node, it is determined whether to assign ants to the first computing node based on the proportion of the data blocks to be read included in the first computing node.

8. A container scheduling device, characterized in that, include: A construction module is used to construct a container utility function and a load balancing objective function, and to determine the pheromone update strategy for a multi-objective ant colony algorithm based on the container utility function and the load balancing objective function. The container utility function indicates the utility of allocating a target container to a target computing node, and the utility of allocating a target container to a target computing node is proportional to the amount of data to be read included in the target computing node. The load balancing objective function indicates the degree of balance in allocating the target container to the target computing node. The initialization module is used to initialize the initial parameters of the multi-target ant colony algorithm. The determination module is used to determine the container scheduling scheme based on the initial parameters of the multi-target ant colony algorithm and the pheromone update strategy.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images