Scheduling method and apparatus

Abstract

Disclosed are a scheduling method and apparatus. The method is applied to an inference system. The method comprises: after acquiring a plurality of inference requests, an inference system grouping the inference requests to obtain a plurality of groups; determining a target tensor parallelism degree of each group; and scheduling an inference request in each group on the basis of the target tensor parallelism degree of each group.

Description

Scheduling methods and devices

[0001] This application claims priority to Chinese Patent Application No. 202510181091.1, filed with the State Intellectual Property Office of China on February 18, 2025, entitled “Scheduling Method and Apparatus”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a scheduling method and apparatus. Background Technology

[0003] With the development of inference technology, inference services have been widely applied in various fields. In related technologies, to improve the performance of inference services provided by inference systems, parallel inference strategies have been proposed, which involve executing inference requests in parallel using multiple processors within the inference system. When adopting parallel inference strategies, related technologies manually determine a fixed tensor parallelism degree for the inference system. However, using this fixed tensor parallelism degree to process inference requests does not effectively improve the performance of the inference service. Summary of the Invention

[0004] This application provides a scheduling method and apparatus that helps improve the performance of inference services provided by an inference system.

[0005] Firstly, a scheduling method is provided for an inference system. The method includes: grouping multiple acquired inference requests into multiple groups; determining the target tensor parallelism of each group, wherein, according to the target tensor parallelism of each group, the inference performance when processing the inference requests in each group meets the target condition, and the latency of processing the inference requests in each group is less than or equal to the target latency indicated by the inference requests in each group; and scheduling the inference requests in each group according to the target tensor parallelism of each group.

[0006] In the above scheme, after the inference system obtains multiple groups, it determines the target tensor parallelism for each group and schedules the inference requests in each group according to the target tensor parallelism. Because a target tensor parallelism is determined for each group, the inference system, when processing the inference requests in each group according to the target tensor parallelism, can not only ensure that the latency for processing the inference requests in each group is less than or equal to the user-specified target latency, but also ensure that the inference performance corresponding to each group meets the target conditions. Thus, by setting appropriate target conditions, high inference performance can be provided to the users to whom the inference requests in each group belong, thereby helping to improve the performance of the inference service provided by the inference system to users.

[0007] In one possible implementation, the inference performance corresponding to the target tensor parallelism of each group includes at least one of the following: the size of the scheduling group corresponding to the target tensor parallelism of each group or the throughput corresponding to the target tensor parallelism of each group, wherein the size of the scheduling group is used to indicate the number of inference requests scheduled each time.

[0008] In this implementation, inference performance can be set to at least one of scheduling group size or throughput, which not only helps to improve the diversity of inference performance, but also helps to improve the performance of inference services for users from multiple dimensions.

[0009] In another possible implementation, the size of the scheduling group for each group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group, wherein the scheduling threshold corresponding to the target tensor parallelism of each group is used to indicate the number of inference requests that the processor can support processing, as indicated by the target tensor parallelism of each group.

[0010] In this implementation, by setting the scheduling group size of each group to be less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group, it helps to ensure that when processing the inference requests in each group according to the target tensor parallelism of each group, the processor indicated by the target tensor parallelism of each group can carry the inference requests in the scheduling group of each group. This helps to ensure the performance stability and reliability of the inference system and avoids data loss due to the inability to carry the data obtained during the inference process, which would affect the accuracy of the inference results.

[0011] In another possible implementation, multiple groups include a target group. Determining the target tensor parallelism of each group in the multiple groups includes: determining the size of the scheduling group corresponding to each tensor parallelism based on the target latency indicated by the inference requests in the target group, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism; and determining the target tensor parallelism of the target group from at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism.

[0012] In this implementation, when inference performance includes the size of the scheduling group, determining the final target tensor parallelism by using the size of the scheduling group corresponding to different tensor parallelisms helps ensure that the size of the scheduling group corresponding to the determined target tensor parallelism meets the target condition, thus helping to ensure the accuracy of the determined target tensor parallelism. When inference performance includes throughput, since the size of the scheduling group is mostly proportional to the throughput, determining the final target tensor parallelism by using the size of the scheduling group corresponding to different tensor parallelisms helps to simplify the process of determining the target tensor parallelism, thereby improving the efficiency of determining the target tensor parallelism.

[0013] In another possible implementation, multiple groups include a target group. Determining the target tensor parallelism of each group in the multiple groups includes: determining the size of the scheduling group corresponding to each tensor parallelism based on the target latency indicated by the inference requests in the target group, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism; determining the throughput corresponding to each tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism; and determining the target tensor parallelism of the target group from at least one tensor parallelism based on the throughput corresponding to each tensor parallelism.

[0014] In this implementation, when inference performance includes throughput, determining the final target tensor parallelism by using the size of the scheduling group corresponding to different tensor parallelisms helps ensure that the throughput corresponding to the determined target tensor parallelism meets the target condition, thereby helping to ensure the accuracy of the determined target tensor parallelism. When inference performance includes the size of the scheduling group, determining the final target tensor parallelism by using the throughput corresponding to different tensor parallelisms helps to increase the diversity of methods for determining the target tensor parallelism.

[0015] In another possible implementation, the size of the scheduling group corresponding to each tensor parallelism satisfies the following relationship (1):

[0016] Among them, BS1 i The size of the scheduling group corresponding to the i-th tensor parallelism in at least one tensor parallelism is used to characterize the size of the scheduling group; TL is used to characterize the target latency of the inference request indication in the target group; NL is used to characterize the target latency of the inference request indication in the target group. i The network communication latency corresponding to the i-th tensor parallelism in at least one tensor parallelism is used; LN is used to characterize the number of hidden layers in the inference model of the inference system; NT is used to characterize the number of communications between sub-tensors in a single hidden layer of the inference model; MMT is used to characterize the total latency required to compute the matrix layer of each hidden layer of the inference model; PAL is used to characterize the network communication latency corresponding to the i-th tensor parallelism in at least one tensor parallelism. i The page attention latency corresponding to the i-th tensor parallelism in at least one tensor parallelism is used to characterize the page attention latency corresponding to the i-th tensor parallelism. This page attention latency indicates the latency of performing page attention on the scheduling group of the target group when processing the scheduling group of the target group according to the i-th tensor parallelism. OT is used to characterize the loss latency when the inference system processes inference requests in the target group. K i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

[0017] In another possible implementation, the throughput corresponding to each tensor parallelism satisfies the following relationship (2):

[0018] Among them, T i LOut is used to characterize the throughput corresponding to the i-th tensor parallelism in at least one tensor parallelism, and K is used to characterize the output load of inference requests in the target group. i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

[0019] In another possible implementation, determining the target tensor parallelism of the target group from at least one tensor parallelism includes: determining a scheduling threshold corresponding to each tensor parallelism in the at least one tensor parallelism, wherein the scheduling threshold corresponding to the target tensor parallelism is used to indicate the number of inference requests that the processor indicated by each tensor parallelism supports processing; and determining the target tensor parallelism of the target group from the at least one tensor parallelism based on the scheduling threshold corresponding to each tensor parallelism, wherein the size of the scheduling group corresponding to the target tensor parallelism of the target group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of the target group.

[0020] In this implementation, by setting a scheduling threshold corresponding to each tensor parallelism when determining the target tensor parallelism from at least one tensor parallelism, it can be ensured that the processor indicated by the target tensor parallelism can carry the scheduling group corresponding to the target tensor parallelism, thereby helping to ensure the reliability and accuracy of the inference service provided by the inference performance.

[0021] In another possible implementation, the scheduling threshold corresponding to each tensor parallelism satisfies the following relationship (3):

[0022] Among them, BS2 i K is a scheduling threshold used to characterize the i-th tensor parallelism in at least one tensor parallelism. i The tensor parallelism is used to characterize the i-th tensor parallelism in at least one tensor parallelism, d is used to characterize the size of the storage space of a single processor of the inference system, MS is used to characterize the size of the storage space required by the inference model, and D is used to characterize the size of the storage space required by a single inference request in the target group.

[0023] In another possible implementation, the acquired multiple inference requests are grouped to obtain multiple groups, including: acquiring the load of each inference request among the multiple inference requests, wherein the load of each inference request is used to indicate the input load and the output load of each inference request; and grouping the multiple inference requests according to the load of each inference request to obtain multiple groups.

[0024] In this implementation, the load of inference requests is obtained through input and output loads. This helps the load of inference requests to more accurately indicate the processing latency of inference requests, enabling subsequent grouping based on inference request loads to group inference requests with similar processing latencies into the same group. Thus, when the inference system processes inference requests in each scheduling group within the same group in parallel, it helps ensure that inference requests in a scheduling group can be processed in similar timeframes. This avoids a situation where some inference requests in a scheduling group have been processed while others still require a longer processing time, forcing the inference system to wait for the latter to complete before scheduling further inference requests. This ultimately improves the performance of the inference system in processing inference requests.

[0025] In another possible implementation, multiple inference requests are grouped according to the load of each inference request to obtain multiple groups, including: grouping multiple inference requests according to the load of each inference request using a clustering algorithm to obtain multiple groups.

[0026] In this implementation, multiple inference requests are clustered using a clustering algorithm to obtain multiple groups, which helps to improve grouping efficiency and accuracy.

[0027] In another possible implementation, the inference model includes an inference model, the target tensor parallelism of the target group is M, the target group includes a first scheduling group, and the inference requests in each group are scheduled according to the target tensor parallelism of each group, including: deploying the M sub-tensors of the inference model on the M processors of the inference system; deploying the M sub-blocks of each inference request in the first scheduling group on the M processors, so that the M sub-tensors on the M processors process the M sub-blocks of each inference request in the first scheduling group in parallel.

[0028] In another possible implementation, the size of the scheduling group corresponding to each tensor parallelism satisfies the following relationship (4):

[0029] In another possible implementation, the throughput corresponding to each tensor parallelism satisfies the following relationship (5): T i =LOut*BS1 i (5)

[0030] In another possible implementation, the method further includes: determining the size of the storage space required for a single inference request in the target group based on the load of a single inference request in the target group.

[0031] In another possible implementation, the storage space required for a single inference request in the target group satisfies the following relationship (6): D=(LIn+LOut)*HS*LN*U*CA (6)

[0032] Where D represents the size of the storage space required for a single inference request in the target group, LIn represents the input load of a single inference request in the target group, LOut represents the output load of a single inference request in the target group, HS represents the dimension of the hidden layer of the inference model, LN represents the number of hidden layers of the inference model, U represents the amount of key-value (KV) data stored in each hidden layer of the inference model, and CA represents the computational precision of the model.

[0033] In another possible implementation, after scheduling inference requests in each group according to the target tensor parallelism of each group, the method may further include: obtaining at least one delay corresponding to the target parallelism of each group; the at least one delay corresponding to the target parallelism of each group includes at least one of the following: network communication delay, total delay required for computation of the matrix layer of each hidden layer of the inference model, delay of page attention execution of the scheduling group of the target group, and loss delay when the inference model processes inference requests in the target group.

[0034] In this implementation, after the inference system processes the inference request in the target group, it obtains the latency required in the above relationship (1), which helps to ensure that when the scheduling group size of the target group is determined based on the newly obtained latency, the accuracy of the determined scheduling group size is improved.

[0035] In another possible implementation, determining the target tensor parallelism of each of the multiple groups includes: determining the inference performance corresponding to each tensor parallelism in at least one tensor parallelism based on the target latency indicated by the inference request in the target group; the inference performance corresponding to each tensor parallelism is used to indicate the inference performance when processing the inference request in the target group according to each tensor parallelism; determining the target tensor parallelism of the target group from at least one tensor parallelism based on the inference performance corresponding to each tensor parallelism; and the inference performance corresponding to the target tensor parallelism of the target group satisfies the target condition.

[0036] In another possible implementation, the inference performance corresponding to the target tensor parallelism of the target group satisfies at least one of the following conditions: the inference performance corresponding to the target tensor parallelism of the target group is the largest among at least one tensor parallelism; the inference performance corresponding to the target tensor parallelism of the target group is greater than or equal to a performance threshold; or the inference performance corresponding to the target tensor parallelism of the target group is ranked less than or equal to a ranking threshold among at least one tensor parallelism. The inference performance corresponding to the target tensor parallelism of the target group is used to indicate the inference performance when processing inference requests in the target group according to the target tensor parallelism of the target group.

[0037] Secondly, a scheduling device is provided, comprising: functional units for executing any of the methods provided in the first aspect, wherein the actions performed by each functional unit are implemented in hardware or by hardware executing corresponding software. For example, the scheduling device includes: a grouping module, a determination module, and a scheduling module; the grouping module is used to group multiple acquired inference requests into multiple groups; the determination module is used to determine the target tensor parallelism of each group; according to the target tensor parallelism of each group, the inference performance when processing the inference requests in each group meets the target condition, and the latency of processing the inference requests in each group is less than or equal to the target latency indicated by the inference requests in each group; the scheduling module is used to schedule the inference requests in each group according to the target tensor parallelism of each group.

[0038] Thirdly, a processor is provided that can be used to execute any of the methods provided in the first aspect above.

[0039] Fourthly, a chip is provided, comprising: a processor and a power supply circuit; the power supply circuit can be used to supply power to the chip; the processor can be used to execute any of the methods provided in the first aspect above.

[0040] Fifthly, a management node is provided, comprising: a processor, a memory, and a computer program / instructions stored in the memory; the processor executes the computer program / instructions to cause the management node to perform any of the methods provided in the first aspect above.

[0041] A sixth aspect provides an inference system, comprising: at least one management node and at least one inference node; the at least one management node includes a processor, a memory, and a computer program / instructions stored in the memory; the processor of the at least one management node executes the computer program / instructions stored in the memory of the at least one management node to enable the at least one management node to implement any of the methods provided in the first aspect above, so that the at least one inference node can process inference requests scheduled by the management node.

[0042] In a seventh aspect, a computer program product is provided, comprising a computer program / instructions that, when executed by a computing device, implement any of the methods provided in the first aspect above.

[0043] Eighthly, a computer-readable storage medium is provided, on which a computer program / instructions are stored, which, when executed by a computing device, implement any of the methods provided in the first aspect above.

[0044] The technical effects of any of the implementation methods in aspects two through eight can be seen in the technical effects of different implementation methods in aspect one above, and will not be repeated here. Attached Figure Description

[0045] Figure 1 is a schematic diagram of one of the system architectures provided in this application;

[0046] Figure 2 is a second schematic diagram of a system architecture provided in this application;

[0047] Figure 3 is a schematic diagram of a system architecture provided in this application;

[0048] Figure 4 is a schematic diagram of a system architecture provided in this application;

[0049] Figure 5 is a flowchart of a scheduling method provided in this application;

[0050] Figure 6 is a schematic diagram of a grouping method provided in this application;

[0051] Figure 7 is a schematic diagram of a reasoning process provided in this application;

[0052] Figure 8 is a schematic diagram of a scheduling device provided in this application;

[0053] Figure 9 is a schematic diagram of a computing device provided in this application;

[0054] Figure 10 is a schematic diagram of a computing device cluster provided in this application;

[0055] Figure 11 is a schematic diagram of the connection of a computing device cluster provided in this application. Detailed Implementation

[0056] To facilitate understanding, a brief introduction to the relevant terms used in this application will be provided first.

[0057] Tensor: A collection of components that satisfy transformation relationships when the coordinate system changes; it is a multidimensional array.

[0058] Tensor parallelism (TP) refers to partitioning a model's tensor (e.g., the weight matrix) into multiple sub-tensors according to their dimensions, and then deploying these sub-tensors across multiple processors. Each processor processes the partitioned sub-tensors, and the computation results from each processor are combined through communication operations. For example, a model employing tensor parallelism could be an inference model.

[0059] Tensor parallelism refers to the granularity of tensor partitioning, that is, the number of sub-quanta obtained after partitioning the model's tensor. For example, if the model's tensor is partitioned into q tensors, the tensor parallelism of the model is q, where q is a positive integer greater than 1.

[0060] The technical solution provided in this application will be described in detail below with reference to the accompanying drawings.

[0061] With the development of inference technology, inference services have been widely applied in various fields. In related technologies, to improve the performance of inference services provided by inference systems, parallel inference strategies have been proposed, which involve executing inference requests in parallel using multiple processors within the inference system. When employing parallel inference strategies, related technologies often manually determine a fixed tensor parallelism degree for the inference system. However, using this fixed tensor parallelism degree to process inference requests does not effectively improve the performance of the inference service.

[0062] In view of this, this application provides a scheduling method applied to an inference system. After acquiring multiple inference requests, the inference system groups these requests into multiple groups. Then, the inference system determines the tensor parallelism for each group and schedules the inference requests within each group according to this tensor parallelism. Since a target tensor parallelism is determined for each group, when the inference system processes the inference requests in each group according to the target tensor parallelism, it can not only ensure that the latency for processing the inference requests in each group is less than or equal to the user-specified target latency, but also ensure that the inference performance corresponding to each group meets the target conditions. Thus, by setting appropriate target conditions, higher inference performance can be provided to the users to whom the inference requests in each group belong, thereby helping to improve the performance of the inference service provided by the inference system to users.

[0063] It should be noted that the various implementations of the above scheduling method will be introduced in the embodiment shown in Figure 5, and will not be described in detail here.

[0064] Next, the system architecture involved in the technical solution provided in this application will be further described with reference to the accompanying drawings.

[0065] This application provides an inference system that applies the above-described scheduling method, wherein the inference system processes inference requests through an inference model.

[0066] Example 1: The inference system includes multiple processors, which process inference requests through an inference model. For example, as shown in Figure 1, inference system 1 includes processor 1, ..., processor J, where J is a positive integer greater than 1.

[0067] It should be noted that in this application, "multiple" includes two or more, which will not be elaborated further.

[0068] Optionally, the processor of the inference system includes artificial intelligence (AI) processors such as central processing unit (CPU), graphics processing unit (GPU), data processing unit (DPU), neural processing unit (NPU), tensor processing unit (TPU), and intelligent processing unit (IPU).

[0069] It should be noted that this application does not limit the specific form of the processor in the inference system; the above is merely an illustrative example.

[0070] For example, the inference model may include at least one of a large language model (LLM), a vision language model (VLM), an audio language model (ALM), or a large multimodal model (LMM).

[0071] It should be noted that this application does not limit the type of inference model; the above is merely an illustrative example.

[0072] For example, processing a reasoning request includes performing a prefill task and a decoding task. The prefill task involves reasoning from the prompt indicated by the reasoning request using a reasoning model to obtain the first token, which can also be called the first token or the initial token. The decoding task involves reasoning from the first token to obtain the second token, from the second token to obtain the third token, and so on, until the f-th token is used to obtain the reasoning result (i.e., the (f+1)-th token). Here, f is a positive integer greater than 1.

[0073] In one example, the pre-filling and decoding tasks can be executed through the same inference node. In another example, the pre-filling and decoding tasks can be executed through different inference nodes.

[0074] Optionally, the inference system also includes the memory of each of the multiple processors. For example, as shown in FIG1, the inference system 1 also includes the memory 11 of processor 1, ..., the memory J1 of processor J.

[0075] Each processor's memory is used to store data. For example, the data stored in the memory may include prompts indicating the inference request, lexical and key-value (KV) data obtained during the inference process, etc.

[0076] Optionally, the memory of the inference system may include dynamic random access memory (DRAM), storage class memory (SCM), static random access memory (SRAM), high bandwidth memory (HBM), etc.

[0077] It should be noted that this application does not limit the type of memory in the inference system; the above is merely an illustrative example.

[0078] Example 2: The inference system includes at least one inference node, which includes multiple processors, and at least one inference node is used to process inference requests through an inference model. For example, as shown in Figure 2, inference system 2 includes inference node 1, ..., inference node P, where P is a positive integer greater than 1.

[0079] It should be noted that in this application, "at least one" includes one or more, which will not be elaborated further.

[0080] For example, at least one inference node includes a first inference node, wherein the first inference node can be any one of the at least one inference nodes. The following description uses the first inference node as an example to illustrate at least one inference node.

[0081] Optionally, the inference node can be a computing device. The computing device may include at least one of a terminal device or a network device.

[0082] For example, the terminal device can be an ultra-mobile personal computer (UMPC), a laptop, a netbook, a desktop computer, an all-in-one computer, etc.

[0083] It should be noted that this application does not impose any restrictions on the device form of the terminal equipment; the above is merely an illustrative example.

[0084] For example, network devices may include servers, bare metal servers, etc. A server can be a single physical server, or it can be two or more physical servers that share different responsibilities and work together to achieve the various functions of the server. For example, a server can be a blade server, a high-density server, a rack server, or a tower server, etc.

[0085] It should be noted that this application does not limit the form factor of the network device; the above is merely an illustrative example.

[0086] Optionally, an inference node includes at least one processor. The processor is used to process inference requests through an inference model. For example, as shown in Figure 3, inference node 1 includes processors 11, ..., 1u. Here, u is a positive integer greater than 1.

[0087] It should be noted that for information on the processors of the inference nodes, please refer to the processor descriptions of the inference system mentioned above; they will not be repeated here.

[0088] In one example, at least one inference node contains different inference nodes that include the same number of processors. In another example, at least one inference node contains different inference nodes that include different numbers of processors.

[0089] In one example, at least one inference node includes different inference nodes that contain processors of the same form factor. In another example, at least one inference node includes different inference nodes that contain processors of different form factors.

[0090] Optionally, the inference node also includes the memory of each of at least one processor. For example, as shown in FIG3, inference node 1 also includes the memory 111 of processor 11, ..., the memory 1u1 of processor 1u.

[0091] It should be noted that for information regarding the memory of the inference node, please refer to the description of the memory of the inference system mentioned above; it will not be repeated here.

[0092] Optionally, the inference system also includes at least one management node. The at least one management node is used to execute the scheduling method and scheduling tasks provided in this application. Scheduling tasks include allocating nodes to perform pre-filling tasks for inference and allocating nodes to perform decoding tasks for inference requests.

[0093] For example, as shown in Figure 1, inference system 1 includes management node 1. As shown in Figure 2, inference system 2 includes management node 2.

[0094] For example, this application does not limit the name of at least one management node, wherein the node in the at least one management node used to perform scheduling tasks may also be called a scheduling node.

[0095] In one example, the management node can be a processor selected from multiple processors. For example, as shown in Figure 1, management node 1 can be a processor selected from processor 1, ..., processor J. Alternatively, the management node can be a processor other than multiple processors. For example, as shown in Figure 1, management node 1 can be a processor other than processor 1, ..., processor J.

[0096] In another example, the management node can be a node selected from the inference nodes. For example, as shown in Figure 2, management node 2 can be a node selected from inference nodes 1, ..., P. Alternatively, the management node can be a node other than the inference nodes. For example, as shown in Figure 2, management node 2 can be a node other than inference nodes 1, ..., P. Other related explanations for management nodes can be found in the explanation of inference nodes above, and will not be repeated here.

[0097] Optionally, the inference system can communicate with an electronic device. For example, as shown in Figure 1, the management node 1 of inference system 1 communicates with the electronic device via a network. The electronic device can send inference requests to the inference system and receive inference results returned by the inference system, etc. As shown in Figure 2, the management node 2 of inference system 2 communicates with the electronic device via a network.

[0098] In one example, a user might send an inference request to the management node of the inference system via an electronic device. In another example, the electronic device might automatically send an inference request to the management node of the inference system while running its business.

[0099] Alternatively, the electronic device may be a mobile phone, tablet computer, handheld computer, personal computer (PC), personal digital assistant (PDA), ultra-mobile personal computer (UMPC), laptop computer, netbook, desktop computer, or all-in-one computer, etc.

[0100] Alternatively, the electronic device may be a network device.

[0101] It should be noted that this application does not limit the form of the electronic device; the above is merely an illustrative example.

[0102] The hardware architecture provided in this application has been described above. The software architecture provided in this application will now be described exemplarily with reference to Figure 4.

[0103] Optionally, the inference system includes an inference service layer, an inference engine layer, and an inference execution layer.

[0104] The inference service layer is used to group multiple inference requests to obtain at least one group. The inference engine layer is used to determine the tensor parallelism for each group in the at least one group. The inference execution layer is used to process inference requests and obtain at least one latency corresponding to the target tensor parallelism.

[0105] It should be noted that in this application, latency is used to indicate duration. For example, the latency for processing an inference request is used to indicate the duration of processing the inference request, which will not be elaborated further hereafter.

[0106] For example, the inference service layer and the inference engine layer can be deployed on the management node. The inference execution layer can also be deployed on the management node.

[0107] Optionally, the inference system provided in this application can be a system provided by infrastructure managed by a cloud service platform. The cloud services provided by the cloud service platform may include the scheduling method provided in this application. For example, the cloud service platform may be a public cloud, a private cloud, a hybrid cloud, etc.

[0108] For example, a user can purchase cloud services provided by a cloud service platform to use the scheduling method provided in this application. For instance, after purchasing cloud services, a user can communicate with the inference system managed by the cloud service platform via an electronic device to instruct the inference system to execute inference requests. During the processing of inference requests by the inference system managed by the cloud service platform, the scheduling method provided in this application can be used to schedule the inference requests, thereby enabling parallel processing of inference requests through multiple sub-tensors of the inference model.

[0109] Optionally, this application also provides target software. The target software is used to implement the scheduling method provided in this application.

[0110] For example, the target software may include the aforementioned inference service layer and inference engine layer. Furthermore, the target software may also include the aforementioned inference execution layer. By installing the target software on the inference system, the user enables the inference system to implement the scheduling method provided in this application.

[0111] It should be noted that the system architecture shown in Figures 1 to 4 does not constitute a limitation on the system architecture for executing the scheduling method provided in this application.

[0112] It should be noted that the system architecture and application scenarios described in this application are for the purpose of more clearly illustrating the technical solutions of this application, and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new application scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0113] For ease of understanding, the scheduling method provided in this application will be described below with reference to the above system architecture and accompanying drawings.

[0114] Figure 5 is a flowchart of a scheduling method provided in this application. Exemplarily, the scheduling method includes steps 501-503.

[0115] It should be noted that the "step" in this application can be abbreviated as "S", and will not be elaborated further hereafter.

[0116] The scheduling method shown in Figure 5 will be described below with reference to the system architecture shown in Figures 1 to 4.

[0117] S501: The management node groups the multiple inference requests it receives, resulting in multiple groups.

[0118] In this application, after the management node of the inference system receives multiple inference requests from users, it groups these requests into multiple groups. For example, as shown in Figure 5, the inference service layer of the management node receives inference request 1, ..., inference request z. The inference service layer of the management node groups inference request 1, ..., inference request z into group 1, ..., group v. Group 1 includes inference request 1, etc., and group v includes inference request z, etc. z is a positive integer greater than 1, and v is a positive integer greater than 1.

[0119] It should be noted that this application does not limit the number of groups obtained by the management node; the above is merely an illustrative example.

[0120] Optionally, S501 includes the following S501a and S501b.

[0121] S501a: The management node obtains the load of each inference request among multiple inference requests.

[0122] Optionally, the load of each inference request may include several cases, which are illustrated below with examples of case a and case b.

[0123] In case a, the load of each inference request is used to indicate the input load and the output load of each inference request.

[0124] The input payload of each inference request indicates the length of the prompt word indicated by each inference request, and the length of the prompt word may include the number of bytes of the prompt word. The output payload of each inference request indicates the length of the multiple tokens obtained from processing each inference request, and the length of the multiple tokens may include the number of bytes of the multiple tokens.

[0125] It should be noted that the input payload of each inference request can also be referred to as the input length (LenInput) of each inference request. The output payload of each inference request can also be referred to as the output length (LenOutput) of each inference request.

[0126] Optionally, S501a includes the following S501a1-S501a2. The load of each inference request can be obtained through the following S501a1-S501a2.

[0127] For example, the plurality of inference requests includes a first inference request. The following description uses the first inference request as an example to illustrate this application.

[0128] S501a1: The management node obtains the input load and output load of each inference request.

[0129] For example, the first inference request includes a prompt word. The management node obtains the input payload of the first inference request based on the prompt word indicated in the first inference request. For instance, the management node uses the number of bytes of the prompt word carried in the first inference request as the input payload of the first inference request.

[0130] For example, the management node obtains the output load of the first inference request based on the input load of the first inference request.

[0131] In one example, after receiving the input payload of the first inference request, the management node obtains the output payload of the first inference request through a statistical algorithm. For instance, if the prompt word indicated by the first inference request is 4000 bytes, the management node uses 4000 bytes as the input parameter of the statistical algorithm and obtains 1000 bytes as the output payload of the statistical algorithm.

[0132] For example, statistical algorithms include time series analysis algorithms, regression algorithms, Bayesian statistical algorithms, etc.

[0133] It should be noted that this application does not limit the type of statistical algorithm; the above is merely an illustrative example.

[0134] In another scenario, the management node obtains the output payload of the first inference request through the first neural network model. For example, if the management node inputs 4000 bytes into the first neural network model and receives 1000 bytes as the output of the first neural network model, then these 1000 bytes constitute the output payload of the first inference request.

[0135] For example, the first neural network model includes fully connected neural network (FCNN), large language model (LLM), etc.

[0136] It should be noted that this application does not limit the type of the first neural network model; the above is merely an illustrative example.

[0137] S501a2: The management node obtains the load of each inference request based on the input load and the output load of each inference request.

[0138] For example, the management node obtains a first array based on the input load and output load of the first inference request. This first array represents the load of the first inference request. The first array includes a first element and a second element, where the first element indicates the input load of the first inference request, and the second element indicates the output load of the first inference request.

[0139] For example, the first array can be either (first element, second element) or (second element, first element). The following example, using (first element, second element) as the first array, will be used to illustrate this application.

[0140] For example, if the input payload of the first inference request is 4000 bytes and the output payload of the first inference request is 1000 bytes, the management node obtains the first array (4000, 1000) based on the input payload and the output payload of the first inference request.

[0141] In this embodiment, the load of the inference request is represented by an array, which can simultaneously reflect the input load and the output load. This allows for grouping from both the input and output load dimensions, thereby improving the comprehensiveness of the parameters used in grouping and thus improving the accuracy of the grouping results.

[0142] It should be noted that this application does not limit the method of obtaining the payload of the inference request; the above is merely an illustrative example.

[0143] In this scenario, obtaining the inference request load through input and output load helps the load more accurately indicate the processing latency of the inference requests. This allows subsequent grouping based on the inference request load to group inference requests with similar processing latencies into the same group. When the inference system processes inference requests in each scheduling group within the same group in parallel, it helps ensure that inference requests in a scheduling group are processed in similar timeframes. This avoids a situation where some inference requests in a scheduling group have been processed while others still require a longer processing time, forcing the inference system to wait for the latter to finish before scheduling further inference requests. This ultimately improves the performance of the inference system in processing inference requests.

[0144] In case b, the load of each inference request is used to indicate the input load of each inference request.

[0145] It should be noted that the explanation of situation b can be found in the explanation of situation a above, and will not be repeated here.

[0146] In this scenario, by setting the load for each inference request to include the input load for each inference request, it not only helps to increase the diversity of load selection for each inference request, but also helps to improve grouping efficiency since it is grouped by only one input load.

[0147] S501b: The management node groups multiple inference requests based on the load of each inference request, resulting in multiple groups.

[0148] For example, after the management node obtains the load of each inference request from multiple inference requests, it groups the inference requests into multiple groups. Within the same group, the difference in load between different inference requests is less than or equal to a difference threshold.

[0149] It should be noted that this application does not impose any restrictions on the specific value of the difference threshold; it can be determined by a clustering algorithm or manually.

[0150] Optionally, grouping inference requests can be implemented in various ways, which are illustrated below by way b1 and way b2.

[0151] In method b1, the management node uses a clustering algorithm to cluster the load of multiple inference requests, resulting in multiple groups.

[0152] For example, after the management node obtains the load of each inference request among multiple inference requests, it clusters the loads of the multiple inference requests using a clustering algorithm to obtain multiple groups.

[0153] For example, clustering algorithms may include K-means clustering, hierarchical clustering, Gaussian mixture models, etc.

[0154] It should be noted that this application does not limit the type of clustering algorithm; the above is merely an illustrative example. The following description uses K-means clustering as an example to illustrate this application.

[0155] For example, when clustering the load of multiple inference requests using the K-means clustering algorithm, clustering coefficients can be determined using the silhouette coefficient, elbow method, dispersion score, and cophenetic correlation coefficient. The clustering coefficient indicates the number of groups obtained when grouping multiple inference requests. For example, a clustering coefficient of 2 means that multiple groups are represented by two groups.

[0156] It should be noted that this application does not impose any restrictions on the method of determining clustering coefficients; the above is merely an illustrative example.

[0157] For example, the loads of multiple inference requests include load 1 of inference request 1, ..., load 6 of inference request 6, where load 1, ..., load 6 are (4000, 1000), (3750, 1000), (4000, 750), (2000, 4000), (2300, 3800), and (2500, 4000) respectively. As shown in Figure 6, in the two-dimensional coordinate system, the horizontal axis indicates the input load, the vertical axis indicates the output load, and "1" on the two-dimensional coordinate system indicates load 1, ..., and "6" indicates load 6. After the management node clusters load 1, ..., load 6 using a clustering algorithm, it obtains group 1 and group 2, that is, the clustering coefficient is 2. Group 1 includes inference request 1, inference request 2, and inference request 3, and group 2 includes inference request 4, inference request 5, and inference request 6. The cluster centers of group 1 are (4000, 1000) and the cluster centers of group 2 are (2300, 3800).

[0158] It should be noted that this application does not impose any restrictions on the cluster centers of Group 1 and Group 2; the above is merely an illustrative example.

[0159] For example, in group 1, the difference in load between different inference requests is less than or equal to a difference threshold. In group 2, the difference in load between different inference requests is less than or equal to a difference threshold.

[0160] In this approach, multiple inference requests are clustered using a clustering algorithm to obtain multiple groups, which helps improve grouping efficiency and accuracy.

[0161] In method b2, the management node groups the load of multiple inference requests based on the difference threshold, resulting in multiple groups.

[0162] For example, after the management node obtains the load of each inference request among multiple inference requests, it groups inference requests whose load differences are less than or equal to a difference threshold into the same group, thus obtaining multiple groups. For instance, if the difference between the input load of inference request 2 and the input load of inference request 1 is less than or equal to the difference threshold, and the difference between the output load of inference request 2 and the output load of inference request 1 is also less than or equal to the difference threshold, then inference request 2 and inference request 1 are grouped into the same group.

[0163] It should be noted that other related explanations for method b2 can be found in the explanations for method b1 above, and will not be repeated here.

[0164] In this approach, the management node groups multiple inference requests based on a difference threshold, which helps to improve the diversity of grouping methods.

[0165] In the above embodiments, after the management node receives an inference request, it dynamically determines the inference request grouping based on the load of the inference request, thereby helping to group inference requests with similar processing latency into the same group. This helps to avoid excessive differences in processing latency among inference requests in each group, thus helping to improve the performance of the inference service provided by the inference system.

[0166] Optionally, S501 includes: the management node groups the acquired R inference requests into multiple groups. Here, R is a positive integer greater than 1.

[0167] For example, when the management node receives R inference requests, it groups the R inference requests into multiple groups.

[0168] Optionally, R can be any value between 500 and 1500.

[0169] In one example, R is 500. This helps to group fewer inference requests, thus helping to avoid assigning inference requests with significantly different loads to the same group, and consequently, helping to avoid assigning inference requests with significantly different processing latency to the same group.

[0170] In another example, R is 1500. This helps control the frequency of grouping operations, thus helping to avoid consuming excessive computing resources.

[0171] In yet another example, R is 1000. This not only helps control the difference in inference request load within the same group, but also helps control the amount of computing resources consumed.

[0172] It should be noted that this application does not impose any restrictions on the specific value of R; the above is merely an illustrative example.

[0173] In this embodiment, by grouping each of the acquired R inference requests, the number of inference requests that need to be grouped each time can be controlled. This helps to avoid assigning inference requests with significantly different loads to the same group, thereby ensuring that the processing latency of inference requests within the same group is similar. This helps to improve the performance of the inference service provided by the inference system.

[0174] S502: The management node determines the target tensor parallelism for each of the multiple groups.

[0175] For example, the multiple groups include a target group, which can be any one of the multiple groups. The following description uses the target group as an example to illustrate this application.

[0176] Tensor parallelism is used to indicate the number of processors that process inference requests in parallel. For example, the tensor parallelism of a target group is used to indicate the number of processors that process inference requests in the target group in parallel.

[0177] For ease of distinction, the tensor parallelism of the target group will be denoted as target tensor parallelism t, and the target delay of the inference request indication in the target group will be denoted as target delay t. These will not be elaborated further.

[0178] In this application, after the management node receives multiple groups, it determines the target tensor parallelism for each group. For example, as shown in Figure 5, the inference engine layer of the management node determines the target tensor parallelism 1 for group 1, ..., and the target tensor parallelism v for group v. The target group can be any one of group 1, ..., or group v.

[0179] In this application, the inference performance when processing inference requests in each group meets the target condition according to the target tensor parallelism of each group. Furthermore, the latency of processing inference requests in each group according to the target tensor parallelism of each group is less than or equal to the target latency indicated by the inference request in each group.

[0180] For example, the inference performance when processing inference requests in the target group according to the target tensor parallelism t meets the target condition. Furthermore, the latency of processing inference requests in the target group according to the target tensor parallelism t is less than or equal to the target latency t.

[0181] In this embodiment, the inference performance when processing inference requests in the target group according to the target tensor parallelism t meets the target condition.

[0182] In this way, by setting appropriate target conditions, high inference performance can be achieved in processing inference requests within the target group, thereby improving the performance of the inference service provided to users by the inference system. Furthermore, by setting the latency for processing inference requests within the target group according to the target tensor parallelism t to be less than or equal to the target latency t, since the target latency t is a user-specified latency, the processing latency of inference performance within the target group is less than or equal to the user-specified latency when the inference system processes inference requests according to the target parallelism t, thus helping to ensure a good user experience.

[0183] Optionally, inference performance includes various cases, which are illustrated below using cases A through C.

[0184] In scenario A, inference performance includes the size of the scheduling group. The size of the scheduling group indicates the number of inference requests scheduled each time. The size of the scheduling group for each group indicates the number of inference requests scheduled from each group each time. For example, the size of the scheduling group for the target group indicates the number of inference requests scheduled from the target group each time.

[0185] The inference performance corresponding to the target tensor parallelism of each group includes the size of the scheduling group corresponding to the target tensor parallelism of each group. The inference performance corresponding to the target tensor parallelism of each group indicates the inference performance when processing inference requests in each group according to the target tensor parallelism of each group. The size of the scheduling group corresponding to the target tensor parallelism of each group indicates the number of inference requests scheduled from each group each time when processing inference requests in each group according to the target tensor parallelism of each group.

[0186] It should be noted that a scheduling group can also be called a batch, and the size of a scheduling group can also be called the batch size (BS).

[0187] For example, the inference performance corresponding to the target tensor parallelism t is used to indicate the inference performance when processing inference requests in the target group according to the target tensor parallelism t. The inference performance corresponding to the target tensor parallelism t includes the size of the scheduling group corresponding to the target tensor parallelism t. The size of the scheduling group corresponding to the target tensor parallelism t is used to indicate the number of inference requests scheduled from the target group each time when processing inference requests in the target group according to the target tensor parallelism t. For example, if the target tensor parallelism t is 8 and the size of the scheduling group corresponding to the target tensor parallelism t is 4, then the management node schedules 4 inference requests from the target group each time and splits each of the 4 inference requests into 8 sub-blocks. Then, the management node deploys the 8 sub-blocks of each inference request on 8 processors of the inference system, so that the 8 sub-tensors on the 8 processors process the 4 inference requests in parallel, and the 8 sub-tensors on the 8 processors process the 8 sub-blocks of each inference request in parallel. That is, the sub-tensors on each processor process 1 sub-block of each of the 4 inference requests in parallel.

[0188] In this scenario, setting inference performance includes the size of the scheduling group. By setting appropriate target conditions, the inference system can process a larger number of inference requests in parallel when processing inference requests in the target group according to the target tensor parallelism t. This helps improve the processing efficiency of inference requests in the target group when there are many requests, thereby enhancing the user experience.

[0189] In scenario B, inference performance includes throughput. Throughput indicates the number of tokens obtained per unit of time. The throughput for each group indicates the number of tokens obtained per unit of time when the inference system processes inference requests within each group. For example, the throughput for the target group indicates the number of tokens obtained per unit of time when the inference system processes inference requests within the target group.

[0190] The inference performance corresponding to the target tensor parallelism of each group includes the throughput corresponding to the target tensor parallelism of each group. The throughput corresponding to the target tensor parallelism of each group is used to indicate the number of tokens obtained by the inference system per unit time when processing inference requests in each group according to the target tensor parallelism of each group.

[0191] For example, the inference performance corresponding to the target tensor parallelism t includes the throughput corresponding to the target tensor parallelism t. The throughput corresponding to the target tensor parallelism t is used to indicate the number of tokens obtained by the inference system per unit time when processing inference requests in the target group according to the target tensor parallelism t. For example, if the target tensor parallelism t is 8, the size of the scheduling group corresponding to the target tensor parallelism t is 4, and the sub-tensor parallelism on each processor of the inference system processes one sub-block of each of the four inference requests, the inference system obtains 500 tokens per unit time, then the throughput of the inference system is 500.

[0192] In this case, inference performance is set to include throughput, thereby enabling the inference system to achieve high throughput when processing inference requests in the target group according to the target tensor parallelism t by setting appropriate target conditions, which helps to improve the performance of the inference service provided by the inference system to users.

[0193] It should be noted that other relevant explanations for situation B can be found in the explanations for situation A above, and will not be repeated here.

[0194] In case C, inference performance includes throughput and the size of the scheduling group.

[0195] It should be noted that the relevant explanations for situation C can be found in the explanations for situations A and B above, and will not be repeated here.

[0196] Optionally, S502 includes multiple implementations, which are illustrated below by way of method 1 and way of method 2.

[0197] In Method 1, the management node determines the target tensor parallelism t based on the size of the scheduling group corresponding to different tensor parallelisms.

[0198] The following example, using the target group as an example, illustrates Method 1 through S502a and S502b.

[0199] S502a: The management node determines the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism based on the target delay t.

[0200] The size of the scheduling group corresponding to each tensor parallelism degree indicates the number of inference requests scheduled each time when processing inference requests in the target group according to each tensor parallelism degree. For example, the size of the scheduling group corresponding to tensor parallelism degree 1 indicates the number of inference requests scheduled from the target group each time when processing inference requests in the target group according to tensor parallelism degree 1. For instance, if the size of the scheduling group corresponding to tensor parallelism degree 1 is 4, then 4 inference requests are scheduled from the target group each time when processing inference requests in the target group according to tensor parallelism degree 1.

[0201] The following provides an example of at least one tensor parallelism.

[0202] Example 1: After obtaining the target group, the management node can determine the first tensor parallelism (hereinafter referred to as the first tensor parallelism) among at least one tensor parallelism based on the number of processors included in the inference system, and determine the inference performance corresponding to the first tensor parallelism. For example, referring to the inference system described in Figure 1, if J is 10, meaning the inference system includes 10 processors, the management node can determine the first tensor parallelism from 2 to 10, that is, 1 < first tensor parallelism ≤ 10 (i.e., the number of processors). For example, the management node can determine the first tensor parallelism using a binary search method. For instance, the management node can determine the first tensor parallelism as 5.

[0203] In one example, after determining the inference performance corresponding to the first tensor parallelism, the management node can determine whether the inference performance corresponding to the first tensor parallelism meets the target condition. If the inference performance corresponding to the first tensor parallelism meets the target condition, the first tensor parallelism is determined to be the target tensor parallelism t.

[0204] In another example, after determining the inference performance corresponding to the first tensor parallelism, the management node determines a second tensor parallelism (hereinafter referred to as the second tensor parallelism) among at least one tensor parallelism based on the number of processors included in the inference system and the first tensor parallelism, and determines the inference performance corresponding to the second tensor parallelism. Based on the same principle, after obtaining the inference performance of multiple tensor parallelisms, the management node determines the target tensor parallelism t from the multiple tensor parallelisms.

[0205] For example, the parallelism of the second tensor can be increased by a first value or decreased by a first value based on the parallelism of the first tensor. For instance, if the first value is 1, then the parallelism of the second tensor can be 6, or the parallelism of the second tensor can be 4.

[0206] It should be noted that this application does not limit the size of the first value; the above is merely an illustrative example.

[0207] Example 2: The inference system stores at least one tensor parallelism. After the management node obtains the target group, it determines the inference performance corresponding to each of the at least one tensor parallelism stored in the inference system.

[0208] In this application, the inference request in the target group includes a user-specified latency (i.e., target latency t), which is the latency specified by the user for processing the inference request. For example, different inference requests in the target group indicate the same target latency t. After obtaining the target latency t, the management node determines the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism based on the target latency t.

[0209] For example, the target latency indicated by the inference request in different groups may be the same or different, and this application does not limit this.

[0210] The following examples 1a to 1c illustrate various implementations of S502a.

[0211] In Example 1a, the management node can determine the size of the scheduling group corresponding to each tensor parallelism based on the second neural network model. For example, the management node inputs the target delay t and tensor parallelism 1 into the second neural network model to obtain the size of the scheduling group corresponding to tensor parallelism 1.

[0212] For example, the second neural network model is a pre-trained neural network model used to determine the size of the scheduling group supported by the inference system based on the target latency and tensor parallelism.

[0213] In this example, the size of the scheduling group corresponding to each tensor parallelism is determined by a neural network model, which helps to improve the diversity of determination methods and determination efficiency.

[0214] Example 1b, the size of the scheduling group corresponding to each tensor parallelism satisfies the following relationship (1):

[0215] Among them, BS1 i Used to characterize the size of the scheduling group corresponding to the i-th tensor parallelism in at least one tensor parallelism. TL (Target Latency) is used to characterize the target latency indicated by the inference request in the target group. NL iNetLatency is used to characterize the network communication latency corresponding to the i-th tensor parallelism in at least one tensor parallelism. LayerN (LN) is used to characterize the number of hidden layers in the inference model of the inference system. NT is used to characterize the number of communications between sub-tensors in a single hidden layer of the inference model; for example, NT can be 2. For example, the communication pattern between sub-tensors in a single hidden layer can be an AllReduce operation. Matrix Multiplication Time (MMT) is used to characterize the total latency required to compute the matrix layer of each hidden layer of the inference model. The computation of the matrix layer in each hidden layer can also be called matrix multiplication computation latency. PAL i This is used to characterize the page attention latency corresponding to the i-th tensor parallelism in at least one tensor parallelism. The page attention latency corresponding to the i-th tensor parallelism is used to indicate the latency of performing page attention on each scheduling group of the target group when processing each scheduling group of the target group according to the i-th tensor parallelism. For example, PAL i It can be a*BS1 i +b. OT(Others) is used to characterize the loss latency of the inference model when processing inference requests in the target group. "Loss latency of the inference model when processing inference requests in the target group" can also be called the other loss latency of a single inference operation in the inference system. K i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

[0216] It should be noted that "the size of the scheduling group corresponding to the parallelism t of the target tensor" can also be called "the batch size under the calibration delay (target delay t)".

[0217] For example, the management node can obtain the above relationship (1) based on cost modeling. Cost modeling is used to indicate that the target delay t satisfies the following relationship (7):

[0218] For example, NL i The unit is milliseconds (ms), the unit of MMT is milliseconds (ms), and the unit of PAL is milliseconds (ms). i The unit for is milliseconds (ms), and the unit for OT is milliseconds (ms).

[0219] For example, the management node can determine the size of the scheduling group corresponding to each tensor parallelism based on the above relationship (1). For example, TL is 50 milliseconds (ms), the network communication latency corresponding to NL1 is 0.02 milliseconds, LN is 96, NT is 2, K1 is 8, MMT is 40 milliseconds, OT is 2 milliseconds, PAL1 is 2*BS11, that is, "a*BS1 i If a is 2 and b is 0 in "+b", then BS11 equals 4.42. After taking the integer part, BS11 is 4.

[0220] In this example, the size of the scheduling group corresponding to the parallelism t of the target tensor is determined by the above relationship (1). Since the relevant latency involved in processing the inference request is taken into account, it helps to improve the accuracy of the determination result.

[0221] Below, regarding NL in the above relationship... i MMT, PAL i The methods for obtaining parameters are illustrated below.

[0222] In one example, the inference system stores NL i LN, NT, MMT, OT, PAL i Parameter values, etc. When the inference system executes the scheduling method provided in this application for the first time, it can use the parameter values ​​stored in the inference system to determine the size of the scheduling group corresponding to each tensor parallelism.

[0223] In another example, after the inference system processes the inference requests in the target group according to the target tensor parallelism t, it can obtain the NL corresponding to the target tensor parallelism t. i MMT, OT, PAL i The parameter values ​​are as follows. In the next round of executing the scheduling method provided in this application, the inference system can determine the size of the scheduling group based on the parameter values ​​obtained during the actual processing of inference requests.

[0224] Example 1c, the size of the scheduling group corresponding to each tensor parallelism satisfies the following relationship (4):

[0225] It should be noted that the relevant explanations for Example 1c can be found in the explanations of Examples 1a and 1b above, and will not be repeated here.

[0226] In this example, the size of the scheduling group corresponding to the parallelism t of the target tensor is determined by the above relationship (4). Since it takes into account the important latency involved in processing the inference request, it not only helps to improve the accuracy of the determination result, but also helps to ensure the efficiency of obtaining the determination result.

[0227] In this approach, by determining the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism, the target tensor parallelism t can be determined from at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism. This helps to ensure that the inference performance corresponding to the target tensor parallelism t can meet the target conditions.

[0228] S502b: The management node determines the target tensor parallelism t from at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism.

[0229] In this application, S502b includes multiple implementation methods, which are exemplarily described below through method 1A and method 1B.

[0230] In mode 1A, the management node determines the target tensor parallelism t based on whether the size of the scheduling group corresponding to each tensor parallelism satisfies the first condition.

[0231] For example, after obtaining the size of the scheduling group corresponding to each tensor parallelism, the management node determines the target tensor parallelism t from at least one tensor parallelism based on whether the size of the scheduling group corresponding to each tensor parallelism satisfies a first condition. The size of the scheduling group corresponding to the target tensor parallelism t satisfies the first condition. If the size of the scheduling group corresponding to the target tensor parallelism t satisfies the first condition, the inference performance when processing inference requests in the target group according to the target tensor parallelism t satisfies the target condition.

[0232] For ease of description, the "inference performance when processing inference requests in the target group according to the target tensor parallelism t" will be referred to as the inference performance corresponding to the target tensor parallelism t.

[0233] The following examples illustrate how the size of the scheduling group corresponding to the parallelism t of the target tensor satisfies the first condition, through cases D to F.

[0234] Case D: The size of the scheduling group corresponding to the target tensor parallelism t is the largest among the sizes of at least one scheduling group corresponding to tensor parallelism.

[0235] Among them, when the inference request includes the size of the scheduling group, if the size of the scheduling group corresponding to the target tensor parallelism t is the largest, then the inference performance corresponding to the target tensor parallelism t is the largest among at least one tensor parallelism.

[0236] For example, the size of the scheduling group corresponding to tensor parallelism 1 is 4, the size of the scheduling group corresponding to tensor parallelism 2 is 3, and the size of the scheduling group corresponding to tensor parallelism 3 is 2. Since the size of the scheduling group corresponding to tensor parallelism 1 is the largest, tensor parallelism 1 is determined as the target tensor parallelism t.

[0237] In this scenario, selecting the maximum tensor parallelism of the corresponding scheduling group as the target tensor parallelism t helps maximize the scheduling group when the inference system processes inference requests in the target group according to the target tensor parallelism t. This helps improve the processor efficiency of inference requests in the target group, thereby maximizing the inference performance of the inference system when providing inference services to users.

[0238] In case E, the size of the scheduling group corresponding to the target tensor parallelism t is less than or equal to the sorting threshold in the order of the sizes of at least one scheduling group corresponding to tensor parallelism.

[0239] Wherein, if the inference request includes the size of the scheduling group, and the order of the scheduling group size corresponding to the target tensor parallelism t is less than or equal to the order threshold, then the order of the inference performance corresponding to the target tensor parallelism t is less than or equal to the order threshold in the inference performance corresponding to at least one tensor parallelism.

[0240] It should be noted that this application does not impose any restrictions on the specific value of the sorting threshold, which can be dynamically set according to the actual scenario.

[0241] For example, the sizes of the scheduling groups corresponding to tensor parallelism 1, tensor parallelism 2, and tensor parallelism 3, in descending order, are: tensor parallelism 1, tensor parallelism 2, and tensor parallelism 3. When the sorting threshold is 1, the order of tensor parallelism 1 equals the sorting threshold, so tensor parallelism 1 is the target tensor parallelism t. When the sorting threshold is 2, the order of tensor parallelism 1 is less than the sorting threshold, and the order of tensor parallelism 2 equals the sorting threshold, so either tensor parallelism 1 or tensor parallelism 2 is the target tensor parallelism t.

[0242] In this case, the tensor parallelism t is selected as the target tensor parallelism t by choosing the tensor parallelism of the corresponding scheduling group whose sorting is less than or equal to the sorting threshold.

[0243] This not only helps to select the tensor parallelism with the larger scheduling group as the target tensor parallelism t from at least one tensor parallelism, but also helps to improve the selection range and diversity of the selection method for the target tensor parallelism t.

[0244] It should be noted that when the sizes of the scheduling groups corresponding to at least one tensor parallelism are sorted in ascending order, the size of the scheduling group corresponding to the target tensor parallelism t is greater than or equal to the first sorting threshold. For a detailed explanation of the first sorting threshold, please refer to the explanation of case E above; it will not be repeated here.

[0245] It should be noted that other relevant explanations for situation E can be found in the explanation of situation D above, and will not be repeated here.

[0246] Case F: The size of the scheduling group corresponding to the parallelism t of the target tensor is greater than or equal to the first threshold.

[0247] Specifically, if the inference request includes the size of the scheduling group, and the size of the scheduling group corresponding to the target tensor parallelism t is greater than or equal to the first threshold, then the inference performance corresponding to the target tensor parallelism t is greater than or equal to the first threshold.

[0248] For example, the size of the scheduling group corresponding to the target tensor parallelism t is greater than or equal to a first threshold. For instance, if the first threshold is 4, since the size of the scheduling group corresponding to tensor parallelism 1 is equal to 4, tensor parallelism 1 is determined as the target tensor parallelism t. As another example, if the first threshold is 3, since the size of the scheduling group corresponding to tensor parallelism 1 is greater than 3 and the size of the scheduling group corresponding to tensor parallelism 2 is equal to 3, either tensor parallelism 1 or tensor parallelism 2 is determined as the target tensor parallelism t.

[0249] It should be noted that this application does not impose a limit on the size of the first threshold, which can be dynamically set according to the actual situation of the inference system.

[0250] It should be noted that other relevant explanations for situation F can be found in the explanations for situations D and E above, and will not be repeated here.

[0251] In this case, by selecting a tensor parallelism degree whose size of the corresponding scheduling group is greater than or equal to the performance threshold as the target tensor parallelism degree t, it is not only helpful to select the tensor parallelism degree with the larger scheduling group from at least one tensor parallelism degree as the target tensor parallelism degree t, but also to improve the selection range and selection method diversity of the target tensor parallelism degree t.

[0252] In this application, the size of the scheduling group corresponding to the parallelism t of the target tensor satisfies the first condition, including satisfying at least one of the above conditions D, E, or F. That is, conditions D to F can be used in combination or individually, and this application does not impose any restrictions on this.

[0253] In the above approach, by setting the target tensor parallelism t based on the size of the scheduling group corresponding to each tensor parallelism, it helps to ensure that the inference performance can meet the target conditions when processing inference requests in the target group according to the determined target tensor parallelism t.

[0254] In Method 1B, the management node can determine the target tensor parallelism t based on the size of the scheduling group corresponding to each tensor parallelism and the threshold of the scheduling group corresponding to each tensor parallelism.

[0255] Hereinafter, S502b1 and S502b2 will be used to illustrate mode 1B.

[0256] S502b1: The management node determines the scheduling threshold corresponding to each tensor parallelism in at least one tensor parallelism.

[0257] The scheduling threshold corresponding to each tensor parallelism is used to indicate the number of inference requests that the processor indicated by each tensor parallelism can handle when inference requests are made in the target group at each tensor parallelism.

[0258] For example, the management node determines the scheduling threshold corresponding to each tensor parallelism based on the storage space size of the processor in the inference system. Here, the processor's storage space size indicates the storage space size of the processor's memory. For instance, referring to Figure 1, the storage space size of processor 1 indicates the storage space size of memory 11 (i.e., the memory of processor 1).

[0259] The following examples, 1d and 1e, illustrate various implementations of S502b1.

[0260] In Example 1d, the management node can determine the scheduling threshold corresponding to each tensor parallelism based on the third neural network model. For example, the management node inputs the processor's storage space size, the load of a single inference request, and tensor parallelism 1 into the third neural network model to obtain the scheduling threshold corresponding to tensor parallelism 1. Here, the third neural network model is a pre-trained neural network model used to determine the scheduling threshold corresponding to the tensor parallelism based on the processor's storage space size, the load of a single inference request, and the tensor parallelism.

[0261] Example 1e, the scheduling threshold corresponding to each tensor parallelism satisfies the following relationship (3):

[0262] Among them, BS2 i K is a scheduling threshold used to characterize the i-th tensor parallelism in at least one tensor parallelism. i Used to characterize the i-th tensor parallelism in at least one tensor parallelism, d is used to characterize the storage space size of a single processor, MS is used to characterize the second storage space size required by the inference model, and D is used to characterize the storage space size required by a single inference request in the target group.

[0263] It should be noted that "the storage space required for a single inference request" can also be called "the KV Cache memory space required for a single inference request". "The scheduling threshold corresponding to the target tensor parallelism t" can also be called "the batch size that can be accommodated under the target tensor parallelism t".

[0264] For example, MS is in gigabytes (GB), d is in GB, and D is in GB.

[0265] For example, the management node can determine the scheduling threshold corresponding to each tensor parallelism based on the above relationship (3). For example, if K1 is 8, d is 55GB, MS is 350GB, D is 21.9GB, then BS21 is 4.

[0266] Optionally, the management node determines the storage space required for a single inference request in the target group based on the load of the individual inference request in the target group. For example, the load of a single inference request in the target group may be the load of the inference request indicated by the cluster center of the target group, or it may be the average load of the individual inference requests in the target group, or it may be the maximum load of the individual inference requests in the target group, or it may be the minimum load of the individual inference requests in the target group.

[0267] In one example, the management node can determine the storage space required for a single inference request within a target group based on a fourth neural network model. For instance, the management node can input the load of a single inference request within the target group into the fourth neural network model to obtain the storage space required for that single inference request. Here, the fourth neural network model is a pre-trained neural network model used to determine the required storage space for each inference request based on its load.

[0268] In another example, the storage space required for inference requests in the target group satisfies the following relationship (6): D=(LIn+LOut)*HS*LN*U*CA (6)

[0269] Where D represents the size of the storage space required for a single inference request in the target group, LIn represents the input load of a single inference request in the target group, LOut represents the output load of a single inference request in the target group, HS (Hidden Size) represents the dimension of the hidden layer of the inference model, LN (LayerNum) represents the number of hidden layers of the inference model, CA (Calculation Accuracy) represents the computational accuracy of the model, and U represents the amount of key-value (KV) data stored in each hidden layer of the inference model. For example, U is 2.

[0270] For example, the management node can determine the storage space required for a single inference request in the target group based on the relationship (6) above. For instance, if LIn is 4000, LOut is 1000, HS is 12888, LN is 96, U is 2, and CA is 2, then the memory space required for a single inference request is 21.9GB.

[0271] S502b2: The management node determines the target tensor parallelism t from at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism and the scheduling threshold corresponding to each tensor parallelism.

[0272] For example, after the management node obtains the size of the scheduling group corresponding to each tensor parallelism, it determines the target tensor parallelism t from at least one tensor parallelism based on whether the size of the scheduling group corresponding to each tensor parallelism satisfies a first condition and whether the size of the scheduling group corresponding to each tensor parallelism is less than or equal to the scheduling threshold corresponding to each tensor parallelism. The size of the scheduling group corresponding to the target tensor parallelism t satisfies the first condition, and the size of the scheduling group corresponding to the target tensor parallelism t is less than or equal to the scheduling threshold corresponding to the target tensor parallelism t.

[0273] For example, if the scheduling threshold corresponding to the target tensor parallelism t is 4, and the size of the scheduling group corresponding to the target tensor parallelism t is 4, then the size of the scheduling group corresponding to the target tensor parallelism t is equal to the scheduling threshold corresponding to the target tensor parallelism t. It should be noted that other related explanations for Method 1B can be found in the explanations for Method 1A above, and will not be repeated here.

[0274] In this approach, the inference system obtains two parameters for each group: the optimal tensor parallelism (e.g., the target tensor parallelism t) at the calibrated latency and the size of the scheduling group (i.e., the batch size). Then, the inference system schedules inference requests within each group based on the optimal tensor parallelism and batch size. For example, if the target group has an optimal tensor parallelism of 8 and a batch size of 4, the management node schedules 4 inference requests from the target group each time, and divides each of these 4 inference requests into 8 sub-blocks. Then, each of the 8 sub-blocks of an inference request is scheduled to be processed on 8 processors.

[0275] In this approach, by setting a scheduling threshold corresponding to each tensor parallelism when determining the target tensor parallelism t from at least one tensor parallelism, it is possible to ensure that the processor indicated by the target tensor parallelism t can carry the scheduling group corresponding to the target tensor parallelism t, thereby helping to ensure the reliability and accuracy of the inference service provided by the inference performance.

[0276] Optionally, in conjunction with method 1B above, when processing inference requests in each group according to the target tensor parallelism of each group, the size of the scheduling group for each group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group. The scheduling threshold corresponding to the target tensor parallelism of each group is used to indicate the number of inference requests that the processor can support processing, as indicated by the target tensor parallelism of each group.

[0277] For example, when processing inference requests in a target group according to the target tensor parallelism t, the size of the scheduling group of the target group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism t. For instance, if the scheduling threshold corresponding to the target tensor parallelism t is 4, then when processing inference requests in a target group according to the target tensor parallelism t, the number of inference requests scheduled from the target group each time is less than or equal to 4.

[0278] In this embodiment, by setting the size of the scheduling group for each group to be less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group, it helps to ensure that when processing the inference requests in each group according to the target tensor parallelism of each group, the processor indicated by the target tensor parallelism of each group can carry the inference requests in the scheduling group of each group. This helps to ensure the performance stability and reliability of the inference system and avoids data loss due to the inability to carry the data obtained during the inference process, which would affect the accuracy of the inference results.

[0279] In the above approach, when inference performance includes the size of the scheduling group, determining the final target tensor parallelism *t* by using the size of the scheduling group corresponding to different tensor parallelisms ensures that the size of the scheduling group corresponding to the determined target tensor parallelism *t* meets the target condition, thus helping to guarantee the accuracy of the determined target tensor parallelism *t*. When inference performance includes throughput, since the size of the scheduling group is generally proportional to the throughput, determining the final target tensor parallelism *t* by using the size of the scheduling group corresponding to different tensor parallelisms not only simplifies the process of determining the target tensor parallelism *t*, thereby improving the efficiency of determining the target tensor parallelism *t*, but also helps to ensure that the inference performance corresponding to the target tensor parallelism *t* meets the target condition.

[0280] In Method 2, the management node determines the target tensor parallelism t based on the throughput corresponding to different tensor parallelisms.

[0281] The following example, using the target group as an example, illustrates Method 2 through S502c and S502e.

[0282] S502c: The management node determines the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism based on the target delay t.

[0283] It should be noted that the relevant explanations for S502c can be found in the explanations for S502a above, and will not be repeated here.

[0284] S502d: The management node determines the throughput corresponding to each tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism.

[0285] The throughput corresponding to each tensor parallelism is used to indicate the throughput when processing inference requests in the target group according to each tensor parallelism.

[0286] For example, after obtaining the size of the scheduling group corresponding to each tensor parallelism, the management node can determine the throughput corresponding to each tensor parallelism by combining the output load of a single inference request in the target group. For instance, the target group includes E inference requests, where the output load of the E inference requests includes the output load 1 of inference request 1, ..., the output load E of inference request E. The output load of a single inference request can be any one of the largest output load among the E output loads, the smallest output load among the E output loads, or the average of the E output loads.

[0287] The following examples, 2a to 2c, illustrate various implementations of S502d.

[0288] In Example 2a, the management node can determine the throughput corresponding to each tensor parallelism based on the fifth neural network model. For example, the management node inputs the output load of a single inference request in the target group and the size of the scheduling group corresponding to tensor parallelism 1 into the fifth neural network model to obtain the throughput corresponding to tensor parallelism 1.

[0289] For example, the fifth neural network model is a pre-trained neural network model used to determine the throughput of the inference system based on the output load and tensor parallelism of a single inference request.

[0290] In this example, the throughput corresponding to each tensor parallelism is determined by a neural network model, which helps to improve the diversity of determination methods and determination efficiency.

[0291] Example 2b, the throughput corresponding to each tensor parallelism satisfies the following relationship (2):

[0292] Among them, T i LOut is used to characterize the throughput corresponding to the i-th tensor parallelism in at least one tensor parallelism, and LOut is used to characterize the output load of a single inference request in the target group. BS1 i K is used to characterize the size of the scheduling group corresponding to the i-th tensor parallelism in at least one tensor parallelism. iUsed to characterize the i-th tensor parallelism in at least one tensor parallelism.

[0293] For example, the management node can determine the throughput corresponding to each tensor parallelism based on the above relationship (2). For example, if BS11 is 4, LOut is 1000, K1 is 8, then T1 is 500 tokens.

[0294] In this example, the throughput corresponding to each tensor parallelism is determined through the aforementioned relationship (2). Since throughput-related parameters are incorporated, this helps improve the accuracy of the determination. Furthermore, the throughput corresponding to each tensor parallelism indicated by formula (2) is the throughput of a single processor, ensuring that the determined target parallelism t guarantees that the throughput of a single processor processing inference requests within the target group meets the target condition. Thus, by setting appropriate target conditions, it is possible to ensure that each processor of the inference system achieves high inference performance, thereby enabling the inference system to achieve high inference performance when processing inference requests within the target group, and ultimately improving the performance of the inference service provided by the inference system.

[0295] Example 2c, the throughput corresponding to each tensor parallelism satisfies the following relationship (5): T i =LOut*BS1 i (5)

[0296] Among them, T i LOut is used to characterize the throughput corresponding to the i-th tensor parallelism in at least one tensor parallelism, and LOut is used to characterize the output load of each inference request in the target group. BS1 i Used to characterize the size of the scheduling group corresponding to the i-th tensor parallelism in at least one tensor parallelism.

[0297] For example, the management node can determine the throughput corresponding to each tensor parallelism based on the above relationship (5). For example, if BS11 is 4 and LOut is 1000, then T1 is 4000 tokens.

[0298] In this example, the throughput corresponding to each tensor parallelism is determined through the aforementioned relation (5). Since throughput-related parameters are incorporated, the accuracy of the determination is improved. Furthermore, the throughput corresponding to each tensor parallelism indicated by relation (5) is the throughput of a set of processors, thus ensuring that the determined target parallelism t guarantees that the throughput of the set of processors processing inference requests in the target group meets the target condition. Here, a set of processors refers to all processors used for each tensor parallelism; for example, if the tensor parallelism is 4, then a set of processors consists of 4 processors. In this way, by setting appropriate target conditions, it is possible to ensure that a set of processors in the inference system achieves high inference performance, thereby helping to improve the performance of the inference service provided to users by the inference system.

[0299] S502e: The management node determines the target tensor parallelism t from at least one tensor parallelism based on the throughput corresponding to each tensor parallelism.

[0300] In this application, S502e includes multiple implementation methods, which are exemplarily described below through method 2A and method 2B.

[0301] In mode 2A, the management node can determine the target tensor parallelism t based on whether the throughput corresponding to each tensor parallelism meets the second condition.

[0302] For example, after the management node obtains the throughput corresponding to each tensor parallelism, it determines the target tensor parallelism t of the target group from at least one tensor parallelism based on whether the throughput corresponding to each tensor parallelism satisfies the second condition. The throughput corresponding to the target tensor parallelism t satisfies the second condition. If the throughput corresponding to the target tensor parallelism t satisfies the first condition, the inference performance when processing inference requests in the target group according to the target tensor parallelism t satisfies the target condition.

[0303] The following sections will introduce how the throughput corresponding to the target tensor parallelism t satisfies the first condition, through cases G to I.

[0304] In case G, the throughput corresponding to the target tensor parallelism t is the largest among at least one tensor parallelism.

[0305] For example, the throughput corresponding to the target tensor parallelism t is the largest among at least one tensor parallelism. For instance, the throughput corresponding to tensor parallelism 1 is 500, the throughput corresponding to tensor parallelism 2 is 400, and the throughput corresponding to tensor parallelism 3 is 450. Since the throughput corresponding to tensor parallelism 1 is the largest, tensor parallelism 1 is determined as the target tensor parallelism t.

[0306] In this scenario, selecting the tensor parallelism with the highest throughput as the target tensor parallelism t helps the inference system maximize its throughput when processing inference requests in the target group according to the target tensor parallelism t. This helps improve the processor efficiency of inference requests in the target group, thereby maximizing the performance of the inference system in providing inference services to users.

[0307] In case H, the throughput corresponding to the target tensor parallelism t is ranked less than or equal to the ranking threshold among the throughputs corresponding to at least one tensor parallelism.

[0308] For example, the throughput corresponding to tensor parallelism 1, tensor parallelism 2, and tensor parallelism 3, in descending order, are: tensor parallelism 1, tensor parallelism 3, tensor parallelism 2. With a sorting threshold of 1, the sorting of tensor parallelism 1 equals the sorting threshold, so tensor parallelism 1 is the target tensor parallelism t.

[0309] It should be noted that when the throughput corresponding to at least one tensor parallelism is sorted in ascending order, the throughput corresponding to the target tensor parallelism t is greater than or equal to the first sorting threshold.

[0310] In this case, by selecting a tensor parallelism with a throughput less than or equal to the sorting threshold as the target tensor parallelism t, it is not only helpful to select a tensor parallelism with a larger throughput from at least one tensor parallelism as the target tensor parallelism t, but also to improve the selection range and diversity of selection methods for the target tensor parallelism t.

[0311] It should be noted that other relevant explanations for situation H can be found in the explanation of situation G above, and will not be repeated here.

[0312] Case I: The throughput corresponding to the target tensor parallelism t is greater than or equal to the second threshold.

[0313] For example, the throughput corresponding to the target tensor parallelism t is greater than or equal to the second threshold. For instance, if the second threshold is 500, and the throughput corresponding to tensor parallelism 1 is equal to 500, then tensor parallelism 1 is determined to be the target tensor parallelism t.

[0314] It should be noted that this application does not impose any restrictions on the size of the second threshold, which can be dynamically set according to the actual situation of the inference system.

[0315] It should be noted that other relevant explanations for situation I can be found in the explanations for situations H and G above, and will not be repeated here.

[0316] In this case, by selecting a tensor parallelism with a throughput greater than or equal to the performance threshold as the target tensor parallelism t, it is not only helpful to select a tensor parallelism with a larger throughput from at least one tensor parallelism as the target tensor parallelism t, but also to improve the selection range and diversity of selection methods for the target tensor parallelism t.

[0317] In this application, the throughput corresponding to the target tensor parallelism t satisfies the second condition, including satisfying at least one of the above conditions G, H, or I. That is, conditions G to I can be used in combination or individually, and this application does not impose any restrictions on this.

[0318] It should be noted that other relevant explanations for situations G to I can be found in the explanations for situations D to F above, and will not be repeated here.

[0319] In the above approach, by setting the target tensor parallelism t based on the throughput corresponding to each tensor parallelism, it helps to ensure that the inference performance can meet the target conditions when processing inference requests in the target group according to the determined target tensor parallelism t.

[0320] In Method 2B, the management node can determine the target tensor parallelism t based on the throughput corresponding to each tensor parallelism and the scheduling group threshold corresponding to each tensor parallelism.

[0321] Hereinafter, S502e1 and S502e2 will be used to illustrate method 2B.

[0322] S502e1: The management node determines a scheduling threshold corresponding to each tensor parallelism in at least one tensor parallelism; the scheduling threshold corresponding to each tensor parallelism is used to indicate the number of inference requests that the processor indicated by each tensor parallelism can handle.

[0323] It should be noted that the relevant explanations for S502e1 can be found in the explanations for S502b1 above, and will not be repeated here.

[0324] S502e2: The management node determines the target tensor parallelism t from at least one tensor parallelism based on the throughput corresponding to each tensor parallelism and the scheduling threshold corresponding to each tensor parallelism.

[0325] For example, after obtaining the throughput corresponding to each tensor parallelism, the management node determines the target tensor parallelism t from at least one tensor parallelism based on whether the throughput corresponding to each tensor parallelism satisfies the second condition and whether the size of the scheduling group corresponding to each tensor parallelism is less than or equal to the scheduling threshold corresponding to each tensor parallelism. Specifically, the throughput corresponding to the target tensor parallelism t satisfies the second condition, and the size of the scheduling group corresponding to the target tensor parallelism t is less than or equal to the scheduling threshold corresponding to the target tensor parallelism t.

[0326] For example, the management node can dynamically adjust the size of the tensor parallelism to obtain the maximum throughput that a single processor can achieve, provided that the size of the scheduling group is less than the scheduling threshold. This enables a single processor to achieve the goal of optimal tensor parallelism, and further enables single-path inference (i.e., inference of a single inference model) to achieve optimal tensor parallelism.

[0327] It should be noted that other relevant explanations for Method 2B can be found in the explanations for Method 2A and Method 1B above, and will not be repeated here.

[0328] In this implementation, when inference performance includes throughput, determining the final target tensor parallelism by the size of the scheduling group corresponding to different tensor parallelisms helps ensure that the throughput corresponding to the determined target tensor parallelism meets the target condition, thereby helping to ensure the accuracy of the determined target tensor parallelism. When inference performance includes the size of the scheduling group, determining the final target tensor parallelism by the throughput corresponding to different tensor parallelisms helps to increase the diversity of methods for determining the target tensor parallelism. Furthermore, by using throughput as a screening parameter for determining the target tensor parallelism t, it helps to ensure that the throughput provided by the inference system to the user meets the target condition, for example, achieving the optimal throughput that the inference system can provide (i.e., the throughput that meets the second condition), which can improve the throughput by at least 25%-50% compared to related technology solutions.

[0329] S503: The management node schedules inference requests in each group based on the target tensor parallelism of each group.

[0330] In this application, after obtaining the target tensor parallelism of each group, the management node schedules the inference requests in each group according to the target tensor parallelism of each group. For example, as shown in Figure 5, the management node's inference engine layer schedules the inference requests in group 1 according to the target tensor parallelism 1, ..., and the management node schedules the inference requests in group v according to the target tensor parallelism v. The management node's inference execution layer processes the inference requests in group 1, ..., and group v.

[0331] For example, as shown in Figure 7, matrix X represents the input data indicating the inference request, which includes prompt words. Matrix A represents the tensor of the inference model, where the number of columns in matrix X equals the number of rows in matrix A. Without a tensor parallel strategy, the inference model processes the inference request as shown in Figure 7(a). For example, this inference process is completed by processor a, yielding the final result matrix Y. With a tensor parallel strategy, the inference model processes the inference request as shown in Figure 7(b). For example, matrix X is divided into two smaller blocks, X1 and X2, and matrix A is divided into two smaller blocks, A1 and A2. Then, one processor processes the calculations of X1 and A1, and another processor processes the calculations of X2 and A2, thus obtaining the final result matrix Y.

[0332] It should be noted that in this application, the node used to execute S501 and S502 and the node used to execute S503 can be the same node or different nodes, and this application does not impose any restrictions on this.

[0333] Optionally, S503 includes the following S503a and S503b.

[0334] S503a: The management node deploys the M subtensors of the inference model on the M processors of the inference system.

[0335] For example, after the management node determines that the parallelism of the target tensor of the target group is M, it divides the tensor of the inference model into M sub-tensors and deploys the M sub-tensors on M processors respectively. Among them, one sub-tensor is deployed on one processor.

[0336] For example, inference requests in the target group can be divided into multiple scheduling groups, including a first scheduling group, which can be any one of the multiple scheduling groups. The following description uses the first scheduling group as an example to illustrate this application.

[0337] S503b: The management node deploys M sub-blocks of each inference request in the first scheduling group onto M processors.

[0338] For example, each scheduling group in the target group is N in size. After the management node determines that the parallelism t of the target tensor is M, it schedules N inference requests from the target group and splits each of the N inference requests into M sub-blocks. Then, the management node deploys the M sub-blocks of each inference request onto M processors, where one sub-block of each inference request is deployed on one processor, thereby enabling the M sub-tensors on the M processors to process the M sub-blocks of each inference request in the first scheduling group in parallel. After the M processors have executed the N inference requests in the first scheduling group, the management node can continue to schedule the next scheduling group of the target group onto the M processors.

[0339] For example, M is 8 and N is 4. After the management node determines that the parallelism t of the target tensor is 8 and the scheduling group size of the target group is 4, it splits the tensor of the inference model into 4 sub-tensors and deploys these 4 sub-tensors on 4 processors, namely processor 1, processor 2, processor 3, and processor 4. The management node schedules 4 inference requests from the target group, namely inference request 1, inference request 2, inference request 3, and inference request 4. Then, the management node splits inference request 1 into 4 sub-blocks, namely sub-block 11, sub-block 12, sub-block 13, and sub-block 14. The management node splits inference request 2 into 4 sub-blocks, namely sub-block 21, sub-block 22, sub-block 23, and sub-block 24. The management node splits inference request 3 into 4 sub-blocks, namely sub-block 31, sub-block 32, sub-block 33, and sub-block 34. The management node divides inference request 1 into 4 sub-blocks, namely sub-block 41, sub-block 42, sub-block 43 and sub-block 44.

[0340] The management node deploys sub-blocks 11, 21, 31, and 41 on processor 1, which processes them in parallel. The management node deploys sub-blocks 12, 22, 32, and 42 on processor 2, which also processes them in parallel. The management node deploys sub-blocks 13, 23, 33, and 43 on processor 3, which also processes them in parallel. The management node deploys sub-blocks 14, 24, 34, and 44 on processor 4, which also processes them in parallel.

[0341] For example, for each of the N inference requests, after each of the M processors has processed the computation of each sub-tensor and each sub-block, each processor can obtain the final computation results of the M sub-blocks and M sub-tensors for each inference request. For example, as shown in Figure 7, each processor can obtain the computation result of X1×A1+X2×A2.

[0342] For example, inference requests in different groups can be scheduled to be processed on the same processor, or they can be scheduled to be processed on different processors, and this application does not limit this.

[0343] Optionally, the scheduling method may further include: obtaining at least one latency corresponding to the target tensor parallelism t. Wherein, the at least one latency corresponding to the target parallelism t includes at least one of the following: network communication latency (NL), total latency (MMT) required to compute the matrix layer of each hidden layer of the inference model, latency of executing page attention on the scheduling group of the target packet (PAL), and loss latency (OT) when the inference model processes inference requests in the target packet.

[0344] For example, when M processors process inference requests in each scheduling group in parallel, they obtain parameter values ​​such as NL, MMT, and PAL. After the M processors have finished processing the inference requests in each scheduling group, the management node can obtain the parameter values ​​such as NL, MMT, and PAL, and use these values ​​as parameters to determine the tensor parallelism of each new group in the next round of grouping. For example, as shown in Figure 5, after the inference execution layer obtains at least one delay, it sends at least one delay to the inference engine layer of the inference system so that the inference engine layer can use it to subsequently determine the tensor parallelism of each group.

[0345] In this embodiment, by obtaining the parameter values ​​obtained from the processing inference, the tensor parallelism of the new group can be determined in combination with the latest obtained parameter values, thereby helping to improve the accuracy of the determined tensor parallelism of the new group.

[0346] In the above scheme, clustering is performed based on the changes in the input and output load of inference requests, and a suitable tensor parallelism strategy is determined for each group under the calibration delay specified by the user. That is, the target tensor parallelism of each group is determined, thereby obtaining the optimal tensor parallelism of each group, so that the inference system can achieve optimal inference performance when processing inference requests in each group, such as maximizing throughput.

[0347] Furthermore, this application establishes Cost Modeling, combining the inference model, the input and output loads of each inference request, and the hardware capabilities of the inference system (e.g., processor storage space size, network communication latency, etc.) to determine the scheduling threshold (i.e., the batch size threshold) and throughput. Under the constraints of Cost Modeling, and in conjunction with the user-specified calibration latency, a tensor parallel scheduling strategy (i.e., the target tensor parallelism for each group) is determined, thereby achieving the goal of optimizing single-path inference performance, for example, maximizing inference performance.

[0348] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the scheduling device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0349] This application embodiment can, based on the above method, exemplarily divide the scheduling device into functional modules. For example, the scheduling device may include functional modules corresponding to each functional division, or two or more functions may be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; in actual implementation, there may be other division methods.

[0350] For example, Figure 8 shows a possible schematic diagram of the scheduling device (i.e., scheduling device 800) involved in the above embodiments. The actions performed by the scheduling device 800 can be executed by the management node of the inference system or implemented by the corresponding software executed by the management node of the inference system. The scheduling device 800 may include: a grouping module 801, a determining module 802, and a scheduling module 803; the grouping module 801 is used to group the acquired multiple inference requests to obtain multiple groups. For example, as shown in S501 of Figure 5. The determining module 802 is used to determine the target tensor parallelism of each group in the multiple groups, wherein, according to the target tensor parallelism of each group, the inference performance when processing the inference request in each group meets the target condition, and the latency of processing the inference request in each group is less than or equal to the target latency indicated by the inference request in each group. For example, as shown in S502 of Figure 5. The scheduling module 803 is used to schedule the inference request in each group according to the target tensor parallelism of each group. For example, as shown in S503 of Figure 5.

[0351] Optionally, the inference performance corresponding to the target tensor parallelism of each group includes at least one of the following: the size of the scheduling group corresponding to the target tensor parallelism of each group or the throughput corresponding to the target tensor parallelism of each group, wherein the size of the scheduling group is used to indicate the number of inference requests scheduled each time.

[0352] Optionally, the size of the scheduling group for each group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group, wherein the scheduling threshold corresponding to the target tensor parallelism of each group is used to indicate the number of inference requests that the processor can support processing, as indicated by the target tensor parallelism of each group.

[0353] Optionally, the multiple groups include a target group, and the determining module 802 is specifically used to: determine the size of the scheduling group corresponding to each tensor parallelism based on the target latency indicated by the inference request in the target group, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism; and determine the target tensor parallelism of the target group from at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism.

[0354] Optionally, the multiple groups include a target group, and the determining module is specifically used to: determine the size of the scheduling group corresponding to each tensor parallelism based on the target latency indicated by the inference request in the target group, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism; determine the throughput corresponding to each tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism; and determine the target tensor parallelism of the target group from at least one tensor parallelism based on the throughput corresponding to each tensor parallelism.

[0355] Optionally, the determining module 802 is specifically configured to: determine a scheduling threshold corresponding to each tensor parallelism in at least one tensor parallelism, wherein the scheduling threshold corresponding to the target tensor parallelism is used to indicate the number of inference requests that the processor indicated by each tensor parallelism supports processing; determine the target tensor parallelism of the target group from the at least one tensor parallelism according to the scheduling threshold corresponding to each tensor parallelism; the size of the scheduling group corresponding to the target tensor parallelism of the target group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of the target group.

[0356] Optionally, the size of the scheduling group corresponding to each tensor parallelism satisfies the following relationship (1):

[0357] Among them, BS1 i The size of the scheduling group corresponding to the i-th tensor parallelism in at least one tensor parallelism is used to characterize the size of the scheduling group; TL is used to characterize the target latency of the inference request indication in the target group; NL is used to characterize the target latency of the inference request indication in the target group. iThe network communication latency corresponding to the i-th tensor parallelism in at least one tensor parallelism is used; LN is used to characterize the number of hidden layers in the inference model of the inference system; NT is used to characterize the number of communications between sub-tensors in a single hidden layer of the inference model; MMT is used to characterize the total latency required to compute the matrix layer of each hidden layer of the inference model; PAL is used to characterize the network communication latency corresponding to the i-th tensor parallelism in at least one tensor parallelism. i The page attention latency corresponding to the i-th tensor parallelism in at least one tensor parallelism is used to characterize the page attention latency corresponding to the i-th tensor parallelism. This page attention latency indicates the latency of performing page attention on the scheduling group of the target group when processing the scheduling group of the target group according to the i-th tensor parallelism. OT is used to characterize the loss latency when the inference system processes inference requests in the target group. K i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

[0358] Optionally, the throughput corresponding to each tensor parallelism satisfies the following relationship (2):

[0359] Among them, T i LOut is used to characterize the throughput corresponding to the i-th tensor parallelism in at least one tensor parallelism. LOut is used to characterize the output load of inference requests in the target group. K i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

[0360] Optionally, the scheduling threshold corresponding to each tensor parallelism satisfies the following relationship (3):

[0361] Among them, BS2 i The scheduling threshold is used to characterize the i-th tensor parallelism in at least one tensor parallelism. d is used to characterize the storage space size of a single processor in the inference system. MS is used to characterize the storage space required by the inference model. D is used to characterize the storage space required by a single inference request in the target group.

[0362] Optionally, the grouping module 801 is specifically used to: obtain the payload of each inference request among multiple inference requests, wherein the payload of each inference request is used to indicate the input payload and the output payload of each inference request; and group the multiple inference requests according to the payload of each inference request to obtain multiple groups. The input payload of each inference request is used to indicate the length of the prompt word indicated by each inference request; and the output payload of each inference request is used to indicate the length of the multiple tokens obtained when processing each inference request.

[0363] Optionally, the grouping module 801 is specifically used to: group multiple inference requests according to the load of each inference request using a clustering algorithm to obtain multiple groups.

[0364] Optionally, the inference model includes an inference model, the target tensor parallelism of the target group is M, the target group includes a first scheduling group, and the scheduling module 803 is specifically used to: deploy the M sub-tensors of the inference model on the M processors of the inference system; deploy the M sub-blocks of each inference request in the first scheduling group on the M processors, so that the M sub-tensors on the M processors can process the M sub-blocks of each inference request in the first scheduling group in parallel.

[0365] Optionally, the size of the scheduling group corresponding to each tensor parallelism satisfies the following relationship (4):

[0366] Optionally, the throughput corresponding to each tensor parallelism satisfies the following relationship (5): T i =LOut*BS1 i (5)

[0367] Optionally, the determining module 802 is specifically used to: determine the size of the storage space required for a single inference request in the target group based on the load of a single inference request in the target group.

[0368] Optionally, the storage space required for a single inference request in the target group satisfies the following relationship (6): D=(LIn+LOut)*HS*LN*U*CA (6)

[0369] Where D represents the size of the storage space required for a single inference request in the target group, LIn represents the input load of a single inference request in the target group, LOut represents the output load of a single inference request in the target group, HS represents the dimension of the hidden layer of the inference model, LN represents the number of hidden layers of the inference model, U represents the amount of key-value (KV) data stored in each hidden layer of the inference model, and CA represents the computational precision of the model.

[0370] Optionally, the grouping module 801 is specifically used to: obtain the input load and output load of each inference request; and obtain the load of each inference request based on the input load and output load of each inference request.

[0371] Optionally, the scheduling device 800 further includes an execution module 804. The execution module 804 is configured to: obtain at least one delay corresponding to the target tensor parallelism of each group; the at least one delay corresponding to the target tensor parallelism of each group includes at least one of the following: network communication delay, delay required for the hidden layer of the inference model to process inference requests, total delay required for the computation of the matrix layer of each hidden layer of the inference model, delay for the scheduling group to execute page attention on the target group, and loss delay when the inference model processes inference requests in the target group.

[0372] Optionally, the determining module 802 is specifically used to: determine the inference performance corresponding to each tensor parallelism in at least one tensor parallelism based on the target latency indicated by the inference request in the target group; the inference performance corresponding to each tensor parallelism is used to indicate the inference performance when processing the inference request in the target group according to each tensor parallelism; determine the target tensor parallelism of the target group from at least one tensor parallelism based on the inference performance corresponding to each tensor parallelism; the inference performance corresponding to the target tensor parallelism of the target group satisfies the target condition.

[0373] Optionally, the inference performance corresponding to the target tensor parallelism of the target group satisfies at least one of the following conditions: the inference performance corresponding to the target tensor parallelism of the target group is the largest among at least one tensor parallelism; the inference performance corresponding to the target tensor parallelism of the target group is greater than or equal to a performance threshold; or the inference performance corresponding to the target tensor parallelism of the target group is ranked less than or equal to a ranking threshold among at least one tensor parallelism. The inference performance corresponding to the target tensor parallelism of the target group is used to indicate the inference performance when processing inference requests in the target group according to the target tensor parallelism of the target group.

[0374] For a detailed description of the above-mentioned optional methods, please refer to the foregoing method embodiments, which will not be repeated here. Furthermore, the explanation of any of the scheduling devices 800 provided above, as well as the description of their beneficial effects, can be found in the corresponding method embodiments described above, and will not be repeated here.

[0375] In this application, the grouping module 801, the determining module 802, the scheduling module 803, and the execution module 804 can all be implemented in software or in hardware. For example, the implementation of the grouping module 801 will be described below. Similarly, the implementation of the determining module 802, the scheduling module 803, and the execution module 804 can refer to the implementation of the grouping module 801.

[0376] As an example of a software functional unit, grouping module 801 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, grouping module 801 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0377] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0378] As an example of a hardware functional unit, the group module 801 may include at least one computing device, such as a server. Alternatively, the group module 801 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0379] The multiple computing devices included in the grouping module 801 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the grouping module 801 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the grouping module 801 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0380] It should be noted that, in other embodiments, the grouping module 801, the determining module 802, the scheduling module 803, and the execution module 804 can all execute any step in the scheduling method. The steps implemented by the grouping module 801, the determining module 802, the scheduling module 803, and the execution module 804 can be specified as needed. By implementing different steps in the scheduling method through the grouping module 801, the determining module 802, the scheduling module 803, and the execution module 804, all functions of the scheduling device can be achieved.

[0381] This application also provides a computing device 900. As shown in FIG9, the computing device 900 includes: a bus 902, a processor 904, a memory 906, and a communication interface 908. The processor 904, the memory 906, and the communication interface 908 communicate with each other via the bus 902. The computing device 900 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 900.

[0382] Bus 902 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 9, but this does not imply that there is only one bus or one type of bus. Bus 902 can include pathways for transmitting information between various components of computing device 900 (e.g., memory 906, processor 904, communication interface 908).

[0383] Processor 904 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0384] The memory 906 may include volatile memory, such as random access memory (RAM). The processor 904 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0385] The memory 906 stores executable program code, and the processor 904 executes this executable program code to implement the functions of the aforementioned grouping module 801, determining module 802, scheduling module 803, and execution module 804, thereby implementing the scheduling method. That is, the memory 906 stores instructions for executing the scheduling method.

[0386] The communication interface 908 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 900 and other devices or communication networks.

[0387] For example, the computing device 900 described above may be a management node as shown in Figure 1, a management node as shown in Figure 2, an inference node, etc.

[0388] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0389] As shown in Figure 10, the computing device cluster 1000 includes at least one computing device 900. The memory 906 of one or more computing devices 900 in the computing device cluster 1000 may store the same instructions for executing the scheduling method.

[0390] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster 1000 may also store a portion of the instructions for executing the scheduling method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the scheduling method.

[0391] It should be noted that the memory 906 in different computing devices 900 within the computing device cluster 1000 can store different instructions, each used to execute a portion of the functions of the scheduling device. That is, the instructions stored in the memory 906 of different computing devices 900 can implement the functions of one or more modules among the grouping module 801, the determination module 802, the scheduling module 803, and the execution module 804.

[0392] For example, the computing device cluster described above may include at least one management node, at least one node of a first type, and at least one node of a second type. The at least one management node includes a processor, memory, and computer programs / instructions stored in the memory; the processor in the at least one management node executes the computer programs / instructions stored in the memory to enable the at least one management node to implement the scheduling method in the above embodiments, thereby switching computing tasks between at least one node of the first type and / or at least one node of the second type.

[0393] In some possible implementations, one or more computing devices in the computing device cluster 1000 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation. As shown in Figure 11, two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device.

[0394] In this type of possible implementation, the memory 906 in computing device 900A stores instructions for executing the functions of grouping module 801 and determining module 802. Meanwhile, the memory 906 in computing device 900B stores instructions for executing the functions of scheduling module 803 and execution module 804.

[0395] The connection method between the computing device clusters shown in Figure 11 can be considered as follows: taking into account that the scheduling method provided in this application needs to perform a large amount of computation, the functions of the scheduling module 803 and the execution module 804 are considered to be performed by the computing device 900B.

[0396] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

[0397] It should be noted that other related descriptions of the computing cluster shown in Figure 11 can be found in the description of the computing cluster shown in Figure 10, and will not be repeated here.

[0398] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster shown in Figures 10 and 11. The difference is that the memory 906 of one or more computing devices 900 in this computing device cluster can store the same instructions for executing the scheduling method.

[0399] In some possible implementations, the memory 906 of one or more computing devices 900 in the computing device cluster may also store a portion of the instructions for executing the scheduling method. In other words, a combination of one or more computing devices 900 can jointly execute the instructions for executing the scheduling method.

[0400] This application also provides a processor that can be used to perform the above-described methods.

[0401] This application also provides a chip, including: a processor and a power supply circuit; the power supply circuit can be used to supply power to the processor; the processor can be used to perform the above-described methods.

[0402] This application also provides a management node, including: a processor, a memory, and a computer program / instructions stored in the memory; the processor executes the computer program / instructions to cause the management node to perform any of the methods provided in the first aspect above.

[0403] This application also provides an inference system, comprising: at least one management node and at least one inference node; the at least one management node includes a processor, a memory, and a computer program / instructions stored in the memory; the processor of the at least one management node executes the computer program / instructions stored in the memory of the at least one management node to enable the at least one management node to implement any of the methods provided in the first aspect above, so that the at least one inference node can process inference requests scheduled by the management node.

[0404] This application also provides a computer program product. The computer program product includes a computer program / instructions, which are software or program products capable of running on a computing device or stored on any usable medium. When the computer program / instructions are executed on at least one computing device, the at least one computing device can perform the methods described above.

[0405] This application also provides a computer-readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The computer-readable storage medium stores a computer program / instructions that, when executed on at least one computing device, enable the at least one computing device to perform the described method.

[0406] For example, the available media may be magnetic media (e.g., floppy disks, magnetic disks, magnetic tapes), optical media (e.g., digital video discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0407] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A scheduling method characterized by, Applied to a reasoning system, the method includes: The obtained inference requests are grouped to obtain multiple groups; Determine the target tensor parallelism for each of the plurality of groups, wherein, according to the target tensor parallelism of each group, the inference performance when processing inference requests in each group meets the target condition, and the latency of processing inference requests in each group is less than or equal to the target latency indicated by the inference requests in each group; The inference requests in each group are scheduled according to the target tensor parallelism of each group.

2. The method according to claim 1, characterized in that, The inference performance corresponding to the target tensor parallelism of each group includes at least one of the size of the scheduling group corresponding to the target tensor parallelism of each group or the throughput corresponding to the target tensor parallelism of each group, wherein the size of the scheduling group is used to indicate the number of inference requests scheduled each time.

3. The method according to claim 1 or 2, characterized in that, The size of the scheduling group for each group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group, wherein the scheduling threshold corresponding to the target tensor parallelism of each group is used to indicate the number of inference requests that the processor can support processing, as indicated by the target tensor parallelism of each group.

4. The method according to any one of claims 1 to 3, characterized in that, The plurality of groups includes target groups, and determining the target tensor parallelism of each of the plurality of groups includes: Based on the target latency indicated by the inference request in the target group, determine the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism. The target tensor parallelism of the target group is determined from the at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism.

5. The method according to any one of claims 1-3, characterized in that, The plurality of groups includes target groups, and determining the target tensor parallelism of each of the plurality of groups includes: Based on the target latency indicated by the inference request in the target group, determine the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism. The throughput corresponding to each tensor parallelism is determined based on the size of the scheduling group corresponding to each tensor parallelism. The target tensor parallelism of the target group is determined from the at least one tensor parallelism based on the throughput corresponding to each tensor parallelism.

6. The method of claim 5, wherein, The throughput corresponding to each tensor parallelism degree satisfies the following relationship: Among them, T i LOut is used to characterize the throughput corresponding to the i-th tensor parallelism in the at least one tensor parallelism, and LOut is used to characterize the output load of the inference request in the target group. BS1 i K is used to characterize the size of the scheduling group corresponding to the i-th tensor parallelism in at least one tensor parallelism. i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

7. The method according to any one of claims 4-6, characterized in that, The size of the scheduling group corresponding to each tensor parallelism degree satisfies the following relationship: Among them, BS1 i TL is used to characterize the size of the scheduling group corresponding to the i-th tensor parallelism in the at least one tensor parallelism; TL is used to characterize the target latency of the inference request indication in the target group; NL i The network communication latency corresponding to the i-th tensor parallelism in the at least one tensor parallelism is used to characterize the network communication latency; LN is used to characterize the number of hidden layers in the inference model of the inference system; NT is used to characterize the number of communications between sub-tensors in a single hidden layer of the inference model; MMT is used to characterize the total latency required for computation of the matrix layer of each hidden layer of the inference model; PAL is used to characterize the network communication latency corresponding to the i-th tensor parallelism in the at least one tensor parallelism; PN is used to characterize the number of hidden layers in the inference model; PN is used to characterize the number of hidden layers in the inference model; PN is used to characterize the number of hidden layers in the inference system; LN is used to characterize the number of hidden layers in the inference model ... i The page attention latency corresponding to the i-th tensor parallelism in the at least one tensor parallelism is used to characterize the page attention latency corresponding to the i-th tensor parallelism, which is used to indicate the latency of performing page attention on the scheduling group of the target group when processing the scheduling group of the target group according to the i-th tensor parallelism; OT is used to characterize the loss latency when the inference system processes the inference request in the target group; K i Used to characterize the i-th tensor parallelism in at least one tensor parallelism.

8. The method according to any one of claims 4-7, characterized in that, Determining the target tensor parallelism of the target group from the at least one tensor parallelism includes: Determine a scheduling threshold corresponding to each of the at least one tensor parallelisms, wherein the scheduling threshold corresponding to the target tensor parallelism is used to indicate the number of inference requests that the processor indicated by each tensor parallelism can process. Based on the scheduling threshold corresponding to each tensor parallelism, the target tensor parallelism of the target group is determined from the at least one tensor parallelism, wherein the size of the scheduling group corresponding to the target tensor parallelism of the target group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of the target group.

9. The method of claim 8, wherein, The scheduling threshold corresponding to each tensor parallelism degree satisfies the following relationship: Among them, BS2 i K is a scheduling threshold used to characterize the i-th tensor parallelism in at least one tensor parallelism. i The tensor parallelism is used to characterize the i-th tensor parallelism in at least one tensor parallelism, d is used to characterize the size of the storage space of a single processor of the inference system, MS is used to characterize the size of the storage space required by the inference model, and D is used to characterize the size of the storage space required by a single inference request in the target group.

10. The method according to any one of claims 1-9, characterized in that, The process of grouping the acquired multiple inference requests yields multiple groups, including: Obtain the load of each inference request among the plurality of inference requests, wherein the load of each inference request is used to indicate the input load and the output load of each inference request; The plurality of inference requests are grouped according to the load of each inference request to obtain the plurality of groups.

11. The method of claim 10, wherein, The step of grouping the plurality of inference requests according to the load of each inference request to obtain the plurality of groups includes: Based on the load of each inference request, the multiple inference requests are grouped using a clustering algorithm to obtain the multiple groups.

12. A scheduling apparatus characterized by comprising: The device, applied to a reasoning system, includes: The grouping module is used to group the multiple inference requests obtained into multiple groups; The determination module is used to determine the target tensor parallelism of each group in the plurality of groups, wherein, according to the target tensor parallelism of each group, the inference performance when processing the inference request in each group meets the target condition, and the latency of processing the inference request in each group is less than or equal to the target latency indicated by the inference request in each group; The scheduling module is used to schedule inference requests in each group according to the target tensor parallelism of each group.

13. The apparatus according to claim 12, characterized in that, The inference performance corresponding to the target tensor parallelism of each group includes at least one of the size of the scheduling group corresponding to the target tensor parallelism of each group or the throughput corresponding to the target tensor parallelism of each group, wherein the size of the scheduling group is used to indicate the number of inference requests scheduled each time.

14. The apparatus according to claim 12 or 13, characterized in that, The size of the scheduling group for each group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of each group, wherein the scheduling threshold corresponding to the target tensor parallelism of each group is used to indicate the number of inference requests that the processor can support processing, as indicated by the target tensor parallelism of each group.

15. The apparatus of any one of claims 12-14, wherein, The plurality of groups includes a target group, and the determining module is specifically used for: Based on the target latency indicated by the inference request in the target group, determine the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism. The target tensor parallelism of the target group is determined from the at least one tensor parallelism based on the size of the scheduling group corresponding to each tensor parallelism.

16. The apparatus of any one of claims 12-14, wherein, The plurality of groups includes a target group, and the determining module is specifically used for: Based on the target latency indicated by the inference request in the target group, determine the size of the scheduling group corresponding to each tensor parallelism in at least one tensor parallelism, wherein the size of the scheduling group corresponding to each tensor parallelism is used to indicate the number of inference requests scheduled each time when processing the inference requests in the target group according to each tensor parallelism. The throughput corresponding to each tensor parallelism is determined based on the size of the scheduling group corresponding to each tensor parallelism. The target tensor parallelism of the target group is determined from the at least one tensor parallelism based on the throughput corresponding to each tensor parallelism.

17. The apparatus of claim 15 or 16, wherein, The determining module is also used for: Determine a scheduling threshold corresponding to each tensor parallelism in at least one tensor parallelism, wherein the scheduling threshold corresponding to the target tensor parallelism is used to indicate the number of inference requests that the processor indicated by each tensor parallelism can process. Based on the scheduling threshold corresponding to each tensor parallelism, the target tensor parallelism of the target group is determined from the at least one tensor parallelism, wherein the size of the scheduling group corresponding to the target tensor parallelism of the target group is less than or equal to the scheduling threshold corresponding to the target tensor parallelism of the target group.

18. The apparatus of any one of claims 12-17, wherein, The grouping module is specifically used for: Obtain the load of each inference request among the plurality of inference requests, wherein the load of each inference request is used to indicate the input load and the output load of each inference request; The plurality of inference requests are grouped according to the load of each inference request to obtain the plurality of groups.

19. The apparatus of claim 18, wherein, The grouping module is specifically used for: Based on the load of each inference request, the multiple inference requests are grouped using a clustering algorithm to obtain the multiple groups.

20. A management node, characterized in that, The management node includes a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to cause the management node to implement the method as described in any one of claims 1-11.

21. A reasoning system, characterized in that, The inference system includes at least one management node and at least one inference node; The at least one management node includes a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor of the at least one management node executes the computer program / instructions stored in the memory of the at least one management node to enable the at least one management node to implement the method as described in any one of claims 1-11, so that the at least one inference node can process inference requests scheduled by the management node.

22. A computer program product, characterized in that, The computer program product includes a computer program / instruction, which, when executed by a computing device, implements the method as described in any one of claims 1-11.

Images