Method and apparatus for inference scheduling, cluster, medium, and product

Abstract

Embodiments of the present disclosure relate to a method and apparatus for inference scheduling, a cluster, a medium, and a product. The method comprises: in response to tokens to be computed existing in a token pool, selecting, on the basis of a first time threshold, a first number of tokens from the token pool for forming a first batch, and selecting, on the basis of a second time threshold, a second number of user requests from a request pool for forming a second batch, the token pool comprising tokens generated by performing inference on a plurality of user requests, and the request pool comprising the plurality of user requests. The method further comprises feeding the first batch and the second batch into at least one server of a cluster for inference. In this way, an optimized inference scheduling policy for concurrent loads is provided, which enables more rational and efficient allocation of various inference resources, thereby achieving high resource utilization, high inference throughput, and high bandwidth utilization under concurrent requests.

Description

Methods, apparatus, clusters, media, and products for inference scheduling

[0001] This application claims priority to Chinese Patent Application No. 202510066749.4, filed on January 15, 2025, entitled "Method and Apparatus, Cluster, Medium and Product for Reasoning Scheduling", the entire contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments of this application generally relate to the field of computers, and more specifically, to methods and apparatus for inference scheduling, computing device clusters, computer-readable storage media, and computer program products. Background Technology

[0003] The service-oriented approach to inference in artificial intelligence (AI) models provides the inference process of AI models as a service. Users can invoke the deployed model through application programming interfaces (APIs) or other interfaces, enabling them to infer results from input data. This service-oriented inference process is a crucial step in applying abstract algorithmic models to real-world business applications, demonstrating performance optimization and application benefits across multiple dimensions, including ease of use, scalability (dynamically adjusting computing resources), cost control, and cross-platform compatibility. Summary of the Invention

[0004] This application provides a scheme for inference scheduling. The scheme provides an optimized inference scheduling strategy for concurrent loads, which can allocate various inference resources more reasonably and efficiently to achieve high resource utilization, high inference throughput, and high bandwidth utilization under concurrent requests.

[0005] In a first aspect, a method for inference scheduling is provided. The method includes, in response to existing lexical units to be computed in a lexical pool, selecting a first number of lexical units from the lexical pool based on a first time threshold to form a first batch, and selecting a second number of user requests from a request pool based on a second time threshold to form a second batch. The lexical pool includes lexical units generated by inference from multiple user requests, and the request pool includes multiple user requests. The method also includes feeding the first and second batches into at least one server in the cluster for inference. This method enables more reasonable and efficient allocation of inference resources under concurrent load scenarios, thereby improving resource utilization and inference throughput, bandwidth utilization, etc.

[0006] In some embodiments of the first aspect, the method further includes selecting a third number of user requests from a request pool to form a third batch in response to the absence of any tokens to be computed in the token pool, based on a second time threshold. The method also includes feeding the third batch into at least one server in the cluster for inference. Thus, optimized inference scheduling can be achieved by selecting a reasonable batch size from the load input that satisfies time constraints (e.g., time constraints in a Service Level Agreement (SLA)).

[0007] In some embodiments of the first aspect, in response to the current number of lexical units included in the lexical unit pool being greater than or equal to the maximum number of lexical units that satisfy a first time threshold, the first number of lexical units is the maximum number of lexical units; or in response to the current number of lexical units included in the lexical unit pool being less than the maximum number of lexical units that satisfy the first time threshold, the first number of lexical units is the current number of lexical units. Thus, the batch size of lexical units can be dynamically determined based on the number of lexical units in the lexical unit pool.

[0008] In some embodiments of the first aspect, in response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests satisfying the second time threshold, the second number of user requests is the maximum number of user requests; or in response to the current number of user requests included in the request pool being less than the maximum number of user requests satisfying the second time threshold, the second number of user requests is the current number of user requests. Thus, the request batch size can be dynamically determined based on the number of user requests in the request pool.

[0009] In some embodiments of the first aspect, in response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests satisfying the second time threshold, the third number of user requests is the maximum number of user requests; or in response to the current number of user requests included in the request pool being less than the maximum number of user requests satisfying the second time threshold, the third number of user requests is the current number of user requests. Thus, the request batch size can be dynamically determined based on the number of user requests in the request pool.

[0010] In some embodiments of the first aspect, the second time threshold is greater than the first time threshold.

[0011] In some embodiments of the first aspect, the cluster includes a homogeneous cluster, and the method further includes: dividing user requests in the second batch into user request slices based on unit token processing time; selecting a fourth number of user request slices based on a first time threshold to form a fourth batch; and merging the first batch and the fourth batch to obtain a merged batch, wherein inference includes: assigning the inference of the merged batch to inference resources based on the computing power of inference resources in the homogeneous cluster. Thus, for concurrent load scenarios, the inference stages of multiple requests can be merged, and the inference tasks of each request can be reasonably assigned to the corresponding inference resources based on computing power, ensuring that the inference tasks of each request are running while avoiding waiting and idle inference resources, thereby improving the overall resource utilization.

[0012] In some embodiments of the first aspect, if there are no lexes to be computed in the lexical pool, the current inference includes inference in the pre-filling stage; or if there are lexes to be computed in the lexical pool, the current inference includes inference in the pre-filling stage and inference in the decoding stage.

[0013] In some embodiments of the first aspect, the cluster includes a heterogeneous cluster, wherein inference includes: assigning inference during the pre-filling phase to compute-intensive resources in the heterogeneous cluster, and assigning inference during the decoding phase to memory-intensive resources in the heterogeneous cluster. Thus, taking into account the computational and memory access differences between the pre-filling and decoding phases, assigning inference during the pre-filling phase of compute-intensive tasks to compute-intensive resources, and assigning inference during the decoding phase of memory-intensive tasks to memory-intensive resources, optimizes and balances resource scheduling.

[0014] In some embodiments of the first aspect, the cluster includes a heterogeneous cluster, wherein inference includes: assigning non-attention computations in the pre-filling phase and the decoding phase to compute-intensive resources in the heterogeneous cluster, and assigning attention computations in the decoding phase to memory-intensive resources in the heterogeneous cluster. Thus, taking into account the computation and memory access differences within the decoding phase itself, assigning non-attention computations to compute-intensive resources and attention computations to memory-intensive resources further optimizes and balances resource scheduling.

[0015] In some embodiments of the first aspect, the method further includes updating the lexical pool based on non-terminal words in the inference results for generating subsequent lexical units.

[0016] In some embodiments of the first aspect, the first time threshold is the time between tokens (TBT); or the second time threshold is the time to generate the first token (TTFT).

[0017] In a second aspect, an apparatus for inference scheduling is provided. The apparatus includes a batch forming module configured to, in response to the existence of lexical units to be computed in a lexical pool, select a first number of lexical units from the lexical pool based on a first time threshold to form a first batch, and select a second number of user requests from a request pool based on a second time threshold to form a second batch, the lexical pool including lexical units generated by inference on multiple user requests, and the request pool including multiple user requests. The apparatus also includes a batch feeding module configured to feed the first and second batches to at least one server in a cluster for inference.

[0018] In some embodiments of the second aspect, the apparatus further includes a second batch forming module configured to select a third number of user requests from a request pool to form a third batch in response to the absence of any lexical units to be computed in the lexical pool, based on a second time threshold. The apparatus also includes a second batch feeding module configured to feed the third batch into at least one server in the cluster for inference.

[0019] In some embodiments of the second aspect, in response to the current number of lexical units included in the lexical unit pool being greater than or equal to the maximum number of lexical units that satisfy the first time threshold, the first number of lexical units is the maximum number of lexical units; or in response to the current number of lexical units included in the lexical unit pool being less than the maximum number of lexical units that satisfy the first time threshold, the first number of lexical units is the current number of lexical units.

[0020] In some embodiments of the second aspect, in response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests that satisfy the second time threshold, the second number of user requests is the maximum number of user requests; or in response to the current number of user requests included in the request pool being less than the maximum number of user requests that satisfy the second time threshold, the second number of user requests is the current number of user requests.

[0021] In some embodiments of the second aspect, in response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests that satisfy the second time threshold, the third number of user requests is the maximum number of user requests; or in response to the current number of user requests included in the request pool being less than the maximum number of user requests that satisfy the second time threshold, the third number of user requests is the current number of user requests.

[0022] In some embodiments of the second aspect, the second time threshold is greater than the first time threshold.

[0023] In some embodiments of the second aspect, the cluster includes a homogeneous cluster, and the apparatus further includes: a request segmentation module configured to segment user requests in the second batch into user request slices based on unit token processing time; a third batch formation module configured to select a fourth number of user request slices based on a first time threshold to form a fourth batch; and a batch fusion module configured to fuse the first batch and the fourth batch to obtain a fused batch. The apparatus also includes an assignment module configured to assign the inference of the fused batch to inference resources based on the computing power of the inference resources in the homogeneous cluster.

[0024] In some embodiments of the second aspect, if there are no words to be computed in the word pool, the current reasoning includes reasoning in the pre-filling stage; or if there are words to be computed in the word pool, the current reasoning includes reasoning in the pre-filling stage and reasoning in the decoding stage.

[0025] In some embodiments of the second aspect, the cluster includes a heterogeneous cluster, and the apparatus further includes an assignment module configured to assign inference during the pre-filling phase to compute-intensive resources in the heterogeneous cluster and to assign inference during the decoding phase to memory-intensive resources in the heterogeneous cluster.

[0026] In some embodiments of the second aspect, the cluster includes a heterogeneous cluster, and the apparatus further includes an assignment module configured to assign non-attention computations in the pre-filling phase inference and in the decoding phase inference to compute-intensive resources in the heterogeneous cluster, and to assign attention computations in the decoding phase inference to memory-intensive resources in the heterogeneous cluster.

[0027] In some embodiments of the second aspect, the apparatus further includes a lexical pool update module configured to update the lexical pool based on non-terminal words in the inference results for generating subsequent lexical units.

[0028] In some embodiments of the second aspect, the first time threshold is the time between tokens (TBT); or the second time threshold is the time to generate the first token (TTFT).

[0029] According to a third aspect of this disclosure, a computing device is provided. The computing device includes a processor and a memory storing instructions that, when executed by the processor, cause the computing device to perform the method described according to a first aspect of this disclosure.

[0030] According to a fourth aspect of this disclosure, a computing device cluster is provided. The computing device cluster includes at least one computing device, each computing device including a processor and memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method described in the first aspect of this disclosure. In some embodiments, the computing device cluster includes a single computing device. In other embodiments, the computing device cluster includes multiple computing devices. In some embodiments, the computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0031] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program instructions that, when executed by a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method described according to a first aspect of this disclosure.

[0032] According to a sixth aspect of this disclosure, a computer program product is provided. The computer program product includes instructions that, when executed by a computing device or cluster of computing devices, cause the computing device or cluster of computing devices to perform the method described in accordance with a first aspect of this disclosure. In some embodiments, the computer program product may include one or more software installation packages that can be downloaded or copied and executed on the computing device or cluster of computing devices when the method provided in the first aspect or its possible variations is required.

[0033] Please note that the Summary of the Invention is provided to introduce a series of concepts in a simplified form, which will be further described below in the Detailed Description. The Summary of the Invention is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0034] The above and other objects, features, and advantages of this disclosure will become clearer through a more detailed description of the embodiments thereof in conjunction with the accompanying drawings, in which:

[0035] Figure 1 is a schematic diagram illustrating an exemplary reasoning environment in which methods and / or processes according to embodiments of the present disclosure may be implemented;

[0036] Figure 2 is a flowchart schematically illustrating a method for inference scheduling according to an embodiment of the present disclosure;

[0037] Figure 3 illustrates an inference scheduling process for concurrent loads according to an embodiment of the present disclosure;

[0038] Figure 4 illustrates an inference scheduling process for concurrent loads under a homogeneous architecture according to an embodiment of the present disclosure;

[0039] Figure 5 illustrates an inference scheduling process for concurrent loads under a heterogeneous architecture according to an embodiment of the present disclosure;

[0040] Figure 6 illustrates another inference scheduling process for concurrent loads under a heterogeneous architecture according to an embodiment of the present disclosure;

[0041] Figure 7 is a schematic diagram illustrating the inference scheduling process under homogeneous and heterogeneous architectures according to embodiments of the present disclosure;

[0042] Figure 8 illustrates a diagram of the pre-filled-decode (PD) fusion result according to an embodiment of the present disclosure;

[0043] Figure 9 schematically illustrates a schematic block diagram of a cache management apparatus according to an embodiment of the present disclosure;

[0044] Figure 10 schematically illustrates a schematic block diagram of an example device that can be used to implement exemplary implementations of this disclosure;

[0045] Figure 11 schematically illustrates a schematic block diagram of an example device cluster that can be used to implement exemplary implementations of this disclosure; and

[0046] Figure 12 schematically illustrates a schematic block diagram of another example device that can be used to implement exemplary implementations of this disclosure.

[0047] In all the accompanying drawings, the same or similar reference numerals usually indicate the same or similar elements. Detailed Implementation

[0048] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0049] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0050] As mentioned above, AI model inference service encapsulates the inference process of an AI model into a service accessible externally, allowing users to invoke the deployed AI model via APIs or other interfaces. Within this framework, the model can be used to infer results from input data. The inference service process enables abstract algorithmic models to be effectively applied to real-world business scenarios, with language model-based inference services being a key application.

[0051] Language models, especially the large language models (LLMs) that have emerged in recent years, have developed into an influential and widely used technology. One of their core functions is to greatly simplify the previously complex and cumbersome process of knowledge acquisition. Users can overcome previously high knowledge barriers to achieve accurate and fast data question answering, information retrieval, and more, allowing ordinary users to easily enjoy the convenience brought by technological advancements.

[0052] The inference process of a large language model is divided into three stages: prefill, cache transfer, and decoding. The computation process for generating the first token is called the prefill stage. After generation, the first token is fed back into the large language model as input to predict the next token, continuing until the end word is generated. This process of continuously using previously generated tokens to generate new tokens is the decoding stage. Furthermore, the cache transfer stage (KV cache transfer stage) refers to the process where, during the inference process of the large language model, the keys and values ​​of the attention mechanism calculated in each autoregressive iteration are stored. These stored cached values ​​are then directly accessed in subsequent iterations to avoid redundant computation, thereby improving inference efficiency. In the current scheme, these stages are executed sequentially.

[0053] However, numerous problems and challenges remain regarding the scheduling of the inference process. For example, streaming serial inference suffers from low resource utilization under multi-user concurrent loads. Inference is performed in a streaming-serial manner, leading to low resource utilization due to the bottleneck effect in both homogeneous and heterogeneous server scenarios. Considering that real-world inference scenarios primarily handle multi-user concurrent requests, processing a large number of such requests can easily cause some inference resources to wait and become idle, thus reducing overall resource utilization. Another example is the existence of time bubbles in the pre-fill-decode (PD) pipeline construction. The pre-fill stage is typically a computationally intensive task, while the decoding stage is a memory-intensive task; these two stages differ in computation and memory access. Furthermore, in addition to the aforementioned differences between the pre-fill and decoding stages, the multiplication and attention operations in the decoding stage also differ in computation and memory access, with attention operations being more memory-intensive. When performing pipelined masking, if the computational and memory access differences between the pre-filling and decoding stages, as well as the computational and memory access differences within the decoding stage itself, are not fully considered, it may lead to an imbalance in execution time, resulting in time bubbles. These problems and challenges make it difficult to fully leverage the value of cluster-scale resources in inference scenarios, often resulting in high costs but only low performance gains.

[0054] To address at least some of the aforementioned problems and other potential issues, a scheme for inference scheduling according to embodiments of this disclosure is proposed. The scheme includes, in response to the existence of lexical units to be computed in a lexical pool, selecting a first number of lexical units from the lexical pool based on a first time threshold to form a first batch, and selecting a second number of user requests from a request pool based on a second time threshold to form a second batch. The lexical pool includes lexical units generated by inference on multiple user requests, and the request pool includes multiple user requests. The scheme also includes feeding the first and second batches into at least one server in the cluster for inference. In this manner, an optimized inference scheduling strategy for concurrent loads is provided, enabling more reasonable and efficient allocation of inference resources to achieve high resource utilization, high inference throughput, and high bandwidth utilization under concurrent requests.

[0055] The basic principles and several example implementations of this disclosure are described below with reference to Figures 1 to 12. It should be understood that these exemplary embodiments are given only to enable those skilled in the art to better understand and implement the embodiments of this disclosure, and are not intended to limit the scope of this disclosure in any way.

[0056] Figure 1 is a schematic diagram illustrating an exemplary inference environment 100 in which methods and / or processes according to embodiments of the present disclosure may be implemented. As shown in Figure 1, the inference environment 100 includes an inference system 120 and one or more users interacting with it (e.g., users 111, 112, and 113 shown in Figure 1). As shown in Figure 1, the inference system 120, i.e., the set of inference resources, schematically includes inference resources 121, 122, and 123, which are deployed in the cloud for dynamic access or allocation. Figure 1 illustrates the implementation environment of an embodiment of the present disclosure using a distributed inference system (i.e., the corresponding inference resources may not be geographically located but are associated through cloud computing or distributed technologies). It should be understood that this is merely an exemplary and not limiting description, and other different types of inference systems are also possible, such as centralized inference systems (i.e., the corresponding inference resources are uniformly deployed locally). A suitable inference environment configuration can be selected according to actual usage needs.

[0057] As illustrated in Figure 1, multiple users 111, 112, and 113 can concurrently submit their respective user requests, such as queries or questions, to the inference system. These user requests can be based on natural language descriptions. Examples of the inference system 120 may include a cluster of computing devices or a cluster of servers, and the inference resources may correspond to the computing devices or servers within the cluster. In this example, the language model used for inference (e.g., a Large Language Model (LLM)) can be deployed on at least one computing device or server to handle the user requests. Appropriate inference resources in the inference system 120 can be assigned to respond to these user requests, such as retrieving data corresponding to a query or generating an answer corresponding to a question.

[0058] According to embodiments of this disclosure, the inference system 120 can refer to a cloud-based computing and service model that couples inference resources such as computing resources and memory access resources together via a network to provide inference services to users. The inference system 120 is not merely a simple collection of resources, but a highly integrated, intelligently scheduled, and flexibly scalable inference environment. In this environment, key elements such as various resources and applications are tightly coupled, allowing users to dynamically acquire, configure, and use these resources as needed without having to own or manage them locally.

[0059] Within the framework of Inference System 120, high-performance computing systems such as supercomputers and large-scale computing clusters are not only deployed efficiently, but also achieve a high level of flexibility and scalability. These high-performance computing resources, which were originally limited by high deployment costs and complex operation and maintenance management, are now seamlessly integrated into the cloud through cloud computing technology and elastic scheduling technology, forming a powerful and flexible inference ecosystem.

[0060] In the example in Figure 1, the inference resources 121, 122, and 123 included in the inference system 120 can be coupled to each other via a network. Taking a server cluster as an example, the servers can interconnect to communicate, for example, to achieve data synchronization or sharing between servers. In this configuration scenario, the servers can process tasks in parallel through distributed computing. This parallel computing capability enables the cloud computing platform to handle more complex and massive tasks. Furthermore, each server in the cluster can have redundancy and fault tolerance mechanisms to ensure reliable task execution. When a node fails, the system can automatically migrate the job to other normally functioning nodes, ensuring task continuity and availability.

[0061] In the example configuration of the inference system 120 as a server cluster, examples of inference resources or resource nodes may include a central server, an edge server, or a local server in a local data center. Besides the servers mentioned above, other computing devices can serve as resource nodes in the inference system 120, such as supercomputers, personal computers, laptops, in-vehicle computing devices, mobile devices (such as smartphones, tablets, etc.), wearable electronic devices, multimedia devices, personal digital assistants (PDAs), smart home devices, consumer electronics, or combinations thereof. It should be understood that the resource nodes described herein are merely exemplary and not limiting; for example, other different types of resource nodes may exist as technology evolves.

[0062] It should be understood that, for ease of illustration and depiction, only one inference system (i.e., inference system 120) and three inference resources (i.e., inference resources 121, 122, and 123) are shown in Figure 1. However, this is merely exemplary, and other inference systems and inference resources with different configurations and numbers may exist, which can be customized according to specific use cases and needs, as will be described below. The example environment in which the methods and / or processes according to embodiments of this disclosure can be implemented has been described above with reference to Figure 1. The method 200 for inference scheduling according to embodiments of this disclosure is described below with reference to Figure 2.

[0063] Figure 2 is a flowchart schematically illustrating a method 200 for inference scheduling according to an embodiment of the present disclosure. According to the method 200 described in conjunction with Figure 2, an optimized inference scheduling strategy for concurrent loads is provided, enabling more reasonable and efficient allocation of inference resources in concurrent load scenarios, thereby improving resource utilization and inference throughput, bandwidth utilization, etc.

[0064] At point 210, in response to the existence of lexical units to be computed in the lexical unit pool, a first number of lexical units are selected from the lexical unit pool based on a first time threshold to form the first batch, and a second number of user requests are selected from the request pool based on a second time threshold to form the second batch. The lexical unit pool includes lexical units generated by reasoning from multiple user requests, and the request pool includes multiple user requests. The lexical unit pool can be implemented by constructing a memory pool, which can be accessed by servers in the cluster or by the memory in the servers. The lexical unit pool can be established to store lexical units to be computed (also called, lexical units to be computed), which can be the first word (unit) generated in the pre-filling stage or the intermediate word (unit) generated in the decoding stage. If the generated lexical unit is not the end word, it will be placed in the lexical unit pool for subsequent reasoning. The request pool can also be implemented by constructing a memory pool, and the request pool can be established to store user requests, where user requests can be questions or queries based on language descriptions submitted by users, and certain preprocessing can be performed on them (e.g., simplification and deduplication).

[0065] According to embodiments of this disclosure, when there are already tokens to be computed in the token pool, the current inference includes inference in the pre-filling phase and inference in the decoding phase. By way of example and not limitation, this may be because, in a concurrent load scenario, the inference in the pre-filling phase of a previously requested request has been completed and is currently in the inference in the decoding phase, while subsequent requests are in the inference in the pre-filling phase.

[0066] To concurrently process multiple user requests and avoid waiting and idle inference resources, a certain number of terms to be computed and a certain number of user requests can be selected by the servers or processors on the servers in the inference system (e.g., inference system 120 in Figure 1) based on time constraints. This allows them to be processed simultaneously by the inference resources in the cluster. For example, multiple servers in the cluster can process these selected terms to be computed and user requests simultaneously, or multiple processors on the servers can perform such simultaneous processing. Specifically, a first number of terms to be computed that meet a first time threshold are selected from the term pool to form a term batch (also referred to as the first batch in this document), and a second number of user requests that meet a second time threshold are selected from the request pool to form a request batch (also referred to as the second batch in this document). The time constraints such as the first and second time thresholds can be time constraints in the SLA. Furthermore, in addition to time constraints, the terms to be computed and user requests for each inference process can also be selected based on resource constraints, such as the available inference resources in the cluster.

[0067] At point 220, the first and second batches are fed into at least one server in the cluster for inference. Unlike existing schemes that execute inference tasks sequentially for each request and serially execute the pre-filling and decoding phases of the same request, embodiments of this disclosure select a number of terms generated for inference of multiple user requests, and select a number of user requests for multiple user requests, then send them to the cluster for inference on a language model (e.g., a Large Language Model (LLM)) deployed at the cluster. The language model can be deployed on one server in the cluster or distributed across multiple servers. The selected terms to be computed and user requests (i.e., the first and second batches) can be fed into the inference resources in the cluster by, for example, servers in inference system 120 or processors on servers, ensuring that the inference resources can process each request simultaneously. For example, multiple servers in the cluster can process these selected terms to be computed and user requests simultaneously, or multiple processors in the servers can also perform such simultaneous processing. In other words, the inference tasks for each request are made running on the inference resources, thereby avoiding resource waiting and idleness in serial scheduling.

[0068] The method 200 for inference scheduling according to embodiments of this disclosure provides an inference service-oriented scheduling method for concurrent loads. It can select a reasonable number of terms to be computed and user requests based on time constraints. The selected terms to be computed and user requests cover the pre-filling stage, decoding stage, etc. of multiple concurrent requests, so that each request can run on inference resources. This promotes the fuller and more efficient allocation of various inference resources and avoids waiting and idleness, thereby achieving high resource utilization, high inference throughput, high bandwidth utilization, and high bandwidth efficiency under concurrent requests.

[0069] Figure 3 illustrates an inference scheduling process 300 for concurrent loads according to an embodiment of the present disclosure. As exemplarily shown in Figure 3, multiple concurrent user requests 311, 312, and 313 can be placed into a request pool 322. The request pool 322 can be configured to store and manage sequences of requests for the language model for efficient batch processing and scheduling. After selection, processing, etc., the user requests in the request pool 322 can be further fed into cluster 330 for inference. Taking a server cluster as an example, a language model (Large Language Model (LLM)) is deployed on at least one server in cluster 330. Cluster 330 can be configured to process these user requests using the language model and provide responses to these user requests.

[0070] As described above, when there are already words to be calculated in the word pool 321, the current inference includes inference in the pre-filling stage and inference in the decoding stage. The word pool 321 can be configured to transfer words to be calculated. In other words, as long as it is not the end word (meta), that is, the first word (meta) generated in the pre-filling stage or the intermediate word (meta) generated in the decoding stage, it will be temporarily placed in the word pool 321 for subsequent feeding into the cluster 330 for continued inference. For example, the first word or intermediate word generated from the cluster 330 will return to the word pool 321 to continue as words to be calculated. According to the embodiments of this disclosure, in response to the current number of words included in the word pool 321 being greater than or equal to the maximum number of words that satisfy the first time threshold, the first number of words can be the maximum number of words, or in response to the current number of words included in the word pool 321 being less than the maximum number of words that satisfy the first time threshold, the first number of words can be the current number of words.

[0071] In some embodiments, the first time threshold may include, but is not limited to, time between tokens (TBT), and time per output token (TPOT) may also be considered. It is desirable that the token processing time of the selected tokens meets or is as close as possible to the first time threshold; however, when the number of tokens to be computed in the token pool 321 is insufficient, all currently processed tokens within it are selected as a token batch. In an exemplary, non-limiting example, TBT in the SLA is used as an example of the first time threshold, and it is assumed that TBT is 1.5 seconds and the unit token processing time is 0.4 seconds. If there are 3 or more tokens in the token pool 321, the 3 tokens with higher priority or higher ranking are selected as a token batch; however, if there are only 2 tokens in the token pool 321, these 2 tokens are selected as a token batch. This selection process can be performed on the sequence length dimension for each of multiple user requests.

[0072] According to embodiments of this disclosure, in response to the current number of user requests included in request pool 322 being greater than or equal to the maximum number of user requests that satisfy the second time threshold, the second number of user requests can be the maximum number of user requests, or in response to the current number of user requests included in request pool 322 being less than the maximum number of user requests that satisfy the second time threshold, the second number of user requests can be the current number of user requests.

[0073] The second time threshold differs from the first time threshold; for example, the second time threshold may be greater than the first time threshold. In some embodiments, the second time threshold may be, for example, but not limited to, the first token generation time (TTFT). It is desirable that the processing time of the selected user requests should meet or be as close as possible to the second time threshold; however, when the number of user requests in request pool 322 is insufficient, all currently active user requests within it are selected as a request batch. In an exemplary, non-limiting example, the TTFT in the SLA is used as the second time threshold, and it is assumed that the TTFT is 5 seconds and the processing time per user request is 1.5 seconds. If there are 3 or more user requests in request pool 322, the 3 highest priority or highest-ranked user requests are selected as a request batch; however, if there are only 2 user requests in request pool 322, these 2 user requests are selected as a request batch.

[0074] Another scenario exists where there are no lexes to be computed in the lexical pool 321. This could be due to inference not yet starting, or the pre-filling phase still in progress, i.e., the first word has not yet been generated. In the case where there are no lexes to be computed in the lexical pool, the current inference includes the inference of the pre-filling phase. According to embodiments of this disclosure, in response to the absence of lexes to be computed in the lexical pool, a third number of user requests can be selected from the request pool 322 based on a second time threshold to form a third batch. The second time threshold can differ from the first time threshold, for example, the second time threshold may be greater than the first time threshold. For example, such a time constraint could be another time constraint in the SLA, such as, but not limited to, the first lexical generation time (TTFT). Furthermore, in addition to time constraints, user requests for each inference process can also be selected based on resource constraints, such as available inference resources in cluster 330. After forming the request batch, the request batch can be fed into the at least one server in cluster 330 for inference.

[0075] According to embodiments of this disclosure, in response to the current number of user requests included in request pool 322 being greater than or equal to the maximum number of user requests satisfying the second time threshold, the third number of user requests can be the maximum number of user requests; or in response to the current number of user requests included in request pool 322 being less than the maximum number of user requests satisfying the second time threshold, the third number of user requests can be the current number of user requests. That is, it is desirable that the request processing time of the selected user requests should meet or be as close as possible to the second time threshold. However, when the number of user requests to be calculated in user request pool 322 is insufficient, all currently existing user requests within it are selected as a request batch. In this case, when selecting the third number of user requests, there are no user tokens to be calculated in token pool 321, therefore, the selection of user tokens to be calculated does not need to be considered at this time.

[0076] Figure 4 illustrates an inference scheduling process 400 for concurrent load under a homogeneous architecture according to an embodiment of the present disclosure. As exemplarily shown in Figure 4, multiple concurrent user requests 411, 412, and 413 can be placed into a request pool 422. After certain preprocessing, the user requests in the request pool 422 can be further fed into the cluster 430 for inference. The first word or intermediate word generated from the cluster 430 is returned to the lexical pool 421 to continue as lexical units to be computed for subsequent inference. The cluster 430 can be a homogeneous server cluster (also known as a homogeneous cluster architecture), in which each server has consistent inference resources, meaning that there may be no essential difference in terms of, for example, computational performance or memory access performance during the inference process.

[0077] In the homogeneous cluster architecture scenario shown in Figure 4, if there are no tokens to be computed in the token pool 421, user requests can be selected from the request pool 422 at 432 based on, for example, TTFT or other time constraints to form a request batch, and the process can proceed to 433 for inference. At 434, if the generated token is a terminator, the inference for that user request ends; if the generated token is a non-terminal terminator, it is placed in the token pool 421 as a token to be computed to update the token pool 421.

[0078] According to embodiments of this disclosure, when cluster 430 is a homogeneous cluster and there are already words to be computed in the word pool 421, word batches and request slice batches can be formed at 431. Words to be computed can be selected from the word pool 421 to form word batches based on, for example, TBT or other time constraints. Furthermore, user requests in the second batch can be segmented into user request slices based on unit word processing time. As described above, a second number of user requests are selected based on a second time threshold to form a request batch (i.e., the second batch). User request slices (i.e., user request chunks) can be obtained by segmenting user requests with reference to unit word processing time, wherein a user request slice is a part or subset of a user request.

[0079] The processing of a lexical unit has a unit lexical processing time metric, that is, the processing time required for a single lexical unit. The unit lexical processing time can be characterized by the average or minimum / maximum processing time of the lexical unit. User request slices can be obtained by segmenting user requests based on the unit lexical processing time. In some embodiments, a user request can be segmented such that the unit request slice processing time is equal to the unit lexical processing time, or other proportional relationships between the two are also possible. By way of example and not limitation, assuming the unit lexical processing time is 0.4 seconds and the unit user request processing time is 1.6 seconds, one user request can be segmented into four user request slices, each with a processing time of 0.4 seconds; or one user request can be segmented into two user request slices, each with a processing time of 0.8 seconds, and so on.

[0080] According to embodiments of this disclosure, a fourth number of user request slices can be selected from the aforementioned user request slices based on a first time threshold to form a fourth batch. To adapt to the lexical batch (i.e., the first batch), a certain number of user request slices can be selected based on the first time threshold to form a request slice batch (i.e., the fourth batch). The first time threshold, as described above, can be a time constraint in the SLA, examples of which include TBT, TPOT, or other time constraints. It is desirable that the request slice processing time of the selected user request slices meets or is as close as possible to the first time threshold. In an exemplary, non-limiting example, TBT in the SLA is used as an example of the first time threshold, and it is assumed that TBT is 1.5 seconds and the unit request slice processing time is 0.4 seconds. If there are three or more user request slices in the request pool 322, the three user request slices with higher priority or higher sorting are selected as the request slice batch.

[0081] According to embodiments of this disclosure, the first batch and the fourth batch can be merged to obtain a merged batch. The obtained merged batch can be assigned to the corresponding inference resources based on the computing power of the inference resources in the cluster. The merged batch can indicate one or more requests or portions thereof in concurrent user requests, and can also indicate one or more lexical units in the lexical units to be computed corresponding to the concurrent user requests. For example, a server or processor on a server in inference system 120 can merge lexical batches and request slice batches together, enabling the inference resources to concurrently process multiple user requests after being sent to the inference resources in the cluster. By way of example and not limitation, the matrix corresponding to the selected first number of lexical units to be computed and the matrix corresponding to the second number of user request slices can be merged into a single composite matrix based on the sequence length dimension and / or the feature layer dimension (or, the hidden dimension). A lexical batch can be considered as a matrix containing features of a specific number of lexical units, indicating the feature vector of each lexical unit in the batch; similarly, a request slice batch can also be represented as a matrix containing features of a specific number of user request slices, indicating the feature vector of each user request slice in the batch.

[0082] Hidden dimensions can indicate, for example, the dimension of the feature representation or vector representation formed by encoding text input (such as tokens, user request slices, etc.) through a transformer (e.g., the embedding length). Sequence length can indicate the number of elements (such as tokens, characters) in a data sample. Request slice batches can be BSH dimensions (i.e., batch size, sequence length, hidden dimension), and token batches can be B1H dimensions (i.e., batch size, sequence length of 1, hidden dimension). By way of example and not limitation, assuming the request slice batch matrix is ​​[[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15]] (i.e., B=1, S=5, H=3) and the lexical batch matrix is ​​[16,17,18], then the fusion matrix for fusion along the sequence length dimension could be [[1,2,3],[4,5,6],[7,8,9],[10,11,12],[13,14,15],[16,17,18]]. Furthermore, assuming the lexical batch matrix is... The requested slice matrix is If both have a hidden dimension of 3, then the fusion matrix for fusing them along the hidden dimension can be: Other fusion dimensions or fusion methods are also feasible, and will not be elaborated here. For ease of understanding and illustration, a simplified example is given here. It should be understood that the dimensions involved in real-world implementations can be larger than the dimensions shown in this example.

[0083] After the fusion matrix is ​​generated, inference can proceed to point 433. At point 434, if the generated lexical is a terminator, the inference for that user request ends; if the generated lexical is a non-terminal terminator, it is placed in lexical pool 421 as a lexical to be computed to update the lexical pool 421. According to embodiments of this disclosure, the corresponding parts of the inference in the fusion batch can be assigned to resources for inference in a homogeneous cluster architecture (e.g., cluster 430 shown in Figure 4), such as a Neural Processing Unit (NPU), so that each stage of the inference process for each user request in multiple user requests (e.g., inference in the pre-filling stage and inference in the decoding stage, etc.) is assigned to the corresponding inference resources in the cluster. For example, the assignment process can be based on the computing power of the inference resources. For example, assuming an NPU can process n sub-inference processes per unit time, the sub-matrix corresponding to each of the n sub-inference processes (e.g., all pre-filling stage inference, decoding stage inference, or both pre-filling and decoding stage inference) from the fusion batch matrix can be assigned to the NPU for inference computation. That is, the assigned sub-matrix can correspond to multiple inference stages. In this way, under concurrent load scenarios, the pre-filling stage inference and decoding stage inference for each user request can be processed in batches, ensuring that each user request runs on inference resources, preventing inference resources from being in a waiting or idle state, thereby improving the overall resource utilization efficiency.

[0084] Figure 5 illustrates an inference scheduling process 500 for concurrent loads under a heterogeneous architecture according to an embodiment of the present disclosure. Similar to Figure 4, as exemplarily shown in Figure 5, multiple concurrent user requests 511, 512, and 513 can be placed into a request pool 522. After selection, processing, etc., the user requests in the request pool 522 can be further fed into cluster 530 for inference. The first word or middle word generated from cluster 530 is returned to the lexical pool 521 to continue as lexical units to be computed for subsequent inference. The difference is that cluster 530 can be a heterogeneous server cluster (also known as a heterogeneous cluster architecture), in which the servers or server resources have their own advantages in inference performance. For example, some have performance advantages in handling computationally intensive tasks (such resources can be called computationally intensive resources), while others may have performance advantages in handling memory-intensive tasks (such resources can be called memory-intensive resources), and so on.

[0085] In the heterogeneous cluster architecture scenario shown in Figure 5, if there are no tokens to be computed in the token pool 521, user requests can be selected from the request pool 522 at 532 based on, for example, TTFT or other time constraints to form a request batch, and can proceed to 534 to be assigned to computationally intensive resources for inference. At 534, if the generated token is a non-terminal word (meta), the token can be added to the token pool 521 as a token to be computed to update the token pool 421.

[0086] In the case where cluster 530 is a heterogeneous cluster and there are already words to be computed in the lexicon pool 521, i.e., the current inference includes inference in the pre-filling phase and inference in the decoding phase, according to embodiments of this disclosure, at 531, words to be computed can be selected from the lexicon pool 521 based on, for example, TBT or other time constraints to form a lexicon batch, and at 532, user requests can be selected from the request pool 522 based on, for example, TTFT or other time constraints to form a request batch. The formed lexicon batch and request batch can be assigned to memory-intensive resources (i.e., at 533) and compute-intensive resources (i.e., at 534) respectively for inference (i.e., at 535). In other words, inference in the pre-filling phase is assigned to compute-intensive resources at 534 and inference in the decoding phase is assigned to memory-intensive resources at 533 and proceeds to 533 for inference. At point 536, if the generated lexical unit is a terminator, the reasoning for the user request ends; if the generated lexical unit is a non-terminal terminator, the lexical unit is placed into the lexical unit pool 521 as a lexical unit to be calculated to update the lexical unit pool 521.

[0087] According to embodiments of this disclosure, inference in the pre-filling stage can be assigned to compute-intensive resources in a heterogeneous cluster (e.g., 534 shown in Figure 5), and inference in the decoding stage can be assigned to memory-intensive resources in the heterogeneous cluster (e.g., 533 shown in Figure 5). Thus, given the differences in computational and memory access characteristics between the pre-filling and decoding stages, to optimize and balance resource scheduling, inference in the pre-filling stage of compute-intensive tasks is assigned to compute-intensive resources, and inference in the decoding stage of memory-intensive tasks is assigned to memory-intensive resources. Each stage can execute efficiently in an adapted resource environment, thereby further improving overall resource utilization efficiency.

[0088] Figure 6 illustrates another inference scheduling process 600 for concurrent loads under a heterogeneous architecture according to an embodiment of the present disclosure. As exemplarily shown in Figure 6, multiple concurrent user requests 611, 612, and 613 can be placed into a request pool 622. After selection, processing, etc., the user requests in the request pool 622 can be further fed into cluster 630 for inference. The first or middle words generated from cluster 630 are returned to the lexical pool 621 to continue as lexical units to be computed for subsequent inference. Similar to Figure 5, cluster 630 can be a heterogeneous server cluster. In the example of Figure 6, cluster 630 can include compute-intensive resources and memory-intensive resources. The difference is that the inference in the decoding phase of Figure 6 is further differentiated to be assigned to compute-intensive resources and memory-intensive resources respectively.

[0089] In the heterogeneous cluster architecture scenario shown in Figure 6, if there are no lexical units to be computed in the lexical pool 621, user requests can be selected from the request pool 622 at 631 based on, for example, TTFT or other time constraints to form a request batch. This request batch can be used for pre-filled inference at 633, which can be assigned to computationally intensive resources for execution (i.e., 635). After the pre-filled inference is completed, the generated first word will be placed into the lexical pool 621 at 636 as a lexical unit to be computed for subsequent inference.

[0090] In the case where cluster 630 is a heterogeneous cluster and there are already words to be computed in the lexicon pool 621, i.e., the current inference includes inference in the pre-filling phase and inference in the decoding phase, according to embodiments of this disclosure, at 632, words to be computed can be selected from the lexicon pool 621 based on, for example, TBT or other time constraints to form a lexicon batch, and at 631, user requests can be selected from the request pool 622 based on, for example, TTFT or other time constraints to form a request batch. This request batch can be used for pre-filling inference at 633, which can be assigned to computationally intensive resources for execution (i.e., 635). After pre-filling inference is completed, the generated first word is placed into the lexicon pool 621 at 636 as a word to be computed for subsequent inference. Furthermore, this lexicon batch can be used for inference in the decoding phase, where non-attentional computations (e.g., Multilayer Perceptron (MLP) computations) in the decoding phase (i.e., 634) are assigned to computationally intensive resources in the heterogeneous cluster (i.e., 635). Next, attention calculations during the decoding phase of inference can be performed via KV cache transfer 637 (i.e., 639). These attention calculations can be assigned to memory-intensive resources in a heterogeneous cluster (i.e., 640). At 641, if the generated term is a term ending in a word, the inference for that user request ends; if the generated term is a term not ending in a word, it is placed in term pool 621 as a term to be calculated to update the term pool 621.

[0091] Figure 7 is a schematic diagram illustrating the inference scheduling process 700 under homogeneous and heterogeneous architectures according to embodiments of the present disclosure. Figure 7 is shown to more intuitively illustrate the difference between the inference scheduling strategy according to embodiments of the present disclosure under homogeneous and heterogeneous architectures. The left side of Figure 7 schematically illustrates the inference scheduling process under a homogeneous architecture. As shown in the example on the left side of Figure 7, the largest possible batches of terms and requests, satisfying, for example, TBT and TTFT, can be selected from the request pool 702 and the term pool 704, respectively. User requests in a request batch can be segmented into user request slices based on unit processing time, for example, the unit processing time of the segmented user request slice is equal to the unit term processing time. The largest possible batch of request slices, satisfying, for example, TBT, can be selected for fusion with the term batch. In this example, one user request slice and two terms to be computed are selected as an illustrative example of a fusion batch, where the processing time of a single user slice can be equal to the processing time of a single term, for example, 0.05 s, and they satisfy TBT in the sequence length dimension, where the TTFT is, for example, 2 s, and the TBT is, for example, 0.05 seconds.

[0092] Furthermore, the right side of Figure 7 schematically illustrates the inference scheduling process under a heterogeneous architecture. In this example, the NPU suitable for computationally intensive tasks (hereinafter referred to as NPU 1) is used to perform computations in the pre-filling phase and non-attentional computations in the decoding phase, while the NPU suitable for memory-intensive tasks (hereinafter referred to as NPU 2) is used to perform attentional computations in the decoding phase. In addition, this example utilizes memory pooling technology to achieve pre-filling-decoding (PD) separation and PMV separation (further separating attentional and non-attentional computations in the decoding phase). Memory pooling technology reduces frequent memory request calls and improves the efficiency of memory allocation and deallocation.

[0093] As illustrated on the right side of Figure 7, the pre-fill pool in the memory pool can be used to cache intermediate results and data from the inference phase of the pre-filling stage for request batches or request slice batches, facilitating access and transmission in subsequent stages. In the case of PD separation, the decoding pool in the memory pool can be used to cache inference data for the decoding stage of lexical batches, such as the keys and values ​​of the attention mechanism computed in each autoregressive iteration step, and these cached values ​​can be directly accessed in subsequent iteration steps. Furthermore, in the case of PMV separation, the decoding pool in the memory pool utilizing Near-data processing (NDP) / Processing in Memory (PIM) can be used to cache the keys and values ​​of the attention mechanism for direct access later. Near-data processing reduces data transmission latency, while processing in memory improves bandwidth utilization. It should be understood that the pre-fill pool can be used in conjunction with compute-intensive resources, and the decoding pool can be used in conjunction with memory-intensive resources.

[0094] In this example of a heterogeneous server architecture, the largest possible batch of requests that satisfies, for example, a TTFT (Time-To-Value) can be selected from request pool 706, where TTFT is, for example, 2 seconds, and "T" represents a possible delay or interval. After the corresponding calculations, the generated first word can be placed into lexical pool 708. Next, non-attentional computations (e.g., MLP computations) in the decoding phase can be assigned to computationally intensive resources. After transfer via KV cache, attentional computations in the decoding phase can be assigned to memory-intensive resources, where TBT (Time-To-Value) is, for example, 0.05 seconds. In some embodiments, non-attentional computations in the decoding phase can be fused with inference in the pre-filling phase for fused inference.

[0095] Figure 8 illustrates a diagram of a pre-fill-decode (PD) fusion result 800 according to an embodiment of the present disclosure. Figure 8 exemplarily illustrates the fusion result of inference from multiple pre-fill stages and multiple decoding stages in one iteration step of the inference process. As shown in Figure 8, "L" can be a linear matrix indicating the fusion between the inference from the pre-fill stage and the inference from the decoding stage (non-attention computation), such fusion may be based on the sequence length dimension and / or the feature layer dimension. The "L" matrix can be assigned to a computationally intensive resource (e.g., NPU 1 above) with computational performance advantages for inference computation. Furthermore, "A" in the dashed box can indicate a matrix of attention computations that failed to be fused due to different sequence length layer dimensions. The "A" matrix can be assigned to a memory-intensive resource (e.g., NPU 2 above) with memory access performance advantages for inference computation.

[0096] Figure 9 illustrates an apparatus 900 for inference scheduling according to some embodiments of the present disclosure. Each module in the apparatus 900 can be configured to perform corresponding steps in a method for inference scheduling according to embodiments of the present disclosure. As shown in Figure 9, the apparatus 900 for inference scheduling includes a batch formation module 910 and a batch feeding module 920. According to embodiments of the present disclosure, the apparatus 900 includes a batch formation module 910 configured to, in response to the existence of lexicals to be computed in a lexical pool, select a first number of lexicals from the lexical pool based on a first time threshold to form a first batch, and select a second number of user requests from a request pool based on a second time threshold to form a second batch, the lexical pool including lexicals generated by inference on multiple user requests, and the request pool including multiple user requests. The apparatus 900 also includes a batch feeding module 920 configured to feed the first batch and the second batch into at least one server in a cluster for inference. This enables more reasonable and efficient allocation of inference resources under concurrent load scenarios, thereby improving resource utilization and inference throughput, bandwidth utilization, etc.

[0097] In some embodiments, the apparatus 900 further includes a second batch forming module configured to select a third number of user requests from the request pool to form a third batch in response to the absence of any lexical units to be computed in the lexical pool, based on a second time threshold. The apparatus 900 also includes a second batch feeding module configured to feed the third batch into at least one server in the cluster for inference. Thus, a reasonable batch size that satisfies time constraints (e.g., time constraints in the SLA) can be selected from the load input, thereby achieving optimized inference scheduling.

[0098] In some embodiments, in response to the current number of lexical units included in the lexical unit pool being greater than or equal to the maximum number of lexical units that satisfy a first time threshold, the first number of lexical units is the maximum number of lexical units; or in response to the current number of lexical units included in the lexical unit pool being less than the maximum number of lexical units that satisfy the first time threshold, the first number of lexical units is the current number of lexical units. Thus, the batch size of lexical units can be dynamically determined based on the number of lexical units in the lexical unit pool.

[0099] In some embodiments, in response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests that satisfy the second time threshold, the second number of user requests is the maximum number of user requests; or in response to the current number of user requests included in the request pool being less than the maximum number of user requests that satisfy the second time threshold, the second number of user requests is the current number of user requests. Thus, the request batch size can be dynamically determined based on the number of user requests in the request pool.

[0100] In some embodiments, in response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests that satisfy the second time threshold, the third number of user requests is the maximum number of user requests; or in response to the current number of user requests included in the request pool being less than the maximum number of user requests that satisfy the second time threshold, the third number of user requests is the current number of user requests. Thus, the request batch size can be dynamically determined based on the number of user requests in the request pool.

[0101] In some embodiments, the second time threshold is greater than the first time threshold.

[0102] In some embodiments, the cluster includes a homogeneous cluster. The device 900 further includes: a request segmentation module configured to segment user requests in the second batch into user request slices based on unit token processing time; a third batch formation module configured to select a fourth number of user request slices based on a first time threshold to form a fourth batch; and a batch fusion module configured to fuse the first batch and the fourth batch to obtain a fused batch. The device 900 also includes an assignment module configured to assign the inference of the fused batch to the inference resources based on the computing power of the inference resources in the homogeneous cluster. Thus, for concurrent load scenarios, the inference stages of multiple requests can be fused, and the inference can be reasonably assigned to the corresponding inference resources based on computing power, ensuring that the inference tasks of each request are running while avoiding waiting and idle inference resources, thereby improving the overall resource utilization.

[0103] In some embodiments, if there are no lexes to be computed in the lexe pool, the current inference includes inference in the pre-filling stage; or if there are lexes to be computed in the lexe pool, the current inference includes inference in the pre-filling stage and inference in the decoding stage.

[0104] In some embodiments, the cluster includes a heterogeneous cluster, and the apparatus 900 further includes an assignment module configured to assign the inference of the pre-filling phase to compute-intensive resources in the heterogeneous cluster, and to assign the inference of the decoding phase to memory-intensive resources in the heterogeneous cluster. Thus, taking into account the computational and memory access differences between the pre-filling and decoding phases, assigning the inference of the pre-filling phase of compute-intensive tasks to compute-intensive resources, and assigning the inference of the decoding phase of memory-intensive tasks to memory-intensive resources, optimizes and balances resource scheduling.

[0105] In some embodiments, the cluster includes a heterogeneous cluster, and the apparatus 900 further includes an assignment module configured to assign non-attention computations in the inference of the pre-filling phase and the inference of the decoding phase to compute-intensive resources in the heterogeneous cluster, and to assign attention computations in the inference of the decoding phase to memory-intensive resources in the heterogeneous cluster. Thus, taking into account the computational and memory access differences within the decoding phase itself, assigning non-attention computations to compute-intensive resources and attention computations to memory-intensive resources further optimizes and balances resource scheduling.

[0106] In some embodiments, the apparatus 900 further includes a lexical pool update module configured to update the lexical pool based on non-terminal words in the inference results for generating subsequent lexical units.

[0107] In some embodiments, the first time threshold is the time between tokens (TBT); or the second time threshold is the time to generate the first token (TTFT).

[0108] As an example of a software functional unit, the batch forming module 910 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the batch forming module 910 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0109] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0110] As an example of a hardware functional unit, the batch forming module 910 may include at least one computing device, such as a server. Alternatively, the batch forming module 910 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0111] The batch forming module 910 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the batch forming module 910 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the batch forming module 910 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0112] It should be noted that, in other embodiments, the batch forming module 910 can be used to perform any step in the method for inference scheduling according to embodiments of the present disclosure, and the batch feeding module 920 can be used to perform any step in the method for inference scheduling according to embodiments of the present disclosure. Furthermore, the aforementioned second batch forming module, second batch feeding module, request segmentation module, third batch forming module, batch fusion module, assignment module, and lexical pool update module can also be used to perform any step in the method for inference scheduling according to embodiments of the present disclosure.

[0113] The steps implemented by the batch forming module 910, batch feeding module 920, second batch forming module, second batch feeding module, request segmentation module, third batch forming module, batch fusion module, assignment module, and word pool update module can be specified as needed. The batch forming module 910, batch feeding module 920, second batch forming module, second batch feeding module, request segmentation module, third batch forming module, batch fusion module, assignment module, and word pool update module respectively implement different steps in the method for inference scheduling according to the embodiments of this disclosure to realize all the functions of the apparatus 900 for inference scheduling.

[0114] This disclosure also provides a computing device 1000. FIG10 schematically illustrates a schematic block diagram of an example computing device 1000 that can be used to implement exemplary implementations of this disclosure. The computing device 1000 can be used to perform a method for inference scheduling according to embodiments of this disclosure. As shown in FIG10, the computing device 1000 includes: a bus 1002, a processor 1004, a memory 1006, and a communication interface 1008. The processor 1004, the memory 1006, and the communication interface 1008 communicate with each other via the bus 1002. The computing device 1000 can be a server or a terminal device. It should be understood that this disclosure does not limit the number of processors and memories in the computing device 1000.

[0115] Bus 1002 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 10, but this does not imply that there is only one bus or one type of bus. Bus 1002 can include pathways for transmitting information between various components of computing device 1000 (e.g., memory 1006, processor 1004, communication interface 1008).

[0116] The processor 1004 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0117] The memory 1006 may include volatile memory, such as random access memory (RAM). The processor 1004 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0118] The memory 1006 stores executable program code, and the processor 1004 executes the executable program code to implement the functions of the aforementioned batch forming module 910, batch feeding module 920, etc., thereby realizing the inference scheduling method according to the embodiments of this disclosure. That is, the memory 1006 stores instructions for executing the inference scheduling method according to the embodiments of this disclosure.

[0119] Alternatively, the memory 1006 may store executable code, which the processor 1004 executes to implement the functions of the aforementioned inference scheduling apparatus 1100, thereby implementing the inference scheduling method according to embodiments of the present disclosure. That is, the memory 1006 stores instructions for executing the inference scheduling method according to embodiments of the present disclosure.

[0120] The communication interface 1008 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1000 and other devices or communication networks.

[0121] This disclosure also provides a computing device cluster. Figure 11 shows a schematic block diagram of an example computing device cluster 1100 that can be used to implement exemplary implementations of this disclosure. The computing device cluster 1100 enables the inference scheduling method provided in this disclosure to be executed in a distributed manner by multiple servers in the cluster. The computing device cluster 1100 includes at least one computing device. The computing device may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0122] As shown in FIG11, the computing device cluster 1100 includes at least one computing device 1000. The memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may store the same instructions for executing the method for inference scheduling according to embodiments of the present disclosure.

[0123] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may also store portions of the instructions for executing the inference scheduling method according to embodiments of the present disclosure. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for executing the inference scheduling method according to embodiments of the present disclosure.

[0124] It should be noted that the memory 1006 in different computing devices 1000 within the computing device cluster 1100 can store different instructions, which are used to execute certain functions of the device 1100 for inference scheduling. That is, the instructions stored in the memory 1006 of different computing devices 1000 can implement the functions of one or more modules such as the batch forming module 910 and the batch feeding module 920.

[0125] In some possible implementations, one or more computing devices in the computing device cluster 1100 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 12 illustrates one possible implementation. As shown in Figure 12, two computing devices 1000A and 1000B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1006 in computing device 1000A stores instructions for executing the functions of the batch forming module 910. Simultaneously, the memory 1006 in computing device 1000B stores instructions for executing the functions of the batch feeding module 920.

[0126] The connection method between the computing device cluster 1100 shown in Figure 12 may be based on the fact that the inference scheduling method provided in this disclosure may be for distributed storage devices, and therefore considers that the functions implemented by the batch feed module 920 are executed by the computing device 1000B.

[0127] It should be understood that the functions of computing device 1000A shown in Figure 12 can also be performed by multiple computing devices 1000. Similarly, the functions of computing device 1000B can also be performed by multiple computing devices 1000.

[0128] This disclosure also provides another computing device cluster 1100. The connection relationships between the computing devices in this computing device cluster 1100 can be similarly referred to in FIG12 and the connection method of the computing device cluster 1100 described in FIG12. The difference is that the memory 1006 in one or more computing devices 1000 in this computing device cluster 1100 may store the same instructions for executing the inference scheduling method according to the embodiments of this disclosure.

[0129] In some possible implementations, the memory 1006 of one or more computing devices 1000 in the computing device cluster 1100 may also store portions of the instructions for executing the inference scheduling method according to embodiments of the present disclosure. In other words, a combination of one or more computing devices 1000 can jointly execute the instructions for executing the inference scheduling method according to embodiments of the present disclosure.

[0130] This disclosure also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform a method for inference scheduling according to embodiments of this disclosure.

[0131] This disclosure also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a method for inference scheduling according to embodiments of this disclosure.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for inference scheduling, comprising: In response to the existence of lexical units to be calculated in the lexical unit pool, a first number of lexical units are selected from the lexical unit pool based on a first time threshold to form a first batch, and a second number of user requests are selected from the request pool based on a second time threshold to form a second batch, wherein the lexical unit pool includes lexical units generated by reasoning from multiple user requests, and the request pool includes multiple user requests. as well as The first and second batches are fed into at least one server in the cluster for inference.

2. The method according to claim 1, further comprising: In response to the absence of any lexical to be computed in the lexical pool, a third number of user requests are selected from the request pool based on the second time threshold to form a third batch; as well as The third batch is fed into at least one server in the cluster for inference.

3. The method according to claim 1, wherein: In response to the current number of lexical units included in the lexical pool being greater than or equal to the maximum number of lexical units that satisfy the first time threshold, where the first number of lexical units is the maximum number of lexical units, or In response to the current number of lexical units included in the lexical pool being less than the maximum number of lexical units that satisfy the first time threshold, the first number of lexical units is the current number of lexical units.

4. The method according to claim 1, wherein: In response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests that satisfy the second time threshold, where the second number of user requests is the maximum number of user requests, or In response to the current number of user requests included in the request pool being less than the maximum number of user requests that satisfy the second time threshold, the second number of user requests being the current number of user requests.

5. The method according to claim 2, wherein: In response to the current number of user requests included in the request pool being greater than or equal to the maximum number of user requests that satisfy the second time threshold, the third number of user requests is the maximum number of user requests, or In response to the current number of user requests included in the request pool being less than the maximum number of user requests that satisfy the second time threshold, the third number of user requests is the current number of user requests.

6. The method according to claim 1, wherein: The second time threshold is greater than the first time threshold.

7. The method of claim 1, wherein the cluster comprises a homogeneous cluster, and the method further comprises: Based on the unit word processing time, the user requests in the second batch are divided into user request slices; A fourth number of user request slices are selected based on the first time threshold to form a fourth batch; as well as The first batch and the fourth batch are merged to obtain a merged batch, and The reasoning includes: Based on the computing power of the inference resources in the homogeneous cluster, the inference of the fused batch is assigned to the inference resources.

8. The method according to claim 1, wherein: If the lexicon pool already contains lexicons to be computed, the current inference process includes inference in the pre-filling phase and inference in the decoding phase.

9. The method of claim 8, wherein the cluster comprises a heterogeneous cluster, and wherein the inference comprises: The inference in the pre-filling phase is assigned to compute-intensive resources in the heterogeneous cluster, and the inference in the decoding phase is assigned to memory-intensive resources in the heterogeneous cluster.

10. The method of claim 8, wherein the cluster comprises a heterogeneous cluster, and wherein the inference comprises: The non-attention computations in the inference of the pre-filling phase and the inference of the decoding phase are assigned to the compute-intensive resources in the heterogeneous cluster, and the attention computations in the inference of the decoding phase are assigned to the memory-intensive resources in the heterogeneous cluster.

11. The method according to claim 1, further comprising: The lexical pool is updated based on the non-ending words in the reasoning results to generate subsequent lexical units.

12. The method according to any one of claims 1-5, wherein: The first time threshold is the time taken between word segments (TBT); or The second time threshold is the first word generation time (TTFT).

13. An apparatus for inference scheduling, comprising: A batch forming module is configured to, in response to the existence of lexical units to be calculated in the lexical unit pool, select a first number of lexical units from the lexical unit pool based on a first time threshold to form a first batch, and select a second number of user requests from the request pool based on a second time threshold to form a second batch, wherein the lexical unit pool includes lexical units generated by reasoning from multiple user requests, and the request pool includes multiple user requests. A batch feed module is configured to feed the first batch and the second batch into at least one server in the cluster for inference.

14. A computing device cluster comprising at least one computing device, each computing device including a processor and a memory, wherein the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that, The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform the method according to any one of claims 1 to 12.

Images