Model allocation method and management platform

Abstract

The present application provides a model allocation method, a management platform, and a system. The method comprises: allocating a plurality of models to a plurality of computing nodes, wherein the plurality of models are in one-to-one correspondence to a plurality of tasks, at least one model among the plurality of models is allocated to each computing node among the plurality of computing nodes, and the computing node is used for using the allocated model to execute a task corresponding to the model; acquiring execution quality of the task corresponding to the allocated model and executed by each computing node among the plurality of computing nodes, to obtain service quality of the plurality of tasks; and reallocating the plurality of models to the plurality of computing nodes on the basis of the service quality of the plurality of tasks, wherein at least one model among the plurality of models is reallocated to each computing node among the plurality of computing nodes, and the computing node is used for using the reallocated model to execute a task corresponding to the model. In the method, the model allocated to the computing node can be continuously adjusted, so as to ensure load balance among computing nodes, thereby improving the service quality of the tasks and reducing waste of resources.

Description

A model allocation method and management platform

[0001] This application claims priority to Chinese Patent Application No. 202510125886.0, filed on January 26, 2025, entitled "A Model Allocation Method and Management Platform", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model allocation method and management platform. Background Technology

[0003] Due to the outstanding performance of AI models such as large language models (LLMs), AI models have been applied to various tasks, including chatbot dialogue, code generation, text processing, and embodied intelligence. Typically, multiple models are used to handle different tasks, with different models handling different types of tasks. In cloud technology, to improve resource utilization, these multiple models are usually distributed across multiple computing nodes. The computing nodes then use their assigned models to execute the tasks they are meant to process.

[0004] The workload of a task may change continuously, and the load on the computing node where the corresponding model resides will also change accordingly, leading to load imbalance among multiple computing nodes. Current allocation schemes that distribute multiple models to multiple computing nodes cannot guarantee load balancing among the nodes, resulting in poor task performance or wasted resources. Summary of the Invention

[0005] This application provides a model allocation method and management platform that can continuously adjust the models allocated to computing nodes to ensure load balancing among computing nodes, thereby improving the service quality of tasks and reducing resource waste.

[0006] Firstly, a model allocation method executable by a management platform is provided. This management platform manages multiple computing nodes in an infrastructure. For example, the computing nodes among these multiple computing nodes can be one of a neural network processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field-programmable gate array (FPGA), etc., and the multiple computing nodes can be composed of any one or more of GPUs, NPUs, TPUs, and FPGAs.

[0007] The infrastructure may include at least one data center, and each data center includes multiple servers. The servers in each data center constitute the servers in the infrastructure. The multiple compute nodes are deployed on at least one server in the infrastructure.

[0008] The method may include the following steps.

[0009] First, the management platform can assign multiple models to multiple computing nodes. Each model corresponds one-to-one with a different task, and each computing node is assigned at least one of the models. The computing node then uses its assigned model to execute the corresponding task. For example, these multiple models can be large models, such as large language models or large visual models.

[0010] After assigning multiple models to multiple computing nodes, the management platform can detect the quality of each computing node executing the tasks corresponding to its assigned models, thus obtaining the service quality of each task. By analyzing the quality of each computing node's execution of its assigned tasks, the service quality of various tasks can be obtained. For example, the service quality of various tasks can be the sum or average of the quality of each computing node's execution of its assigned tasks.

[0011] Next, the management platform can reassign multiple models to multiple computing nodes based on the service quality of various tasks; wherein each computing node is reassigned to at least one model among the multiple models, and the computing node is used to execute the task corresponding to the model using the reassigned model.

[0012] The quality of service (GHS) of a task reflects the load on a computing node, and the GHS of multiple tasks reflects whether there is a load imbalance among multiple computing nodes. Redistributing multiple models across multiple computing nodes based on the GHS of various tasks can ensure load balancing among these nodes, thereby improving the GHS of multiple tasks and reducing resource waste.

[0013] In one possible implementation, the multiple models may include a first model. A redistribution scheme that reallocates multiple models to multiple computing nodes based on the service quality of multiple tasks may include: confirming that the service quality of the task corresponding to the first model among the multiple tasks is lower than the service quality of other tasks among the multiple tasks, and confirming that the available resources of the first computing node among the multiple computing nodes are greater than the available resources of other computing nodes among the multiple computing nodes; and assigning the first model to the first computing node.

[0014] In this implementation, models with low service quality are assigned to computing nodes with abundant available resources. By redistributing multiple models to multiple computing nodes, the appropriate computing nodes can be assigned to them efficiently, thereby achieving load balancing among multiple computing nodes.

[0015] In one possible implementation, the scheme for confirming that the available resources of the first computing node are greater than the available resources of other computing nodes may include: obtaining the request frequency of the tasks corresponding to the models allocated to each of the multiple computing nodes; when the request frequency of the tasks corresponding to the models allocated to the first computing node is less than the request frequency of the tasks allocated to other nodes, confirming that the available resources of the first computing node are greater than the available resources of other computing nodes.

[0016] In this implementation, the request frequency of the tasks corresponding to the models that have been allocated to the computing nodes can be used to efficiently identify the available resources of the computing nodes, thereby improving the efficiency of reallocation and enabling multiple computing nodes to achieve load balancing as soon as possible.

[0017] In one possible implementation, the service quality of each task can be predicted based on historical execution information, resulting in a predicted service quality for multiple tasks. Furthermore, the actual service quality of each task during execution can be measured, yielding a measured service quality for the multiple tasks. The cycle length of the model's redistribution period can be adjusted based on both the predicted and measured service qualities of the multiple tasks.

[0018] The greater the gap between the expected service quality of multiple tasks and the actual service quality of multiple tasks, the shorter the cycle length of the model redistribution period, so as to redistribute multiple models to multiple computing nodes as soon as possible and ensure load balancing among multiple computing nodes.

[0019] The smaller the gap between the expected service quality of multiple tasks and the measured service quality of multiple tasks, the longer the cycle length of the model redistribution period should be, so as to reduce the frequency of redistribution of multiple models to multiple computing nodes and reduce the resource overhead caused by redistribution.

[0020] In one possible implementation, at least two of the multiple models are simultaneously assigned to a second computing node among the multiple computing nodes; the method further includes: configuring a priority for each of the at least two models based on the expected execution time of the task corresponding to each of the at least two models; wherein the expected execution time of the task corresponding to the model is negatively correlated with the priority of the model; and instructing the second computing node to execute the task corresponding to the model among the at least two models according to the priority of the model among the at least two models.

[0021] In this implementation, computing nodes can prioritize executing tasks with shorter execution times, and then execute tasks with longer execution times. This reduces the overall latency of various tasks and improves the overall service quality for all tasks.

[0022] In one possible implementation, the multiple models include a second model, and the method further includes: reducing the priority of the second model when the actual time required for the second computing node to execute the task corresponding to the second model is greater than the expected execution time of the task.

[0023] If the actual execution time of a task exceeds its estimated execution time, it means the task will not be completed within the estimated timeframe. In this case, the task is considered to have poor service quality and should be abandoned. Lowering the priority of this task in this situation can ensure the service quality of other tasks, thereby guaranteeing the overall service quality of multiple tasks.

[0024] In one possible implementation, the required unit storage space for different models in the multiple models is different; the method further includes: obtaining a unified storage space based on the required unit storage space for different models in the multiple models; wherein the size of the unified storage space is an integer multiple of the size of the unit storage space required by each model in the multiple models; instructing each computing node in the multiple computing nodes to execute the task corresponding to the model allocated to the computing node using the unified storage space as the unit storage space.

[0025] In this implementation, the storage space can be unified as a unit storage space, which can meet the unit storage space requirements of different models, enabling computing nodes to execute different models, thereby ensuring the implementation of the redistribution of multiple models to multiple computing nodes.

[0026] Secondly, a management platform is provided for managing multiple computing nodes in an infrastructure. The infrastructure includes at least one data center, each of which includes multiple servers, and the multiple computing nodes are deployed on at least one server within the infrastructure. The management platform includes: an allocation module for allocating multiple models to the multiple computing nodes; wherein each model corresponds one-to-one with multiple tasks, and each computing node is allocated to at least one model among the multiple models, and the computing node executes the task corresponding to the allocated model; an acquisition module for acquiring the quality of service of each computing node executing the task corresponding to the allocated model, thus obtaining the service quality of the multiple tasks; and a reassignment module for reassigning the multiple models to the multiple computing nodes based on the service quality of the multiple tasks; wherein each computing node is reassigned to at least one model among the multiple models, and the computing node executes the task corresponding to the reassigned model.

[0027] In one possible implementation, the multiple models include a first model; the reallocation module is used to: confirm that the service quality of the task corresponding to the first model is lower than the service quality of other tasks among the multiple tasks, and confirm that the available resources of the first computing node are greater than the available resources of other computing nodes among the multiple computing nodes; wherein, the other tasks are tasks other than the task corresponding to the first model among the multiple tasks, and the other computing nodes are computing nodes other than the first computing node among the multiple computing nodes; and allocate the first model to the first computing node.

[0028] In one possible implementation, the reallocation module is used to: obtain the request frequency of the tasks corresponding to the models allocated to each of the multiple computing nodes; when the request frequency of the tasks corresponding to the models allocated to the first computing node is less than the request frequency of the tasks corresponding to the models allocated to other computing nodes, confirm that the available resources of the first computing node are greater than the available resources of other computing nodes.

[0029] In one possible implementation, the service quality of multiple tasks includes the expected service quality and the measured service quality of multiple tasks; the reassignment module is used to: obtain the model reassignment period based on the gap between the expected service quality and the measured service quality; wherein the length of the model reassignment period is negatively correlated with the gap; and reassign multiple models to multiple computing nodes according to the model reassignment period.

[0030] In one possible implementation, at least two of the multiple models are simultaneously assigned to a second computing node among the multiple computing nodes; the acquisition module is further configured to: configure a priority for each of the at least two models based on the expected execution time of the task corresponding to each of the at least two models; wherein the expected execution time of the task corresponding to the model is negatively correlated with the priority of the model; and instruct the second computing node to execute the task corresponding to the model among the at least two models according to the priority of the model among the at least two models.

[0031] In one possible implementation, the multiple models include a second model, and the acquisition module is further configured to: reduce the priority of the second model when the actual time required for the second computing node to execute the task corresponding to the second model is greater than the expected execution time of the task.

[0032] In one possible implementation, the required unit storage space for different models in the multiple models is different; the acquisition module is also used to: obtain a unified storage space based on the required unit storage space for different models in the multiple models; wherein the size of the unified storage space is an integer multiple of the size of the unit storage space required by each model in the multiple models; and instruct each computing node in the multiple computing nodes to execute the task corresponding to the model allocated to the computing node using the unified storage space as the unit storage space.

[0033] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, such that the computing device cluster performs the method provided in the first aspect.

[0034] Fourthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, perform the method as provided in the first aspect.

[0035] Fifthly, a computer program product containing instructions is provided, which, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method provided in the first aspect.

[0036] Sixthly, a cloud system is provided, comprising a management platform and infrastructure; wherein the infrastructure includes at least one data center, each of the at least one data center includes multiple servers, and multiple computing nodes are deployed in the at least one server in the infrastructure; the management platform is used to: allocate multiple models to multiple computing nodes; wherein the multiple models correspond one-to-one with multiple tasks, each of the multiple computing nodes is allocated to at least one of the multiple models, and the computing nodes are used to execute the tasks corresponding to the allocated models; obtain the quality of each computing node in executing the tasks corresponding to the allocated models, thereby obtaining the service quality of multiple tasks; and based on the service quality of multiple tasks, reassign the multiple models to the multiple computing nodes; wherein each of the multiple computing nodes is reassigned to at least one of the multiple models, and the computing nodes are used to execute the tasks corresponding to the reassigned models.

[0037] The beneficial effects of aspects two through six can be found in the above description of the beneficial effects of aspect one, and will not be repeated here. Attached Figure Description

[0038] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0039] Figure 2 is a schematic diagram of a management platform provided in an embodiment of this application;

[0040] Figure 3 is a schematic diagram of a model allocation method provided in an embodiment of this application;

[0041] Figure 4 is a flowchart of a model allocation method provided in an embodiment of this application;

[0042] Figure 5 is a schematic diagram of the latency of different model running strategies;

[0043] Figure 6 is a flowchart of a model allocation method provided in an embodiment of this application;

[0044] Figure 7 is a schematic diagram of the structure of a management platform provided in an embodiment of this application;

[0045] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0046] Figure 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0047] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0048] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" refers to two or more objects, and "various types" refers to two or more types. Terms such as "first," "second," etc., are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0049] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.

[0050] Model: Composed of multiple neural network layers. Typically, these layers are arranged sequentially, with the output of one layer serving as the input to the next. A neural network layer can be simply referred to as a network layer or layer. Different models may differ in one or more of the following: model architecture, model size, and unit storage space.

[0051] Model architecture: also known as model structure, includes the number of layers in the model, the number of neurons in each layer, activation functions, etc.

[0052] Model size: refers to the size of the model, including the model's parameter scale, etc.

[0053] Unit storage space: refers to the unit of storage space required to store data related to the model; it is indivisible storage space. The unit storage space is usually determined by the model's structure. This storage can be cache, memory, or persistent storage.

[0054] Key-value cache: refers to the storage space used to cache key (K) matrices and value (V) matrices. A key-value cache can consist of multiple cache blocks, where a cache block is the unit of storage space used for caching.

[0055] The key (K) matrix is ​​a matrix representing the input features in machine learning algorithms, used to compare similarity with the query (Q) vector.

[0056] Value (V) matrix: In machine learning algorithms, this is a matrix representing the actual information of the input. By weighting the value matrix using similarity, the output, i.e., the model's computational result, can be obtained.

[0057] Large AI models, often simply called large models, refer to "large parameter" models trained using massive amounts of data and powerful computing capabilities. These models typically possess high versatility and generalization ability, and can be applied to fields such as natural language processing, image recognition, and speech recognition. Common large models include large language models (LLMs) and visual large models (VLMs).

[0058] Large language models (LLMs) are language models trained using deep learning algorithms and possessing a large number of parameters. Compared to ordinary language models, LLMs have stronger language processing capabilities and wider applicability, enabling them to handle more complex and abstract language tasks, such as question answering and text generation.

[0059] Visual large models (VLMs) are deep learning-based machine learning models designed to process and analyze large-scale image and video data. VLMs simulate the human brain's image processing by constructing deep neural networks, using multiple layers of neurons to process input data layer by layer, extracting increasingly abstract feature representations. The principle behind VLMs is to use deep learning techniques and neural networks with a large number of parameters to analyze and understand visual content.

[0060] Model reasoning refers to the process of using a trained model to handle tasks, such as analyzing and predicting data to generate new content or classifications.

[0061] Parallelism refers to distributed computation of the model, dividing it into multiple model blocks and assigning these blocks to multiple computing nodes. Each computing node is assigned one model block, and each node performs the computations required for its assigned block. The number of model blocks in this process is called the parallelism factor.

[0062] Tensor parallelism (TP) is a distributed model computing technique that involves splitting the tensor parameters of a model along a specific dimension to obtain multiple model blocks, and then distributing these blocks across multiple computing nodes for parallel computation to accelerate model computation.

[0063] Pipeline parallelism (PP) is a distributed model computing technique where different computing nodes run different layers of the model, representing inter-layer parallelism. In other words, pipeline parallelism divides the model at the layer level, resulting in multiple interconnected model blocks. During model computation (e.g., model inference), the related calculations for different model blocks are performed by different computing nodes.

[0064] Data parallelism (DP) is a distributed model computing technique that achieves distributed model computation through a parallel computing paradigm. In data parallelism, each of multiple computing nodes deploys a model, and different computing nodes execute model computations for different input data, thereby breaking down the model computation task into multiple smaller, independent, parallel subtasks.

[0065] Sequence parallelism (SP) is a distributed computing technique for processing extremely long texts (such as 64k or 128k texts), designed to extend the ability of models to handle long texts. It improves computational efficiency and model performance by segmenting long texts along a sequence dimension and processing them in parallel.

[0066] Model block: A neural network block is obtained by dividing a model. A model can be divided into multiple model blocks.

[0067] A task refers to the business that the model is meant to accomplish. The execution of a task includes the computational process from model input to model output. Common tasks include: chatbot dialogue, code generation, text processing, and embodied intelligence.

[0068] A request, also known as a task request, is used to request the model to process a task. One task request corresponds to one task, and is used to request the model to process that task. The number of requests reflects the workload. The more requests, the larger the workload, and the heavier the load on the computing nodes executing the task.

[0069] Request frequency refers to the frequency at which task requests arrive, which is the number of requests that need to be processed per unit of time. Request frequency reflects the workload of a task per unit of time. The higher the request frequency, the more requests need to be processed per unit of time, the larger the workload, and the heavier the load on the computing nodes executing the task. In the following description, the request frequency for a model allocation period is obtained by dividing the total number of requests that need to be processed within a model allocation period by the period length of that model allocation period.

[0070] Quality of Service (QoS) refers to the quality of a model's task processing, or in other words, the quality of task execution by the computing node where the model resides. It is an indicator used to reflect the quality of the task execution process or its results. For example, QoS can be represented by latency or unallocated index.

[0071] Service level objective (SLO) is a key metric for average service quality, used to measure service reliability and performance, and ensure user satisfaction. A common SLO is the latency objective, which is the allowable latency.

[0072] Latency: The ratio between the actual execution time of a task and its theoretical execution time. The theoretical execution time refers to the time required to execute the task under ideal conditions (e.g., sufficient computing node resources).

[0073] Normalized latency: This refers to the sum of the latencies of all requests over a certain period of time, divided by the average latency of the task. The average latency of the task is the average of the latencies of all requests historically associated with that task.

[0074] The unallocated index is calculated by adding the number of unexecuted requests, the number of executed requests that do not meet service level objectives, and a multiple of the normalized latency of executed requests, where the multiple is less than 1. In some embodiments, the unallocated index is calculated using the following formula: Where s is the model, R s For all requests of the task corresponding to model s, I is a preset index, and L is a variable. r SLO is the delay for request r. r Let α be the service level objective (SLO) for request r, where α is a coefficient ranging from 0 to 1. This represents the normalized latency of the task corresponding to model s. Since using only normalized latency as the unallocated index might result in models with insufficient resources processing too few requests, leading to falsely low normalized latency that fails to reflect true service quality, the unallocated index now considers the number of requests for which the SLO (Service Level Order) is not met (i.e., latency is greater than the target latency).

[0075] Actual service quality: refers to the service quality that is actually detected.

[0076] Expected service quality (SQS): Also known as projected service quality, this refers to the service quality calculated based on relevant information. Expectations are not always entirely accurate; therefore, expected service quality may not equal actual service quality.

[0077] Iteration time: refers to the time required for a model such as LLM to generate and output a token.

[0078] Model allocation cycle: Consists of the execution time of a model allocation and the interval between two adjacent model allocation execution times. The start time of a model allocation cycle is the execution time of the preceding model allocation in two separate allocations, and the cycle length is the interval between two adjacent model allocation execution times.

[0079] Computational nodes, often simply referred to as nodes, are nodes that possess computing resources and are capable of performing model computations (such as model inference). Common computing nodes include neural network processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), and field-programmable gate arrays (FPGAs).

[0080] Computing resources, often simply referred to as resources, refer to the hardware, software, and data that support program execution and enable related functions. For example, resources can include processor resources, storage resources, and network resources. Processor resources can include a central processing unit (CPU) or a GPU. Storage resources can include memory and video memory. Network resources can include a network interface controller (NIC) and bandwidth.

[0081] Available computing resources: also known as idle computing resources or computing resource reserves, refer to unused computing resources.

[0082] Management platform: A platform provided by the resource provider for interaction with users. Users can register accounts on the management platform and rent resources in the infrastructure managed by the platform. For example, users can deploy models in the infrastructure through the management platform. In cloud technology, the management platform can be called a cloud management platform.

[0083] Infrastructure: refers to the facilities that provide resources for computing services. Infrastructure may include at least one data center, each data center comprising multiple servers, where each server can provide one or more computing nodes.

[0084] To improve infrastructure resource utilization and reduce costs, multiple models for handling different types of tasks are typically deployed within the same infrastructure. Since different types of tasks have varying complexities, the resources required to process them may differ, and the model structures and sizes used for different tasks may also vary, differentiated allocation of models for different tasks is necessary. To improve task performance and resource utilization, the allocation scheme for distributing multiple models to computing nodes within the infrastructure needs to be optimized.

[0085] The allocation optimization schemes in related technologies are static optimization schemes. Typically, allocation strategies are configured according to the resources required by the model to process the task, the model structure, and the model size. After multiple models are allocated to multiple computing nodes according to the allocation strategy, the computing nodes where the models reside are generally not adjusted.

[0086] The request frequency of a task may vary over time, with different tasks having different peak request periods, and one task's peak request period potentially being another task's low request period. Tasks with peak request frequencies require higher computing resources, and the computing nodes hosting the corresponding models may lack sufficient resources, resulting in lower service quality. Conversely, tasks with low request frequencies have lower computing resource requirements, and the computing nodes hosting the corresponding models may have idle resources, leading to resource waste.

[0087] Furthermore, to improve the resource utilization of computing nodes, two or more models may be assigned to the same computing node. In related technologies, computing nodes execute tasks corresponding to two or more models according to a first-come, first-served (FCFS) or round-robin strategy. Some tasks are highly complex and require longer execution times, while others are less complex and require shorter execution times. When tasks of different complexities arrive simultaneously, the computing node may execute the more complex tasks first, while the less complex tasks may have to wait longer, which affects the overall performance of various tasks.

[0088] This application provides a model allocation method. This method can allocate multiple models to multiple computing nodes. The computing nodes can then use the allocated models to execute the tasks corresponding to those models. The method can obtain the quality of service (QoS) of the multiple computing nodes executing the tasks corresponding to the models, and reassign the multiple models to the multiple computing nodes according to their QoS. In this way, dynamic adjustment of the models allocated to computing nodes is achieved, ensuring load balancing among the multiple computing nodes, guaranteeing task QoS while avoiding resource waste.

[0089] Next, the model allocation method provided in the embodiments of this application will be described.

[0090] Figure 1 illustrates a system architecture that can be used to implement this method. This system architecture may include a management platform 100 and infrastructure 200.

[0091] Infrastructure 200 includes multiple computing nodes, such as computing node 210, computing node 220, and computing node 230. Each of these computing nodes can be used to deploy a model and execute the tasks corresponding to that model. Infrastructure 200 may include at least one data center, and each data center may contain multiple servers. The servers deployed in each data center may be referred to as servers in infrastructure 200. The multiple computing nodes may be deployed within at least one server in infrastructure 200.

[0092] The management platform 100 can manage the multiple computing nodes, for example, by assigning multiple models to the multiple computing nodes according to a model allocation method, so as to utilize the multiple computing nodes to execute the tasks corresponding to the multiple models. In some embodiments, the management platform 100 can be connected to the infrastructure 200. In some embodiments, the management platform 100 can be located within the infrastructure 200, for example, in one or more data centers of the infrastructure 200.

[0093] In some embodiments, the multiple models may be of the same type, for example, all of the multiple models belong to LLM. In some embodiments, the multiple models may include different types of models, for example, at least one of the multiple models may be LLM, and at least another model may be VLM.

[0094] In some embodiments, as shown in FIG2, the management platform 100 may include a global manager 110. The global manager 110 is used to allocate execution models and distribute task requests. To implement the aforementioned functions, the global manager 110 may include an allocation module 111, an acquisition module 112, and a request dispatcher module 113.

[0095] The allocation module 111, also known as the placement controller, is used to generate allocation schemes for multiple models. Each model corresponds one-to-one with a different task.

[0096] The allocation module 111 can allocate multiple models to multiple computing nodes according to the initial allocation scheme. For example, it can allocate model 310 to computing node 210, model 320 to computing node 220, and model 330 to computing node 230.

[0097] As shown in Figure 3, the request distribution module 113 is used to send the task request to the computing node where the corresponding model of the task is located, so that the computing node can perform related operations based on the request, thereby realizing task execution.

[0098] The acquisition module 112 can acquire the quality of task execution for each computing node, obtaining the measured service quality of that task. The acquisition module 112 can also acquire the request frequency for each type of task. As shown in Figure 3, the request frequency of a task may vary. With changes in request frequency, the load on the computing nodes also changes, leading to changes in the service quality of the task. The initial allocation scheme may struggle to adapt to changes in request frequency, resulting in poor service quality or wasted resources.

[0099] The allocation module 111 can generate a redistribution scheme and reassign the multiple models to the multiple computing nodes according to the redistribution scheme. The redistribution scheme can be generated based on the measured service quality of various tasks. For example, if the acquisition module 112 obtains that model 310 has a low service quality, while models 320 and 330 have high service quality, then the generated redistribution scheme could be to assign model 310 to computing nodes 210 and 220, and to assign models 320 and 330 to computing node 230.

[0100] In some embodiments, as shown in FIG2, the management platform 100 may further include at least one local engine 120. The local engine 120, also known as a distributed engine, is used to control the execution of tasks by computing nodes. The aforementioned plurality of computing nodes can be divided into at least one node group, wherein each node group includes at least one computing node. The at least one local engine 120 corresponds one-to-one with the at least one node group, and the local engine 120 is used to control the computing nodes in the node group corresponding to the local engine 120 to execute tasks.

[0101] As shown in Figure 2, the local engine 120 may include a scheduling module 121, an execution module 122, and a storage management module 123.

[0102] The scheduling module 121, also known as the request scheduler, can schedule requests. The node group corresponding to the local engine may be assigned to two or more models. The scheduling module 121 can schedule the requests corresponding to these two or more models to schedule the tasks corresponding to those models. Here, the request corresponding to a model refers to the request of the task corresponding to that model. Detailed scheduling will be described in the following embodiments and will not be repeated here.

[0103] The execution module 122 is used to instruct computing nodes to perform relevant tasks based on the scheduling results of the scheduling module 121.

[0104] Storage management module 123 can manage unit storage space. The same node group may be allocated to two or more models. These two or more models may require different unit storage spaces. For example, different models may have different key matrix sizes and / or different value matrix sizes, requiring different unit storage spaces for storage. To enable the compute nodes in the node group to execute tasks for these two or more models, storage management module 123 can calculate the common multiple of the unit storage spaces required by the two or more models and use this common multiple as a unified storage space. The compute nodes use this unified storage space as the unit storage space to execute the tasks corresponding to the two or more models. For example, the storage space here refers to the space used to cache the key matrix and value matrix; the unified storage space can be called a unified key-value cache.

[0105] In some embodiments, the at least one local engine 120 is managed by a global manager 110. For example, the global manager 110 may communicate with the local engine 120 via inter-process communication technology. For example, the global manager 110 may use a ray cluster manager to manage the at least one local engine 120.

[0106] The foregoing embodiments described the system architecture provided by this application. Next, in conjunction with this system architecture, the flow of the model allocation method provided by this application will be described. As shown in Figure 4, the method includes the following steps.

[0107] The allocation module 111 executes step 401, allocating multiple models to multiple computing nodes according to the initial allocation scheme. Each computing node can be assigned to at least one of the multiple models, and the computing node is used to execute the task corresponding to the assigned model.

[0108] In some embodiments, the initial allocation scheme may be to distribute multiple models equally among multiple compute nodes, that is, different models are assigned to the same number of compute nodes, or different compute nodes are assigned to the same number of models. In some embodiments, the initial allocation scheme may be to randomly allocate multiple models among multiple compute nodes. For example, the initial allocation scheme may be to assign a model to one or more compute nodes capable of supporting the operation of that model, and the model exclusively uses those one or more compute nodes.

[0109] The requesting end can perform step 402, issuing a request. This request is for one or more tasks. In some embodiments, the requesting end can be a user. The user can instruct the model to perform the task by issuing a task request.

[0110] The request distribution module 113 can receive a request and execute step 403 to distribute the request. The request distribution module 113 can identify the task to which the request belongs and, based on the task, identify the computing node where the corresponding model resides. The request distribution module 113 can then send the request to that computing node.

[0111] In one embodiment, the same model may have at least two instances, meaning at least two copies of the same model are deployed. Correspondingly, there are at least two compute nodes executing the tasks corresponding to that model. In this case, the distribution module 113 can send the request to the compute node with the shorter request queue among the at least two compute nodes. Here, the request queue refers to a queue of requests waiting to be executed. A shorter request queue indicates a smaller number of requests waiting to be executed.

[0112] In some embodiments, the local engine 120 corresponding to the compute node is used to control the compute node to execute related tasks. The request distribution module 113 can send requests to the local engine 120. The local engine 120 can execute step 404 to control the compute node to execute tasks. As mentioned above, a request represents a task, and the compute node executing a task can also be called executing a request, which means executing the task represented by the request. The compute node uses the model corresponding to the request to execute the task represented by the request. The compute node can support the model to perform related calculations on the task represented by the request in order to execute the task represented by the request.

[0113] In some embodiments, the local engine 120 may execute step 405, recording the input length, output length, and actual execution time. The input length, also known as the request input length or task input length, refers to the length of the content input to the model by the task represented by the request. The output length, also known as the request output length, is the length of the content output by the model for the task represented by the request. The length of the content can be represented by the string length. The actual execution time, also known as the actual execution time of the request, is the actual time taken for the computing node where the model resides to execute the task represented by the request using the model.

[0114] The acquisition module 112 can execute step 406 to acquire the measured service quality for each task. The measured service quality of a task is the quality of the actual execution of the task by the computing node where the corresponding model resides. The measured service quality is obtained by detecting the process and results of the computing node executing the task using the model. Among them, the local engine 120 can detect the quality of the actual task execution by the computing nodes it manages to obtain the measured service quality of the task.

[0115] In step 407, the acquisition module 112 can send the measured service quality of each task to the allocation module 111. Through step 407, the allocation module 111 can obtain the measured service quality of multiple tasks. For example, the measured service quality of multiple tasks can be the sum or average of the measured service quality of each task in the multiple tasks.

[0116] The allocation module 111 can execute step 408 to generate a redistribution scheme based on the measured service quality of various tasks.

[0117] The reallocation scheme can include the target computing node of the model. The target computing node of the model refers to the computing node to which the model is to be allocated.

[0118] In some embodiments, the allocation module 111 can generate a reallocation scheme based on the principle of allocating models corresponding to tasks with low measured service quality to computing nodes with large available resources. Specifically, target computing nodes can be allocated to the models corresponding to tasks in descending order of their measured service quality. During the generation of the reallocation scheme, models that have received target computing nodes are referred to as allocated models, and models that have not received target computing nodes are referred to as unallocated models.

[0119] For example, it can be set that in the tasks corresponding to unassigned models (initially, all models are unassigned), the measured quality of service (QoS) of the task corresponding to model 310 is lower than that of other tasks. It can also be set that the available resources of compute node 210 are greater than those of other compute nodes. Then, compute node 210 is used as the target compute node for model 310, model 310 becomes an assigned model, model 310 is removed from the unassigned models, and the resources required to execute the task corresponding to model 310 are deducted from the available resources of compute node 210. Here, "other tasks" refers to tasks other than the task corresponding to model 310, and "other compute nodes" refers to compute nodes other than compute node 210. The aforementioned process is repeated until all models obtain target compute nodes.

[0120] In some embodiments, the available resources of a computing node include storage resources (e.g., video memory). When allocating a target computing node to a model, it is possible to first identify which computing nodes have storage resources that meet the model's storage resource requirements, and then use the computing nodes that meet the model's storage resource requirements as the model's target computing nodes. For example, if there are at least two computing nodes that meet the model's storage resource requirements, these at least two computing nodes can be configured to include computing node 210, and the available resources of computing node 210 can be greater than the available resources of the other computing nodes among the at least two computing nodes, excluding computing node 210. In this case, computing node 210 can be used as the target computing node for the model.

[0121] In some embodiments, the available resources of a compute node are negatively correlated with the request frequency of the models allocated to that compute node. That is, the lower the request frequency of the models allocated to a compute node, the greater the available resources of that compute node. Specifically, when generating a redistribution strategy, if a compute node becomes the target compute node for a certain model, then that model is the model already allocated to that compute node. If a compute node has not yet become the target compute node for any model, then the available resources of that compute node are equal to its resource specifications. If a compute node has been allocated to at least two models, then the request frequency of the models allocated to that compute node is the sum of the request frequencies of those at least two models.

[0122] The resource specifications of the multiple computing nodes mentioned above can be set to be the same. If the request frequency of the model allocated to a certain computing node is less than the request frequency allocated to other computing nodes, then the available resources of that computing node are greater than the available resources of other computing nodes.

[0123] The request frequency of the models allocated to the computing nodes refers to the request frequency of the tasks corresponding to those models. The request frequency of a model can be obtained by statistically analyzing the historical requests of the tasks corresponding to that model.

[0124] In some embodiments, the redistribution scheme may include parallel configuration. Parallel configuration refers to splitting the model into multiple model blocks and deploying the different model blocks to different computing nodes to run the model in parallel across multiple computing nodes. For example, parallel configuration may be any of tensor parallelism, pipelined parallelism, data parallelism, or sequence parallelism.

[0125] Parallel configuration can include the number of parallel nodes, which refers to the number of target computing nodes for the model. Multiple models can be configured to have the same number of parallel nodes. Multiple computing nodes can be divided into at least one node group based on the number of parallel nodes, where the number of computing nodes in each node group equals the number of parallel nodes. Target computing nodes can be assigned to models at the node group level. The target computing nodes for a model are all the computing nodes in the corresponding node group. For example, in tasks corresponding to unassigned models, the measured quality of service (QoS) of the task corresponding to model 310 in the unassigned models can be set to be lower than the measured QoS of tasks corresponding to other models in the unassigned models. If the available resources of the first node group in at least one node group are identified as greater than the available resources of the other node groups in that at least one node group, then the computing nodes in the first node group are used as the target computing nodes for model 310. This process is repeated until all models obtain target computing nodes. Here, the available resources of a node group refer to the sum of the available resources of all computing nodes in that node group.

[0126] In some embodiments, the model can have multiple parallel configurations, each with a different number of parallel operations. All possible parallel configurations of the model can be enumerated, and multiple computing nodes can be divided according to the number of parallel operations for each configuration, resulting in node groups for that configuration. Then, target computing nodes are allocated for each parallel configuration. Specifically, target computing nodes are allocated to the model at the node group level of the parallel configuration, resulting in a redistribution scheme for that configuration. This yields multiple redistribution schemes for different parallel configurations. Each redistribution scheme for a parallel configuration can be referred to as a candidate redistribution scheme. Thus, multiple candidate redistribution schemes can be obtained.

[0127] In some embodiments, the number of parallel nodes in a possible parallel configuration of the model is a multiple of two, and may include 2. In some embodiments, if a server in infrastructure 200 deploys at most 8 compute nodes, then the number of parallel nodes in a possible parallel configuration of the model is no greater than 8, to ensure that the target compute nodes of the model are located on the same server. In some embodiments, a model may be large in size and require significant storage (e.g., video memory) resources from the compute nodes. A single node group may not be sufficient to meet the deployment requirements of the model. In this case, two or more node groups can be merged, and the compute nodes in the merged node group can be used as the target compute nodes of the model.

[0128] The service quality of each task under each candidate reassignment scheme can be predicted, resulting in the predicted service quality of multiple tasks under that candidate reassignment scheme. Then, the candidate reassignment scheme with the highest predicted service quality across the multiple tasks is selected as the reassignment scheme generated in step 408. For example, the predicted service quality of the multiple tasks can be the sum or average of the predicted service quality of each task among the multiple tasks.

[0129] In some embodiments, the expected quality of service (QS) of a task can be estimated based on model information of the model corresponding to the task, task request information, etc. Taking latency as an example of expected QS, model information may include iteration duration, average length of requested input, and average length of requested output; latency = iteration duration × average length of requested input × average length of requested output. Furthermore, iteration duration may include the iteration duration of the model's prefill phase and the iteration duration of the decoding phase. The model information, task request information, etc., can be obtained by the acquisition module 112 from the corresponding local engine 120.

[0130] In this way, a redistribution scheme can be generated.

[0131] The allocation module 111 can also perform step 409 to obtain the expected quality of service for various tasks.

[0132] In some embodiments, as described above, when selecting a redistribution scheme, the expected quality of service (QoS) for multiple tasks under each candidate redistribution scheme is anticipated. The expected QoS for multiple tasks under the candidate redistribution schemes can be used as the expected QoS for the multiple tasks obtained in step 409.

[0133] In some embodiments, if step 408 does not predict the expected quality of service for multiple tasks, then in step 409, the expected quality of service for each task is predicted to obtain the expected quality of service for multiple tasks. The relevant methods can be found in the above description and will not be repeated here.

[0134] The allocation module 111 executes step 410, which reassigns multiple models to multiple computing nodes according to the redistribution strategy. Each computing node is reassigned to at least one of the multiple models, and the computing node uses the reassigned model to execute the task corresponding to that model.

[0135] The request distribution module 113 can receive the request sent by the requesting end through step 411, and distribute the request to the corresponding computing node through step 412, so that the computing node can use the corresponding model to execute the task represented by the request.

[0136] In some embodiments, the local engine 120 may perform step 413, controlling the computing node to execute a task based on the model's priority. Here, the task is the one represented by the request.

[0137] At least two models may be assigned to the same compute node. The local engine 120 corresponding to this compute node can set different priorities for these at least two models. When both models have tasks to execute, the local engine 120 can control the compute node to execute the tasks corresponding to the two models based on their priorities. Specifically, the compute node executes the tasks of the model with higher priority first, and then executes the tasks of the model with lower priority. In other words, when there are tasks corresponding to models with different priorities waiting to be executed, the tasks corresponding to the model with higher priority are executed first, followed by the tasks corresponding to the model with lower priority.

[0138] In some embodiments, the local engine 120 can estimate the execution time of the task corresponding to the model based on the actual execution time, input length, and output length of the task in the past, thus obtaining the estimated execution time of the task corresponding to the model. This execution time is the estimated execution time of a single task, also known as the estimated execution time of a request. As described above, in step 405, the local engine 120 records the input length, output length, and actual execution time of the task executed by the computing node. Specifically, the average and standard deviation of the input lengths of multiple requests for the task recorded in step 405 can be calculated to obtain the average and standard deviation of the historical input length of the task. The average and standard deviation of the output lengths of multiple requests recorded in step 405 can be calculated to obtain the average and standard deviation of the historical output length of the task. The average and standard deviation of the actual execution times of multiple requests recorded in step 405 can be calculated to obtain the average and standard deviation of the historical actual execution time of the task. Then, based on the average and standard of the historical input length of the task corresponding to the model, the average and standard deviation of the historical output length of the task corresponding to the model, the average and standard deviation of the historical actual execution time of the task corresponding to the model, and the input and output length of the currently waiting task corresponding to the model, the estimated execution time of the currently waiting task is calculated.

[0139] In some embodiments, the local engine 120 can obtain the estimated execution time of the current task to be executed corresponding to the model based on the historical actual execution time of the task corresponding to the model. Specifically, the local engine 120 can add the average and standard deviation of the historical actual execution time of the task to obtain the estimated execution time of the current task to be executed.

[0140] The priority of a task can be derived based on its estimated execution time. Priority and estimated execution time are negatively correlated. The priority of the current task to be executed corresponding to the model can be used as the priority of the model.

[0141] In some embodiments, as shown in Figure 5, computing nodes may execute tasks corresponding to models at a time slice granularity. The execution duration of a task can include multiple time slices. To avoid some model-related tasks not being executed, the model priority is dynamic, not static. For example, the model priority = 1 / (estimated execution time × remaining execution time). Where, remaining execution time = estimated execution time - already executed time. Thus, as tasks are executed, the priority of the corresponding model gradually increases. The task execution time refers to the execution time of a single task, that is, the execution time of the task represented by a particular request.

[0142] In some embodiments, when the actual execution time of a task corresponding to a compute node execution model reaches the expected execution time of that task, the task may not yet be completed. That is, the actual required execution time of the task corresponding to the model is longer than the expected execution time. In this case, the priority of the model can be reduced. Here, the actual execution time of a task corresponding to a compute node execution model refers to the actual execution time of a specific task within the compute node execution model, and the expected execution time refers to the expected execution time of that specific task.

[0143] In one example of this embodiment, when the actual required execution time of the task corresponding to the model is greater than the expected execution time of the task, the expected execution time of the task is doubled, and then the priority is recalculated using the doubled expected execution time, thereby reducing the priority of the model.

[0144] Prioritizing computation nodes based on models to execute tasks corresponding to those models can reduce the overall latency of multiple tasks. For example, as shown in Figure 5, under a first-come, first-served (FFS) strategy, the latency of the task corresponding to model 310 is 5 time slices, the latency of the task corresponding to model 320 is 7 time slices, and the overall latency is 12 time slices. Under a round-robin strategy, the latency of the task corresponding to model 310 is 7 time slices, the latency of the task corresponding to model 320 is 4 time slices, and the overall latency is 11 time slices. According to the embodiment of this application (i.e., the priority-based execution of computation nodes based on models as described above), the latency of the task corresponding to model 310 is 7 time slices, the latency of the task corresponding to model 320 is 3 time slices, and the overall latency is 10 time slices.

[0145] In some embodiments, multiple models can be periodically reassigned to multiple computing nodes according to a redistribution cycle. If the redistribution cycle is too long, redistribution cannot be performed in a timely manner when the request frequency of tasks changes, resulting in poor service quality and resource waste for various tasks. If the redistribution cycle is too short, it leads to large additional resource overhead. Therefore, in this embodiment, the length of the redistribution cycle can be dynamically adjusted based on the gap between the actual service quality and the expected service quality of various tasks under the previous redistribution strategy to determine when the next redistribution will occur.

[0146] As shown in Figure 4, the acquisition module 112 can execute step 414 to obtain the measured service quality of each task after model reallocation. The service quality of a task is the quality of the actual execution of the task by the computing node where the corresponding model resides. The measured service quality is the service quality obtained by detecting the process and results of the computing node executing the task using the model. Among them, the local engine 120 can detect the quality of the actual task execution by the computing nodes it manages to obtain the measured service quality of the task.

[0147] In step 415, the acquisition module 112 can send the measured service quality of each task to the allocation module 111. Through step 407, the allocation module 111 can obtain the measured service quality of the various tasks after model reassignment. For example, the actual service quality of the various tasks can be the sum or average of the measured service quality of each task among the various tasks.

[0148] In step 416, the allocation module 111 can adjust the redistribution period based on the measured service quality of various tasks obtained in step 415 and the expected service quality of various tasks obtained in step 409. Specifically, the length of the redistribution period is adjusted. The difference between the measured service quality of various tasks and the expected service quality of various tasks can be calculated, and then the length of the redistribution period is adjusted based on this difference. This difference is negatively correlated with the length of the redistribution period. A large difference indicates that the previous redistribution strategy did not match reality, and redistribution needs to be performed again as soon as possible; therefore, the length of the redistribution period is reduced. A small difference indicates that the previous redistribution strategy was consistent with reality, and redistribution does not need to be performed again as soon as possible; therefore, the length of the redistribution period is increased.

[0149] After adjusting the redistribution period, multiple models can be redistributed to multiple compute nodes based on the adjusted period. Subsequent redistribution may include generating a new redistribution strategy and then reallocating multiple models to multiple compute nodes according to the new strategy. Details can be found in the description above and will not be repeated here.

[0150] In summary, the model allocation method provided in this application can obtain the service quality of multiple tasks corresponding to multiple models after allocating multiple models to multiple computing nodes, and reallocate the multiple models to multiple computing nodes based on the service quality of multiple tasks, thereby ensuring load balancing among multiple computing nodes, improving the service quality of tasks, and reducing resource waste.

[0151] In some embodiments, a large-scale model service management system is proposed. The goal is to perceive the impact of load changes on the system's quality of service (SLO) within a computing node cluster, and dynamically adjust the service deployment and request scheduling of the inference system based on real-time impact results to improve the overall SLO. From a system state perspective, after the inference system completes its initial deployment, execution data of the load (i.e., requests) and SLO satisfaction status are collected during system operation. Based on the collected data, a relocation simulation analysis is performed on the deployed inference services in the system to attempt to derive a new deployment method with a better overall SLO for historical load conditions.

[0152] This embodiment supports a service-aware, low-latency large model service sharing system. The key observation behind this system is that large model services typically have unique characteristics, including iteration time and input / output length distribution. Based on these characteristics, the placement and request scheduling of large model services can mitigate the head-of-line blocking problem caused by the first-come, first-served scheduling strategy and the surge in memory demand caused by the round-robin scheduling strategy.

[0153] I. To place multiple large model services in the cluster, this embodiment employs a placement algorithm based on search and heuristic rules. By monitoring historical load and system SLO, the placement of the inference service is periodically simulated to see if the system SLO can be optimized by adjusting the service deployment. Since the time complexity of finding an optimal placement scheme increases exponentially with cluster size and the number of large model services, finding an optimal solution within an acceptable timeframe is difficult. To reduce the complexity of placement, this embodiment divides the placement process into two stages: computation node group partitioning and large model service allocation, and proposes some heuristic strategies to further reduce the search space. Using parallel execution, this embodiment can reduce the execution time of the placement algorithm to the minute level, and it can overlap with the service execution process, avoiding additional system overhead. Furthermore, a fixed placement strategy cannot adapt to fluctuating request traffic over time. The common practice is to replace services at a fixed period, but the fixed replacement period is difficult to determine manually and may affect service performance. This embodiment proposes an adaptive replacement algorithm that can dynamically adjust the placement period based on estimated and actual performance.

[0154] II. This embodiment proposes a request scheduling algorithm with better service awareness and latency for shared large model services. Priorities are assigned to requests for each model based on the execution time and service characteristics of historical system requests, and preemptive scheduling is adopted according to these priorities. However, in practice, request lengths may deviate from service characteristics, thus affecting the performance of the scheduling algorithm. This embodiment proposes a doubling quota scheduling algorithm, setting the execution time quota as the expected execution time. For requests exceeding the execution time quota, a priority reduction scheme is adopted, and the execution time quota is doubled.

[0155] This embodiment includes a cluster manager to coordinate large model services and requests, and multiple local executors to execute shared large model services. Managing the video memory of key-value caches for shared large model services presents challenges. Sharing video memory resources across different large model services can improve memory utilization; however, the varying key-value cache block sizes across different large model services pose challenges to unified management. This embodiment proposes a unified key-value cache management technique based on merging cache blocks, achieving a balance in terms of cache block table size, locality, and fragmentation.

[0156] 1) System composition:

[0157] This embodiment proposes SeaLLM, a large model service management system that supports efficient sharing of multiple large models, automatic placement and relocation of large model services, and low-latency request scheduling. The system architecture of this embodiment includes a global manager and a set of local engines on compute nodes.

[0158] Global Manager: In this embodiment, the global manager is responsible for generating placement schemes for large model services, distributing requests, and monitoring large model services. The global manager includes three modules: placement controller, service monitor, and request dispatcher.

[0159] Placement Control Module: This module generates placement schemes for all large model services in the cluster. A placement scheme includes the compute nodes and parallel configuration used by each large model service. Since the request traffic of large model services fluctuates over time, a fixed placement scheme cannot adapt to changes in request traffic in a timely manner. Therefore, the placement control module periodically regenerates placement schemes based on historical characteristics. For new large model services, the placement control module allocates a smaller number of dedicated compute nodes that the service can execute on. Note that a fixed relocation period can affect service performance and is difficult to determine manually; for example, a small relocation period may introduce excessive system overhead, while a large relocation period cannot adapt to changes in request traffic in a timely manner. Therefore, this module uses an adaptive relocation algorithm to automatically change the relocation period.

[0160] Service Monitoring Module: This module monitors large model services and continuously collects their feature data. The collected data includes model information (including iteration durations for the pre-filling and decoding phases under different parallel configurations), request information (including the average and standard deviation of input / output lengths, and the number of requests), and execution information (including latency for each request and SLO satisfaction rate for each service). This data is sent to the Placement Control Module for periodic repositioning and adjusting the repositioning cycle.

[0161] Request dispatch module: When a new request arrives, this module determines the local engine to execute it based on the service type of the new request. If multiple local engines can serve the newly arrived request, the local engine with the fewer queued requests is selected.

[0162] Local Engine: In this embodiment, the local engine is responsible for scheduling and executing requests distributed by the global manager. Each local engine manages at most two large model services, as sharing more large model services would place a significant burden on the GPU memory of existing hardware. As hardware evolves, this technology can be extended to scenarios where more large model services are shared. The local engines share large model services over time. Each local engine contains three modules: a request scheduler, a multi-LLM executor, and a unified key-value cache.

[0163] Request Scheduler: The request scheduler formulates a scheduling strategy for the distributed requests. The scheduling algorithm has two basic requirements: firstly, the scheduling strategy must achieve high service performance; secondly, the scheduling time complexity must be low. Considering the characteristics of large-model services, this embodiment proposes a doubling quota scheduling algorithm to achieve superior service performance by leveraging these service characteristics.

[0164] Multi-model executor: The request scheduler selects a batch of requests to be executed and sends them to the multi-model executor for execution. The multi-model executor manages shared models and calls the corresponding models to perform inference services.

[0165] Unified Key-Value Cache: In practice, the size of key-value cache blocks is determined by the architecture of the large model; therefore, shared large models may have different cache block sizes. This embodiment proposes a unified key-value cache based on merged blocks to solve the problem of heterogeneous cache blocks. Tests show that the performance of this unified key-value cache is close to that of a single-model key-value cache.

[0166] (2) Implementation principle

[0167] The operational process of a service-aware and latency-optimized resource-sharing system for large-scale model inference consists of three phases: the new service initialization phase, the global manager service placement phase, and the request execution phase.

[0168] The first stage is the initial phase of a new service. Upon arrival of a new service, the global manager's placement control module allocates a limited number of dedicated executable compute nodes as the initial placement strategy. During this phase, requests are scheduled according to a first-come, first-served (FIFO) policy. The global manager's service monitoring module then begins collecting the service's model information, request information, and execution information for subsequent relocation and request scheduling.

[0169] Next comes the service placement phase generated by the global manager. The global manager initiates a global relocation at regular intervals, with the placement cycle adjusted based on estimated and actual performance. Each relocation decision includes the placement plan for each service in the next cycle, including the compute nodes used and parallel configuration. Each relocation requires the service monitoring module to collect model information, request information, and execution information for each service, and generates a placement plan based on the large model service placement algorithm based on search and heuristic rules proposed in this embodiment.

[0170] Next comes the request execution phase. For arriving requests, the global manager's request distribution module distributes the request to the local engine corresponding to the service type. If multiple engines can serve the request, the engine with the fewer queued requests is selected for distribution. If the service corresponding to the request is a new service, a first-come, first-served (FFS) scheduling strategy is used. Otherwise, based on the collected model and request information, the doubling quota scheduling algorithm proposed in this embodiment is used to set priorities and quotas, and the priority is reduced and the quota is doubled when the execution time exceeds the quota. This scheduling algorithm adopts a batch execution strategy, generating a batch of requests to be executed each time. The large model executor calls the corresponding large model for inference according to the generated scheduling strategy. The key-value cache generated or used during inference is managed using the unified key-value caching technology proposed in this embodiment.

[0171] This system is implemented based on the vLLM large model inference architecture. The main additions include a global manager (comprising a placement control module, a service monitoring module, and a request distribution module), and modifications to the local engine's request scheduler, large model executor, and unified key-value cache. This embodiment is deployed on a deep learning cluster composed of compute nodes.

[0172] Global Manager

[0173] The cluster contains only one global manager. This includes a request-accepting module, a placement algorithm, an adaptive replacement algorithm, a request dispatcher, and a service monitor. The global manager communicates with other roles in the cluster via inter-process communication technology and uses Ray Cluster Manager to support the distributed engine. The request-accepting module is not innovative in this embodiment. The request dispatcher allocates requests based on the service type and the number of requests in the request queue. The service monitor continuously monitors model information (including iteration durations of the pre-filling and decoding phases under different parallel configurations), request information (including the average and standard deviation of input and output lengths), and execution information (including latency for each request and SLO satisfaction rate for each service) using statistical tools such as timestamps and counters. The placement and replacement algorithms innovatively proposed in this embodiment will be detailed below.

[0174] Service Placement Algorithm

[0175] A service placement scheme comprises a shared large model service group, compute nodes used by each shared group, and parallel configurations for each shared group. The number of all feasible placement schemes grows exponentially with the number of compute nodes and services. Therefore, searching all placement schemes and selecting the optimal one cannot be achieved within a reasonable timeframe. This embodiment proposes a two-stage placement algorithm. First, this embodiment divides the entire cluster into multiple compute node groups by enumerating parallel configurations, and then assigns the large model service to these compute node groups to form shared groups. Based on historical request information, a simulator is used to simulate the performance of the compute services (including normalized latency, SLO satisfaction rate, etc.), and a placement scheme with a high SLO satisfaction rate and low normalized latency is selected.

[0176] Computation Node Group Partitioning: In the computation node group partitioning phase, this embodiment enumerates all possible parallel configurations to partition the entire cluster. Existing parallel solutions for large model services are mainly tensor parallelism, so this embodiment only considers tensor parallelism. Note that the placement algorithm in this embodiment can be extended by enumerating more types of parallel dimensions. To reduce the number of possible partitioning schemes and thus reduce the execution time of the placement algorithm, this embodiment introduces two heuristic rules: (1) Consistent with common settings, this embodiment assumes that the tensor parallel dimension is a power of 2 and will not exceed the number of computation nodes on a single server to avoid system overhead caused by cross-node communication; (2) This embodiment sets the same tensor parallel dimension for all groups. If some large model services require more computation nodes, two small-scale computation node groups are merged into a large-scale computation node group, and this process is repeated until all services can be placed.

[0177] Service Allocation: In the service allocation phase, this embodiment allocates large model services to the compute node groups partitioned in the first phase. Similar to the previous phase, enumerating all possible allocation schemes is very time-consuming; therefore, this embodiment uses heuristic rules to accelerate the process. This embodiment iteratively allocates large model services to compute node groups, with the allocation rule being to allocate services with higher unserved indices to compute node groups with lower request frequencies among multiple compute node groups that meet GPU memory requirements. The unserved index is defined as follows: Where s represents the large model service currently under consideration, and R s For all requests to service s, I represents the exponent, where I is greater than SLO. r Lr is the object being calculated; values ​​smaller than L are not considered. r SLO is the delay for request r. r To request the SLO of r, α is a coefficient between 0 and 1. The normalized latency of service s is used. Note that using only normalized latency as the unallocated index may result in some large model services with insufficient resources being able to serve too few requests, leading to falsely low normalized latency and affecting subsequent resource allocation. Therefore, the unallocated index considers the number of requests with unmet SLOs and uses it as the primary indicator. When allocating a large model service to a compute node group, this embodiment first checks whether the compute node group's GPU memory can accommodate the new large model service. For each group, this embodiment shares at most two large model services, because sharing more large model services requires excessive GPU memory to store model parameters, activation values, etc. For compute nodes with larger GPU memory in the future, this embodiment can be extended to sharing more large model services.

[0178] Service Relocation Algorithm

[0179] The request traffic for large model services changes dynamically over time. Fixed placement schemes cannot respond to changes in request traffic in a timely manner, leading to performance degradation caused by sudden requests. To address this issue, this embodiment periodically relocates large model services in the cluster. Existing methods typically use a fixed relocation period; however, there is still room for optimization in fixed relocation periods. Specifically, too short a relocation period increases the system overhead of migrating services, while too long a relocation period cannot provide a timely relocation solution. This embodiment proposes an adaptive relocation algorithm to automatically select an appropriate relocation period. This algorithm adjusts the relocation period based on the difference between the actual achieved performance and the estimated performance. First, an initial relocation period is set. In each round of relocation, if the actual normalized latency and SLO satisfaction rate in the previous round differ significantly from the estimated metrics in the previous round, the relocation period is reduced; otherwise, the relocation period is increased.

[0180] Local engine

[0181] Each shared large model service has a local engine on a specific node. The large model executor is implemented based on the vLLM large model execution engine, supporting the execution of multiple large models. Model parameters for the shared large models are stored in the compute node's memory for quick switching and execution. This embodiment uses Ray workers for parallel execution and NCCL for communication. The local engine simultaneously supports characterizing and collecting service features (such as request input / output length, model execution time, etc.) and sending this feature data to the global manager. This implementation also includes a node worker on each node, used to connect the global manager and the local engine, monitor the local engine status, report anomalies, and assist the global manager in restoring the local engine. During relocation, the node worker handles the process of stopping the old service and starting the new service.

[0182] Request scheduler

[0183] Given a placement scheme for large model services, it is also necessary to determine the execution order of each large model service request, i.e., scheduling the requests of each large model service to reduce normalized latency. The scheduling algorithm has two main requirements: efficiency and performance. First, because the execution time of large model service requests is usually short, the scheduling algorithm needs to generate a simple and fast scheduling scheme; second, the scheduling algorithm needs to provide a high-performance execution order. Existing scheduling schemes (first-come, first-served and round-robin) face problems such as head-of-line blocking and memory demand explosion. To solve these problems, this embodiment proposes a doubling quota scheduling algorithm. First, the key observation on which this algorithm is based is that different large model services have different characteristics, including the distribution of input and output lengths. Based on the above observation, this embodiment first executes requests with shorter expected execution times to reduce normalized latency. Specifically, we set the priority of requests to the product of the estimated execution time and the estimated remaining time, and at the same time, we use preemptive scheduling to avoid head-of-line blocking. It can be proven that the above scheduling algorithm can achieve globally superior normalized latency under certain assumptions (requests can be preempted at any time; all requests have the same execution time and the estimation is accurate). Intuitively, this scheduling algorithm presents a weighted queuing problem, demonstrating superior performance when sharing services from at most two large models. However, in practice, these assumptions may not be met. Regarding the first assumption, considering the short iteration time in the decoding phase, it can be approximated as satisfied. The second assumption, however, is difficult to satisfy in practice.

[0184] To address the difficulty in satisfying the second assumption, this embodiment proposes a doubling quota scheduling algorithm to achieve a better normalized latency. Specifically, this embodiment assigns an execution time quota to each request, initially set at the average execution time of the large model service to which the request belongs plus the standard deviation of that execution time. Note that in this algorithm, higher priority values ​​indicate lower priority and later execution, while higher priority values ​​gradually decrease as execution progresses, meaning they are executed earlier. If a request exceeds its quota and is still not completed, its quota is doubled, and its priority is adjusted accordingly. This algorithm repeats the above process for each request until it is completed. This scheduling algorithm uses batch processing, selecting a batch of requests at the same stage for execution at a time. For the pre-filling stage, the batch size limits the number of input symbols in the request; for the decoding stage, the batch size limits the number of requests.

[0185] Unified key-value caching

[0186] Sharing multiple large models requires an efficient and flexible mechanism for managing resources, especially compute node memory. Compute node memory is primarily used to store three parts: model parameters, activation values, and key-value cache. Model parameters and activation values ​​are relatively stable given model information and batch size; this embodiment allocates sufficient space for peak usage of model parameters and activation values. However, managing the shared key-value cache presents challenges. First, the key-value cache is dynamic during model execution; allocating independent compute memory to each large model for its key-value cache can lead to memory fragmentation or insufficient memory. Simultaneously, the size of the key-value cache block is affected by the model structure, and different large models serve different model structures, making it difficult to use the same key-value cache blocks.

[0187] To efficiently share and manage key-value caches, this embodiment adopts a unified key-value cache mechanism, which changes the size of the key-value cache blocks to adapt to the shared large model service. Specifically, the commonly used hidden layer size of large models is usually 128, so this dimension does not need to be changed. For other dimensions, there are two main solutions: splitting the cache block into smaller blocks (the block size is a common divisor of different model dimensions, called the partitioning-based solution) and merging cache blocks to form larger cache blocks (the block size is a common multiple of different model dimensions, called the merging-based solution). This embodiment adopts the merging-based solution, mainly for the following considerations: (1) the merging solution requires less space to store the cache block table; (2) the merging solution requires fewer read and write operations to obtain the same number of key-value cache pairs and has better locality; (3) even if the merging solution may cause some resource fragmentation, large model inference often uses batch processing, which can alleviate this resource fragmentation problem.

[0188] The solution provided in this embodiment can be applied to more types of devices. Currently, this embodiment has been tested and evaluated for a certain homogeneous computing node. It is possible to apply this method to other types of computing nodes or heterogeneous computing node clusters, such as heterogeneous clusters that combine NPUs and GPUs. This method does not have strict requirements on the type of device.

[0189] The solution provided in this embodiment can be applied to other distributed parallel scenarios. Currently, this embodiment focuses on the implementation and evaluation of tensor parallelism, but the method may be applied to data parallelism, pipelined parallelism, sequence parallelism, and other parallel scenarios.

[0190] The solution provided in this embodiment can be implemented on other deep learning frameworks. Currently, this embodiment is implemented and evaluated based on the virtual large language model (vLLM) inference service framework, but it may be implemented using other deep learning frameworks (such as Megatron, DeepSpeed, etc.).

[0191] The solution provided in this embodiment can be applied to clusters of other AI services. Currently, this implementation is only for large model inference services, but the method may be applied to traditional deep learning inference services, big data services, and other service scenarios.

[0192] In summary, the solution provided in this embodiment is based on a large model service relocation algorithm using search and heuristic rules, a similar algorithm not yet found in existing inference systems. After model relocation, the overall system SLO is optimized. The large model request scheduling algorithm with superior latency provided in this embodiment, compared to first-come, first-served and random allocation algorithms, is a preemptive scheduling algorithm based on historical request data. This scheduling algorithm reduces the probability of long requests blocking short requests, decreases the overall memory consumption for processing requests, and lowers the overall latency.

[0193] Based on the above description, this application provides a model allocation method. This method can be applied to a management platform for managing multiple computing nodes in an infrastructure; wherein the infrastructure includes at least one data center, each of the at least one data center includes multiple servers, and the multiple computing nodes are deployed in the at least one server in the infrastructure. In some embodiments, the computing nodes can be any of a GPU, NPU, TPU, or FPGA.

[0194] As shown in Figure 6, the method includes the following steps.

[0195] First, in step 601, multiple models are assigned to the multiple computing nodes; wherein, the multiple models correspond one-to-one with multiple tasks, and each computing node is assigned to at least one of the multiple models, and the computing node is used to execute the task corresponding to the assigned model. For example, the implementation of step 601 can be referred to the above description of step 401 or step 410 in Figure 4, and will not be repeated here.

[0196] Secondly, in step 602, the quality of each computing node executing the task corresponding to the assigned model is obtained, thus obtaining the service quality of the various tasks. For example, the implementation of step 602 can be found in the description of steps 406 and 407 in Figure 4 above, and will not be repeated here.

[0197] Next, in step 603, based on the service quality of the various tasks, the multiple models are reassigned to the multiple computing nodes; wherein each computing node is reassigned to at least one of the multiple models, and the computing node is used to execute the task corresponding to the reassigned model. For example, the implementation of step 603 can be referred to the above description of steps 408-410 in Figure 4, and will not be repeated here.

[0198] In some embodiments, the plurality of models includes a first model; the step of reallocating the plurality of models to the plurality of computing nodes based on the service quality of multiple tasks includes: confirming that the service quality of the task corresponding to the first model among the multiple tasks is lower than the service quality of other tasks, and confirming that the available resources of the first computing node among the plurality of computing nodes are greater than the available resources of other computing nodes; wherein, the other tasks are tasks other than the task corresponding to the first model among the multiple tasks, and the other computing nodes are computing nodes other than the first computing node among the plurality of computing nodes; and the first model is allocated to the first computing node. The relevant implementation method can be referred to the above description of step 408 in Figure 4, and will not be repeated here.

[0199] In one example of this embodiment, confirming that the available resources of the first computing node among the plurality of computing nodes are greater than the available resources of the other computing nodes includes: obtaining the request frequency of the task corresponding to the model allocated to each of the plurality of computing nodes; when the request frequency of the task corresponding to the model allocated to the first computing node is less than the request frequency of the task corresponding to the model allocated to the other computing nodes, confirming that the available resources of the first computing node are greater than the available resources of the other computing nodes.

[0200] In some embodiments, the service quality of the multiple tasks includes the expected service quality and the measured service quality of the multiple tasks; the step of reallocating the multiple models to the multiple computing nodes based on the service quality of the multiple tasks includes: obtaining a model reallocation period based on the gap between the expected service quality and the measured service quality; wherein the length of the model reallocation period is negatively correlated with the gap; and reallocating the multiple models to the multiple computing nodes according to the model reallocation period. The relevant implementation method can be referred to the above description of step 416 in Figure 4, and will not be repeated here.

[0201] In some embodiments, at least two of the plurality of models are simultaneously assigned to a second computing node among the plurality of computing nodes; the method further includes: configuring a priority for each of the at least two models based on the estimated execution time of the task corresponding to each of the at least two models; wherein the estimated execution time of the task corresponding to the model is negatively correlated with the priority of the model; and instructing the second computing node to execute the task corresponding to the model among the at least two models according to the priority of the model among the at least two models. The relevant implementation method can be referred to the above description of step 413 in Figure 4, and will not be repeated here.

[0202] In one example of this embodiment, the plurality of models includes a second model, and the method further includes: when the actual time required for the second computing node to execute the task corresponding to the second model is greater than the expected execution time of the task, reducing the priority of the second model.

[0203] In some embodiments, the required unit storage space for different models among the plurality of models is different; the method further includes: obtaining a unified storage space based on the required unit storage space for different models among the plurality of models; wherein the size of the unified storage space is an integer multiple of the required unit storage space for each model among the plurality of models; instructing each computing node among the plurality of computing nodes to execute the task corresponding to the model allocated to the computing node using the unified storage space as the unit storage space.

[0204] The model allocation method provided in this application can redistribute multiple models to multiple computing nodes based on the service quality of multiple tasks, which can ensure load balancing among multiple computing nodes, thereby improving the service quality of multiple tasks and reducing resource waste.

[0205] Based on the above description, this application provides a management platform 700. The management platform 700 is used to manage multiple computing nodes in an infrastructure; wherein the infrastructure includes at least one data center, each of the at least one data center includes multiple servers, and the multiple computing nodes are deployed in the at least one server in the infrastructure. As shown in Figure 7, the management platform 700 includes:

[0206] The allocation module 710 is used to allocate multiple models to the multiple computing nodes; wherein, the multiple models correspond one-to-one with multiple tasks, each computing node is allocated to at least one of the multiple models, and the computing node is used to execute the task corresponding to the allocated model.

[0207] The acquisition module 720 is used to acquire the quality of the tasks assigned to the model executed by each of the multiple computing nodes, and to obtain the service quality of the multiple tasks.

[0208] The reallocation module 730 is used to reallocate the multiple models to the multiple computing nodes based on the service quality of the multiple tasks; wherein each computing node is reallocated to at least one of the multiple models, and the computing node is used to execute the task corresponding to the model using the reallocated model.

[0209] In some embodiments, the plurality of models includes a first model; the reallocation module 730 is configured to: confirm that the service quality of the task corresponding to the first model among the plurality of tasks is lower than the service quality of other tasks, and confirm that the available resources of the first computing node among the plurality of computing nodes are greater than the available resources of other computing nodes; wherein, the other tasks are tasks other than the task corresponding to the first model among the plurality of tasks, and the other computing nodes are computing nodes other than the first computing node among the plurality of computing nodes; and allocate the first model to the first computing node.

[0210] In one example of this embodiment, the reallocation module 730 is configured to: obtain the request frequency of the task corresponding to the model that has been allocated to each of the plurality of computing nodes; when the request frequency of the task corresponding to the model that has been allocated to the first computing node is less than the request frequency of the task corresponding to the model that has been allocated to the other computing nodes, confirm that the available resources of the first computing node are greater than the available resources of the other computing nodes.

[0211] In some embodiments, the service quality of the multiple tasks includes the expected service quality and the measured service quality of the multiple tasks; the reallocation module 730 is configured to: obtain a model reallocation period based on the gap between the expected service quality and the measured service quality; wherein the length of the model reallocation period is negatively correlated with the gap; and reallocate the multiple models to the multiple computing nodes according to the model reallocation period.

[0212] In some embodiments, at least two of the plurality of models are simultaneously assigned to a second computing node among the plurality of computing nodes; the acquisition module 720 is further configured to: configure a priority for each of the at least two models based on the estimated execution time of the task corresponding to each of the at least two models; wherein the estimated execution time of the task corresponding to the model is negatively correlated with the priority of the model; and instruct the second computing node to execute the task corresponding to the model among the at least two models according to the priority of the model among the at least two models.

[0213] In one example of this embodiment, the plurality of models includes a second model, and the acquisition module 720 is further configured to: reduce the priority of the second model when the actual time required for the second computing node to execute the task corresponding to the second model is greater than the expected execution time of the task.

[0214] In some embodiments, the required unit storage space for different models among the plurality of models is different; the acquisition module 720 is further configured to: obtain a unified storage space based on the required unit storage space for different models among the plurality of models; wherein the size of the unified storage space is an integer multiple of the required unit storage space for each model among the plurality of models; and instruct each computing node among the plurality of computing nodes to execute the task corresponding to the model allocated to the computing node using the unified storage space as the unit storage space.

[0215] The allocation module 710, acquisition module 720, and reallocation module 730 can all be implemented in software or in hardware. For example, the implementation of allocation module 710 will be described below. Similarly, the implementation of acquisition module 720 and reallocation module 730 can refer to the implementation of allocation module 710.

[0216] As an example of a software functional unit, the allocation module 710 may include code running on a compute instance. The compute instance may include at least one of a physical host (compute device), a virtual machine, or a container. Further, the compute instance may be one or more. For example, the allocation module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0217] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0218] As an example of a hardware functional unit, the allocation module 710 may include at least one computing device, such as a server. Alternatively, the allocation module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0219] The multiple computing devices included in the allocation module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the allocation module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the allocation module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0220] It should be noted that, in other embodiments, the allocation module 710 can be used to execute any step in the method shown in FIG. 6, the acquisition module 720 can be used to execute any step in the method shown in FIG. 6, and the reallocation module 730 can be used to execute any step in the method shown in FIG. 6. The steps implemented by the allocation module 710, the acquisition module 720, and the reallocation module 730 can be specified as needed. By implementing different steps in the method shown in FIG. 6 through the allocation module 710, the acquisition module 720, and the reallocation module 730, all the functions of the management platform 700 can be realized.

[0221] This application also provides a computing device 800. As shown in FIG8, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other via the bus 802. The computing device 800 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0222] Bus 802 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 802 can include pathways for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, communication interface 808).

[0223] Processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0224] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0225] The memory 806 stores executable program code, and the processor 804 executes this executable program code to implement the functions of the aforementioned allocation module 710, acquisition module 720, and reallocation module 730, thereby implementing the method shown in FIG6. That is, the memory 806 stores instructions for executing the method shown in FIG6.

[0226] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0227] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0228] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method shown in Figure 6.

[0229] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG. 6. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG. 6.

[0230] It should be noted that the memory 806 in different computing devices 800 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the management platform 700. That is, the instructions stored in the memory 806 of different computing devices 800 can implement the functions of one or more modules among the allocation module 710, the acquisition module 720, and the reallocation module 730.

[0231] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 806 in computing device 800A stores instructions for executing the functions of the allocation module 710. Simultaneously, the memory 806 in computing device 800B stores instructions for executing the functions of the acquisition module 720 and the reallocation module 730.

[0232] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0233] This application embodiment also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster described in Figures 9 and 10. The difference is that the memory 806 of one or more computing devices 800 in this computing device cluster can store the same instructions for executing the method shown in Figure 6.

[0234] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG. 6. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG. 6.

[0235] This application also provides a cloud system. The cloud system includes a management platform and infrastructure. The infrastructure includes at least one data center, each of which includes multiple servers, and the multiple computing nodes are deployed on the at least one server within the infrastructure. The management platform is used to: allocate multiple models to the multiple computing nodes; wherein each of the multiple models corresponds one-to-one with multiple tasks, and each of the multiple computing nodes is allocated to at least one of the multiple models, and the computing node is used to execute the task corresponding to the allocated model; obtain the quality of service of each of the multiple computing nodes in executing the task corresponding to the allocated model, thus obtaining the service quality of the multiple tasks; and based on the service quality of the multiple tasks, reassign the multiple models to the multiple computing nodes; wherein each of the multiple computing nodes is reassigned to at least one of the multiple models, and the computing node is used to execute the task corresponding to the reassigned model.

[0236] The management platform can also be implemented by referring to the introduction of management platform 100 above, and the infrastructure can also be implemented by referring to the introduction of infrastructure 200 above.

[0237] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, the at least one computing device performs the method shown in FIG6.

[0238] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a host migration device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method shown in FIG6.

[0239] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Images