Model deployment method and management platform

Abstract

A model deployment method and a management platform. The method comprises: acquiring a plurality of model blocks, wherein the plurality of model blocks are obtained by means of splitting a model to be deployed, and different model blocks from among the plurality of model blocks have different resource requirements; creating a computing instance corresponding to each of the plurality of model blocks by using resources of at least one of a plurality of servers, wherein the computing instance corresponding to the model block is assigned, from the resources of the at least one server, to resources that match the resource requirements of the model block; and deploying each model block in the computing instance corresponding to the model block, such that each model block runs in the computing instance corresponding to the model block. The method can avoid resource mismatch, thereby improving resource utilization and the running efficiency of models.

Description

A model deployment method and management platform

[0001] This application claims priority to Chinese Patent Application No. 202411718113.5, filed on November 27, 2024, entitled "A Model Deployment Method and Management Platform", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a model deployment method and apparatus. Background Technology

[0003] With the development of AI, large models such as large language models (LLM) and large visual models (LSM) have emerged. These large models have a large number of parameters, and this number continues to grow. Therefore, running large models requires significant resources (such as computing power and storage). A single device can no longer meet the operational needs of large models.

[0004] The industry has proposed a distributed computing solution that breaks down a large model into multiple model blocks and then deploys these different model blocks to different devices, allowing different model blocks to run on different devices and enabling multiple devices to collaboratively run the same large model.

[0005] Currently, the resource specifications of the devices used to deploy different model blocks are the same, but the resource requirements of different model blocks are different. This results in resource mismatch, which leads to problems such as low resource utilization and low model running efficiency. Summary of the Invention

[0006] This application provides a model deployment method and management platform that can avoid resource mismatch and improve resource utilization and model operating efficiency.

[0007] Firstly, a model deployment method is provided, which is applied to a management platform connected to infrastructure, the infrastructure including at least one data center, the at least one data center having multiple servers, each of the multiple servers having resources, and the management platform being able to create computing instances using the resources of each of the multiple servers; the method includes: obtaining multiple model blocks; wherein the multiple model blocks are obtained by splitting a model to be deployed, and the resource requirements of different model blocks are different; using the resources of at least one of the multiple servers, creating computing instances corresponding to each model block; wherein the computing instances corresponding to each model block are allocated resources from the resources of at least one server that match the resource requirements of each model block; and deploying each model block to the computing instances corresponding to each model block, so that each model block runs in the computing instances corresponding to each model block.

[0008] This method, when deploying a model, creates separate computing instances with different resource requirements for each model block within the model. Specifically, the resources of the computing instance created for each model block are matched to its resource requirements. Each model block is then deployed to a computing instance that matches its resource requirements, ensuring that each model block can run within that matched instance. This avoids resource mismatch and improves resource utilization and model operating efficiency.

[0009] In one possible implementation, the method further includes: adjusting the number of computing instances and / or the resources allocated to the first model block; wherein the first model block is any model block among a plurality of model blocks. For example, the adjustment of the number of computing instances and / or the resources allocated to the first model block is unaffected by the number of computing instances and / or the resources allocated to other model blocks.

[0010] This implementation allows for adjustments to computational instances at the model block level, enabling fine-grained adjustments to computational instances and fine-grained allocation of resources, thereby further improving resource utilization and model operating efficiency.

[0011] In one possible implementation, the multiple model blocks include a first model block and a second model block, where the first model block and the second model block are any one of the multiple model blocks. The process involves using the resources of at least one of the multiple servers to create a computing instance corresponding to each model block, including: obtaining first deployment instruction information, which indicates that the first model block and the second model block need to be deployed to the same server or different servers; and based on the first deployment instruction information, creating the first computing instance and the second computing instance on the same server or different servers among the multiple servers; wherein the first computing instance corresponds to the first model block, and the second computing instance corresponds to the second model block.

[0012] The first deployment instruction specifically indicates whether the first model block and the second model block have affinity. If the first model block and the second model block have affinity, then the first computing instance and the second computing instance are deployed on the same server. If the first model block and the second model block do not have affinity, then the first computing instance and the second computing instance are deployed on different servers. In this way, flexible deployment of model blocks in the infrastructure can be achieved to meet different business needs.

[0013] In one possible implementation, after deploying each model block to the corresponding computing instance, the method further includes: when the computing instance corresponding to the first model block among the multiple model blocks fails, creating a third computing instance corresponding to the first model block using the resources of several servers among the multiple servers; wherein the third computing instance allocates resources from the resources of the several servers that match the resource requirements of the first model block; the first model block is any model block among the multiple model blocks; and deploying the first model block to the third computing instance so that the first model block runs in the third computing instance.

[0014] In this implementation, when the computing instance corresponding to a model block fails, the fault can be repaired simply by recreating the computing instance corresponding to that model block. This does not affect the operation of other model blocks, thereby reducing the impact on related services, such as reducing the duration of service interruption and reducing the scope of the fault, thus improving service availability.

[0015] In one possible implementation, after deploying each model block to the corresponding computing instance, the method further includes: when the resource requirements of the first model block among the multiple model blocks change, creating a fourth computing instance corresponding to the first model block using the resources of several servers among the multiple servers; wherein the fourth computing instance allocates resources from the resources of the several servers to match the changed resource requirements of the first model block; the first model block is any model block among the multiple model blocks; and deploying the first model block to the fourth computing instance so that the first model block runs in the fourth computing instance.

[0016] In this implementation, when the resource requirements of a model block change, the corresponding computing instance of the model block can be recreated to adapt to the changed resource requirements without affecting the operation of other model blocks. This reduces the impact on related business operations, such as reducing the duration of business interruption and the scope of failure, thereby improving the availability of the model's business operations.

[0017] In one possible implementation, each of the multiple model blocks is deployed to a computation instance corresponding to that model block, including: when the size of the first model block is less than a threshold, deploying the program code and parameters of the first model block to the computation instance corresponding to the first model block simultaneously; when the size of the first model block is greater than or equal to the threshold, deploying the program code of the first model block to the computation instance corresponding to the first model block; wherein the computation instance corresponding to the first model block obtains the parameters of the first model block by executing the program code; wherein the first model block is any model block among the multiple model blocks.

[0018] The size of a model block refers to the amount of data in the model block. The amount of data in a model block is obtained by adding the amount of data in the program code of the model block and the amount of data in the parameters of the model block.

[0019] In this implementation, different deployment methods can be selected based on the size of the model block, thus enabling flexible deployment of the model.

[0020] In one possible implementation, the method further includes: obtaining second deployment instruction information, the second deployment instruction information being used to instruct the deployment of the model to be deployed as multiple replicas; wherein each replica of the multiple replicas includes multiple model blocks; utilizing the resources of at least one of the multiple servers to create a computing instance corresponding to each model block in the multiple model blocks, including: creating a computing instance corresponding to each model block in each replica; deploying each model block to the computing instance corresponding to each model block, including: deploying each model block in each replica to the computing instance corresponding to each model block in each replica.

[0021] In this context, a copy of the model is also called an instance of the model. A copy of the model is a specific model.

[0022] In this implementation, the user can instruct the deployment of multiple replicas of the model, each replica comprising the multiple model blocks. To this end, a compute instance matching the resource requirements can be created for each model block of each replica, and each model block of each replica can be deployed to the compute instance matching the resource requirements. This improves the resource utilization of the compute instances in each replica and the operational efficiency of each replica.

[0023] Secondly, a management platform is provided, which is connected to an infrastructure including at least one data center. The at least one data center contains multiple servers, each with resources. The management platform can utilize the resources of each of the multiple servers to create computing instances. The management platform includes: an acquisition module for acquiring multiple model blocks, wherein the multiple model blocks are obtained by splitting a model to be deployed, and the resource requirements of different model blocks are different; a creation module for using the resources of at least one of the multiple servers to create computing instances corresponding to each model block, wherein the computing instance corresponding to each model block is allocated resources from the resources of at least one server that match the resource requirements of each model block; and a deployment module for deploying each model block to the computing instance corresponding to each model block, so that each model block runs in the computing instance corresponding to each model block.

[0024] In one possible implementation, the creation module is also used to: adjust the number of computing instances corresponding to the first model block and / or the resources allocated to it; wherein the first model block is any model block among a plurality of model blocks.

[0025] In one possible implementation, the multiple model blocks include a first model block and a second model block, where the first model block and the second model block are any one of the multiple model blocks; the creation module is used to: obtain first deployment instruction information, which indicates that the first model block and the second model block need to be deployed to the same server or different servers; based on the first deployment instruction information, create a first computing instance and a second computing instance on the same server or different servers among the multiple servers; wherein, the first computing instance corresponds to the first model block, and the second computing instance corresponds to the second model block.

[0026] In one possible implementation, after deploying each model block to the corresponding computing instance, the creation module is further configured to: when the computing instance corresponding to the first model block among multiple model blocks fails, utilize the resources of several servers among multiple servers to create a third computing instance corresponding to the first model block; wherein, the third computing instance is allocated resources from the resources of the several servers that match the resource requirements of the first model block; the first model block is any model block among multiple model blocks; the deployment module is further configured to: deploy the first model block to the third computing instance so that the first model block runs in the third computing instance.

[0027] In one possible implementation, after deploying each model block to the corresponding computing instance, the creation module is further configured to: when the resource requirements of the first model block among multiple model blocks change, utilize the resources of several servers among multiple servers to create a fourth computing instance corresponding to the first model block; wherein, the fourth computing instance allocates resources from the resources of the several servers to match the changed resource requirements of the first model block; the first model block is any model block among multiple model blocks; the deployment module is further configured to: deploy the first model block to the fourth computing instance so that the first model block runs in the fourth computing instance.

[0028] In one possible implementation, the deployment module is further configured to: when the size of the first model block is less than a threshold, simultaneously deploy the program code and parameters of the first model block to the computation instance corresponding to the first model block; when the size of the first model block is greater than or equal to the threshold, deploy the program code of the first model block to the computation instance corresponding to the first model block; wherein the computation instance corresponding to the first model block obtains the parameters of the first model block by executing the program code; wherein the first model block is any model block among multiple model blocks.

[0029] In one possible implementation, the acquisition module is further configured to: acquire second deployment instruction information, the second deployment instruction information being used to instruct the model to be deployed as multiple replicas; wherein each of the multiple replicas includes multiple model blocks; the creation module is further configured to: create a computing instance corresponding to each model block in each replica; the deployment module is further configured to: deploy each model block in each replica to the computing instance corresponding to each model block in each replica.

[0030] Thirdly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method provided in the first aspect.

[0031] Fourthly, a computing device cluster is provided, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method provided in the second aspect.

[0032] Fifthly, a computer-readable storage medium is provided, including computer program instructions that, when executed by a cluster of computing devices, execute the method provided in the first aspect.

[0033] The beneficial effects of the second to fifth aspects can be referred to the introduction of the beneficial effects of the first aspect above, and will not be repeated here. Attached Figure Description

[0034] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0035] Figure 2 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0036] Figure 3 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0037] Figure 4 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0038] Figure 5 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0039] Figure 6 is a flowchart of a model deployment method provided in an embodiment of this application;

[0040] Figure 7 is a schematic diagram of the structure of a management platform provided in an embodiment of this application;

[0041] Figure 8 is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0042] Figure 9 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0043] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0044] The solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings. In the embodiments of this application, "multiple" refers to two or more objects, and "various types" refers to two or more types. Terms such as "first," "second," etc., are only used to distinguish similar objects and are not necessarily used to describe a specific order or number of objects.

[0045] To facilitate understanding of the solutions provided in the embodiments of this application, the technical terms that may be involved in the embodiments of this application will be introduced first.

[0046] Cloud technology refers to a managed service that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing.

[0047] Infrastructure refers to the facilities that provide computing resources, storage resources, and / or communication resources for computing services. Infrastructure may include at least one data center, and each data center includes multiple servers. These servers act as hosts for creating computing instances. In cloud technology, infrastructure is also called cloud infrastructure, used to provide computing resources, storage resources, and / or communication resources for cloud computing services.

[0048] Management platform: A platform provided by the computing service provider for interaction with users. Users can register accounts on the management platform and rent computing services using those accounts, thus becoming tenants of the computing services. Users can manage computing instances in the infrastructure through the management platform. In scenarios where the computing service is a cloud computing service, the management platform can be called a cloud management platform.

[0049] A compute instance, also known as a virtual computing device or virtual computing instance, refers to a complete computer system simulated by software, possessing a complete hardware system and running in a completely isolated environment. The hardware system of a compute instance is physical hardware virtualized into a virtual hardware system. A complete compute instance has an independent virtual computing system (e.g., a central processing unit (CPU)) and a virtual disk. Compute instances that need to communicate with the outside world also have independent virtual network interface cards (NICs). Typical compute instances include virtual machines (VMs), containers, and elastic cloud servers (ECS).

[0050] A model is typically constructed from artificial neural networks, consisting of multiple layers. Generally, these layers are arranged sequentially, with the output of one layer serving as the input to the next. A neural network layer can be simply referred to as a network layer or layer. Additionally, a model can be divided into program code and model parameters.

[0051] The program code of a model refers to the code that describes or represents the structure of the model, the data interaction relationships between the model's operators, etc.

[0052] Model parameters are numerical values ​​that describe or represent the various variables and relationships in a model, such as weights and biases. Model parameters are learned through model training.

[0053] Operator: A computational unit in a model. Generally speaking, an operator is the smallest indivisible computational unit in a model.

[0054] Large models (LMs) are machine learning models with a large number of parameters and complex computational structures. They are typically built from deep neural networks and have billions or even hundreds of billions of parameters. Large models can be applied in various fields, such as natural language processing, computer vision, and speech recognition.

[0055] Model blocks are obtained by breaking down a model. For example, they can be obtained by splitting the neural network that makes up the model into layers, or by splitting the model at the operator level, or by splitting the model according to stages.

[0056] A stage refers to a phase in which a model processes data. Typically, data processing by a model involves multiple stages. For example, a large language model's data processing includes a context filling (P) stage and a decoding (D) stage. The model block used to perform the prefill stage is called the prefill model block, and the model block used to perform the decode stage is called the decode model block.

[0057] Due to the massive number of parameters in large models, a single computing instance can hardly meet the resource requirements for running such a large model. One approach is to split a large model into multiple model blocks when it needs to be run. These different model blocks are then deployed to different computing instances, allowing different model blocks to run on different instances. While this approach achieves distributed execution of large models, it may lead to resource mismatches.

[0058] Specifically, different model blocks within the same model may have different resource requirements. Some model blocks require more computing power, some more storage resources, and some more communication resources. For example, some vendors in the industry break down large models into prefill model blocks, decode model blocks, and scheduling model blocks. The prefill model block has a very long input information length, and computation latency is a decisive factor, resulting in a computational bottleneck. The decode model block has a fixed input information length of 1, with lower computation latency, but suffers from a memory access bottleneck. In other words, the prefill model block requires more computing power, and the decode model block requires more storage resources. The scheduling model block is used to exchange computation results between the prefill and decode model blocks, requiring more communication resources.

[0059] The above approach, when deploying model blocks to compute instances, does not consider the resource requirements of the model blocks, potentially leading to resource mismatch. Multiple compute instances used to deploy multiple model blocks of the same model may have identical resource specifications. When running model blocks, some compute instances may have an excess of one type of resource while another is in short supply. Therefore, resource mismatch not only wastes resources but also reduces the model's operational efficiency.

[0060] In view of this, embodiments of this application provide a model deployment method. This method can create computing instances with different resources based on the different resource requirements of different model blocks of the model to be deployed, and deploy the model blocks to computing instances whose resource requirements match those of the model blocks. This avoids resource mismatch and improves resource utilization and model operating efficiency.

[0061] Next, the model deployment method provided in the embodiments of this application will be described in detail.

[0062] Figure 1 illustrates a system architecture that can be used to implement this method. The system architecture includes a management platform 100 and infrastructure 200 connected to the management platform 100. As shown in Figure 1, infrastructure 200 may include at least one data center, each data center housing multiple servers. Each server has resources, such as computing resources, storage resources, and communication resources. The management platform 100 can manage the resources of the servers in infrastructure 200, for example, by using the resources of the servers in infrastructure 200 to create computing instances.

[0063] Users can deploy model 300 in infrastructure 200 through management platform 100. Model 300 can also be referred to as model 300 to be deployed. Model 300 can be a large model, such as a large language model or a large vision model. Model 300 can be broken down into multiple model blocks, such as model block 310, model block 320, model block 330, etc. For example, the neural network constituting model 300 can be split into multiple model blocks by layer. For example, model 300 can be split into multiple model blocks by operators at the granularity. For example, model 300 can be split into multiple model blocks according to stages. For example, model 300 can be a large language model, and these multiple model blocks can include prefill model blocks, decode model blocks, and scheduling model blocks.

[0064] The resource requirements of different model blocks obtained from the same model may be different. For example, the prefill model block requires more computing resources, the decode model block requires more storage resources, and the scheduling model block requires more communication resources.

[0065] Different model blocks derived from the same model perform different functions. In other words, when processing the business logic of the model, different model blocks within the model have different roles. Therefore, model blocks can be referred to as roles, and different model blocks have different roles.

[0066] When deploying model 300, management platform 100 can obtain the resource requirements of each model block from the multiple model blocks derived from model 300. Then, based on the resource requirements of each model block, management platform 100 uses the server resources in infrastructure 200 to create computing instances for deploying each model block, ensuring that these instances are allocated resources matching their respective resource requirements. Next, management platform 100 deploys each model block to the computing instance matching its resource requirements, allowing each model block to run on that instance. This avoids resource mismatch.

[0067] In the following text, for ease of description, the computational instance used to deploy the model block can be referred to as the computational instance corresponding to the model block.

[0068] In some embodiments, the management platform 100 may include a model management layer 110 and a compute instance management layer 120. The model management layer 110 may be referred to as level 1 (L1), and the compute instance management layer 120 may be referred to as level 2 (L2). The model management layer 110 may include a model description module 111 and a controller module 112. The compute instance management layer 120 may include a compute instance creation module 121 and a compute instance adjustment module 122.

[0069] The model description module 111 is used to describe the resource requirements of multiple model blocks of model 300, the deployment instructions of model blocks, the life cycle of model 300, the services provided by the model, and other information.

[0070] There can be multiple deployment indications for a model block. For example, deployment indication A1 indicates whether a model block has affinity with other model blocks, and deployment indication A2 indicates the number of deployment replicas of model 300. A replica of model 300 is also called an instance of model 300, and the number of deployment replicas of model 300 is also called the number of model instances, which is the number of deployments of model 300. For example, if the number of deployment replicas is 3, then 3 instances of model 300 are deployed in infrastructure 200. If two model blocks have affinity, it means that the two model blocks need to be deployed on the same server. If two model blocks do not have affinity, it means that the two model blocks need to be deployed on different servers.

[0071] Among them, deployment instruction information A1 and deployment instruction information A2 can be issued by the user, and the model description module 111 can obtain and record the deployment instruction information issued by the user.

[0072] The control module 112 can manage the computation instances corresponding to the model blocks of model 300 according to the model's lifecycle, such as creating and destroying them.

[0073] For example, the model description module 111 can describe relevant information through a custom resource definition (CRD), and the relevant information described by the CRD can be called a custom resource (CR).

[0074] Control module 112 can control computing instance management layer 120 based on the information described by model description module 111. Specifically, according to deployment instruction information A2, control module 112 can instruct computing instance creation module 121 to select a server for creating the model block based on the affinity of the model block. According to deployment instruction information A1, control module 112 can instruct computing instance creation module 121 to create a corresponding number of computing instances based on the number of deployment replicas of model 300. According to the lifecycle of model 300, control module 112 can instruct computing instance adjustment module 122 to create and destroy computing instances corresponding to model blocks of model 300. Control module 112 can instruct computing instance adjustment module 122 to upgrade, expand, or shrink the computing instances corresponding to model blocks of model 300. For example, when the resource requirements of a model block change, control module 112 can instruct computing instance adjustment module 122 to upgrade, expand, or shrink the computing instances corresponding to that model block. For example, when the resource requirements of a model block increase, the control module 112 can instruct the computation instance adjustment module 122 to increase the number of computation instances corresponding to that model block or allocate more resources to the computation instances corresponding to that model block. When the resource requirements of a model block decrease, the control module 112 can instruct the computation instance adjustment module 122 to reduce the number of computation instances corresponding to that model block or reduce the resources allocated to the computation instances corresponding to that model block.

[0075] The computation instance creation module 121 can utilize the server resources in infrastructure 200 to create computation instances corresponding to model blocks. One or more computation instances can be created for the same model block. For example, two computation instances can be created for model blocks 310, 320, and 330 respectively. Exemplarily, the computation instance creation module 121 can also record network topology information between different model blocks of model 300. This network topology information can be recorded in a ConfigMap or a custom file. The computation instance creation module 121 then mounts the ConfigMap or custom file containing the network topology information between model blocks to the computation instance corresponding to the model block, enabling the computation instance to obtain the network topology information between model blocks and perform data interaction.

[0076] In some embodiments, referring to FIG3, the management platform 100 may include a management component, a control component, and an operator component. The management component can receive a model to be deployed, such as model 300, from a user, and then instruct the control component to deploy model 300 to the infrastructure 200. The control component can deploy the model by invoking the operator component according to the model deployment method provided in this embodiment. The operator component consists of the aforementioned model management layer 110 and computing instance management layer 120; thus, the control component can deploy the model by invoking the operator component according to the model deployment method provided in this embodiment.

[0077] In some embodiments, the model block is deployed to the computing instance corresponding to the model block by distributing the image to that instance. That is, the computing instance corresponding to the model block can download the image of the model block to achieve deployment. The appropriate deployment method is selected based on the size of the model block. The size of the model block refers to the amount of data in the model block; the amount of data in model block 310 is obtained by adding the amount of data in the model block's program code and the amount of data in the model block's parameters. The following description uses model block 310 as an example.

[0078] In one example of this embodiment, as shown in Figure 3, when the size of model block 310 is less than a preset threshold B1, the image of model block 310 includes the program code and parameters of model block 310. The program code of model block 310 refers to the code corresponding to that model block in the program code of the model to which model block 310 belongs (i.e., model 300), and the parameters of model block 310 refer to the parameters corresponding to model block 310 in the model parameters of the model to which model block 310 belongs. In this example, both the program code and parameters of model block 310 are contained in the image of model block 310. By downloading the image of model block 310, the computing instance corresponding to the model block can simultaneously deploy the program code and parameters of model block 310 to that computing instance.

[0079] For example, the threshold B1 can be preset empirically or experimentally. For example, the threshold B1 can be set based on the memory size of the computing instance.

[0080] In another example of this embodiment, as shown in Figure 4, when the size of model block 310 is greater than or equal to threshold B1, the image of model block 310 includes the program code of the model block but not the parameters of model block 310. In this example, the program code of model block 310 is deployed to the computing instance corresponding to model block 310 by distributing the image to the computing instance corresponding to model block 310. The computing instance corresponding to model block 310 downloads the parameters of model block 310 from the model file by running the program code of model block 310. This completes the deployment of model block 310 in the computing instance. Specifically, a model file including the parameters of model block 310 can be mounted to the computing instance corresponding to model block 310, allowing the computing instance to download the parameters of model block 310 from the model file.

[0081] In another example of this embodiment, when the size of the program code of model 300 is less than the threshold B1, the program codes of multiple model blocks can be integrated into the same image. Then, this image is distributed to the computation instance corresponding to each of the multiple model blocks. The computation instance corresponding to the model block selects the program code of that model in the image and executes the program code to implement the function of that model block. In one example of this embodiment, when the size of model 300 is less than the threshold B1, the image also includes parameters of multiple model blocks. The computation instance corresponding to the model block selects the parameters of that model block in the image by executing the program of that model block and uses the parameters to perform related calculations. In another example of this embodiment, when the size of model 300 is greater than or equal to the threshold B1, the image does not include the parameters of the model blocks. The computation instance corresponding to the model block downloads the parameters of that model block by executing the parameters of that model block and uses the parameters to perform related calculations. Here, the size of the program code refers to the amount of data in the program code. The size of model 300 is obtained by adding the amount of data in the program code of model 300 and the amount of data in the parameters of model 300.

[0082] In another example of this embodiment, the image of model block 310 can be a pre-built image, requiring no user provision. Specifically, the program code of some model blocks in model 300 may be generic code; for example, the program code of model blocks used to implement model block input or output functions is typically generic. This generic code can be pre-built in the management platform 100, which can include this generic code in the image to obtain the pre-built image. In one example, a model file including the parameters of model block 310 can be mounted to the computing instance corresponding to model block 310, allowing the computing instance to download the parameters of model block 310 from the model file. Here, the model file is provided by the user. In another example, the parameters of model block 310 can be copied to the pre-built image of model block 310, thereby including the parameters of model block 310 in the image of model block 310. The parameters of model block 310 are provided by the user.

[0083] The above example illustrates the system architecture provided in this application embodiment. Next, taking implementation within this system architecture as an example, the implementation process of the model deployment method provided in this application embodiment is described. This method is executed by the management platform 100, as shown in Figure 6, and includes the following steps.

[0084] Step 601: Obtain multiple model blocks; these multiple model blocks are obtained by splitting model 300, and the resource requirements of different model blocks are different. Model 300 can be referred to as the model to be deployed.

[0085] In step 601, the management platform 100 can acquire and record the resource requirements of each of the multiple model blocks. For example, the different resource requirements of different model blocks specifically refer to the different types of resources required. For instance, some model blocks require more computing resources than others, some require more storage resources, some require more communication resources, and so on. For example, step 601 can be executed by the model management layer 110.

[0086] In some embodiments, the neural network constituting model 300 can be split into layers to obtain the plurality of model blocks. In some embodiments, model 300 can be split at the granularity of an operator to obtain the plurality of model blocks. In some embodiments, model 300 can be split according to stages to obtain the plurality of model blocks. In some embodiments, model 300 can be a large language model, and the plurality of model blocks can include a prefill model block, a decode model block, and a scheduling model block.

[0087] In some embodiments, the user can split model 300 into multiple model blocks. The user can also evaluate or test the resource requirements of each model block. Then, the user can send the multiple model blocks and the resource requirements of each model block to the management platform 100, so that the management platform 100 can obtain the multiple model blocks and the resource requirements of each model block.

[0088] In some embodiments, the management platform 100 can split the model 300 into multiple model blocks, and evaluate or test the resource requirements of each model block. Typically, when a model is to be deployed, the management platform 100 can experiment to find a splitting scheme for the model. For example, it can split the model using a splitting strategy (e.g., random or average splitting), then deploy and run the model blocks obtained under that splitting strategy to obtain the model's running results. The splitting strategy can be adjusted, and the model can be split using the adjusted strategy. Then, the model blocks obtained under the adjusted splitting strategy can be deployed and run to obtain the model's running results. This process is repeated until a better or optimal splitting strategy is found. The model blocks obtained by this better or optimal splitting strategy are the aforementioned multiple model blocks. The resource requirements of each model block are obtained based on its running results.

[0089] Step 602: Using the resources of at least one server in infrastructure 200, create a computing instance corresponding to each of the plurality of model blocks; wherein, the computing instance corresponding to each model block is allocated resources from the resources of the at least one server that match the resource requirements of each model block. For example, step 602 can be performed by the computing instance management layer 120.

[0090] As described above, the management platform 100 can acquire and record the resource requirements of each of the multiple model blocks. In step 602, the management platform 100 can create a computing instance corresponding to the model block according to its recorded resource requirements. When creating the computing instance corresponding to the model block, resources matching the resource requirements of the model block are allocated to the computing instance. For example, if computing power resources are likely to become a bottleneck for the operation of a model block, and the model block requires more computing power resources, then more computing power resources are allocated to the computing instance corresponding to that model block. If storage resources (e.g., memory) are likely to become a bottleneck for the operation of a model block, and the model block requires more storage resources, then more storage resources are allocated to the computing instance corresponding to that model block. And so on.

[0091] In some embodiments, the management platform 100 creates a computation instance template corresponding to a model block according to the resource requirements of the model block it records. The computation instance template has resources that match the resource requirements of the model block. In step 602, the management platform 100 can use the resources of the at least one server to copy the computation instance template corresponding to the model block to obtain a computation instance corresponding to the model block.

[0092] In some embodiments, the number of computing instances and / or the resources allocated can be adjusted at the model block level. Taking model block 310 as an example, the number of computing instances and / or the resources allocated to model block 310 can be adjusted. The adjustment of the computing instances corresponding to model block 310 is unaffected by the computing instances corresponding to other model blocks. That is, the number of computing instances and / or the resources allocated to model block 310 can be adjusted while keeping the number of computing instances and / or the resources allocated to other model blocks unchanged. For example, if there were originally N computing instances corresponding to model block 310, the number can be adjusted to M through computing instance destruction or creation operations. Here, N and M are both integers greater than or equal to 1, and N and M are not equal. For example, if the original resource allocated to the computing instance corresponding to model block 310 is resource C1, the resource allocated to the computing instance corresponding to model block 310 can be adjusted to resource C2 through resource reallocation operations. Resource C1 and resource C2 are different. Thus, model 300 allows for refined resource allocation, improving resource utilization.

[0093] In some embodiments, the management platform 100 can create compute instances corresponding to different model blocks on the same or different servers based on whether the model blocks have affinity. Specifically, compute instances corresponding to model blocks with affinity are deployed on the same server, while compute instances corresponding to model blocks without affinity are deployed on different servers.

[0094] Taking model blocks 310 and 320 as examples, the management platform 100 can obtain deployment instruction information A1 for model block 310 or model block 320. This deployment instruction information A1 indicates whether model blocks 310 and 320 need to be deployed to the same server or different servers. Deployment instruction information A1 can be issued by the user or determined by the management platform 100 based on the relationship between the model blocks. For example, if model blocks 310 and 320 frequently interact (the output of model block 310 is the input of model block 320, or vice versa), then model blocks 310 and 320 have affinity, and their corresponding computational instances need to be deployed on the same server. As another example, to improve the fault tolerance of model 300 and prevent the failure of all or most model blocks of model 300 from becoming unusable due to a single server failure, the computational instances corresponding to model block 310 and model block 320 need to be deployed on different servers.

[0095] Based on the deployment instruction information A1, the management platform 100 can deploy the computing instances corresponding to model block 310 and model block 320 on the same server or different servers. Specifically, if the deployment instruction information indicates that model block 310 and model block 320 need to be deployed on the same server, the management platform 100 will deploy the computing instances corresponding to model block 310 and model block 320 on the same server. If the deployment instruction information indicates that model block 310 and model block 320 need to be deployed on different servers, the management platform 100 will deploy the computing instances corresponding to model block 310 and model block 320 on different servers.

[0096] In some embodiments, the management platform 100 can also obtain deployment instruction information A2. As described above, deployment instruction information A2 is used to indicate the number of deployment replicas of the model. The number of deployment replicas indicated by deployment instruction information A2 can be set to multiple, that is, deployment instruction information A2 indicates that model 300 is deployed as multiple replicas. Each replica of model 300 includes the multiple model blocks mentioned above. In step 602, a computing instance corresponding to each model block in each replica can be created, wherein the computing instance corresponding to each model block in each replica is allocated resources that match the resource requirements of each model block in that replica.

[0097] Step 603: Deploy each of the multiple model blocks to the corresponding computation instance, so that each model block runs in the corresponding computation instance. The running of the model block can be either training or inference.

[0098] In some embodiments, as described above, there are multiple deployment replicas of model 300. In step 602, a computing instance corresponding to each model block of each replica is created, and each model block of each replica can be deployed to the computing instance corresponding to each model block of each replica.

[0099] Deploying a model block to its corresponding computation instance specifically involves deploying the model block's program code and parameters to that instance. The following explanation will use model block 310 as an example.

[0100] In some embodiments, in step 603, when the size of model block 310 is less than threshold B1, the program code and parameters of model block 310 can be deployed simultaneously to the computing instance corresponding to model block 310. For example, the program code and parameters of model block 310 can be included in the image of model block 310, and then the image of model block 310 can be distributed to the computing instance corresponding to model block 310 to deploy the program code and parameters of model block 310 to the computing instance corresponding to model block 310. For details, please refer to the description of the embodiment shown in Figure 3 above, which will not be repeated here.

[0101] In some embodiments, in step 603, when the size of model block 310 is greater than or equal to threshold B1, the program code of model block 310 can be deployed to the computing instance corresponding to model block 310; wherein, the computing instance corresponding to model block 310 obtains the parameters of model block 310 by executing the program code. For example, the program code of model block 310 can be included in the image of model block 310, and then the image of model block 310 can be distributed to the computing instance corresponding to model block 310. The computing instance obtains the program code of model block 310 from the image and executes the program code of model block 310. The execution of the program code of model block 310 by the computing instance enables the computing instance to download the parameters of model block 310, thereby obtaining the parameters of the model block. For details, please refer to the description of the embodiment shown in Figure 4 above, which will not be repeated here.

[0102] In some embodiments, the program code for the model blocks can be pre-configured by the management platform 100. Some model block program codes are generic, and the management platform 100 can pre-configure these model block program codes. In step 603, the management platform 100 can deploy the pre-configured model block program code to the computing instance corresponding to the model block. Therefore, the user is not required to provide the program code for the model block, saving user effort and improving the user experience.

[0103] In some embodiments, after deploying the model block to the corresponding computing instance, maintenance can be performed at the computing instance level, such as fault management and recovery. Taking the computing instance corresponding to model block 310 as an example, after step 603, when the computing instance corresponding to model block 310 fails, the management platform 100 can utilize the resources of several servers in the infrastructure 200 to recreate the computing instance corresponding to model block 310; wherein, the recreated computing instance allocates resources from the resources of the several servers to match the resource requirements of model block 310. Here, the recreated computing instance can be referred to as the third computing instance. Then, the management platform 100 deploys model block 310 to the recreated computing instance (i.e., the third computing instance) so that model block 310 runs in the recreated computing instance. Here, the several servers can be servers other than the at least one of the aforementioned servers, or servers among the at least one of the aforementioned servers.

[0104] In this way, when a computing instance corresponding to a certain model block fails, the fault can be repaired simply by recreating the computing instance corresponding to that model block, without affecting the operation of other model blocks. This reduces the impact on the business of model 300, such as reducing the duration of business interruption and reducing the scope of the fault, thereby improving the availability of the model's business.

[0105] In some embodiments, after deploying a model block to a corresponding computing instance, maintenance can be performed at the model block level, such as resource adjustments. Taking model block 310 as an example, the program code or parameters of model block 310 may change, such as through upgrades, bug fixes, or parameter corrections. Changes in the program code or parameters of model block 310 may lead to changes in its resource requirements. In this scenario, the computing instance corresponding to model block 310 can be recreated to adapt to the changes in its resource requirements. Specifically, after step 603, if the resource requirements of model block 310 change, the management platform 100 can utilize the resources of several servers in the infrastructure 200 to recreate the computing instance corresponding to model block 310; wherein, the recreated computing instance is allocated resources from the resources of these servers that match the changed resource requirements of model block 310. This recreated computing instance can be referred to as the fourth computing instance. Next, the management platform 100 deploys the model block 310 to the newly created computing instance (i.e., the fourth computing instance) so that the model block 310 runs in the newly created computing instance. The plurality of servers can be servers other than the at least one of the aforementioned servers, or servers among the at least one of the aforementioned servers.

[0106] In this way, when the resource requirements of a certain model block change, only the computing instance corresponding to that model block needs to be recreated to adapt to the changed resource requirements, without affecting the operation of other model blocks. This reduces the impact on the business of Model 300, such as reducing the duration of business interruption and reducing the scope of the failure, thereby improving the availability of the model's business.

[0107] Furthermore, the above description, using model block 310 as an example, illustrates the specific operation of the model deployment method provided in this application embodiment. Any of the aforementioned multiple model blocks can be operated in accordance with the content described above. In other words, model block 310 can be any of the aforementioned multiple model blocks.

[0108] In summary, the model deployment method provided in this application can create computing instances with different resources according to the different resource requirements of different model blocks of the model to be deployed, and deploy the model blocks to computing instances that match the resource requirements of the model blocks. In this way, resource mismatch is avoided, and resource utilization and model running efficiency are improved.

[0109] Based on the above description, this application embodiment also provides a management platform 700. The management platform 700 is connected to infrastructure, which includes at least one data center. The at least one data center has multiple servers, each of which has resources. The management platform can utilize the resources of each of the multiple servers to create computing instances. As shown in Figure 7, the management platform 700 includes:

[0110] The acquisition module 710 is used to acquire multiple model blocks; wherein, the multiple model blocks are obtained by splitting the model to be deployed, and the resource requirements of different model blocks are different;

[0111] A creation module 720 is used to create a computing instance corresponding to each model block among the plurality of server resources; wherein, the computing instance corresponding to each model block is allocated resources from the resources of the at least one server that match the resource requirements of each model block;

[0112] The deployment module 730 is used to deploy each model block to the computing instance corresponding to each model block, so that each model block runs in the computing instance corresponding to each model block.

[0113] In some embodiments, the creation module 720 is further configured to: adjust the number of computing instances corresponding to the first model block and / or the resources allocated to it; wherein the first model block is any model block among the plurality of model blocks.

[0114] In some embodiments, the plurality of model blocks includes a first model block and a second model block, wherein the first model block and the second model block are any one of the plurality of model blocks; the creation module 720 is further configured to: obtain first deployment instruction information, wherein the first deployment instruction information is used to indicate that the first model block and the second model block need to be deployed to the same server or different servers; based on the first deployment instruction information, create a first computing instance and a second computing instance on the same server or different servers among the plurality of servers; wherein the first computing instance corresponds to the first model block, and the second computing instance corresponds to the second model block.

[0115] In some embodiments, after deploying each model block to the computing instance corresponding to each model block, the creation module 720 is further configured to: when the computing instance corresponding to the first model block among the plurality of model blocks fails, utilize the resources of several servers among the plurality of servers to create a third computing instance corresponding to the first model block; wherein, the third computing instance is allocated resources from the resources of the several servers that match the resource requirements of the first model block; the first model block is any model block among the plurality of model blocks; the deployment module 730 is further configured to: deploy the first model block to the third computing instance so that the first model block runs in the third computing instance.

[0116] In some embodiments, after deploying each model block to the computing instance corresponding to each model block, the creation module 720 is further configured to: when the resource requirements of the first model block among the plurality of model blocks change, utilize the resources of several servers among the plurality of servers to create a fourth computing instance corresponding to the first model block; wherein, the fourth computing instance is allocated resources from the resources of the several servers that match the changed resource requirements of the first model block; the first model block is any model block among the plurality of model blocks; the deployment module 730 is further configured to: deploy the first model block to the fourth computing instance so that the first model block runs in the fourth computing instance.

[0117] In some embodiments, the deployment module 730 is further configured to: when the size of the first model block is less than a threshold, simultaneously deploy the program code and parameters of the first model block to the computing instance corresponding to the first model block; when the size of the first model block is greater than or equal to the threshold, deploy the program code of the first model block to the computing instance corresponding to the first model block; wherein the computing instance corresponding to the first model block obtains the parameters of the first model block by executing the program code; wherein the first model block is any model block among the plurality of model blocks.

[0118] In some embodiments, the acquisition module 710 is further configured to: acquire second deployment instruction information, the second deployment instruction information being used to instruct the model to be deployed as multiple replicas; wherein each of the multiple replicas includes the multiple model blocks; the creation module 720 is further configured to: create a computing instance corresponding to each model block in each replica; the deployment module 730 is further configured to: deploy each model block in each replica to the computing instance corresponding to each model block in each replica.

[0119] The acquisition module 710, creation module 720, and deployment module 730 can all be implemented in software or hardware. For example, the implementation of the acquisition module 710 will be described below. Similarly, the implementation of the creation module 720 and deployment module 730 can refer to the implementation of the acquisition module 710.

[0120] As an example of a software functional unit, module 710 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, module 710 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same Availability Zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0121] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0122] As an example of a hardware functional unit, the acquisition module 710 may include at least one computing device, such as a server. Alternatively, the acquisition module 710 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0123] The multiple computing devices included in the acquisition module 710 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the acquisition module 710 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the acquisition module 710 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0124] It should be noted that, in other embodiments, the acquisition module 710 can be used to execute any step in the method shown in FIG. 6, the creation module 720 can be used to execute any step in the method shown in FIG. 6, and the deployment module 730 can be used to execute any step in the method shown in FIG. 6. The steps implemented by the acquisition module 710, the creation module 720, and the deployment module 730 can be specified as needed. By implementing different steps in the method shown in FIG. 6 through the acquisition module 710, the creation module 720, and the deployment module 730, all functions of the management platform 700 can be realized.

[0125] This application also provides a computing device 800. As shown in FIG8, the computing device 800 includes: a bus 802, a processor 804, a memory 806, and a communication interface 808. The processor 804, the memory 806, and the communication interface 808 communicate with each other via the bus 802. The computing device 800 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 800.

[0126] Bus 802 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 8, but this does not imply that there is only one bus or one type of bus. Bus 802 can include pathways for transmitting information between various components of computing device 800 (e.g., memory 806, processor 804, communication interface 808).

[0127] Processor 804 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0128] Memory 806 may include volatile memory, such as random access memory (RAM). Memory 806 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0129] The memory 806 stores executable program code, which the processor 804 executes to implement the functions of the aforementioned acquisition module 710, creation module 720, and deployment module 730, thereby realizing the method shown in FIG6. That is, the memory 806 stores instructions for executing the method shown in FIG6.

[0130] The communication interface 808 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 800 and other devices or communication networks.

[0131] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0132] As shown in Figure 9, the computing device cluster includes at least one computing device 800. The memory 806 of one or more computing devices 800 in the computing device cluster may store the same instructions for executing the method shown in Figure 6.

[0133] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG. 6. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG. 6.

[0134] It should be noted that the memory 806 in different computing devices 800 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the management platform 700. That is, the instructions stored in the memory 806 of different computing devices 800 can implement the functions of one or more modules among the acquisition module 710, creation module 720, and deployment module 730.

[0135] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, two computing devices 800A and 800B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 806 in computing device 800A stores instructions for executing the functions of the acquisition module 710. Simultaneously, the memory 806 in computing device 800B stores instructions for executing the functions of the creation module 720 and the deployment module 730.

[0136] It should be understood that the functions of computing device 800A shown in Figure 10 can also be performed by multiple computing devices 800. Similarly, the functions of computing device 800B can also be performed by multiple computing devices 800.

[0137] This application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similarly referred to the connection method of the computing device cluster shown in Figures 9 and 10. The difference is that the memory 806 of one or more computing devices 800 in this computing device cluster can store the same instructions for executing the method shown in Figure 6.

[0138] In some possible implementations, the memory 806 of one or more computing devices 800 in the computing device cluster may also store partial instructions for executing the method shown in FIG. 6. In other words, a combination of one or more computing devices 800 can jointly execute the instructions for executing the method shown in FIG. 6.

[0139] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, the at least one computing device performs the method shown in FIG6.

[0140] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a host migration device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform the method shown in FIG6.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model deployment method, characterized in that, The method is applied to a management platform connected to an infrastructure, the infrastructure including at least one data center, wherein multiple servers are configured in the at least one data center, each of the multiple servers having resources, and the management platform is capable of creating computing instances using the resources of each of the multiple servers; the method includes: Obtain multiple model blocks; wherein, the multiple model blocks are obtained by splitting the model to be deployed, and the resource requirements of different model blocks are different; Using the resources of at least one of the plurality of servers, a computing instance corresponding to each of the plurality of model blocks is created; wherein, the computing instance corresponding to each model block is allocated resources from the resources of the at least one server that match the resource requirements of each model block; Each model block is deployed to the computing instance corresponding to each model block, so that each model block runs in the computing instance corresponding to each model block.

2. The method according to claim 1, characterized in that, The method further includes: adjusting the number of computing instances corresponding to the first model block and / or the resources allocated to it; wherein the first model block is any model block among the plurality of model blocks.

3. The method according to claim 1 or 2, characterized in that, The plurality of model blocks includes a first model block and a second model block, wherein the first model block and the second model block are any one of the plurality of model blocks; The step of utilizing the resources of at least one of the multiple servers to create a computational instance corresponding to each of the multiple model blocks includes: Obtain first deployment instruction information, which is used to indicate whether the first model block and the second model block need to be deployed to the same server or different servers; Based on the first deployment instruction information, the first computing instance and the second computing instance are created on the same server or different servers among the multiple servers; wherein, the first computing instance corresponds to the first model block, and the second computing instance corresponds to the second model block.

4. The method according to any one of claims 1-3, characterized in that, After deploying each model block to the computation instance corresponding to each model block, the method further includes: When the computing instance corresponding to the first model block among the plurality of model blocks fails, a third computing instance corresponding to the first model block is created using the resources of several servers among the plurality of servers; wherein, the third computing instance is allocated resources from the resources of the several servers that match the resource requirements of the first model block; the first model block is any model block among the plurality of model blocks; The first model block is deployed to the third computing instance so that the first model block runs in the third computing instance.

5. The method according to any one of claims 1-4, characterized in that, After deploying each model block to the computation instance corresponding to each model block, the method further includes: When the resource requirements of the first model block among the plurality of model blocks change, a fourth computing instance corresponding to the first model block is created using the resources of several servers among the plurality of servers; wherein, the fourth computing instance is allocated resources from the resources of the several servers that match the changed resource requirements of the first model block; the first model block is any model block among the plurality of model blocks; The first model block is deployed to the fourth computing instance so that the first model block runs in the fourth computing instance.

6. The method according to any one of claims 1-5, characterized in that, The step of deploying each of the plurality of model blocks to the computing instance corresponding to each model block includes: When the size of the first model block is less than the threshold, the program code and parameters of the first model block are deployed to the computing instance corresponding to the first model block at the same time. When the size of the first model block is greater than or equal to the threshold, the program code of the first model block is deployed to the computing instance corresponding to the first model block; wherein, the computing instance corresponding to the first model block obtains the parameters of the first model block by executing the program code; The first model block is any model block among the plurality of model blocks.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: obtaining second deployment instruction information, the second deployment instruction information being used to instruct the model to be deployed as multiple replicas; wherein each of the multiple replicas includes the multiple model blocks; The step of using the resources of at least one of the multiple servers to create a computing instance corresponding to each model block in the multiple model blocks includes: creating a computing instance corresponding to each model block in each replica; The step of deploying each model block to the computing instance corresponding to each model block includes: deploying each model block in each replica to the computing instance corresponding to each model block in each replica.

8. A management platform, characterized by The management platform is connected to an infrastructure, which includes at least one data center. The at least one data center houses multiple servers, each of which has resources. The management platform can utilize the resources of each of the multiple servers to create computing instances. The management platform includes: The acquisition module is used to acquire multiple model blocks; wherein, the multiple model blocks are obtained by splitting the model to be deployed, and the resource requirements of different model blocks are different; A creation module is used to create a computing instance corresponding to each model block among the plurality of server blocks by utilizing the resources of at least one of the server blocks; wherein, the computing instance corresponding to each model block is allocated resources from the resources of the at least one server that match the resource requirements of each model block; The deployment module is used to deploy each model block to the computing instance corresponding to each model block, so that each model block runs in the computing instance corresponding to each model block.

9. The management platform according to claim 8, characterized in that, The creation module is further configured to: adjust the number of computing instances corresponding to the first model block and / or the resources allocated to it; wherein the first model block is any model block among the plurality of model blocks.

10. The management platform according to claim 8 or 9, characterized in that, The plurality of model blocks includes a first model block and a second model block, wherein the first model block and the second model block are any one of the plurality of model blocks; The creation module is used for: Obtain first deployment instruction information, which is used to indicate whether the first model block and the second model block need to be deployed to the same server or different servers; Based on the first deployment instruction information, the first computing instance and the second computing instance are created on the same server or different servers among the multiple servers; wherein, the first computing instance corresponds to the first model block, and the second computing instance corresponds to the second model block.

11. The management platform according to any one of claims 8-10, characterized in that, After deploying each model block to the corresponding computing instance, the creation module is further configured to: when the computing instance corresponding to the first model block among the plurality of model blocks fails, utilize the resources of several servers among the plurality of servers to create a third computing instance corresponding to the first model block; wherein, the third computing instance is allocated resources from the resources of the several servers that match the resource requirements of the first model block; the first model block is any model block among the plurality of model blocks; The deployment module is further configured to: deploy the first model block to the third computing instance, so that the first model block runs in the third computing instance.

12. The management platform according to any one of claims 8-11, characterized in that, After deploying each model block to the corresponding computing instance, the creation module is further configured to: when the resource requirements of the first model block among the plurality of model blocks change, utilize the resources of several servers among the plurality of servers to create a fourth computing instance corresponding to the first model block; wherein, the fourth computing instance is allocated resources from the resources of the several servers that match the changed resource requirements of the first model block; the first model block is any model block among the plurality of model blocks; The deployment module is further configured to: deploy the first model block to the fourth computing instance, so that the first model block runs in the fourth computing instance.

13. The management platform according to any one of claims 8-12, characterized in that, The deployment module is also used for: When the size of the first model block is less than the threshold, the program code and parameters of the first model block are deployed to the computing instance corresponding to the first model block at the same time. When the size of the first model block is greater than or equal to the threshold, the program code of the first model block is deployed to the computing instance corresponding to the first model block; wherein, the computing instance corresponding to the first model block obtains the parameters of the first model block by executing the program code; The first model block is any model block among the plurality of model blocks.

14. The management platform according to any one of claims 8-13, characterized in that, The acquisition module is further configured to: acquire second deployment instruction information, the second deployment instruction information being used to instruct the model to be deployed as multiple replicas; wherein, each of the multiple replicas includes the multiple model blocks; The creation module is also used to: create a computation instance corresponding to each model block in each copy; The deployment module is further configured to: deploy each model block in each replica to the computing instance corresponding to each model block in each replica.

15. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-7.

17. A computer program product containing instructions, characterized in that, When the instruction is executed by a cluster of computer devices, the cluster of computer devices performs the method as described in any one of claims 1-7.

Images