Task processing method based on cloud management platform, and cloud management platform

Abstract

Disclosed in the present application are a task processing method based on a cloud management platform, and a cloud management platform, which can improve the user experience. The method in the present application comprises: a user being able to send a task processing request to a cloud management platform, wherein the request is used for indicating a first session task including a plurality of first subtasks and a second session task including a plurality of second subtasks; after acquiring a third subtask among the plurality of first subtasks and a fourth subtask among the plurality of second subtasks, the cloud management platform determining, on the basis of the processing order of the third subtask among the plurality of first subtasks, the duration required for obtaining an inference result of the first session task, and determining, on the basis of the processing order of the fourth subtask among the plurality of second subtasks, the duration required for obtaining an inference result of the second session task; and if the duration required for obtaining the inference result of the first session task is less than the duration required for obtaining the inference result of the second session task, the cloud management platform being capable of notifying a large language model that the third subtask should be preferentially processed.

Description

A task processing method based on a cloud management platform and the cloud management platform

[0001] This application claims priority to Chinese Patent Application No. 202411983782.5, filed with the State Intellectual Property Office of China on December 30, 2024, entitled "A Task Processing Method Based on a Cloud Management Platform and a Cloud Management Platform", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of cloud technology, and in particular to a task processing method based on a cloud management platform and a cloud management platform. Background Technology

[0003] With the rapid development of cloud technology, cloud providers can offer task processing services to users. These services can assist users in completing their conversational tasks by deploying large language models in the cloud, thereby meeting users' job requirements.

[0004] In related technologies, when a user has job requirements, they can provide multiple session tasks to the cloud management platform of a cloud vendor. Each session task can consist of multiple subtasks. Since the cloud management platform schedules the large language model to process these multiple session tasks on a subtask-by-subtask basis, when the large language model is processing a subtask of a session task, if subtasks of that session task or other session tasks arrive at the cloud management platform for processing, the cloud management platform can add these subtasks to a queue and schedule them for processing by the large language model according to their arrival time.

[0005] In the above process, the cloud management platform only schedules the large language model to process these subtasks sequentially based on the arrival time of these subtasks in the queue. Since these subtasks in the queue often belong to different session tasks, this will cause different session tasks to be processed by the large language model in an interleaved manner, resulting in mutual interference in the processing of different session tasks, which in turn leads to excessive processing time for each session task and affects the user experience. Summary of the Invention

[0006] This application provides a task processing method and a cloud management platform based on a cloud management platform, which can reduce the time users wait for task processing, thereby improving the user experience to a certain extent.

[0007] A first aspect of this application provides a task processing method based on a cloud management platform, wherein the cloud management platform implementing the method can manage infrastructure providing cloud services, which is used to run large language models. The method includes:

[0008] When a user needs to handle a first session task and a second session task, the user can input a task processing request into the task processing interface provided by the cloud management platform. The cloud management platform can then receive the task processing request sent by the user through the task processing interface. This request indicates the first session task, which can be broken down into multiple first sub-tasks, and the second session task, which can be broken down into multiple second sub-tasks.

[0009] Next, the cloud management platform can start processing the first session task and the second session task based on the request. Suppose that the third subtask of the first subtask of the first session task and the fourth subtask of the second subtask of the second session task arrive at the cloud management platform, the cloud management platform can determine whether to execute the third subtask or the fourth subtask.

[0010] Then, the cloud management platform can use the processing order of the third subtask in multiple first subtasks to determine the time required to obtain the inference result of the first session task, and use the processing order of the fourth subtask in multiple second subtasks to determine the time required to obtain the inference result of the second session task.

[0011] Subsequently, if the time required to obtain the reasoning result of the first session task is less than the time required to obtain the reasoning result of the second session task, it indicates that the priority of the first session task is higher than the priority of the second session task, and the cloud management platform can call the task processing module to prioritize the processing of the third sub-task.

[0012] As can be seen from the above method, the cloud management platform can automatically assess the time required to obtain the inference result of the first session task and the second session task based on the third subtask to be processed in the first session task and the fourth subtask to be processed in the second session task, i.e., the priority of the first session task and the priority of the second session task. If the priority of the first session task is higher (i.e., the time required to obtain the inference result of the first session task is shorter), the cloud management platform can prioritize scheduling the large language model to process the third subtask to be processed in the first session task. In this way, when new subtasks to be processed in the first session task arrive at the cloud management platform, the cloud management platform can still assess the priority of the first session task, and usually the priority of the first session task is still higher. This allows the cloud management platform to continuously schedule the large language model to process the subsequent subtasks in the first session task, enabling the cloud management platform to complete the first session task quickly and preventing the first and second session tasks from being processed interchangeably, thereby reducing the user's waiting time for task processing and improving the user experience to a certain extent.

[0013] In one possible implementation, the cloud management platform determines the time required to obtain the inference result of the first session task based on the processing order of the third subtask among multiple first subtasks. This includes: the cloud management platform, based on the processing order of the third subtask among multiple first subtasks, determining the fifth subtask whose processing order follows the third subtask; and the cloud management platform, based on the feature information of the third subtask and the feature information of the fifth subtask, determining the time required to obtain the inference result of the first session task. In the aforementioned implementation, when the third subtask of the first session task arrives at the cloud management platform, the cloud management platform can independently determine the processing order of the third subtask among multiple first subtasks. Then, based on the processing order of the third subtask among multiple first subtasks, the cloud management platform determines all fifth subtasks whose processing order follows the third subtask. Subsequently, the cloud management platform can obtain the feature information of the third subtask and the feature information of these fifth subtasks, and based on the feature information of the third subtask and the feature information of these fifth subtasks, determine the time required to obtain the inference result of the first session task. Therefore, the cloud management platform can perform feature analysis on each subtask of the first session task (the currently pending third subtask and the subsequent fifth subtask) to obtain the feature information of each subtask in the first session task. This allows for an accurate determination of the remaining time required to complete the first session task (i.e., the time required to obtain the inference result of the first session task), which helps to shorten the remaining time and further reduce the time users spend waiting for task processing, thereby further improving the user experience.

[0014] In one possible implementation, the characteristic information of the third subtask includes at least one of the following: the amount of resources required to process the third subtask and the number of third subtasks; the characteristic information of the fifth subtask includes at least one of the following: the amount of resources required to process the fifth subtask, the number of fifth subtasks, and the waiting time required before processing the fifth subtask. In the aforementioned implementation, the cloud management platform can accurately determine the remaining time required to complete the first session task by analyzing the characteristic information such as the resources required to process each subtask to be processed in the first session task, the number of these subtasks, and the waiting time required before processing these subtasks.

[0015] In one possible implementation, determining that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and then the cloud management platform notifying the large language model to process the third sub-task, includes: determining that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and that the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, and then the cloud management platform notifying the large language model to process the third sub-task. In the aforementioned implementation, after obtaining the time required to obtain the inference result of the first session task and the time required to obtain the inference result of the second session task, since the cloud management platform records the maximum time required to complete the first session task, the cloud management platform can typically first determine whether the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, and then compare the time required to obtain the inference result of the first session task with the time required to obtain the inference result of the second session task. If the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, and the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, it means that the processing of the first session task has not exceeded its completion time limit, and the first session task has a higher priority. Therefore, the cloud management platform prioritizes scheduling the large language model to process the third sub-task.

[0016] In one possible implementation, the method further includes: determining that the time required to obtain the inference result of the first session task is greater than the maximum time required to complete the first session task, and then the cloud management platform notifies the large language model to process the third sub-task. In the aforementioned implementation, if the time required to obtain the inference result of the first session task is greater than the maximum time required to complete the first session task, the time required to obtain the inference result of the first session task is no longer compared with the time required to obtain the inference result of the second session task, because the processing of the first session task has exceeded its completion time limit. Therefore, the cloud management platform can directly notify the large language model to prioritize the processing of the third sub-task.

[0017] In one possible implementation, the infrastructure includes compute nodes running a large language model. The compute nodes' GPU memory is used to store the inference results obtained by the large language model from processing the second session task. The cloud management platform instructs the large language model to process the third sub-task by: instructing the compute nodes to migrate the inference results obtained by the large language model from processing the second session task from GPU memory to the compute node's RAM or hard disk, thus obtaining adjusted GPU memory; and instructing the compute nodes to use the large language model to process the third sub-task and store the inference results obtained by the large language model from processing the third sub-task in the adjusted GPU memory. In the aforementioned implementation, the GPU memory of the compute node where the large language model resides is used to store the inference results obtained by the large language model from processing the first and second session tasks. When the cloud management platform needs to schedule the large language model to prioritize processing the third sub-task of the first session task, the cloud management platform can instruct the compute node to migrate the inference results obtained by the large language model from processing the second session task from the compute node's GPU memory to the compute node's RAM or hard disk, thereby obtaining adjusted GPU memory. Then, the cloud management platform can instruct the computing node to use the large language model to process the third subtask, so that the computing node can directly store the inference results obtained by the large language model from processing the third subtask in the adjusted GPU memory of the computing node. Thus, since the first session task has a higher priority than the second session task, the cloud management platform can release the GPU memory resources on the computing node where the large language model resides that were occupied by the second session task, and provide this GPU memory to the first session task, thereby reducing the latency of the large language model processing the first session task and accelerating its processing speed.

[0018] In one possible implementation, the method further includes: the cloud management platform instructing the computing node to retrieve the inference result obtained by the large language model from the adjusted GPU memory for processing the third sub-task; the cloud management platform instructs the target tool to process the inference result obtained by the large language model from processing the third sub-task to obtain a new inference result, which is then used by the large language model to process the fifth sub-task. In the aforementioned implementation, the cloud management platform can also instruct the computing node where the large language model resides to read the inference result obtained by the large language model from processing the third sub-task of the first session task from its adjusted GPU memory, and provide this inference result to the target tool for further processing to obtain a new inference result. Subsequently, the cloud management platform can invoke the large language model to continue processing the fifth sub-task of the first session task based on the new inference result. Therefore, since the first session task has a higher priority than the second session task, the cloud management platform can prioritize allocating resources such as the target tool to the first session task, thereby further accelerating the processing speed of the first session task.

[0019] In one possible implementation, compute nodes comprise physical servers, bare-metal servers, virtual machines, containers, or microvirtual machines.

[0020] A second aspect of this application provides a cloud management platform for managing infrastructure providing cloud services. The infrastructure is used to run a large language model. The cloud management platform includes: a receiving module for receiving a task processing request sent by a user, the task processing request indicating a first session task and a second session task, the first session task including multiple first subtasks and the second session task including multiple second subtasks; an acquisition module for acquiring, based on the task processing request, a third subtask to be processed among the multiple first subtasks and a fourth subtask to be processed among the multiple second subtasks; a determining module for determining, based on the processing order of the third subtask among the multiple first subtasks, the time required to obtain the inference result of the first session task, and based on the processing order of the fourth subtask among the multiple second subtasks, the time required to obtain the inference result of the second session task; and a first notification module for notifying the large language model to process the third subtask when determining that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task.

[0021] In one possible implementation, the determining module is configured to: determine, based on the processing order of the third subtask among the multiple first subtasks, a fifth subtask whose processing order is after the third subtask; and determine, based on the feature information of the third subtask and the feature information of the fifth subtask, the time required to obtain the inference result of the first session task.

[0022] In one possible implementation, the characteristic information of the third subtask includes at least one of the following: the amount of resources required to process the third subtask and the number of third subtasks; the characteristic information of the fifth subtask includes at least one of the following: the amount of resources required to process the fifth subtask, the number of fifth subtasks, and the waiting time required before processing the fifth subtask.

[0023] In one possible implementation, the first notification module is configured to: determine that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and that the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, and notify the large language model to process the third subtask.

[0024] In one possible implementation, the cloud management platform further includes a second notification module, which determines that the time required to obtain the inference result of the first session task is greater than the maximum time required to complete the first session task, and notifies the large language model to process the third subtask.

[0025] In one possible implementation, the infrastructure includes compute nodes running a large language model, the compute nodes' GPU memory being used to store the inference results obtained by the large language model processing the second session task; a first notification module is used to: notify the compute nodes to migrate the inference results obtained by the large language model processing the second session task from GPU memory to the compute nodes' RAM or hard disk, obtaining adjusted GPU memory; and notify the compute nodes to use the large language model to process a third subtask, and store the inference results obtained by the large language model processing the third subtask in the adjusted GPU memory.

[0026] In one possible implementation, the cloud management platform further includes: a third notification module for notifying computing nodes to retrieve the inference results obtained by the large language model from the adjusted video memory for processing the third subtask; and a fourth notification module for notifying the target tool to process the inference results obtained by the large language model from processing the third subtask to obtain new inference results, which are then used by the large language model to process the fifth subtask.

[0027] In one possible implementation, compute nodes comprise physical servers, bare-metal servers, virtual machines, containers, or microvirtual machines.

[0028] A third aspect of this application provides a computing device cluster, which includes at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to cause the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect according to the instructions.

[0029] A fourth aspect of this application provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method described in the first aspect or any possible implementation of the first aspect.

[0030] A fifth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0031] In this embodiment, when a user needs to process a first session task and a second session task, the user can send a task processing request to the task processing interface provided by the cloud management platform. This request indicates the first and second session tasks. The first session task may contain multiple first sub-tasks, and the second session task may contain multiple second sub-tasks. If a third sub-task pending from among the multiple first sub-tasks and a fourth sub-task pending from among the multiple second sub-tasks arrive at the cloud management platform, the platform can determine the time required to obtain the inference result of the first session task based on the processing order of the third sub-task among the multiple first sub-tasks, and determine the time required to obtain the inference result of the second session task based on the processing order of the fourth sub-task among the multiple second sub-tasks. If the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, the cloud management platform can notify the large language model to prioritize processing the third sub-task of the first session task. In the aforementioned process, the cloud management platform can automatically assess the time required to obtain the inference result of the first session task and the second session task based on the third subtask to be processed in the first session task and the fourth subtask to be processed in the second session task, i.e., the priority of the first session task and the priority of the second session task. If the priority of the first session task is higher (i.e., the time required to obtain the inference result of the first session task is shorter), the cloud management platform can prioritize scheduling the large language model to process the third subtask to be processed in the first session task. In this way, when new subtasks to be processed in the first session task arrive at the cloud management platform, the cloud management platform can still assess the priority of the first session task, and usually the priority of the first session task is still higher. This allows the cloud management platform to continuously schedule the large language model to process subsequent subtasks in the first session task, enabling the cloud management platform to quickly complete the first session task and prevent the first and second session tasks from being processed interchangeably, thereby reducing the user's waiting time for task processing and improving the user experience to a certain extent. Attached Figure Description

[0032] Figure 1 is a schematic diagram of a communication system provided in an embodiment of this application;

[0033] Figure 2 is a schematic diagram of the dependencies between subtasks provided in an embodiment of this application;

[0034] Figure 3 is a schematic diagram of the structure of a cloud management platform provided in an embodiment of this application;

[0035] Figure 4 is a schematic diagram of the prior information provided in an embodiment of this application;

[0036] Figure 5 is another schematic diagram of the prior information provided in the embodiments of this application;

[0037] Figure 6 is another schematic diagram of the prior information provided in the embodiments of this application;

[0038] Figure 7 is another schematic diagram of the prior information provided in the embodiments of this application;

[0039] Figure 8 is another schematic diagram of the prior information provided in the embodiments of this application;

[0040] Figure 9 is a flowchart illustrating a task processing method based on a cloud management platform provided in an embodiment of this application.

[0041] Figure 10 is another structural schematic diagram of the cloud management platform provided in the embodiment of this application;

[0042] Figure 11 is a schematic diagram of a computing device provided in an embodiment of this application;

[0043] Figure 12 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0044] Figure 13 is a schematic diagram of computer devices in a computer cluster connected via a network according to an embodiment of this application. Detailed Implementation

[0045] This application provides a task processing method and a cloud management platform based on a cloud management platform, which can reduce the time users wait for task processing, thereby improving the user experience to a certain extent.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0047] With the rapid development of cloud technology, cloud providers can offer task processing services to users. These services can assist users in completing their conversational tasks by deploying large language models in the cloud, thereby meeting users' job requirements.

[0048] In related technologies, when a user has job requirements, they can provide multiple session tasks to the cloud management platform of a cloud vendor. Each session task can consist of multiple subtasks. Since the cloud management platform schedules the large language model to process these multiple session tasks on a subtask-by-subtask basis, when the large language model is processing a subtask of a session task, if subtasks of that session task or other session tasks arrive at the cloud management platform for processing, the cloud management platform can add these subtasks to a queue and schedule them for processing by the large language model according to their arrival time. For example, suppose the user provides the cloud management platform with session task 1, session task 2, and session task 3. Session task 1 contains subtasks 1 to 10, session task 2 contains subtasks 11 to 20, and session task 3 contains subtasks 21 to 30. Suppose that the cloud management platform first schedules the large language model to process subtask 21, and during this process, subtask 1 and subtask 11 arrive at the cloud management platform in sequence. Therefore, after subtask 21 is completed, the cloud management platform can schedule the large language model to process subtask 1 first, and then process subtask 11.

[0049] In the above process, the cloud management platform schedules the large language model to process these subtasks sequentially based solely on their arrival times in the queue. Since these subtasks often belong to different session tasks, this causes the large language model to process different session tasks interleaved, leading to interference between their processing and resulting in excessively long processing times for each session task, thus impacting user experience. For example, if the large language model processes subtask 31 first, then subtask 1, and then subtask 11, this interleaving of session tasks 1, 2, and 3 will significantly increase the processing time for these three session tasks. This results in longer waiting times for the user to receive the results of any of these session tasks, degrading the user experience.

[0050] To address the aforementioned problems, this application provides a task processing method based on a cloud management platform. This method can be implemented through a cloud service system. Figure 1 is a schematic diagram of the structure of the cloud service system provided in this application embodiment. As shown in Figure 1, the cloud service system includes infrastructure that can provide cloud services and a cloud management platform that manages this infrastructure. The cloud management platform and the infrastructure are described in detail below:

[0051] A cloud management platform can centrally manage the infrastructure of the entire cloud service system. (For example, the cloud management platform can create multiple computing nodes from the infrastructure to provide cloud services to a user, according to the user's instructions. These computing nodes can run large language models, which can process multiple session tasks under the guidance of the cloud management platform to meet the user's job requirements, etc.) The cloud management platform can also be open to users outside the cloud service system and respond to their requests. For example, the cloud management platform can provide various interfaces such as login interfaces and task processing interfaces for user clients (e.g., the user's terminal device or the browser on the terminal device, etc.) to access. Specifically, the cloud management platform can authenticate a user's client through the login interface, allowing the user's client to log in after successful authentication. Similarly, the cloud management platform can also allow the user's client to send task processing requests to the cloud management platform through the task processing interface. These task processing requests indicate multiple session tasks for the user, each session task may contain multiple sub-tasks. Based on the task processing request, a subtask of a certain session task and a subtask of another session task arrive at the cloud management platform. The cloud management platform can determine the (remaining) time required to complete the session task based on the processing order of the subtask of the session task among the multiple subtasks of the session task (it should be noted that the time required to complete the session task can also be referred to as the (remaining) time required to obtain the (final) inference result of the session task). And based on the processing order of the subtask of another session task among the multiple subtasks of another session task, the cloud management platform can determine the (remaining) time required to complete the other session task (it should be noted that the time required to complete the other session task can also be referred to as the (remaining) time required to obtain the (final) inference result of the other session task). If the time required to complete the current session task is less than the time required to complete another session task (i.e., the priority of the current session task is higher than the priority of another session task), the cloud management platform can notify the large language model to prioritize the processing of the subtasks of the current session task. If the time required to complete the current session task is greater than the time required to complete another session task (i.e., the priority of the current session task is lower than the priority of another session task), the cloud management platform can notify the large language model to prioritize the processing of the subtasks of the other session task.

[0052] The infrastructure includes a computing cluster that provides cloud services to the user. This cluster may contain multiple computing nodes created by a cloud management platform, and these nodes can run large language models (LLMs). It's important to note that the cloud management platform can allocate certain computing resources (e.g., central processing unit (CPU) and graphics processing unit (GPU), storage resources (e.g., hard drives, memory, and video memory), and network resources (e.g., network interface cards) to each of these computing nodes. For any one of these computing nodes, the GPU of that node can store the large language model and the inference results obtained from processing the conversation tasks using the large language model (including intermediate and final inference results) in its video memory. In other words, the GPU of that computing node can read and run the large language model from its video memory, then use the large language model to process the user's various conversation tasks. The resulting inference results can be stored in the video memory of that computing node's GPU, thus reducing the latency of conversation task processing (because the video memory of that computing node is owned by that node's GPU).

[0053] Furthermore, in some cases, due to limited video memory space, the GPU of this compute node can migrate data stored in its video memory (e.g., inference results of some low-priority session tasks) to the compute node's main memory or hard disk to free up video memory space, resulting in adjusted video memory. In this way, the GPU of this compute node can have more available adjusted video memory space, supporting the processing of high-priority session tasks for large language models; this will not be elaborated upon here.

[0054] Furthermore, for any one of a user's multiple session tasks (corresponding to multiple applications), the cloud management platform can break down the session task into multiple subtasks (also called multiple stages) with a certain processing order. The processing order of these subtasks can also be understood as the dependencies between them. Specifically, for two subtasks with adjacent processing orders, the subtask processed earlier is dependent on the subtask processed later. In other words, the inference result obtained by the large language model from the subtask processed earlier (i.e., the output of the subtask processed earlier) can be used as the input for the large language model when the subtask processed later is processed. For example, as shown in Figure 2 (Figure 2 is a schematic diagram of the dependency relationship between subtasks provided in the embodiments of this application), suppose the cloud management platform receives a user's session task 1. In the process of processing session task 1, it can split session task 1 into subtasks 1 to 6. The input of subtask 1 is the user data provided by the cloud management platform to the large language model. After the input of subtask 1 is processed by the large language model, the output of subtask 1 can be obtained (equivalent to subtask 1 being processed by the large language model). After certain processing (some target tools), the output of subtask 1 can be used as the input of subtask 2. After the input of subtask 2 is processed by the large language model, the output of subtask 2 can be obtained. ... After certain processing, the output of subtask 5 can be used as the input of subtask 6. After the input of subtask 6 is processed by the large language model, the output of subtask 6 can be obtained. It is understandable that the output of subtask 6 is the final inference result obtained by the large language model in processing conversation task 1, and the outputs of subtasks 1 to 5 are the intermediate inference results obtained by the large language model in processing conversation task 1. Moreover, the processing order of these 6 subtasks is subtask 1 → subtask 2 → subtask 3 → subtask 4 → subtask 5 → subtask 6. Among them, the parallelism of subtask 1 and subtask 2 is 3, the parallelism of subtask 3 and subtask 6 is 1, and the parallelism of subtask 4 and subtask 5 is 2.

[0055] It is worth noting that when user data arrives at the cloud management platform (as a request), it is equivalent to subtask 1 arriving at the cloud management platform. Furthermore, since the large language model and target tools typically run on compute nodes, when the compute node returns the processed output of subtask 1 to the cloud management platform (as a request), it is equivalent to subtask 2 arriving at the cloud management platform. The same applies to subsequent subtasks, which will not be elaborated further here.

[0056] Furthermore, as shown in Figure 3 (Figure 3 is a structural schematic diagram of a cloud management platform provided in an embodiment of this application), the cloud management platform may include a session task feature analysis module, a session task scheduling module, and a session task resource provisioning module. These three modules are described below:

[0057] (1) The session task feature analysis module includes a pre-built database containing prior information on various applications. It should be noted that "application" here can be understood as the type of user's session task. The task processing interface provided by the cloud management platform displays various application options (which can also be understood as a sub-interface of the task processing interface). Therefore, the user selects the application option corresponding to their session task in the task processing interface and enters the user's pending session task at that option. In this way, the task processing request sent by the user through the task processing interface not only indicates the user's pending (one or more) session tasks but also indicates which application these session tasks correspond to.

[0058] In this way, after receiving a task processing request from the task processing interface, the session task feature analysis module can determine that the user's session task belongs to a certain application based on the request. When a subtask in the session task arrives at the cloud management platform, the session task feature analysis module of the cloud management platform can use the prior information of the application in the database to perform feature analysis on the subtask of the session task. Thus, the feature information of the subtask in the session task (including the amount of resources required to process the subtask and the number of subtasks, etc.) and the feature information of the other subtasks whose processing order is after the subtask (including the amount of resources required to process the other subtasks, the number of other subtasks, and the waiting time before processing the other subtasks, etc.) are provided to the session task scheduling module.

[0059] For example, suppose the database contains prior information for applications 1 to n. The prior information for application 1 includes:

[0060] In the aforementioned prior information, application-1 refers to application 1. The prior information of application 1 can be obtained by the cloud management platform calling the large language model, which is based on several session tasks belonging to application 1 previously provided by the user. Specifically, stage-1 refers to the prior information of the first subtask of these session tasks, prompt_tokens refers to the length of the input of the first subtask of these session tasks (for example, in these session tasks, the input length of the first subtask of session task 1 is 500, the input length of the first subtask of session task 2 is 540, etc.), completion_tokens refers to the length of the output of the first subtask of these session tasks, and requests... `t_parallelism` refers to the parallelism of the first subtask of these session tasks; `loop_times` refers to the number of loops required for these session tasks from the current stage (i.e., the total number of subtasks, including the first subtask and those processed after it); `next_stage` refers to the next stage after the first subtask, i.e., the second subtask; `gap_to_next_stage` refers to the interval between the first and second subtasks (i.e., the waiting time before processing the second subtask). Similarly, `stage-2`, `stage-3`, etc., follow the same logic and will not be elaborated further here.

[0061] Therefore, when the session task feature analysis module receives a user's pending session task, and this session task belongs to application 1, if the first subtask of this session task arrives at the session task feature analysis module (i.e., the cloud management platform is currently preparing to call the large language model to process the first subtask of this session task), the session task feature analysis module can read the prior information of the first subtask in application 1 from the database, and use the prior information of the first subtask to predict the length of the input of the first subtask of this session task (generally, the length of the input of the first subtask is usually the data provided by the user, so this length does not need to be predicted), the length of the output of the first subtask of this session task, the total number of the first subtask and the subtasks processed after the first subtask in this session task (i.e., the number of loops), and the parallelism of the first subtask of this session task.

[0062] Next, the conversation task feature analysis module can read the prior information of the second subtask (stage-2) in application 1 from the database, and use the prior information of the second subtask to predict the length of the input of the second subtask of the conversation task, the length of the output of the second subtask of the conversation task, the total number of the second subtask and the subtasks whose processing order is after the second subtask in the conversation task, the parallelism of the second subtask of the conversation task, and the waiting time required before processing the second subtask of the conversation task (because the output of the first subtask of the conversation task needs to be processed by some target tools (including other models, algorithms, software, etc. besides the large language model) before the new output of the first subtask can be used as the input of the second subtask of the conversation task. Therefore, the large language model needs to wait for the output of the first subtask to be processed by the target tool before processing the second subtask of the conversation task).

[0063] ...

[0064] Finally, the session task feature analysis module can read the prior information of the nth subtask (stage-n) in application 1 from the database, and use the prior information of the nth subtask to predict the length of the input of the nth subtask of the session task, the length of the output of the nth subtask of the session task, the total number of the nth subtask and the subtasks whose processing order is after the nth subtask in the session task, the parallelism of the nth subtask of the session task, and the waiting time required before processing the nth subtask of the session task.

[0065] Based on this, the session task feature analysis module can ultimately analyze the following feature information of the first to nth subtasks in the session task: the amount of resources required to process the first session task (determined by the length of the input of the first subtask, the length of the output of the first subtask, the parallelism of the first subtask, etc.), the amount of resources required to process the second subtask, ... the amount of resources required to process the nth subtask, the first subtask in the session task, and its processing order. The final total number of subtasks after the first subtask (for example, the final total number is n, which is determined by the total number of the first subtask and the subtasks processed after the first subtask, the second subtask and the subtasks processed after the second subtask, ..., the nth subtask and the subtasks processed after the nth subtask, etc.), the waiting time required before processing the second subtask of the session task, ..., the waiting time required before processing the nth subtask of the session task.

[0066] In another example, if the second subtask of the session task reaches the session task feature analysis module (i.e., the cloud management platform is currently preparing to call the large language model to process the second subtask of the session task, and the first subtask of the session task has already been processed by the large language model), the session task feature analysis module can use the prior information of the second subtask, the prior information of the third subtask, and the prior information of the mth subtask in application 1 (m can be equal to n or not equal to n) to perform feature analysis, thereby analyzing the feature information of the second to mth subtasks in the session task, which will not be elaborated here.

[0067] It should be understood that the above examples are only used to introduce the prior information of each stage (subtask) in Application 1. In actual applications, the prior information of the same stage or different stages of different applications is usually different. For example, in Figures 4 to 8 (Figure 4 is a schematic diagram of the prior information provided in the embodiment of this application, Figure 5 is another schematic diagram of the prior information provided in the embodiment of this application, Figure 6 is another schematic diagram of the prior information provided in the embodiment of this application, Figure 7 is another schematic diagram of the prior information provided in the embodiment of this application, and Figure 8 is another schematic diagram of the prior information provided in the embodiment of this application), the length of the input and the length of the output of a certain stage in Application 2 are shown in Figure 4, the length of the input and the length of the output of a certain stage in Application 3 are shown in Figure 5, the length of the input and the length of the output of a certain stage in Application 4 are shown in Figure 6, the length of the input and the length of the output of a certain stage in Application 5 are shown in Figure 7, and the length of the input and the length of the output of a certain stage in Application 6 are shown in Figure 8.

[0068] (2) The scheduling module between sessions can determine the (remaining) time required to complete the session task based on the feature information of the subtask of the session task and the feature information of the other subtasks whose processing order is after the subtask. The time required to complete the session task can also be understood as the priority of the session task.

[0069] Generally, not only do subtasks of the current session task arrive at the cloud management platform, but subtasks of other session tasks also arrive one after another. Therefore, these subtasks of the current session task will form a queue. In order to determine which subtask of the current session task will be executed first, the inter-session task scheduling module can also obtain the time required to complete the other session tasks. It can then compare the time required to complete the current session task with the time required to complete the other session tasks. If the time required to complete the current session task is less than the time required to complete the other session tasks, the inter-session task scheduling module can notify the resource supply module within the current session task to prioritize scheduling the large language model to process the subtask of the current session task.

[0070] Furthermore, the inter-session task scheduling module also records the maximum duration for completing the current session task and the maximum duration for completing other session tasks. If the duration required to complete the current session task exceeds the maximum duration, regardless of whether the duration required to complete the current session task is less than the duration required to complete other session tasks, the inter-session task scheduling module can directly notify the resource provisioning module within the session task to prioritize scheduling the large language model to process that subtask of the current session task. Similarly, the same applies to other session tasks, which will not be elaborated further here.

[0071] (3) The resource provisioning module within a session task can, upon notification from the scheduling module between session tasks, adjust the memory space of the computing node where the large language model resides. For example, it can notify the computing node to migrate the inference results obtained by the large language model from processing other session tasks from the computing node's memory to the computing node's RAM or hard disk to obtain the adjusted memory. Then, the resource provisioning module within a session task can notify the large language model on the computing node to process the subtask of the session task and store the inference results of the subtask in the adjusted memory, thereby reducing the latency of task processing.

[0072] Furthermore, since this session task has a high priority, the resources within the session task can also prioritize notifying the computing node where the target tool is located, so that the computing node can trigger the target tool to further process the inference result of the subtask, thereby obtaining a new inference result of the subtask, which is then returned to the session task feature analysis module of the cloud management platform. This is equivalent to the next subtask in the session task whose processing order is after this subtask reaching the session task feature analysis module.

[0073] Furthermore, for one or more computing nodes serving users, these computing nodes can take various forms. For example, these computing nodes can be physical servers (containing certain specifications of computing resources, storage resources, and network resources, etc.) selected by the cloud management platform in the infrastructure. Alternatively, they can be bare metal servers (containing certain specifications of computing resources, storage resources, and network resources, etc.) selected by the cloud management platform in the infrastructure. Furthermore, these computing nodes can be virtual machines (VMs) created by the cloud management platform on physical servers or bare metal servers using virtualization technology. They can also be containers (Docker) created by the cloud management platform on physical servers or bare metal servers using virtualization technology. Finally, they can be microVMs created by the cloud management platform on physical servers or bare metal servers using virtualization technology, and so on.

[0074] Furthermore, these computing nodes can be deployed in the same site or different sites. Sites can be presented in various forms, such as a region in the infrastructure, an availability zone in the infrastructure, a data center (DC) in the infrastructure, a room in the infrastructure, or a rack (also known as a physical server group) in the infrastructure, etc.

[0075] Based on the aforementioned cloud service system, when a user needs to handle multiple session tasks, the user can send a task processing request to the task processing interface provided by the cloud management platform. This request indicates the user's multiple session tasks, each of which may contain multiple subtasks with a certain processing order. If any unprocessed subtasks from one session task and other unprocessed subtasks from other session tasks arrive at the cloud management platform, the platform can determine the time required to complete the session task based on the processing order of the unprocessed subtasks within that session task's multiple subtasks, and similarly determine the time required to complete the remaining session tasks based on the processing order of the unprocessed subtasks within the remaining session tasks' multiple subtasks. If the time required to complete the session task is less than the time required to complete the remaining session tasks, the cloud management platform can instruct the large language model to prioritize processing the unprocessed subtasks. In this process, the cloud management platform can independently assess the time required to complete these multiple session tasks based on the unprocessed subtasks, i.e., the priority of these multiple session tasks. If a certain session task has the highest priority (i.e., requires the least time to complete), the cloud management platform can prioritize scheduling the large language model to process the subtasks pending in that session task. In this way, when new subtasks pending in the same session task arrive at the cloud management platform, the platform can still assess the priority of that session task, and typically, the highest priority remains. This allows the cloud management platform to continuously schedule the large language model to process the various subtasks in the session task, enabling the platform to quickly complete each session task and preventing overlapping processing, thus reducing user waiting time and improving user experience. To further understand the workflow of the cloud management platform, Figure 9 is provided below. Figure 9 is a flowchart illustrating a task processing method based on a cloud management platform provided in this embodiment. As shown in Figure 9, this method can be implemented through the cloud service system shown in Figure 1. The cloud service system includes infrastructure providing cloud services to users and a cloud management platform managing this infrastructure, which runs a large language model. The method includes:

[0076] 901. The cloud management platform receives a task processing request sent by the user. The task processing request is used to indicate the first session task and the second session task. The first session task contains multiple first sub-tasks, and the second session task contains multiple second sub-tasks.

[0077] In this embodiment, when a user needs to handle a first session task and a second session task, the cloud management platform can provide a task processing interface to the user's client (e.g., a task processing bar in the user interface). The user can then input their configured task processing request into the task processing interface through their client. In this way, the cloud management platform can receive the task processing request sent by the user's client through the task processing interface. The task processing request may include the first and second session tasks defined by the user. The first and second session tasks are usually two different session tasks, but they can belong to the same application or different applications. For example, the first session task might be to create a 500-word essay on a certain topic, and the second session task might be to obtain a speech manuscript on another topic; therefore, both the first and second session tasks belong to text generation applications. Alternatively, the first session task might be to create a 500-word essay on a certain topic, and the second session task might be to answer a question; therefore, the first session task belongs to a text generation application, and the second session task belongs to a question-and-answer application.

[0078] It should be noted that the first session task can be broken down into multiple first sub-tasks, and the second session task can be broken down into multiple second sub-tasks. The multiple first sub-tasks within the first session task typically have a certain processing order. These multiple first sub-tasks arrive at the cloud management platform sequentially according to this order. Therefore, the cloud management platform can process these multiple first sub-tasks in turn. For example, the first sub-task with the earliest processing order arrives at the cloud management platform first and is processed first; the second-earliest first sub-task arrives next and is processed next, and so on, until the last first sub-task arrives last and is processed last, thus completing the processing of the first session task. Similarly, the multiple second sub-tasks within the second session task are processed in the same way, and will not be elaborated further here.

[0079] It should be understood that in this embodiment, the number of multiple first subtasks is usually uncertain. The cloud management platform can predict the number of these multiple first subtasks during the processing of each first subtask, which will not be elaborated here. Similarly, the number of multiple second subtasks is also like this, and will not be described in detail here.

[0080] It should also be understood that this embodiment is only used as an example to illustrate the two session tasks that the user needs to handle, namely the first session task and the second session task. It does not limit the number of session tasks that the user needs to handle. In actual applications, the user can also handle more session tasks. For example, the user needs to handle the first session task, the second session task, and the third session task.

[0081] 902. Based on the task processing request, the cloud management platform obtains the third subtask to be processed from multiple first subtasks and the fourth subtask to be processed from multiple second subtasks.

[0082] Upon receiving a task processing request, the cloud management platform can begin processing the first session task and the second session task indicated by the task processing request. If a third subtask pending from among the multiple first subtasks of the first session task and a fourth subtask pending from among the multiple second subtasks of the second session task arrive at the cloud management platform, the cloud management platform can determine whether to prioritize executing the third or fourth subtask.

[0083] It should be noted that the third subtask is the specific first subtask that the cloud management platform is currently preparing to process. In other words, among these multiple first subtasks, the third subtask could be the first subtask in the order of processing, a subtask in the middle of the processing order, or even the last subtask in the processing order. Similarly, the fourth subtask is the specific second subtask that the cloud management platform is currently preparing to process. In other words, among these multiple second subtasks, the fourth subtask could be the first second subtask in the order of processing, a subtask in the middle of the processing order, or even the last second subtask in the processing order.

[0084] For example, suppose a user needs to handle session task 1 (the aforementioned first session task) and session task 2 (the aforementioned second session task). The user can log in to the cloud management platform, which provides a user interface including a task processing bar. The user can then input a task processing request into the task processing bar, and the cloud management platform can receive the request through this bar. The task processing request indicates which session task is which, which is which, and whether it belongs to application 1 or application 2.

[0085] Based on the task processing request, the cloud management platform can determine that session task 1 and session task 2 need to be completed. Since session task 1 can be broken down into subtasks 1.1, 1.2, 1.3, etc., and session task 2 can be broken down into subtasks 2.1, 2.2, 2.3, etc., after initiating the processing of session task 1 and session task 2, subtasks 1.1 and 2.1 can arrive at the cloud management platform one after another, forming a task queue. Since a large language model can usually process one subtask at a time, the cloud management platform needs to determine whether to prioritize the execution of subtask 1.1 or subtask 2.1.

[0086] 903. The cloud management platform determines the time required to obtain the inference result of the first session task based on the processing order of the third subtask among multiple first subtasks, and determines the time required to obtain the inference result of the second session task based on the processing order of the fourth subtask among multiple second subtasks.

[0087] After obtaining the third and fourth subtasks, the cloud management platform can obtain the processing order of the third subtask among the multiple first subtasks, and determine the time required to obtain the inference result of the first session task (i.e. the (remaining) time required to complete the first session task) based on the processing order. It can also obtain the processing order of the fourth subtask among the multiple second subtasks, and determine the time required to obtain the inference result of the second session task (i.e. the (remaining) time required to complete the second session task) based on the processing order.

[0088] Specifically, the cloud management platform can determine the time required to obtain the inference result of the first session task in the following ways:

[0089] When the third subtask of the first session task arrives at the cloud management platform, the cloud management platform can determine the processing order of the third subtask among multiple first subtasks based on the tag that the third subtask carries within its own unique tag in the first session task.

[0090] Next, since the first session task belongs to a certain application, the cloud management platform can obtain the prior information of that application. Then, based on the prior information of that application, the cloud management platform determines the fifth subtask, whose processing order is after the third subtask, from among the multiple first subtasks. The number of fifth subtasks can be one or more; there is no specific restriction here.

[0091] Subsequently, after obtaining the prior information of the application, the cloud management platform can acquire the feature information of the third subtask and the fifth subtask, and based on the feature information of the third and fifth subtasks, determine the time required to obtain the inference result of the first session task. The feature information of the third subtask may include one or more of the following: the amount of resources required to process the third subtask (which may be determined by the length of the input of the third subtask, the length of the output of the third subtask, the parallelism of the third subtask, etc.) and the number of third subtasks (which is usually one). The feature information of the fifth subtask may include one or more of the following: the amount of resources required to process the fifth subtask (which may be determined by the length of the input of the fifth subtask, the length of the output of the fifth subtask, the parallelism of the fifth subtask, etc.), the number of fifth subtasks (which may be one or more), and the waiting time required before processing the fifth subtask.

[0092] As in the example above, after it is determined that subtask 1.1 (the aforementioned third subtask) and subtask 2.1 (the aforementioned fourth subtask) have arrived at the cloud management platform, since session task 1 belongs to application 1 and session task 2 belongs to application 2, the cloud management platform can obtain the prior information of application 1 and the prior information of application 2. Based on the prior information of Application 1, the cloud management platform can obtain the characteristic information of subtask 1.1 and the remaining subtasks whose processing order follows subtask 1.1 (e.g., subtask 1.2, subtask 1.3, etc.). This characteristic information may include: the total number of subtask 1.1 and the remaining subtasks whose processing order follows subtask 1.1 (this total number is n, that is, the cloud management predicts that after subtask 1.1, there are subtasks 1.2 to 1.n (i.e., the aforementioned fifth subtask)), the amount of resources required to process subtask 1.1, the amount of resources required to process subtask 1.2, ..., the amount of resources required to process subtask 1.n, the waiting time required before processing subtask 1.2, ..., the waiting time required before processing subtask 1.n.

[0093] Therefore, the cloud management platform can determine the remaining time required to complete session task 1 based on the feature information of subtasks 1.1 to 1.n.

[0094] More specifically, the cloud management platform can determine the time required to obtain the inference result of the second session task in the following ways:

[0095] When the fourth subtask of the second session task arrives at the cloud management platform, the cloud management platform can determine the processing order of the fourth subtask among multiple second subtasks based on the tag that the fourth subtask carries its own unique tag in the second session task.

[0096] Next, since the second session task belongs to a certain application, the cloud management platform can obtain the prior information of that application. Then, based on the prior information of that application, the cloud management platform determines the sixth subtask, whose processing order is after the fourth subtask, from among the multiple second subtasks. The number of sixth subtasks can be one or more; there is no specific restriction here.

[0097] Subsequently, following the prior information of the application, the cloud management platform can obtain the feature information of the fourth subtask and the sixth subtask, and based on the feature information of the fourth and sixth subtasks, determine the time required to obtain the inference result of the second session task. The feature information of the fourth subtask may include one or more of the following: the amount of resources required to process the fourth subtask (which can be determined by the length of the input of the fourth subtask, the length of the output of the fourth subtask, the parallelism of the fourth subtask, etc.) and the number of fourth subtasks (usually one). The feature information of the sixth subtask may include one or more of the following: the amount of resources required to process the sixth subtask (which can be determined by the length of the input of the sixth subtask, the length of the output of the sixth subtask, the parallelism of the sixth subtask, etc.), the number of sixth subtasks (which can be one or more), and the waiting time required before processing the sixth subtask.

[0098] As in the example above, based on the prior information of Application 2, the cloud management platform can also obtain the characteristic information of subtask 2.1 and the remaining subtasks whose processing order is after subtask 2.1 (e.g., subtask 2.2, subtask 2.3, etc.). This characteristic information may include: the total number of subtask 2.1 and the remaining subtasks whose processing order is after subtask 2.1 (this total number is k, that is, the cloud management predicts that after subtask 2.1, there are subtasks 2.2 to 2.k (i.e., the aforementioned sixth subtask)), the amount of resources required to process subtask 2.1, the amount of resources required to process subtask 2.2, ... the amount of resources required to process subtask 2.k, the waiting time required before processing subtask 2.2, ... the waiting time required before processing subtask 2.k.

[0099] Therefore, the cloud management platform can determine the remaining time required to complete session task 2 based on the feature information of subtasks 2.1 to 2.k.

[0100] 904. If the time required to obtain the reasoning result of the first session task is less than the time required to obtain the reasoning result of the second session task, the cloud management platform notifies the large language model to process the third sub-task.

[0101] After obtaining the inference results for the first session task and the second session task, the cloud management platform can compare the two. If the time required to obtain the inference results for the first session task is less than the time required to obtain the inference results for the second session task, it indicates that the priority of the first session task is higher than that of the second session task, and the cloud management platform can call the task processing module to prioritize the processing of the third subtask. If the time required to obtain the inference results for the first session task is greater than the time required to obtain the inference results for the second session task, it indicates that the priority of the first session task is lower than that of the second session task, and the cloud management platform can call the task processing module to prioritize the processing of the fourth subtask.

[0102] Continuing with the example above, after obtaining the remaining time required to complete session task 1 and the remaining time required to complete session task 2, assuming that the remaining time required to complete session task 1 is less than the remaining time required to complete session task 2, the cloud management platform will call the large language model to prioritize processing subtask 1.1. Therefore, subtask 1.1 of session task 1 has been processed, and subtask 1.2 of session task 1 will arrive at the cloud management platform. At this point, the cloud management platform can re-evaluate whether to execute subtask 1.2 or subtask 2.1 first. This process is similar to the previous process and will not be repeated here.

[0103] Specifically, the cloud management platform can also perform the following operations:

[0104] After obtaining the inference result of the first session task and the inference result of the second session task, since the cloud management platform records the maximum time to complete the first session task and the maximum time to complete the second session task, the cloud management platform can usually first determine whether the time required to obtain the inference result of the first session task is less than or equal to the maximum time to complete the first session task, and whether the time required to obtain the inference result of the second session task is less than or equal to the maximum time to complete the second session task.

[0105] If the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, and the time required to obtain the inference result of the second session task is less than or equal to the maximum time required to complete the second session task, the cloud management platform will continue to compare the time required to obtain the inference result of the first session task with the time required to obtain the inference result of the second session task. If the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, the cloud management platform will prioritize scheduling the large language model to process the third subtask. If the time required to obtain the inference result of the first session task is greater than the time required to obtain the inference result of the second session task, the cloud management platform may call the task processing module to process the fourth subtask.

[0106] If the time required to obtain the inference result of the first session task is greater than the maximum time required to complete the first session task, it indicates that the processing of the first session task may have exceeded the completion time limit and needs to be expedited. Therefore, the cloud management platform schedules the large language model to process the third sub-task first.

[0107] If the time required to obtain the inference result of the second session task exceeds the maximum time required to complete the second session task, it indicates that the processing of the second session task may have exceeded the completion time limit and needs to be expedited. The cloud management platform will then schedule the large language model to process the fourth sub-task first.

[0108] Continuing with the example above, after obtaining the remaining time required to complete session task 1 and the remaining time required to complete session task 2, the cloud management platform can obtain the maximum time required to complete session task 1 and the maximum time required to complete session task 2. If the remaining time required to complete session task 1 is less than or equal to the maximum time required to complete session task 1, and the remaining time required to complete session task 2 is less than or equal to the maximum time required to complete session task 2, it means that the processing of session tasks 1 and 2 has not exceeded the completion time limit. Therefore, the cloud management platform can compare the remaining time required to complete session task 1 and the remaining time required to complete session task 2. If the remaining time required to complete session task 1 is less than the remaining time required to complete session task 2, the cloud management platform will call the large language model to prioritize the processing of subtask 1.1.

[0109] More specifically, the cloud management platform can also perform the following operations:

[0110] Since the large language model runs on compute nodes, the GPU memory of these compute nodes is used to store the inference results obtained by the large language model from processing the first and second session tasks. When the cloud management platform needs to schedule the large language model to prioritize processing the third subtask of the first session task, the cloud management platform can instruct these compute nodes to migrate the inference results obtained by the large language model from processing the second session task from the GPU memory of these compute nodes to their RAM or hard disk, thereby freeing up GPU memory space and obtaining adjusted GPU memory for these compute nodes. Then, the cloud management platform can instruct these compute nodes to use the large language model to process the third subtask, and these compute nodes can directly store the inference results obtained by the large language model from processing the third subtask in their adjusted GPU memory.

[0111] More specifically, the cloud management platform can also perform the following operations:

[0112] The cloud management platform can also instruct these compute nodes to read the inference result (i.e., the output of the third subtask) obtained by the large language model from the adjusted GPU memory and provide this inference result to the compute node where the target tool resides. Then, the cloud management platform can instruct the compute node where the target tool resides to further process this inference result using the target tool, thereby obtaining a new inference result. In this way, the compute node where the target tool resides can return the new inference result to the cloud management platform. This is equivalent to a fifth subtask in the first session task (the subtask processed after and adjacent to the third subtask) arriving at the cloud management platform. If the cloud management platform determines that this fifth subtask needs to be processed subsequently, it can use the new inference result as the input to this fifth subtask and provide it to the compute node where the large language model resides, so that these compute nodes can use the large language model to process the fifth subtask based on its input.

[0113] Continuing with the example above, after calling the large language model to prioritize processing subtask 1.1, the inference result obtained by the large language model from processing subtask 1.1 can be obtained. The cloud management platform can trigger the target tool to further process this inference result, causing the target tool to return a new inference result to the cloud management platform. Since the new inference result can be used as input for subtask 1.2, it is equivalent to subtask 1.2 reaching the cloud management platform. Therefore, the cloud management platform will then continue to determine whether to prioritize executing subtask 1.2 or subtask 2.1. If it determines to prioritize executing subtask 1.2, it can provide the new inference result to the large language model, enabling the large language model to complete subtask 1.2 based on the new inference result. This process will not be elaborated further.

[0114] In this embodiment, when a user needs to process a first session task and a second session task, the user can send a task processing request to the task processing interface provided by the cloud management platform. This request indicates the first and second session tasks. The first session task may contain multiple first sub-tasks, and the second session task may contain multiple second sub-tasks. If a third sub-task pending from among the multiple first sub-tasks and a fourth sub-task pending from among the multiple second sub-tasks arrive at the cloud management platform, the platform can determine the time required to obtain the inference result of the first session task based on the processing order of the third sub-task among the multiple first sub-tasks, and determine the time required to obtain the inference result of the second session task based on the processing order of the fourth sub-task among the multiple second sub-tasks. If the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, the cloud management platform can notify the large language model to prioritize processing the third sub-task of the first session task. In the aforementioned process, the cloud management platform can automatically assess the time required to obtain the inference result of the first session task and the second session task based on the third subtask to be processed in the first session task and the fourth subtask to be processed in the second session task, i.e., the priority of the first session task and the priority of the second session task. If the priority of the first session task is higher (i.e., the time required to obtain the inference result of the first session task is shorter), the cloud management platform can prioritize scheduling the large language model to process the third subtask to be processed in the first session task. In this way, when new subtasks to be processed in the first session task arrive at the cloud management platform, the cloud management platform can still assess the priority of the first session task, and usually the priority of the first session task is still higher. This allows the cloud management platform to continuously schedule the large language model to process subsequent subtasks in the first session task, enabling the cloud management platform to quickly complete the first session task and prevent the first and second session tasks from being processed interchangeably, thereby reducing the user's waiting time for task processing and improving the user experience to a certain extent.

[0115] Furthermore, in this embodiment of the application, the cloud management platform has pre-set prior information of each application. By determining the prior information of the application to which the first session task belongs, characteristic information such as the amount of resources required to process each sub-task in the first session task can be obtained, so as to accurately determine the remaining time required to complete the first session task (i.e. the time required to obtain the reasoning result of the first session task), which is conducive to shortening the remaining time, thereby further reducing the time for users to wait for task processing, and thus further improving the user experience.

[0116] Furthermore, in this embodiment, since the priority of the first session task is higher than that of the second session task, the cloud management platform can release the video memory resources occupied by the second session task in the computing node where the large language model is located (used to store the inference results obtained by the large language model in processing the second session task), and provide these video memory resources to the first session task for use, so as to reduce the latency of the large language model in processing the first session task and speed up the processing speed of the first session task.

[0117] Furthermore, in this embodiment, since the priority of the first session task is higher than that of the second session task, the cloud management platform can prioritize scheduling resources such as target tools for the first session task, thereby further accelerating the processing speed of the first session task.

[0118] The above is a detailed description of the task processing method based on a cloud management platform provided in the embodiments of this application. The cloud management platform provided in the embodiments of this application will be described below. Figure 10 is another structural schematic diagram of the cloud management platform provided in the embodiments of this application. As shown in Figure 10, the cloud management platform is used to manage the infrastructure providing cloud services. The infrastructure is used to run a large language model. The cloud management platform includes:

[0119] The receiving module 1001 is used to receive a task processing request sent by the user. The task processing request is used to indicate a first session task and a second session task. The first session task includes multiple first sub-tasks, and the second session task includes multiple second sub-tasks.

[0120] The acquisition module 1002 is used to acquire, based on the task processing request, the third subtask to be processed among multiple first subtasks and the fourth subtask to be processed among multiple second subtasks;

[0121] The determination module 1003 is used to determine the time required to obtain the inference result of the first session task based on the processing order of the third subtask in multiple first subtasks, and to determine the time required to obtain the inference result of the second session task based on the processing order of the fourth subtask in multiple second subtasks.

[0122] The first notification module 1004 is used to determine that the time required to obtain the reasoning result of the first session task is less than the time required to obtain the reasoning result of the second session task, and to notify the large language model to process the third subtask.

[0123] In one possible implementation, the determining module 1003 is configured to: determine, based on the processing order of the third subtask among the multiple first subtasks, a fifth subtask whose processing order is after the third subtask; and determine, based on the feature information of the third subtask and the feature information of the fifth subtask, the time required to obtain the inference result of the first session task.

[0124] In one possible implementation, the characteristic information of the third subtask includes at least one of the following: the amount of resources required to process the third subtask and the number of third subtasks; the characteristic information of the fifth subtask includes at least one of the following: the amount of resources required to process the fifth subtask, the number of fifth subtasks, and the waiting time required before processing the fifth subtask.

[0125] In one possible implementation, the first notification module 1004 is configured to: determine that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and that the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, and notify the large language model to process the third subtask.

[0126] In one possible implementation, the cloud management platform further includes a second notification module, which determines that the time required to obtain the inference result of the first session task is greater than the maximum time required to complete the first session task, and notifies the large language model to process the third subtask.

[0127] In one possible implementation, the infrastructure includes a computing node running a large language model, the compute node's GPU memory being used to store the inference results obtained by the large language model processing the second session task; a first notification module 1004 is used to: notify the compute node to migrate the inference results obtained by the large language model processing the second session task from GPU memory to the compute node's memory or hard disk, obtaining adjusted GPU memory; notify the compute node to use the large language model to process a third subtask, and store the inference results obtained by the large language model processing the third subtask in the adjusted GPU memory.

[0128] In one possible implementation, the cloud management platform further includes: a third notification module for notifying computing nodes to retrieve the inference results obtained by the large language model from the adjusted video memory for processing the third subtask; and a fourth notification module for notifying the target tool to process the inference results obtained by the large language model from processing the third subtask to obtain new inference results, which are then used by the large language model to process the fifth subtask.

[0129] In one possible implementation, compute nodes comprise physical servers, bare-metal servers, virtual machines, containers, or microvirtual machines.

[0130] It should be noted that the information interaction and implementation process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in the embodiments of this application, and will not be repeated here.

[0131] Please refer to Figure 11, which is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 11, the computing device 1100 (which can be used to present the aforementioned cloud management platform) includes: a processor 1101, a memory 1102, a communication interface 1103, and a bus 1104. The processor 1101, the memory 1102, and the communication interface 1103 are coupled through the bus (not labeled in the figure). The memory 1102 stores instructions. When the execution instructions in the memory 1102 are executed, the computing device 1100 executes the method performed by the cloud management platform in the above method embodiment.

[0132] The computing device 1100 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together and implemented as a system-on-a-chip (SOC).

[0133] The processor 1101 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0134] The memory 1102 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0135] The memory 1102 stores executable program code, and the processor 1101 executes the executable program code to implement the functions of the aforementioned receiving module, acquiring module, determining module, and first notification module, thereby realizing the aforementioned task processing method based on the cloud management platform. That is, the memory 1102 stores instructions for executing the aforementioned task processing method based on the cloud management platform.

[0136] The communication interface 1103 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0137] In addition to the data bus, the 1104 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be divided into address bus, data bus, and control bus.

[0138] Please refer to Figure 12, which is a schematic diagram of a computing device cluster provided in an embodiment of this application. As shown in Figure 12, the computing device cluster 1200 includes at least one computing device 1100.

[0139] As shown in Figure 12, the computing device cluster 1200 includes at least one computing device 1100. The memory 1102 of one or more computing devices 1100 in the computing device cluster 1200 may store the same instructions for executing the above-described task processing method based on the cloud management platform.

[0140] In some possible implementations, the memory 1102 of one or more computing devices 1100 in the computing device cluster 1200 may also store partial instructions for executing the aforementioned task processing method based on the cloud management platform. In other words, a combination of one or more computing devices 1100 can jointly execute the aforementioned task processing method based on the cloud management platform.

[0141] It should be noted that the memory 1102 in different computing devices 1100 within the computing device cluster 1200 can store different instructions, which are used to execute certain functions of the aforementioned cloud management platform. That is, the instructions stored in the memory 1102 of different computing devices 1100 can implement the functions of one or more modules, such as the receiving module, the acquiring module, the determining module, and the first notification module.

[0142] In some possible implementations, one or more computing devices 1100 in the computing device cluster 1200 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0143] Please refer to Figure 13, which is a schematic diagram of computer devices in a computer cluster provided in an embodiment of this application being connected via a network. As shown in Figure 13, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0144] In one possible implementation, the memory in computing device 1100A stores instructions for performing the functions of modules such as the receiving module. Meanwhile, the memory in computing device 1100B stores instructions for performing the functions of modules such as the acquisition module, the determination module, and the first notification module.

[0145] It should be understood that the functions of computing device 1100A shown in Figure 13 can also be performed by multiple computing devices. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices.

[0146] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform the steps executed by the cloud management platform in the embodiment shown in FIG9.

[0147] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the steps performed by the cloud management platform in the embodiment shown in FIG9.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0149] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0150] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0151] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0152] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A task processing method based on a cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure that provides cloud services, the infrastructure being used to run large language models, and the method includes: The cloud management platform receives a task processing request sent by the user. The task processing request is used to indicate a first session task and a second session task. The first session task includes multiple first sub-tasks, and the second session task includes multiple second sub-tasks. Based on the task processing request, the cloud management platform obtains the third subtask to be processed from the plurality of first subtasks and the fourth subtask to be processed from the plurality of second subtasks; The cloud management platform determines the time required to obtain the inference result of the first session task based on the processing order of the third subtask among the plurality of first subtasks, and determines the time required to obtain the inference result of the second session task based on the processing order of the fourth subtask among the plurality of second subtasks. If the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, the cloud management platform notifies the large language model to process the third sub-task.

2. The method according to claim 1, characterized in that, The cloud management platform determines the time required to obtain the inference result of the first session task based on the processing order of the third subtask among the multiple first subtasks, including: Based on the processing order of the third subtask among the plurality of first subtasks, the cloud management platform determines the fifth subtask whose processing order is after the third subtask from among the plurality of first subtasks; The cloud management platform determines the time required to obtain the reasoning result of the first session task based on the feature information of the third sub-task and the feature information of the fifth sub-task.

3. The method according to claim 2, characterized in that, The characteristic information of the third subtask includes at least one of the following: the amount of resources required to process the third subtask and the number of the third subtasks; The characteristic information of the fifth subtask includes at least one of the following: the amount of resources required to process the fifth subtask, the number of the fifth subtasks, and the waiting time required before processing the fifth subtask.

4. The method according to any one of claims 1 to 3, characterized in that, The determination that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and the cloud management platform notifying the large language model to process the third sub-task, includes: If the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and the time required to obtain the inference result of the first session task is less than or equal to the maximum time required to complete the first session task, the cloud management platform notifies the large language model to process the third sub-task.

5. The method according to claim 4, characterized in that, The method further includes: If the time required to obtain the inference result of the first session task is greater than the maximum time required to complete the first session task, the cloud management platform notifies the large language model to process the third sub-task.

6. The method according to any one of claims 1 to 5, characterized in that, The infrastructure includes computing nodes running the large language model, and the video memory of the computing nodes is used to store the inference results obtained by the large language model in processing the second conversation task; The cloud management platform notifies the large language model to process the third subtask, including: The cloud management platform notifies the computing node to migrate the inference results obtained by the large language model from the second conversation task from the video memory to the memory or hard disk of the computing node, so as to obtain the adjusted video memory; The cloud management platform notifies the computing node to use the large language model to process the third subtask, and stores the inference result obtained by the large language model in the adjusted video memory.

7. The method according to claim 6, characterized in that, The method further includes: The cloud management platform notifies the computing node to retrieve the inference result obtained by the large language model processing the third subtask from the adjusted video memory; The cloud management platform notifies the target tool to process the inference result obtained by the large language model from the third sub-task, and obtain a new inference result, which is then used by the large language model to process the fifth sub-task.

8. The method according to any one of claims 6 or 7, characterized in that, The computing nodes include physical servers, bare metal servers, virtual machines, containers, or microvirtual machines.

9. A cloud management platform, characterized in that, The cloud management platform is used to manage the infrastructure that provides cloud services, the infrastructure being used to run large language models, and the cloud management platform includes: A receiving module is used to receive a task processing request sent by a user. The task processing request is used to indicate a first session task and a second session task. The first session task includes multiple first sub-tasks, and the second session task includes multiple second sub-tasks. The acquisition module is used to acquire, based on the task processing request, the third subtask to be processed among the plurality of first subtasks and the fourth subtask to be processed among the plurality of second subtasks; The determining module is used to determine the time required to obtain the inference result of the first session task based on the processing order of the third subtask in the plurality of first subtasks, and to determine the time required to obtain the inference result of the second session task based on the processing order of the fourth subtask in the plurality of second subtasks. The first notification module is used to determine that the time required to obtain the inference result of the first session task is less than the time required to obtain the inference result of the second session task, and then notify the large language model to process the third sub-task.

10. The cloud management platform according to claim 9, characterized in that, The determining module is used for: Based on the processing order of the third subtask among the plurality of first subtasks, a fifth subtask whose processing order is after the third subtask is determined from among the plurality of first subtasks. Based on the feature information of the third subtask and the feature information of the fifth subtask, the time required to obtain the reasoning result of the first session task is determined.

11. The cloud management platform according to claim 10, characterized in that, The characteristic information of the third subtask includes at least one of the following: the amount of resources required to process the third subtask and the number of the third subtasks; The characteristic information of the fifth subtask includes at least one of the following: the amount of resources required to process the fifth subtask, the number of the fifth subtasks, and the waiting time required before processing the fifth subtask.

12. The cloud management platform according to any one of claims 9 to 11, characterized in that, The first notification module is used for: If it is determined that the time required to obtain the inference result of the first conversation task is less than the time required to obtain the inference result of the second conversation task, and the time required to obtain the inference result of the first conversation task is less than or equal to the maximum time required to complete the first conversation task, the large language model is notified to process the third subtask.

13. The cloud management platform according to claim 12, characterized in that, The cloud management platform also includes: The second notification module is used to determine that the time required to obtain the reasoning result of the first conversation task is greater than the maximum time required to complete the first conversation task, and to notify the large language model to process the third sub-task.

14. The cloud management platform according to any one of claims 9 to 13, characterized in that, The infrastructure includes computing nodes running the large language model, and the video memory of the computing nodes is used to store the inference results obtained by the large language model in processing the second conversation task; The first notification module is used for: The computing node is notified to migrate the inference results obtained from the second conversation task by the large language model from the video memory to the memory or hard disk of the computing node to obtain the adjusted video memory; The computing node is notified to use the large language model to process the third subtask, and the reasoning result obtained by the large language model in processing the third subtask is stored in the adjusted video memory.

15. The cloud management platform according to claim 14, characterized in that, The cloud management platform also includes: The third notification module is used to notify the computing node to obtain the inference result obtained by the large language model processing the third subtask from the adjusted video memory; The fourth notification module is used to notify the target tool to process the inference result obtained by the large language model in the third sub-task to obtain a new inference result, which is used by the large language model to process the fifth sub-task.

16. The cloud management platform according to any one of claims 14 or 15, characterized in that, The computing nodes include physical servers, bare metal servers, virtual machines, containers, or microvirtual machines.

17. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 8.

18. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method of any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 8.

Images