Method and apparatus for controlling node, device, medium, and product

Abstract

The present application relates to the technical field of artificial intelligence, and provides a method and apparatus for controlling a node, a device, a medium, and a product. The method comprises: acquiring target traffic input into a machine learning model, the machine learning model being deployed across a plurality of computing nodes, and the plurality of computing nodes comprising a plurality of prefill nodes and a plurality of decoding nodes. The method further comprises: on the basis of the target traffic and historical traffic processed by the machine learning model, predicting at least one predicted traffic corresponding to at least one predetermined duration to be processed by the machine learning model. The method further comprises: on the basis of the at least one predicted traffic, adjusting the number of the plurality of prefill nodes and the number of the plurality of decoding nodes. By means of the method, the computing nodes can be adjusted in advance to handle unexpected burst traffic, thereby avoiding overload of disaggregated inference nodes, achieving elastic adjustment of computing resources, increasing resource utilization, and improving user experience.

Description

Methods, apparatus, devices, media, and products for controlling nodes.

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202510206679.8, entitled "Method, Apparatus, Device, Medium and Product for Controlling Nodes", filed on February 21, 2025, which is incorporated herein by reference in its entirety. Technical Field

[0003] The embodiments of this application primarily relate to the field of artificial intelligence technology. More specifically, the embodiments of this application relate to methods, apparatuses, devices, media, and products for controlling nodes. Background Technology

[0004] With the advancement of computer technology, artificial intelligence models, especially neural network models, have developed rapidly. AI models are now being increasingly applied across various fields to process complex information. For example, in the field of artificial intelligence technology, AI models are gradually becoming the primary means of information processing.

[0005] With the development of artificial intelligence model technology, the amount of data that needs to be processed is increasing, and the types of data are becoming more complex, resulting in increasingly large computational resources required for the inference process of current artificial intelligence models. For example, inference in large language models based on transformers consists of a prefill (P) stage and a decoding (D) stage. In related technologies, the prefill and decoding stages are generally deployed separately. However, due to the significant differences in the execution characteristics of prefill and decoding, many problems still need to be solved in the execution of prefill and decoding tasks. Summary of the Invention

[0006] The embodiments of this application provide a scheme for controlling nodes.

[0007] According to a first aspect of this application, a method for controlling nodes is provided. The method includes acquiring target traffic input to a machine learning model, the machine learning model being deployed across multiple computing nodes, the multiple computing nodes including multiple pre-filled nodes and multiple decoding nodes; predicting at least one predicted traffic to be processed by the machine learning model, corresponding to at least one predetermined duration, based on the target traffic and historical traffic processed by the machine learning model; and adjusting the number of the multiple pre-filled nodes and the multiple decoding nodes based on the at least one predicted traffic.

[0008] This approach allows for the prediction of traffic to be processed by the machine learning model by utilizing current and historical traffic processed by the model. By adjusting pre-filled and decoding nodes, adaptive control of nodes used for inference tasks is achieved. This enables the pre-adjustment of computing nodes to handle unexpected surges in traffic, avoids overloading of discrete inference nodes, and achieves elastic adjustment of computing resources, improving resource utilization and enhancing the user experience.

[0009] In some embodiments, acquiring the target traffic input to the machine learning model includes: acquiring traffic data by collecting the traffic input to the machine learning model; and determining the target traffic based on the acquired traffic data. This approach allows for the rapid and accurate determination of the current traffic to be processed by the machine learning model.

[0010] In some embodiments, adjusting the number of multiple pre-filled nodes and multiple decoded nodes includes: adjusting the number of multiple pre-filled nodes based on at least one predicted traffic; determining the load of the multiple pre-filled nodes; and adjusting the number of multiple decoded nodes based on the load and at least one predicted traffic. This method allows for rapid and accurate dynamic adjustment of the number of pre-filled nodes and decoded nodes, enabling the pre-allocation of new computing nodes to handle sudden traffic surges and preventing overload of discrete inference nodes.

[0011] In some embodiments, adjusting the number of multiple pre-filled nodes includes: determining whether the multiple pre-filled nodes can handle a first predicted traffic in at least one predicted traffic; if it is determined that the multiple pre-filled nodes cannot handle the first predicted traffic, adding a target pre-filled node for handling the first predicted traffic; and using the multiple pre-filled nodes and the target pre-filled node to process the first predicted traffic. In this way, dynamic adjustment of nodes can be achieved by dynamically adding pre-filled nodes, thereby improving traffic processing efficiency.

[0012] In some embodiments, determining whether a plurality of pre-filled nodes can handle a first predicted traffic in at least one predicted traffic includes: determining whether the plurality of pre-filled nodes can handle the first predicted traffic based on at least one of the following: whether the memory space of the plurality of pre-filled nodes can accommodate the activation value for the first predicted traffic; or whether a first service level objective for the plurality of pre-filled nodes is met. In this way, it is possible to quickly and accurately determine whether existing pre-filled nodes can handle predicted traffic.

[0013] In some embodiments, processing the first prediction traffic using multiple pre-filled nodes and a target pre-filled node includes: providing a first set of model parameters for a machine learning model from the multi-level memory layers in a memory pool to a target pre-filled node based on the order of the multi-level memory layers in the memory pool; and processing the first prediction traffic using the target pre-filled node with the first set of model parameters and multiple pre-filled nodes. This approach utilizes the order of the multi-level memory layers to load model parameters from different layers, thereby improving parameter loading efficiency and enabling rapid application of nodes to processing tasks.

[0014] In some embodiments, where at least one predicted traffic is at least one first predicted traffic, adjusting the number of multiple decoding nodes includes: determining at least one second predicted traffic to be provided to the multiple decoding nodes by multiple pre-filled nodes based on the load and at least one predicted traffic; and adjusting the number of multiple decoding nodes based on at least one second predicted traffic. In this way, the adjustment of decoding nodes can be dynamically controlled by predicting the traffic to the decoding nodes, thereby ensuring sufficient resources to handle bursts of traffic.

[0015] In some embodiments, adjusting the number of multiple decoding nodes based on at least one second predicted traffic includes: determining whether the multiple decoding nodes are capable of handling the second predicted traffic in at least one second predicted traffic; if it is determined that the multiple decoding nodes are unable to handle the second predicted traffic in at least one second predicted traffic, adding a target decoding node for executing the second predicted traffic; and using the multiple decoding nodes and the target decoding node to process the second predicted traffic. This method allows for the rapid addition of decoding nodes, thereby speeding up request processing.

[0016] In some embodiments, determining whether a plurality of decoding nodes can handle at least one second predicted traffic includes: determining whether the plurality of decoding nodes can handle at least one second predicted traffic based on at least one of the following: whether the memory space of the plurality of decoding nodes can accommodate a key-value cache for the second predicted traffic; or whether a second service level objective for the plurality of decoding nodes is met. In this way, it is possible to quickly determine whether existing decoding nodes can handle incoming traffic.

[0017] In some embodiments, processing the second predicted traffic using multiple decoding nodes and a target decoding node includes: providing a second set of model parameters for the machine learning model from the multi-level memory layers in the memory pool to the target decoding node based on the order of the multi-level memory layers in the memory pool; and controlling the transfer of key-value caches from the multiple decoding nodes to the target decoding node based on the target duration corresponding to the second predicted traffic. In this way, for predicted traffic with longer durations, the migration of key-value caches can be performed in advance, avoiding newly added nodes from being idle, and achieving load balancing among nodes.

[0018] In some embodiments, providing a second set of model parameters for a machine learning model from a multi-level memory layer to a target decoding node includes: providing the target decoding node with parameters for the decoding task located in a first-level memory layer of the multi-level memory layer, which are used by the decoding nodes among a plurality of decoding nodes; and in response to determining that the parameters provided by the first-level memory layer are part of the second set of model parameters, providing the target decoding node with the remaining parameters from the second set of model parameters from other levels of the multi-level memory layer. In this manner, by receiving data from the multi-level memory layer, the loading time of model weights in the extended node is shortened.

[0019] In some embodiments, controlling the transfer of key-value cache from multiple decoding nodes to a target decoding node based on a target duration corresponding to the second predicted traffic includes: determining whether the target duration is greater than a threshold duration; and in response to the target duration being greater than the threshold duration, determining the source decoding node from among the multiple decoding nodes from which the key-value cache blocks are to be migrated; and during the decoding process at the source decoding node, migrating the key-value cache in the source decoding node to the target decoding node block by block. This method uses migration requests to perform load balancing, improving the resource utilization of each node in the cluster inference system.

[0020] In some embodiments, the method further includes: determining the number of requests to be processed by a first pre-filled node among a plurality of pre-filled nodes; and in response to the number being less than a threshold number, transferring the number of requests to a second pre-filled node among the plurality of pre-filled nodes; and removing the first pre-filled node from the plurality of pre-filled nodes. This approach enables node merging, reduces cluster size, and minimizes resource idleness.

[0021] In some embodiments, the method further includes: determining the utilization rate of a first decoding node among a plurality of decoding nodes; and, in response to the utilization rate being lower than a threshold utilization rate, transferring the key-value cache in the first decoding node to a second decoding node among the plurality of decoding nodes in blocks; and, in response to the key-value cache in the first decoding node being transferred to the second decoding node, removing the first decoding node from the plurality of decoding nodes. This approach enables hot merging of nodes, reduces cluster size, and minimizes resource idleness.

[0022] In some embodiments, determining the utilization of a first decoding node among a plurality of decoding nodes includes: determining model computing power utilization, model bandwidth utilization, and model memory utilization for the first decoding node; and determining the utilization of the first decoding node based on at least one of model computing power utilization, model bandwidth utilization, and model memory utilization. This method allows for accurate determination of node resource utilization.

[0023] According to a second aspect of this application, an apparatus for controlling nodes is provided. The apparatus includes: a target traffic acquisition unit configured to acquire target traffic input into a machine learning model, the machine learning model being deployed across multiple computing nodes, the multiple computing nodes including multiple pre-filled nodes and multiple decoding nodes; a traffic prediction unit configured to predict at least one predicted traffic to be processed by the machine learning model, corresponding to at least one predetermined duration, based on the target traffic and historical traffic processed by the machine learning model; and a node adjustment unit configured to adjust the number of the multiple pre-filled nodes and the multiple decoding nodes based on the at least one predicted traffic.

[0024] In some embodiments, the target traffic determination unit includes: a traffic acquisition unit configured to acquire traffic data by acquiring traffic input to a machine learning model; and a first traffic determination unit configured to determine target traffic based on the acquired traffic data.

[0025] In some embodiments, the node adjustment unit includes: a first adjustment unit configured to adjust the number of a plurality of pre-filled nodes based on at least one predicted traffic; a load determination unit configured to determine the load of the plurality of pre-filled nodes; and a second adjustment unit configured to adjust the number of a plurality of decoding nodes based on the load and at least one predicted traffic.

[0026] In some embodiments, the first adjustment control unit includes: a first determination unit configured to determine whether a plurality of pre-filled nodes can carry a first predicted flow in at least one predicted flow; a node addition unit configured to add a target pre-filled node for performing the predicted flow if it is determined that the plurality of pre-filled nodes cannot carry the first predicted flow; and a flow processing unit configured to process the first predicted flow using the plurality of pre-filled nodes and the target pre-filled node.

[0027] In some embodiments, the first determination unit includes: a first traffic carrying determination unit, configured to determine whether a plurality of filling nodes can carry a first predicted traffic in at least one predicted traffic based on at least one of the following: whether the memory space of the plurality of pre-filled nodes can accommodate the activation value for the first predicted traffic; or whether a first service level objective for the plurality of pre-filled nodes is met.

[0028] In some embodiments, the traffic processing unit includes: a first parameter providing unit configured to provide a first set of model parameters for a machine learning model from a multi-level memory layer to a target pre-filled node based on the order of the multi-level memory layers in a memory pool; and a first processing unit configured to process the first predicted traffic using the target pre-filled node having the first set of model parameters and a plurality of pre-filled nodes.

[0029] In some embodiments, at least one predicted traffic is at least one first predicted traffic, and the second adjustment unit includes: a predicted traffic providing unit configured to determine at least one second predicted traffic to be provided by a plurality of pre-filled nodes to a plurality of decoding nodes based on load and at least one first predicted traffic; and a third adjustment unit configured to adjust the number of the plurality of decoding nodes based on at least one second predicted traffic.

[0030] In some embodiments, the third adjustment unit includes: a traffic determination unit configured to determine whether a plurality of decoding nodes can carry at least one second predicted traffic; a target decoding node addition unit configured to add a target decoding node for performing the second predicted traffic if it is determined that the plurality of decoding nodes cannot carry at least one second predicted traffic; and a predicted traffic processing unit configured to process the second predicted traffic using the plurality of decoding nodes and the target decoding node.

[0031] In some embodiments, the traffic determination unit includes: a second traffic carrying determination unit, configured to determine whether a plurality of decoding nodes are capable of carrying the second predicted traffic in at least one second predicted traffic based on at least one of the following: whether the memory space of the plurality of decoding nodes can accommodate the activation value for the second predicted traffic; or whether the second service level objective for the plurality of decoding nodes is met.

[0032] In some embodiments, the predicted traffic processing unit includes: a second parameter providing unit configured to provide a second set of model parameters for a machine learning model from the multi-level memory layers to the target decoding node based on the order of the multi-level memory layers in the memory pool; and a transfer control unit configured to control the transfer of key-value caches from multiple decoding nodes to the target decoding node based on a target duration corresponding to the second predicted traffic.

[0033] In some embodiments, the second parameter providing unit includes: a third parameter providing unit configured to provide the target decoding node with parameters for the decoding task located in a first-level memory layer in a multi-level memory layer, which are used by the decoding nodes among a plurality of decoding nodes; and a remaining parameter providing unit configured to provide the target decoding node with the remaining parameters in the second set of model parameters from other levels of the multi-level memory layer in response to determining that the parameters provided by the first-level memory layer are part of a second set of model parameters.

[0034] In some embodiments, the transfer control unit includes: a duration comparison unit configured to determine whether a target duration is greater than a threshold duration; a source decoding node determination unit configured to determine, in response to the target duration being greater than the threshold duration, a source decoding node among a plurality of decoding nodes to migrate key-value cache blocks; and a block migration unit configured to migrate the key-value cache in the source decoding node to the target decoding node by block during the decoding process of the source decoding node.

[0035] In some embodiments, the apparatus further includes: a request number determination unit configured to determine the number of requests to be processed by a first pre-filled node among a plurality of pre-filled nodes; a request transfer unit configured to transfer the number of requests to a second pre-filled node among the plurality of pre-filled nodes in response to the number being less than a threshold number; and a node removal unit configured to remove the first pre-filled node from the plurality of pre-filled nodes.

[0036] In some embodiments, the apparatus further includes: a utilization determination unit configured to determine the utilization of a first decoding node among a plurality of decoding nodes; a key-value cache transfer unit configured to transfer the key-value cache in the first decoding node in blocks to a second decoding node among the plurality of decoding nodes in response to the utilization being lower than a threshold utilization; and a decoding node removal unit configured to remove the first decoding node from the plurality of decoding nodes in response to the key-value cache in the first decoding node being transferred to the second decoding node.

[0037] In some embodiments, the utilization determination unit includes: a plurality of utilization determination units configured to determine model computing power utilization, model bandwidth utilization, and model memory utilization for a first decoding node; and a decoding node utilization determination module configured to determine the utilization of the first decoding node based on at least one of model computing power utilization, model bandwidth utilization, and model memory utilization.

[0038] According to a third aspect of this application, an electronic device is also provided, comprising: at least one computing unit; and at least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions, when executed by the at least one computing unit, causing the device to perform the method according to the first aspect of this application.

[0039] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the method described according to a first aspect of this application.

[0040] According to a fifth aspect of this application, a computer program product is also provided, including computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method described according to a first aspect of this application.

[0041] Understandably, the apparatus of the second aspect, the electronic device of the third aspect, the computer storage medium of the fourth aspect, or the computer program product of the fifth aspect provided above are used to perform the method provided in the first aspect. Therefore, the explanations or descriptions regarding the first aspect also apply to the second, third, fourth, and fifth aspects. Furthermore, the beneficial effects achieved by the second, third, fourth, and fifth aspects can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0042] The above and other features, advantages and aspects of the embodiments of this application will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description.

[0043] In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0044] Figure 1 shows a schematic diagram of an example environment in which various embodiments of this application can be implemented;

[0045] Figure 2 shows a schematic flowchart of a method for controlling a node according to some embodiments of this application;

[0046] Figure 3 shows a schematic flowchart of a method for adjusting pre-filled nodes according to some embodiments of this application;

[0047] Figure 4 shows a schematic flowchart of a method for adjusting a decoding node according to some embodiments of this application;

[0048] Figure 5 shows a schematic diagram of an example architecture for dynamic scaling of nodes according to some embodiments of this application;

[0049] Figure 6 illustrates a schematic diagram of example functions of node adjustment according to some embodiments of this application;

[0050] Figure 7 illustrates a schematic diagram of an example process for adjusting nodes according to some embodiments of this application;

[0051] Figure 8 shows a block diagram of an apparatus according to some embodiments of this application; and

[0052] Figure 9 shows a block diagram of a computing device capable of implementing several embodiments of the present application. Detailed Implementation

[0053] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0054] In the description of embodiments of this application, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0055] In some traditional solutions, inference tasks for AI models are typically handled by separating the pre-filling and decoding phases. This separation avoids interference between tasks with different characteristics. This deployment can achieve load balancing by promptly rejecting excessive requests based on predicted decoding node load and heuristic automatic hotspot migration strategies. However, this approach only focuses on large language model inference services based on separate deployments. Under sudden traffic surges, servers at full capacity may refuse to provide inference services; during off-peak traffic periods, cluster utilization is low, resulting in resource waste. Other solutions handle inference tasks through separate and hybrid deployments of pre-filling and decoding nodes. However, this approach only focuses on large language model inference services based on separate and hybrid deployments and does not consider the load changes caused by sudden and cyclical traffic surges. Still other solutions utilize rapid multi-level checkpoint loading to fully leverage storage capacity and bandwidth for loading model weights, accelerating new node startup; coordinating Graphics Processing Unit (GPU) servers and cluster controllers for efficient real-time request migration; and employing startup-optimized model scheduling strategies to minimize cold start time. However, this approach only focuses on how to quickly cold-start new nodes when traffic increases, without analyzing the challenges brought about by the inter-node traffic lag and inference characteristics of the PD split inference service.

[0056] However, during the inference service of artificial intelligence models, the number of active users varies throughout the day, and there are also significant differences in user numbers on weekends, holidays, and weekdays, leading to fluctuations in traffic. Furthermore, real-world scenarios also experience sudden traffic spikes. How to handle high traffic volumes under conditions of significant traffic variation and reduce operating costs during low traffic periods becomes a key technical challenge. In the field of split inference services, traffic variations place different load pressures on servers. How to rationally configure the inference cluster to handle high-traffic user requests and improve resource utilization during low-traffic periods presents a significant challenge.

[0057] To address at least some of the aforementioned problems and other potential issues, in embodiments of this application, a computing device can first acquire the target traffic input to a machine learning model, wherein the machine learning model is deployed across multiple computing nodes, including multiple pre-filled nodes and multiple decoding nodes. Then, the computing device can use the target traffic and historical traffic processed by the machine learning model to predict at least one predicted traffic corresponding to at least one predetermined duration to be processed by the machine learning model. Next, the computing device can adjust the number of multiple pre-filled nodes and multiple decoding nodes based on the at least one predicted traffic. In this way, the current and historical traffic processed by the machine learning model can be used to predict the traffic to be processed by the machine learning model, and the pre-filled and decoding nodes can be adjusted. This achieves adaptive control of the nodes used to perform inference tasks, allowing new computing nodes to be requested in advance to handle unexpected bursts of traffic, avoiding overload of discrete inference nodes, and enabling elastic adjustment of computing resources, improving resource utilization and enhancing the user experience.

[0058] Figure 1 illustrates a schematic diagram of an example environment 100 in which various embodiments of this application can be implemented. As shown in Figure 1, the example environment 100 includes a computing device 104, which can be used to control the adjustment of pre-filled nodes and / or decoding nodes to be used by the machine learning model 102. The computing device 104 includes, but is not limited to, personal computers, servers, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0059] The inference task of machine learning model 102 can be divided into a pre-filling task and a decoding task. For example, if machine learning model 102 is a large language model, in the pre-filling task phase, the large language model processes input requests to compute intermediate states (keys and values), which are used to generate the first word. In the decoding task phase, the large language model automatically regresses to generate one output word at a time until a stopping condition is met. To accelerate the execution of the machine learning model, some nodes from multiple computing nodes 106 are assigned to the machine learning model. The nodes in the multiple computing nodes can be dedicated computing devices that can be used to perform inference for artificial intelligence models, such as neural network processing unit (NPU) devices or GPU devices. Additionally, the nodes in the multiple computing nodes 106 can also be personal computers, servers, minicomputers, mainframes, distributed computing environments including any of the above systems or devices, etc.

[0060] To improve the utilization of node resources, a portion of the multiple computing nodes 106 are allocated to the machine learning model 102 for performing pre-filling tasks. For ease of description, these nodes performing pre-filling tasks can also be referred to as multiple pre-filling nodes 108. Additionally, a portion of the multiple computing nodes 106 are allocated to perform decoding tasks for the machine learning model 102. For ease of description, these nodes allocated to perform decoding tasks can also be referred to as multiple decoding nodes 110. After the multiple pre-filling nodes 108 complete the pre-filling tasks in the machine learning model 102, they transmit the data calculated by the pre-filling tasks to the multiple decoding nodes 110 to continue the execution of subsequent decoding tasks. For example, the key-value cache (KV cache) calculated by the multiple pre-filling nodes is transmitted to the multiple decoding nodes 110.

[0061] To cope with sudden surges or changes in traffic to the machine learning model 102 and to improve the utilization of computing resources, the computing device 104 needs to predict the traffic processed by the machine learning model 102 in the future, thereby avoiding situations where it cannot handle sudden surges or changes in traffic. The traffic here can be represented by the number of requests and the length of each request, also known as the request volume. To predict the traffic at different intervals, the computing device 104 can obtain the current traffic being processed by the machine learning model 102, which is also referred to as the target traffic 112. In one example, the computing device 104 can collect traffic data input to the machine learning model 102 at a predetermined frequency, and then calculate the size of the target traffic being processed at predetermined intervals. For example, the computing device collects traffic data at a frequency of 1 Hz and calculates the traffic data every 30 seconds as the target traffic. For example, the average of the traffic data collected over 30 seconds is used as the target traffic 112. Alternatively, the computing device 104 can also receive data collected at a predetermined frequency from other devices to determine the traffic data for the machine learning model 102.

[0062] Additionally, the computing device 104 also needs to use historical traffic 114 processed by the machine learning model 102 when predicting traffic. For example, the task of predicting traffic is implemented using a Long Short-Term Memory (LSTM) model. When the target traffic 112 is input into the LSTM model for prediction, the model also uses some data parameters calculated from the historical traffic 114. Therefore, when predicting at least one predicted traffic 116, both the target traffic 112 and the historical traffic 114 are used. Furthermore, the at least one predicted traffic 116 may include one predicted traffic or multiple predicted traffic.

[0063] Each of the at least one predicted flow 116 corresponds to a different duration, and therefore the at least one predicted flow 116 reflects the flow to arrive after different durations. For example, the at least one predicted flow includes flow predicted 30 seconds later and flow predicted 10 minutes later.

[0064] Then, the computing device 104 can further adjust the plurality of pre-fill nodes 108 for pre-filling tasks and the plurality of decoding nodes 110 for decoding tasks among the plurality of computing nodes 106 based on the at least one predicted traffic 116. For example, based on one of the predicted traffic flows 116, it is determined whether the plurality of pre-fill nodes have sufficient memory space to accommodate activation values ​​or whether a Service Level Objective (SLO) is met. For example, the computing device 104 can calculate the required memory space based on the number of requests and the request length, and then compare this memory space with the memory space that the plurality of pre-fill nodes can provide. If the required memory space is greater than the available memory space, it indicates that the plurality of pre-fill nodes do not have enough space to accommodate activation values. If the plurality of pre-fill nodes do not have enough memory space to accommodate activation values ​​or cannot meet the Service Level Objective, a new pre-fill node needs to be allocated for the machine learning model. This new pre-fill node may also be referred to as the target pre-fill node. Similarly, computing device 104 can further determine whether the multiple decoding nodes have sufficient memory to accommodate the KV cache or whether they can meet service level objectives based on at least one predicted traffic, the number of requests being processed in the pre-filled nodes, the number of requests waiting to be processed, and the length of each of these requests. If the multiple decoding nodes do not have sufficient memory to accommodate the KV cache or cannot meet the service level objectives, a new decoding node can be allocated, which can also be called the target decoding node. The new pre-filled node and the new decoding node are then used for the inference task of machine learning model 102.

[0065] In some embodiments, after a new pre-filled node is allocated, a cold start can be performed on that pre-filled node. During a cold start, model weights can be preloaded layer by layer from the multi-level memory pool. A cold start can also be performed on a new decoding node. During a cold start, in addition to preloading model weights layer by layer from the multi-level memory layer, the KV Cache of some requests from multiple decoding nodes 110 can be migrated to the new decoding node in real time, block by block, in multiple rounds. After the migration is complete, the node can be accessed for inference work, realizing hot migration of requests.

[0066] Additionally, merging operations can be performed on nodes from multiple pre-filled nodes and multiple decoded nodes. Pre-filled nodes with a smaller number of pending requests are merged to release them. Decoding nodes with utilization below a threshold are merged to release them. When releasing a decoding node, the KV cache within that node can be migrated to other decoding nodes in real-time, block-by-block rounds.

[0067] This approach allows for the prediction of traffic to be processed by the machine learning model by utilizing current and historical traffic processed by the model. By adjusting pre-filled and decoding nodes, adaptive control of nodes used for inference tasks is achieved. This enables the advance application of new computing nodes to handle unexpected bursts of traffic, avoids overloading of discrete inference nodes, and achieves elastic adjustment of computing resources, improving resource utilization and enhancing the user experience.

[0068] The above description, with reference to FIG1, illustrates an example environment 100 in which embodiments of this application can be implemented. The following description, with reference to FIG2, is a schematic flowchart of an example 200 of a method for controlling a node according to some embodiments of this application. Example 200 can be executed by the computing device 104 in FIG1 and any suitable computing device.

[0069] At box 202, computing device 104 acquires the target traffic input into a machine learning model, where the machine learning model is deployed across multiple computing nodes 106, including multiple pre-filling nodes 108 and multiple decoding nodes 110. To accelerate the execution of the machine learning model's inference task, the pre-filling and decoding tasks need to be deployed on different nodes, for example, allocating some nodes to perform the pre-filling task and others to perform the decoding task. The set of nodes performing the pre-filling task can be referred to as multiple pre-filling nodes, and the set of nodes performing the decoding task can be referred to as multiple decoding nodes. To handle changing traffic in a timely manner, it is necessary to use the current target traffic to predict future traffic.

[0070] When determining the target traffic input to a machine learning model, traffic data can first be collected. In one example, another computing device collects the traffic input to the machine learning model, and computing device 104 can receive the data collected by the other device to obtain traffic data. In another example, computing device 104 can collect the traffic input to the machine learning model itself to obtain traffic data. Then, computing device 104 further determines the target traffic input to the machine learning model based on the acquired traffic data. For example, traffic data input to the machine learning model can be collected at a predetermined frequency, such as collecting traffic data at a frequency of 1 Hz. Then, the target traffic input to the machine learning model is determined using the data collected within a predetermined period of time, such as using traffic data collected within 30 seconds. In one example, the average value of the data collected within the predetermined period of time is used as the target traffic input to the machine learning model. In another example, a data point is selected from the data collected within the predetermined period of time as the target traffic, or the data is processed using a function to determine the target traffic. The above examples are only for describing this disclosure and are not intended to limit the specific scope of this disclosure. Those skilled in the art can use any suitable method to determine the target traffic from the collected traffic data.

[0071] At box 204, computing device 104 predicts at least one predicted flow corresponding to at least one predetermined duration to be processed by the machine learning model, based on the target flow and historical flow processed by the machine learning model. In predicting the flow to be processed by the machine learning model, in addition to using the target flow, historical flow processed by the machine learning model is also utilized.

[0072] When predicting at least one predicted traffic flow, computing device 104 can use target traffic and historical traffic processed by a machine learning model to predict traffic at least one duration later. For example, computing device 104 can use target traffic and historical traffic to predict a first predicted traffic flow for a first predetermined duration and a second predicted traffic flow for a second predetermined duration, the second predetermined duration being longer than the first predetermined duration. For example, computing device 104 can predict traffic 30 seconds (short-term) later and traffic 10 minutes (long-term). In one example, computing device 104 can input current traffic and historical traffic into a traffic prediction model to predict traffic to be processed after a certain duration; for example, the traffic prediction model could be a Long Short-Term Memory (LSTM) model. In another example, a mapping relationship between traffic flows can be predetermined, and this mapping relationship can be used to determine at least one predicted traffic flow corresponding to the target traffic and historical traffic flows. The above examples are merely for describing this disclosure and are not intended to specifically limit this disclosure.

[0073] At box 206, computing device 104 adjusts the number of multiple pre-filled nodes and multiple decoding nodes based on at least one predicted flow. After obtaining at least one predicted flow at different predicted durations, computing device 104 can adjust the number of multiple pre-filled nodes and multiple decoding nodes according to the at least one predicted flow.

[0074] When the computing device 104 adjusts the number of multiple pre-filled nodes and multiple decoding nodes, it can first control the adjustment of the multiple pre-filled nodes based on at least one predicted traffic. The process of adjusting the pre-filled nodes can be referred to the example described in Figure 3. Then, the computing device 104 further determines the load of the multiple pre-filled nodes, such as the requests being executed and pending in the multiple pre-filled nodes, and the length of the requests. Then, the computing device 104 adjusts the number of multiple decoding nodes based on the load of the multiple pre-filled nodes and the previously obtained at least one predicted traffic. At this point, the computing device 104 can determine the traffic to be executed on the decoding devices based on the number of requests being processed and pending on the pre-filled nodes, the length of the requests, and the subsequent predicted traffic, thereby further adjusting the number of decoding nodes. The process of adjusting the decoding nodes can be referred to the example described in Figure 4.

[0075] When the number of tasks to be processed by a pre-filled node is small, the number of pre-filled nodes can be dynamically reduced. For example, computing device 104 can determine the number of requests to be processed by a first pre-filled node among multiple pre-filled nodes. This first pre-filled node can be any one of the multiple pre-filled nodes. For example, computing device 104 can detect or query the number of requests waiting to be processed at the first pre-filled node. If this number is less than a threshold number, the requests are transferred to a second pre-filled node among the multiple pre-filled nodes. This threshold number is the minimum number of requests a node can process. If the number of requests waiting to be processed in a pre-filled node is less than this threshold number, it indicates that the resources of that pre-filled node will not be fully utilized, potentially leading to resource waste. Therefore, the number of requests waiting to be processed can be transferred to another pre-filled node among the multiple pre-filled nodes for processing. Then, computing device 104 can remove the first pre-filled node from the multiple pre-filled nodes. Therefore, by transferring requests to other nodes when the number of requests allocated to a pre-filled node is relatively small, the number of pre-filled nodes can be reduced, saving computing resources.

[0076] When the number of requests or tasks to be processed by a decoding node is relatively small, the number of decoding nodes can be dynamically reduced. For example, for multiple decoding nodes currently in use, the utilization rate of the first decoding node can be determined first. This first decoding node can be any suitable node among the multiple decoding nodes. Then, this utilization rate is further compared with a threshold utilization rate. If the utilization rate is greater than the threshold utilization rate, it indicates that the resource utilization of the decoding node is high, and the decoding node does not need to be removed. If the utilization rate is lower than the threshold utilization rate, it indicates that the resource utilization of the decoding node is low, and the decoding node can be adjusted. For example, the key-value cache in the first decoding node can be transferred to the second decoding node in real time in multiple rounds. If the key-value cache in the first decoding node has been transferred to the second decoding node, it indicates that all tasks in the first decoding node have been transferred to the second decoding node. At this time, the first decoding node has no data to process, so it can be removed from the multiple decoding nodes.

[0077] In some embodiments, to determine the utilization rate of a first decoding node among multiple decoding nodes, the computing device 104 may acquire and determine the model computing power utilization rate, model bandwidth utilization rate, and model memory utilization rate for the first decoding node. The model computing power utilization rate refers to the ratio of the actual computing power used during model operation to the total computing power provided by the hardware device (such as CPU, GPU, etc.). It reflects the effective utilization of hardware computing resources when processing model tasks. The model bandwidth utilization rate refers to the ratio of the actual data transmission bandwidth used by the model during data transmission to the total bandwidth provided by the hardware device or network. It reflects the efficiency of bandwidth resource utilization when data is transmitted between different components (such as between CPU and memory, between GPU and video memory, between servers, etc.). The model memory utilization rate refers to the ratio of the actual memory space occupied by the model during operation to the total memory space provided by the system or device. It shows the degree of memory resource utilization by the model and reflects the efficiency of the model in memory management. Then, the computing device 104 may use at least one of the model computing power utilization rate, model bandwidth utilization rate, and model memory utilization rate to determine the utilization rate of the first decoding node. In one example, computing device 104 selects the highest utilization rate from model computing power utilization, model bandwidth utilization, and model memory utilization as the utilization rate of the first decoding node. In another example, computing device 104 calculates the average of model computing power utilization, model bandwidth utilization, and model memory utilization as the utilization rate of the first decoding node. The above examples are only used to describe this disclosure and are not intended to limit the specific scope of this disclosure.

[0078] This approach allows for the prediction of traffic to be processed by the machine learning model by utilizing current and historical traffic processed by the model. By adjusting pre-filled and decoding nodes, adaptive control of nodes used for inference tasks is achieved. This enables the advance application of new computing nodes to handle unexpected bursts of traffic, avoids overloading of discrete inference nodes, and achieves elastic adjustment of computing resources, improving resource utilization and enhancing the user experience.

[0079] The above describes a schematic flowchart of an example 200 of a method for controlling nodes according to some embodiments of the present application, with reference to FIG. 2. The following describes a schematic flowchart of a method for adjusting pre-filled nodes according to some embodiments of the present application, with reference to FIG. 3. Example 300 can be executed by the computing device 104 in FIG. 1 and any suitable computing device.

[0080] At box 302, it is determined whether the plurality of pre-filled nodes can handle the first predicted traffic in at least one predicted traffic. After obtaining at least one predicted traffic of different durations for the plurality of pre-filled nodes, computing device 104 needs to determine whether the plurality of pre-filled nodes can handle the predicted traffic in at least one predicted traffic. Determining whether the plurality of pre-filled nodes can handle the predicted traffic in at least one predicted traffic can be determined based on the memory space or computing resources included in the plurality of pre-filled nodes. In one example, computing device 104 can determine whether the plurality of pre-filled nodes can handle the first predicted traffic in at least one predicted traffic by using whether the memory space for the plurality of pre-filled nodes can accommodate the activation value for the first predicted traffic. For example, the required memory space can be calculated based on the number of requests and request length in the first predicted traffic, and further determined whether the high bandwidth memory (HBM) provided by the plurality of pre-filled nodes satisfies the memory space requirement. If the plurality of pre-filled nodes can handle each of the predicted traffic in at least one predicted traffic, it indicates that the resources of the plurality of pre-filled nodes currently allocated are sufficient for future traffic, and no new pre-filled nodes need to be added. In another example, computing device 104 can determine whether the multiple pre-filled nodes are capable of handling the first predicted traffic in at least one predicted traffic stream by utilizing whether a first service level objective for the multiple pre-filled nodes is met. For example, if the first service level objective is that the latency of generating the first token of the pre-filled task does not exceed 1 second, then determining whether the multiple pre-filled nodes are capable of handling the first predicted traffic in at least one predicted traffic stream is done by determining whether the latency of generating the first token of the pre-filled task exceeds 1 second. In yet another example, computing device 104 can determine whether the multiple pre-filled nodes are capable of handling the first predicted traffic stream by utilizing whether the memory space of the multiple pre-filled nodes can accommodate the activation value for the first predicted traffic stream and whether the first service level objective for the multiple pre-filled nodes is met.

[0081] At box 304, if it is determined that multiple pre-filled nodes cannot handle the first predicted traffic, a target pre-filled node is added to execute the predicted traffic. For example, if the high-bandwidth memory provided by multiple pre-filled nodes is insufficient to store the activation value corresponding to the predicted traffic or does not meet the first service level objective, a new node needs to be added for the predicted traffic to accommodate the upcoming increased traffic. In this case, the node allocation management unit can request the allocation of a new node. Then, at box 306, the first predicted traffic is processed using the multiple pre-filled nodes and the target pre-filled node.

[0082] When using multiple pre-filled nodes and a target pre-filled node to process the first prediction traffic, the first set of model parameters for the machine learning model can be provided to the target pre-filled node from the multi-level memory layers in the memory pool, based on the order of the multi-level memory layers. For example, after allocating the target pre-filled node, the model parameters related to the pre-filling task, located in the first-level memory layer and used by the first pre-filled node, can be provided to that target pre-filled node. Next, it can be further determined whether the target pre-filled node has obtained all the parameters from the first-level memory layer. If the target pre-filled node has not obtained all the model parameters from the first-level memory layer, it can obtain the model parameters from the second-level memory layer below the first-level memory layer, and so on, until all the model parameters are obtained. If all the model parameters are obtained from the first-level memory layer or from the multi-level memory layers, the target pre-filled node is allocated these model parameters. After the target pre-filled node has these model parameters, it can be used along with the previous multiple pre-filled nodes to process subsequent prediction traffic.

[0083] This method allows for the pre-allocation of new preprocessing nodes based on predicted traffic, ensuring sufficient preprocessing nodes to handle the predicted traffic when higher traffic volumes arrive later. This accelerates traffic processing speed and capacity, reduces task processing latency, and provides a better user experience.

[0084] The above describes a schematic flowchart of a method for adjusting pre-filled nodes according to some embodiments of this application, with reference to FIG. 3. The following describes a schematic flowchart of a method for adjusting decoded nodes according to some embodiments of this application, with reference to FIG. 4. Example 400 can be performed by the computing device 104 in FIG. 1 and any suitable computing device. For ease of description, the at least one predicted flow described above can be referred to as at least one first predicted flow. The adjustment of multiple decoded nodes is described below with reference to Example 400.

[0085] At box 402, computing device 104 determines at least one second predicted flow to be provided by a plurality of pre-filled nodes to a plurality of decoding nodes based on load and at least one first predicted flow. For example, requests being executed and requests awaiting execution in the plurality of pre-filled nodes can be determined, which, after being processed by the pre-filled nodes, will be processed by the plurality of decoding nodes. Additionally, at least one predicted flow, after being processed by the plurality of pre-filled nodes, also needs to be processed by the plurality of decoding nodes. This load and predicted flow determine the at least one second predicted flow to be processed by the plurality of decoding nodes.

[0086] For example, computing device 104 can determine, based on the load in the plurality of pre-filled nodes and at least one first predicted flow, a second predicted flow for a first predetermined duration and another second predicted flow for a second predetermined duration to be processed by the plurality of decoding nodes. The at least one second predicted flow includes the first second predicted flow and the other second predicted flow. Alternatively or additionally, the at least one second predicted flow may include one predicted flow or may also include predicted flows for other predetermined durations. In one example, the predicted flow to be processed by the plurality of decoding nodes is calculated based on the load in the pre-filled nodes and the predicted flow for different times of the plurality of pre-filled nodes through a predetermined computational disclosure.

[0087] At box 404, the number of multiple decoding nodes is adjusted based on at least one second predicted flow. After determining the at least one second predicted flow to be processed by the multiple decoding nodes, the multiple decoding nodes can be adjusted according to the at least one second predicted flow.

[0088] In some embodiments, when adjusting multiple decoding nodes, the computing device 104 first determines whether the multiple decoding nodes are capable of handling at least one second predicted traffic from a second set of predicted traffic. The second predicted traffic can be any one of the at least one second predicted traffic. For example, it can be determined whether the multiple decoding nodes have sufficient HBM space to accommodate the KV cache corresponding to the second predicted traffic, or whether a second service level objective for the second predicted traffic is met. If it is determined that the multiple decoding nodes are capable of handling at least one second predicted traffic from a second predicted traffic, then the multiple decoding nodes can be used to process the second predicted traffic. If it is determined that the multiple decoding nodes are not capable of handling at least one second predicted traffic from a second predicted traffic, then a target decoding node needs to be added to handle the second predicted traffic. Then, the multiple decoding nodes and the target decoding node are used to process the second predicted traffic.

[0089] In one example, whether multiple decoding nodes can handle at least one second predicted flow can be determined by whether their memory space can accommodate a key-value cache for the second predicted flow. If the memory space of the multiple decoding nodes can accommodate the key-value cache for the second predicted flow, it can be determined that the multiple decoding nodes can handle at least one second predicted flow. If the memory space of the multiple decoding nodes cannot accommodate the key-value cache for the second predicted flow, it can be determined that the multiple decoding nodes cannot handle at least one second predicted flow. In another example, whether multiple decoding nodes can handle at least one second predicted flow can be determined by whether a second service level objective for the multiple decoding nodes is met. For example, the second service level objective is that the latency for generating a token in the decoding task does not exceed 50 milliseconds or 80 milliseconds. If the second service level objective for the multiple decoding nodes is met, it can be determined that the multiple decoding nodes can handle at least one second predicted flow. If the second service level objective for the multiple decoding nodes is not met, it can be determined that the multiple decoding nodes cannot handle the second predicted flow in the second set of predicted flows.

[0090] In some embodiments, when processing the second predicted traffic using multiple decoding nodes and a target decoding node, the second set of model parameters for the decoding task can be provided to the target decoding node from the multi-level memory layers based on the order of the multi-level memory layers in the memory pool. For example, computing device 104 provides the target decoding node with parameters for the decoding task from the first level memory layer in the multi-level memory layers, which are used by the decoding nodes among the multiple decoding nodes. Then, it is determined whether the parameters provided by the first level memory layer are all the parameters in the second set of model parameters for the decoding task of the machine learning model. If they are all the parameters in the second set of model parameters, the parameter loading task for the target decoding node is completed. If it is determined that the parameters provided by the first level memory layer are part of the second set of model parameters, the remaining parameters in the second set of model parameters can be provided to the target decoding node by other levels of memory layers in the multi-level memory layers.

[0091] Then, the computing device 104 can further determine the target duration corresponding to the second predicted traffic. For example, it can determine whether the predicted traffic is short-term (e.g., traffic predicted 30 seconds later) or long-term (e.g., traffic predicted 10 minutes later). Then, based on the target duration corresponding to the second predicted traffic, it controls whether to transfer the key-value cache from multiple decoding nodes to the target decoding node. For example, the computing device 104 can determine whether the target duration is greater than a threshold duration. If the target duration is greater than the threshold duration, it indicates that the predicted traffic will arrive some time later. In this case, the key-value cache from one or more decoding nodes can be migrated in blocks to the target decoding node, thereby reducing the load on the currently used decoding nodes. The decoding node to which the key-value cache blocks are migrated can be called the source decoding node, which can be the node or some of the nodes with the heaviest load among the multiple decoding nodes. If the target duration is less than or equal to the threshold duration, it may not be possible to migrate the key-value cache from multiple decoding nodes to the target decoding node because the predicted traffic arrives too quickly. Therefore, instead of migrating the currently processed request, the target decoding node can be used to receive new requests.

[0092] This approach allows for the prediction of traffic to be processed by the machine learning model by utilizing current and historical traffic processed by the model. This enables the control of adjustments to pre-filled and decoding nodes, achieving adaptive node control. Consequently, new computing nodes can be requested in advance to handle unexpected bursts of traffic, preventing overload of the split inference server. Furthermore, it enables elastic adjustment of computing resources, improving resource utilization and enhancing the user experience.

[0093] The above describes a schematic flowchart of a method for adjusting decoding nodes according to some embodiments of this application, with reference to Figure 4. The following describes a schematic diagram of an example architecture for dynamic scaling of nodes according to some embodiments of this application, with reference to Figure 5.

[0094] In the example architecture 500 of Figure 5, three main modules are included: a traffic predictor 510, a node scaling controller 508, and a node cold start module 506. The traffic predictor 510 can perform short-term predictions 524 and long-term predictions 526. For example, based on the acquired current and historical traffic data, the traffic predictor 510 can use an LSTM model to predict short-term and long-term traffic. For example, predicting the traffic in 30 seconds and 10 minutes. Then, the traffic predictor 510 can provide its predicted data to the service request scheduler 502 and the node scaling controller 508. The service request scheduler 502 is used for request allocation and scheduling. For example, providing the acquired data processing requests to pre-populated nodes 504 for processing. Between pre-fill node 504 and decoding node 512, after the pre-fill task is completed at pre-fill node 504, the KV cache needs to be transferred to the decoding node. This process, shown in box 516, is a layer-by-layer transfer of the KV cache between P / D nodes. First, the KV cache of the paged layer 0 of the model is transferred, then the KV cache of the paged layer 1 is transferred, and so on, until the KV cache of all layers is transferred. The pre-fill node can run multiple instances, such as instance 1, instance 2, and instance 3. For each instance, a new node needs to be requested when the request traffic is too high, and nodes need to be merged and released when the request traffic is too low. For decoding node 512, it can also implement block-by-block migration of the KV cache between hot and cold nodes, as shown in box 522. For example, the KV cache of the paged layer in block 0 can be migrated to other decoding nodes, then the KV cache of the paged layer in block 1 can be migrated to other decoding nodes, and the KV cache of the paged layer in block 2 can be migrated to other decoding nodes. A decoding node can also run multiple instances, such as instance 1, instance 2, and instance 3. For each instance, if the pre-filled traffic is too high, a new node is requested; if the pre-filled traffic is too low, the node is released.

[0095] The node scaling controller 508 can sense the current traffic input to the pre-filled nodes and can also assess the node load. For example, it can assess the load on the pre-filled nodes or decoding nodes. Furthermore, the node scaling controller 508 can perform node load prediction, such as predicting the traffic a node will handle. Then, when the predicted traffic necessitates adding a new node, the node scaling controller 508 performs node pre-request. Furthermore, the node scaling controller 508 can also perform node defragmentation. At this time, it can calculate the decoding node utilization score S. Utilization=max(MFU, MBU, MMU), where MFU represents model computing power utilization, MBU represents model bandwidth utilization, MMU represents model memory utilization, and max() represents the maximum value. If a node's score is below 50%, a loop algorithm is used to find a node that can be merged. Then, the redundant node release module 514 migrates the KV cache in the node in real time in multiple rounds, performs hot switching of inference nodes, and releases node resources. In addition, when the pre-filled node group lacks sufficient HBM space to accommodate activation values ​​for real-time or predicted traffic, a new pre-filled node is requested; when the predicted traffic from the pre-filled nodes continues to grow, and the decoding node cannot accommodate the KV cache under the predicted traffic or cannot meet the service level target, a new decoding node is requested. After requesting a new node 520, the migration of parameters and requests is implemented through the cold start module 506. For example, the model weights are added to the new node 520 through the multi-level memory layer of the multi-level memory pool 518, such as layered transmission.

[0096] When node 520 is a new pre-fill node and decoding node, a multi-level memory pool can be used to preload model weights layer by layer. The memory pool storing parameters is divided into multiple levels or layers, each with a different loading speed. The data processing speed of higher layers is greater than that of lower layers, and the model parameters stored in higher memory layers come from the model parameters stored in lower memory layers. However, each node can allocate a portion of the topmost memory, and parameters can be passed between these portions. Therefore, for a new pre-fill node, the same model parameters used in the topmost memory of other nodes can be directly loaded into the topmost memory used for this new pre-fill node. If the new pre-fill node also needs parameters not present in the topmost memory of other nodes, the remaining parameters can be obtained from the shared lower memory. For example, the multi-level memory layers in multi-level memory pool 518 include an HBM layer, a Dynamic Random Access Memory (DRAM) layer, and a Solid State Drive (SSD) layer. The HBM layer is the first layer, the DRAM layer is the second layer, and the SSD layer is the third layer. If the parameters for the new node exist in the HBM layer used for other nodes, the model parameters are loaded directly from the HBM layer. If the HBM layer does not have the required model parameters, the remaining model parameters are obtained from the DRAM layer or SSD layer.

[0097] Then, the KV caches of some requests from existing nodes 1, 2, and 3 are migrated to the new nodes in real time, in multiple rounds, to achieve load balancing. After the migration is complete, the nodes are connected to the cluster to start inference work, thus achieving hot request migration. If the current cluster cannot handle the predicted short-term traffic, the request migration process is skipped, and the nodes are immediately connected to the cluster after loading the model weights.

[0098] This approach enables the prediction of short-term and long-term traffic trends based on real-time and historical traffic records, and allows for dynamic scaling of nodes. For example, based on the current memory utilization of the split inference cluster and long-term traffic trends, it assesses whether the server can handle sudden traffic surges (SLO meets requirements, nodes are pre-filled to accommodate activation values, and decoding nodes can accommodate KV cache) to determine whether additional nodes need to be requested. Furthermore, it can determine whether to reduce the number of nodes and shrink the cluster size based on the utilization of inference nodes. During node cold start, model weights are pre-loaded onto the new nodes to be connected, accelerating the cold start process; and new nodes are connected in advance based on traffic predictions.

[0099] The above description, with reference to Figure 5, illustrates an example architecture for dynamic scaling of nodes according to some embodiments of this application. The following description, with reference to Figure 6, illustrates an example function for node adjustment according to some embodiments of this application.

[0100] The example function 600 in Figure 6 includes receiving concurrent user requests 602 and then providing them to the traffic prediction module 604, enabling long-term traffic prediction at 606 and short-term traffic prediction at 610. The results of the long-term and short-term traffic predictions are then provided to the scaling control module 608. If a new node needs to be added, a node cold start is performed via the cold start module 614. After the node cold start, node expansion is implemented at box 616. Then, at 618, pre-filling inference is performed using the expanded pre-filled node and the previous pre-filled node. After pre-filling inference, dynamic cache transfer is performed at 620 to transmit to the decoding node. If a new decoding node is added during node expansion, decoding inference is performed at 622 using the new decoding node and the previous decoding node. Then, the inference result is given at 628. Additionally, after node expansion, some decoding inference tasks performed on the original decoding node can be migrated to the expanded node via request migration at 630. Additionally, during pre-filling and decoding inference, load awareness can be performed on the nodes executing tasks at position 624. Based on the perceived load, node hot merging and release can be performed at position 626 when certain conditions are met. For example, for pre-filling nodes, node merging and release are performed when the number of pending tasks is less than a threshold; for decoding nodes, node hot merging and release are determined based on node utilization. Furthermore, concurrent request scheduling is performed at position 612 for pre-filling inference.

[0101] The above description, with reference to FIG. 6, illustrates an example function of node adjustment according to some embodiments of the present application. The following description, with reference to FIG. 7, illustrates an example process for adjusting nodes according to some embodiments of the present application.

[0102] As shown in the example process 700 in Figure 7, at box 702, short-term and long-term request traffic is predicted based on real-time data traffic and historical data, and the corresponding pre-filled traffic is evaluated. For example, in the traffic prediction module, traffic data is collected at a frequency of 1Hz; every 30 seconds, using real-time traffic combined with historical traffic records, the request traffic after 30 seconds (short-term) and 10 minutes (long-term) is predicted through an LSTM network; every 30 seconds, the pre-filled traffic delivered to the decoding node after 30 seconds (short-term) and 10 minutes (long-term) is evaluated based on the predicted request traffic and the pre-filled node load.

[0103] At box 704, nodes with a utilization score below 50% attempt hot merging; and when a pre-filled node cannot handle the requested traffic, or a decoding node cannot handle the pre-filled traffic, a new node is requested. For example, in the scaling control module, the decoding node utilization score S is calculated. Utilization =max(MFU, MBU, MMU); S Utilization For decoding nodes with a score <50%, a cyclic algorithm is used to attempt node merging, migrating the key-value cache in the node to other nodes in real-time, block-by-block rounds, freeing up redundant nodes. For real-time request traffic and predicted request traffic, if a pre-filled node lacks HBM space to accommodate activation values ​​or its SLO (Solution Time Limit) is not met, a new pre-filled node is requested. For pre-filled nodes, their SLO can be set to a token generation latency of no more than 1 second. For real-time pre-filled traffic and expected pre-filled traffic, if a decoding node lacks HBM space to accommodate key-value caches or its SLO is not met, a new decoding node is requested. For decoding nodes, their SLO can be set to a token generation latency of no more than 50 milliseconds or 80 milliseconds.

[0104] At box 706, model weights are preloaded layer by layer. When the current cluster is sufficient to handle the predicted short-term traffic, some ongoing inference requests are hot-migrated to new nodes. In the cold start module, model weights are preloaded layer by layer from a multi-level memory pool. The key-value caches of some requests in existing nodes are migrated to cold nodes in real time in multiple rounds, and after the migration is completed, they are connected to the cluster to start inference work, realizing hot migration of requests. If the current cluster cannot handle the predicted short-term traffic, the request migration process is skipped, the model weights are loaded, and the cluster is immediately connected to accept new requests.

[0105] Figure 8 further illustrates a block diagram of an apparatus 800 for controlling nodes according to an embodiment of this application. The apparatus 800 is applied to a computing device and may include multiple modules for performing the corresponding steps in the method example 200 discussed in Figure 2. As shown in Figure 8, the apparatus 800 includes: a target traffic acquisition unit 802 configured to acquire target traffic input into a machine learning model, the machine learning model being deployed across multiple computing nodes, the multiple computing nodes including multiple pre-filled nodes and multiple decoding nodes; a traffic prediction unit 804 configured to predict at least one predicted traffic corresponding to at least one predetermined duration to be processed by the machine learning model based on the target traffic and historical traffic processed by the machine learning model; and a node adjustment unit 806 configured to adjust the number of multiple pre-filled nodes and multiple decoding nodes based on at least one predicted traffic.

[0106] In some embodiments, the target traffic determination unit 802 includes: a traffic acquisition unit configured to acquire traffic data by acquiring traffic input to a machine learning model; and a first traffic determination unit configured to determine target traffic based on the acquired traffic data.

[0107] In some embodiments, the node adjustment unit 806 includes: a first adjustment unit configured to adjust the number of a plurality of pre-filled nodes based on at least one predicted traffic; a load determination unit configured to determine the load of the plurality of pre-filled nodes; and a second adjustment unit configured to adjust the number of a plurality of decoding nodes based on the load and at least one predicted traffic.

[0108] In some embodiments, the first adjustment unit includes: a first determination unit configured to determine whether a plurality of pre-filled nodes can carry a first predicted traffic in at least one predicted traffic; a node addition unit configured to add a target pre-filled node for performing the first predicted traffic if it is determined that the plurality of pre-filled nodes cannot carry the first predicted traffic; and a traffic processing unit configured to process the first predicted traffic using the plurality of pre-filled nodes and the target pre-filled node.

[0109] In some embodiments, the first determination unit includes: a first traffic carrying determination unit, configured to determine whether a plurality of pre-filled nodes can carry the first predicted traffic in at least one predicted traffic based on at least one of the following: whether the memory space of the plurality of pre-filled nodes can accommodate the activation value for the first predicted traffic; or whether a first service level objective for the plurality of pre-filled nodes is met. In some embodiments, the traffic processing unit includes: a first parameter providing unit, configured to provide a first set of model parameters for a machine learning model from a multi-level memory layer to a target pre-filled node based on the order of the multi-level memory layers in a memory pool; and a first processing unit, configured to process the first predicted traffic using the target pre-filled node having the first set of model parameters and the plurality of pre-filled nodes.

[0110] In some embodiments, at least one predicted traffic is at least one first predicted traffic, and the second adjustment unit includes: a predicted traffic providing unit configured to determine at least one second predicted traffic to be provided by a plurality of pre-filled nodes to a plurality of decoding nodes based on load and at least one first predicted traffic; and a third adjustment unit configured to adjust the number of the plurality of decoding nodes based on at least one second predicted traffic.

[0111] In some embodiments, the third adjustment unit includes: a traffic determination unit configured to determine whether a plurality of decoding nodes can carry at least one second predicted traffic; a target decoding node addition unit configured to add a target decoding node for performing the second predicted traffic if it is determined that the plurality of decoding nodes cannot carry at least one second predicted traffic; and a predicted traffic processing unit configured to process the second predicted traffic using the plurality of decoding nodes and the target decoding node.

[0112] In some embodiments, the traffic determination unit includes: a second traffic carrying determination unit, configured to determine whether the plurality of decoding nodes are capable of carrying at least the second predicted traffic in the second predicted traffic based on at least one of the following: whether the memory space of the plurality of decoding nodes can accommodate the activation value for the second predicted traffic; or whether the second service level objective for the plurality of decoding nodes is met.

[0113] In some embodiments, the predicted traffic processing unit includes: a second parameter providing unit configured to provide a second set of model parameters for a machine learning model from the multi-level memory layers to a target decoding node based on the order of the multi-level memory layers in the memory pool; and a transfer control unit configured to control the transfer of key-value caches from multiple decoding nodes to the target decoding node based on a target duration corresponding to the second predicted traffic.

[0114] In some embodiments, the second parameter providing unit includes: a third parameter providing unit configured to provide the target decoding node with parameters for the decoding task located in a first-level memory layer in a multi-level memory layer, which are used by the decoding nodes among a plurality of decoding nodes; and a remaining parameter providing unit configured to provide the target decoding node with the remaining parameters in the second set of model parameters from other levels of the multi-level memory layer in response to determining that the parameters provided by the first-level memory layer are part of a second set of model parameters.

[0115] In some embodiments, the transfer control unit includes: a duration comparison unit configured to determine whether a target duration is greater than a threshold duration; a source decoding node determination unit configured to determine, in response to the target duration being greater than the threshold duration, a source decoding node among a plurality of decoding nodes to migrate key-value cache blocks; and a block migration unit configured to migrate the key-value cache in the source decoding node to the target decoding node by block during the decoding process of the source decoding node.

[0116] In some embodiments, the apparatus further includes: a request number determination unit configured to determine the number of requests to be processed by a first pre-filled node among a plurality of pre-filled nodes; a request transfer unit configured to transfer the number of requests to a second pre-filled node among a plurality of pre-filled nodes in response to the number being less than a threshold number; and a node removal unit configured to remove the first pre-filled node from the plurality of pre-filled nodes.

[0117] In some embodiments, the apparatus further includes: a utilization determination unit configured to determine the utilization of a first decoding node among a plurality of decoding nodes; a key-value cache transfer unit configured to transfer the key-value cache in the first decoding node in blocks to a second decoding node among the plurality of decoding nodes in response to the utilization being lower than a threshold utilization; and a decoding node removal unit configured to remove the first decoding node from the plurality of decoding nodes in response to the key-value cache in the first decoding node being transferred to the second decoding node.

[0118] In some embodiments, the utilization determination unit includes: a plurality of utilization determination units configured to determine model computing power utilization, model bandwidth utilization, and model memory utilization for a first decoding node; and a decoding node utilization determination module configured to determine the utilization of the first decoding node based on at least one of model computing power utilization, model bandwidth utilization, and model memory utilization.

[0119] Figure 9 shows a schematic block diagram of an example device 900 that can be used to implement embodiments of the present application. For example, the node and computing device 104 in Figure 1 of the present application can be implemented by the example device 900. As shown, the device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 902 or loaded from storage unit 908 into random access memory (RAM) 903. Various programs and data required for the operation of the device 900 may also be stored in RAM 903. The CPU 901, ROM 902, and RAM 903 are interconnected via bus 904. An input / output (I / O) interface 905 is also connected to bus 904.

[0120] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0121] The various processes and handling described above, such as method examples 200, 300, and 400, can be executed by processing unit 901. For example, in some embodiments, method examples 200, 300, and 400 can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by CPU 901, one or more actions of method examples 200, 300, and 400 described above can be performed.

[0122] This application may be a method, apparatus, system, chip, and / or computer program product. A chip may include a processing unit and a communication interface, the processing unit being capable of processing program instructions received from the communication interface. A computer program product may include a computer-readable storage medium on which computer-readable program instructions for performing various aspects of this application are stored.

[0123] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0124] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0125] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing state information from the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0126] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0127] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0128] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0130] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for controlling nodes, characterized in that, The method includes: The target traffic is obtained and input into a machine learning model, which is deployed across multiple computing nodes, including multiple pre-filled nodes and multiple decoding nodes. Based on the target traffic and historical traffic processed by the machine learning model, predict at least one predicted traffic corresponding to at least one predetermined duration to be processed by the machine learning model; and Based on the at least one predicted traffic, adjust the number of the plurality of pre-filled nodes and the plurality of decoding nodes.

2. The method according to claim 1, characterized in that, Obtaining the target traffic input into the machine learning model includes: Traffic data is acquired by collecting traffic input into the machine learning model; and Based on the acquired traffic data, the target traffic is determined.

3. The method according to claim 1, characterized in that, Adjusting the number of the plurality of pre-filled nodes and the plurality of decoding nodes includes: Based on the at least one predicted flow, adjust the number of the plurality of pre-filled nodes; Determine the load of the plurality of pre-filled nodes; and The number of the plurality of decoding nodes is adjusted based on the load and the at least one predicted traffic.

4. The method according to claim 3, characterized in that, Adjusting the number of the plurality of pre-filled nodes includes: Determine whether the plurality of pre-filled nodes are capable of carrying the first predicted traffic in the at least one predicted traffic; If it is determined that the plurality of pre-filled nodes cannot handle the first predicted traffic, a target pre-filled node is added to perform the first predicted traffic; and The first predicted traffic is processed using the plurality of pre-filled nodes and the target pre-filled node.

5. The method according to claim 4, characterized in that, Determining whether the plurality of pre-filled nodes can carry the first predicted traffic in the at least one predicted traffic includes: Whether the plurality of pre-filled nodes are capable of carrying the first predicted flow of the at least one predicted flow is determined based on at least one of the following: Whether the memory space of the multiple pre-filled nodes can accommodate the activation value for the first predicted traffic; or Whether the first service level objective for the plurality of pre-filled nodes is met.

6. The method according to claim 4, characterized in that, Processing the first predicted traffic using the plurality of pre-filled nodes and the target pre-filled node includes: Based on the order of the multi-level memory layers in the memory pool, the first set of model parameters for the machine learning model is provided from the multi-level memory layers to the target pre-filled node; and The first predicted flow is processed using the target pre-filled node and the plurality of pre-filled nodes having the first set of model parameters.

7. The method according to claim 3, characterized in that, The at least one predicted traffic is at least one first predicted traffic, and adjusting the number of the plurality of decoding nodes includes: Based on the load and the at least one first predicted traffic, determine at least one second predicted traffic to be provided by the plurality of pre-filled nodes to the plurality of decoding nodes; and The number of the plurality of decoding nodes is adjusted based on the at least one second predicted traffic.

8. The method according to claim 7, characterized in that, Adjusting the number of the plurality of decoding nodes based on the at least one second predicted traffic includes: Determine whether the plurality of decoding nodes are capable of carrying the second predicted traffic in at least one second predicted traffic; If it is determined that the plurality of decoding nodes cannot carry the second predicted traffic in at least one of the second predicted traffic, a target decoding node is added to execute the second predicted traffic; and The second predicted traffic is processed using the plurality of decoding nodes and the target decoding node.

9. The method according to claim 8, characterized in that, Processing the second predicted traffic using the plurality of decoding nodes and the target decoding node includes: Based on the order of the multi-level memory layers in the memory pool, a second set of model parameters for the machine learning model is provided from the multi-level memory layers to the target decoding node; and Based on the target duration corresponding to the second predicted traffic, the transfer of key-value caches from the plurality of decoding nodes to the target decoding node is controlled.

10. The method according to claim 9, characterized in that, Controlling the transfer of key-value caches from the plurality of decoding nodes to the target decoding node based on the target duration corresponding to the second predicted traffic includes: Determine whether the target duration is greater than the threshold duration; and In response to the target duration being greater than the threshold duration, the source decoding node among the plurality of decoding nodes to be migrated for the key-value cache block is determined; During the decoding process of the source decoding node, the key-value cache in the source decoding node is migrated to the target decoding node in blocks.

11. The method according to any one of claims 1-10, characterized in that, The method further includes: Determine the number of requests to be processed by the first pre-filled node among the plurality of pre-filled nodes; and In response to the number being less than a threshold number, the requests for that number are transferred to a second pre-filled node among the plurality of pre-filled nodes; and Remove the first pre-filled node from the plurality of pre-filled nodes.

12. The method according to any one of claims 1-10, characterized in that, The method further includes: Determine the utilization rate of the first decoding node among the plurality of decoding nodes; and In response to the utilization rate falling below a threshold utilization rate, the key-value cache in the first decoding node is transferred in blocks to the second decoding node among the plurality of decoding nodes; and In response to the key-value cache in the first decoding node being transferred to the second decoding node, the first decoding node is removed from the plurality of decoding nodes.

13. An apparatus for controlling a node, characterized in that, The device includes: The target traffic acquisition unit is configured to acquire target traffic input into a machine learning model, the machine learning model being deployed across multiple computing nodes, the multiple computing nodes including multiple pre-filled nodes and multiple decoding nodes; A traffic prediction unit is configured to predict at least one predicted traffic corresponding to at least one predetermined duration to be processed by the machine learning model, based on the target traffic and historical traffic processed by the machine learning model; and The node adjustment unit is configured to adjust the number of the plurality of pre-filled nodes and the plurality of decoded nodes based on the at least one predicted traffic.

14. The apparatus according to claim 13, characterized in that, The target traffic acquisition unit is configured to include: A traffic acquisition unit is configured to acquire traffic data by acquiring traffic input to the machine learning model; and The first traffic determination unit is configured to determine the target traffic based on the acquired traffic data.

15. The apparatus according to claim 13, characterized in that, The node adjustment unit includes: A first adjustment control unit is configured to adjust the number of the plurality of pre-filled nodes based on the at least one predicted flow. A load determination unit is configured to determine the load of the plurality of pre-filled nodes; and The second adjustment unit is configured to adjust the number of the plurality of decoding nodes based on the load and the at least one predicted traffic.

16. The apparatus according to claim 15, characterized in that, The first adjustment control unit includes: The first determination unit is configured to determine whether the plurality of pre-filled nodes are capable of carrying the first predicted flow in the at least one predicted flow; The node adding unit is configured to add a target pre-filled node for performing the first predicted traffic if it is determined that the plurality of pre-filled nodes cannot handle the first predicted traffic; and The traffic processing unit is configured to process the first predicted traffic using the plurality of pre-filled nodes and the target pre-filled node.

17. The apparatus according to claim 16, characterized in that, The first determination unit includes: The first traffic carrying determination unit is configured to determine whether the plurality of pre-filled nodes can carry the first predicted traffic in the at least one predicted traffic based on at least one of the following: Whether the memory space of the multiple pre-filled nodes can accommodate the activation value for the first predicted traffic; or Whether the first service level objective for the plurality of pre-filled nodes is met.

18. The apparatus according to claim 16, characterized in that, The flow processing unit includes: The first parameter providing unit is configured to provide a first set of model parameters for the machine learning model from the multi-level memory layers in the memory pool to the target pre-filled node based on the order of the multi-level memory layers; and The first processing unit is configured to process the first predicted flow using the target pre-filled node having the first set of model parameters and the plurality of pre-filled nodes.

19. The apparatus according to claim 15, characterized in that, The at least one predicted flow is at least one first predicted flow, and the second adjustment unit includes: A predictive traffic providing unit is configured to determine, based on the load and the at least one first predicted traffic, at least one second predicted traffic to be provided by the plurality of pre-filled nodes to the plurality of decoding nodes; and The third adjustment unit is configured to adjust the number of the plurality of decoding nodes based on the at least one second predicted flow.

20. The apparatus according to claim 19, characterized in that, The third adjustment unit includes: The traffic determination unit is configured to determine whether the plurality of decoding nodes are capable of carrying the second predicted traffic in at least one second predicted traffic; A target decoding node adding unit is configured to add a target decoding node for executing the second prediction traffic if it is determined that the plurality of decoding nodes cannot carry the second prediction traffic in at least one second prediction traffic; and The predicted traffic processing unit is configured to process the second predicted traffic using the plurality of decoding nodes and the target decoding node.

21. An electronic device, comprising: At least one computing unit; At least one memory coupled to the at least one computing unit and storing instructions for execution by the at least one computing unit, the instructions, when executed by the at least one computing unit, causing the device to perform the method according to any one of claims 1-12.

22. A computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method according to any one of claims 1-12.

23. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1-12.

Images