Accelerator memory sharing method and related apparatus

Abstract

An accelerator memory sharing method, which is applied to the technical field of artificial intelligence. In the accelerator memory sharing method, the same accelerator is used to simultaneously execute an inference task and a training task, and a portion of a memory on the accelerator is allocated to the inference task, and the other portion of the memory is allocated to the training task. When an inference load of the accelerator changes, memory amounts on the accelerator that are allocated to the inference task and the training task change accordingly, such that the memory amount on the accelerator that is allocated to the inference task can adaptively meet the requirements of the inference load.

Description

A method and related apparatus for sharing accelerator memory

[0001] This application claims priority to Chinese Patent Application No. 202411629602.3, filed on November 13, 2024, entitled "A Method and Apparatus for Sharing Accelerator Memory", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and apparatus for sharing accelerator memory. Background Technology

[0003] Currently, machine learning platforms comprised of numerous servers equipped with graphics processing units (GPUs) are the mainstream approach for implementing machine learning tasks. Machine learning tasks primarily fall into two categories: inference tasks and training tasks. Generally, inference tasks have real-time requirements, need to meet service-level objectives, and are latency-sensitive. Training tasks typically do not have real-time requirements but often aim to optimize throughput.

[0004] To meet the different needs of inference and training tasks while avoiding performance interference between them, existing machine learning platforms typically deploy separate server clusters for inference and training tasks—namely, inference server clusters and training server clusters. Since the inference requests required are unpredictable, inference server clusters are usually over-provisioned based on peak inference request volumes to avoid service quality degradation due to surges in inference requests. Therefore, when inference request volume is low, the utilization rate of the inference server cluster is low, increasing the operational costs of machine learning services.

[0005] Therefore, how to improve the utilization rate of hardware for performing machine learning tasks has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a method for sharing accelerator memory, which adaptively adjusts the memory allocated to inference and training tasks based on inference load, so that inference and training tasks can run normally on the same accelerator and improve the utilization of the accelerator.

[0007] In a first aspect, a method for sharing accelerator memory is provided. This method is applied to an accelerator on which a first model and a second model are deployed. The accelerator is used to perform inference tasks based on the first model and to perform training tasks based on the second model. The method includes: when the inference load of the accelerator changes, an execution device determines a memory adjustment amount, which indicates the amount of adjustment to the memory on the accelerator used for performing inference tasks.

[0008] Then, the execution device determines the batch size adjustment, the first memory amount, and the second memory amount based on the memory adjustment amount. The batch size adjustment amount indicates the adjustment of the batch size of the training data used by the accelerator when performing the training task; the first memory amount indicates the size of memory reallocated on the accelerator to the inference task; and the second memory amount indicates the size of memory reallocated on the accelerator to the training task. Generally, the total memory amount of the accelerator can be the sum of the first memory amount and the second memory amount.

[0009] Secondly, the execution device adjusts the batch size of the training data used by the accelerator when performing training tasks, based on the batch size adjustment amount.

[0010] Finally, the execution device adjusts the amount of memory used by the accelerator when performing inference tasks to a first amount of memory, and adjusts the amount of memory used by the accelerator when performing training tasks to a second amount of memory.

[0011] In this scheme, a single accelerator is used to execute inference and training tasks simultaneously, with a portion of the accelerator's memory allocated to inference tasks and the remainder to training tasks. As the accelerator's inference load changes, the amount of memory allocated to inference and training tasks changes accordingly. This ensures that the memory allocated to inference tasks adaptively meets the inference load requirements, guaranteeing timely completion of inference tasks while effectively utilizing the accelerator's memory resources. Furthermore, before adjusting the memory allocated to training tasks, the batch size of the training data used by the accelerator during training is adjusted to ensure that the actual memory used by the accelerator during training aligns with the memory adjustment requirements, guaranteeing successful memory adjustments. In this way, adaptively adjusting the memory allocated to inference and training tasks based on inference load allows both tasks to run normally on the same accelerator, ensuring rapid response to inference requests and improving accelerator utilization.

[0012] In one possible implementation, before adjusting the batch size of the training data used by the accelerator when performing a training task, the method for sharing accelerator memory further includes: if the training task currently being performed by the accelerator is a gradient calculation task, the execution device controls the accelerator to stop performing the gradient calculation task so that the accelerator performs a new training task based on the adjusted batch size.

[0013] In other words, when it is determined that the batch size of the training data needs to be adjusted, if the training task being performed by the accelerator is a gradient calculation task, the execution device will control the accelerator to stop performing the gradient calculation task. That is, all intermediate results obtained by the accelerator in the current round of iterative training will be discarded so that a new round of iterative training can be started.

[0014] In this scheme, the execution progress of the training task is used to control whether to stop the training task and execute a new round of training tasks, thereby enabling rapid adjustment of the amount of memory used by the inference task and the training task, and ensuring the rapid response of the inference task.

[0015] In one possible implementation, the gradient calculation task includes multiple subtasks, which are maintained by a task management module in the processor. The task management module is used to submit some of the subtasks to the accelerator sequentially based on the progress of the subtasks being processed by the accelerator.

[0016] The specific ways to stop the accelerator from executing gradient calculation tasks include: clearing all subtasks maintained by the task management module to stop submitting subtasks in the gradient calculation task to the accelerator.

[0017] In this solution, a task management module is installed on the processor. This module maintains multiple subtasks derived from gradient calculation and periodically distributes some subtasks to the accelerator based on its processing progress. This ensures that the accelerator always receives only a portion of the subtasks, allowing it to quickly complete them and stop training. Furthermore, since the task management module is located on the processor, there is no need for hardware management of the accelerator, avoiding uncontrollable hardware management issues. Running a software module on the processor can trigger the accelerator to quickly stop training, reducing the implementation complexity of the solution.

[0018] In one possible implementation, before adjusting the batch size of the training data used by the accelerator when performing a training task, the method for sharing accelerator memory further includes: if the training task currently being performed by the accelerator is a model parameter update task, the execution device waits for the accelerator to finish performing the model parameter update task.

[0019] In this scheme, since the model parameter update task only updates the values ​​of each parameter based on the gradient of each parameter in the model, the execution time of the model parameter update task will be very short. Therefore, the execution device does not need to wait too long for the accelerator to complete the model parameter update task before adjusting the amount of memory used for the inference and training tasks. This ensures that the response of the inference task will not be greatly affected and avoids affecting the normal update of the model parameters.

[0020] In one possible implementation, as the inference load on the accelerator increases, a memory adjustment amount is used to indicate the amount of increase in memory on the accelerator used for performing inference tasks, and a batch size adjustment amount is used to indicate the amount of decrease in the batch size of training data used by the accelerator when performing training tasks.

[0021] Therefore, when adjusting the amount of memory, the execution device increases the amount of memory used by the accelerator to perform inference tasks to a first amount of memory, and decreases the amount of memory used by the accelerator to train the second model to a second amount of memory.

[0022] In one possible implementation, when the inference load on the accelerator decreases, a memory adjustment amount is used to indicate the amount by which the memory on the accelerator used for performing inference tasks is reduced, and a batch size adjustment amount is used to indicate the amount by which the batch size of the training data used by the accelerator when performing training tasks is increased.

[0023] Therefore, when adjusting the amount of memory, the execution device reduces the amount of memory used by the accelerator to a first amount of memory when performing inference tasks, and increases the amount of memory used by the accelerator to a second amount of memory when training the second model.

[0024] In one possible implementation, the execution device determines the target inference task to be executed by the accelerator based on the acquired user request, and determines the target amount of memory required by the accelerator to execute the target inference task. If the difference between the target amount of memory and the amount of memory already allocated by the accelerator to the inference task exceeds a first range, the execution device then triggers a determination of the memory adjustment amount.

[0025] In other words, the execution device determines whether the amount of memory needs to be adjusted by comparing the amount of memory actually required for the inference task with the amount of memory already allocated.

[0026] In this solution, by comparing the amount of memory already allocated to the accelerator with the amount of memory required by the acquired inference tasks, it is determined whether it is necessary to trigger an adjustment to the accelerator's memory allocation. This enables the adjustment of the accelerator's memory allocation based on its inference load, ensuring that the accelerator can allocate a reasonable amount of memory for subsequent inference tasks.

[0027] In one possible implementation, the execution device determines the amount of memory used by the accelerator when performing inference tasks. When the difference between the amount of memory allocated to the inference task and the amount of memory used exceeds a second range, the execution device triggers a determination of a memory adjustment amount.

[0028] In this solution, by comparing the amount of memory allocated to the accelerator with the amount of memory actually used by the accelerator, it is determined whether it is necessary to trigger an adjustment to the accelerator's memory allocation. This enables the adjustment of the accelerator's memory allocation based on the accelerator's real-time inference load, ensuring that the accelerator can allocate a reasonable amount of memory for subsequent inference tasks.

[0029] In one possible implementation, a third model is also deployed on the accelerator. If the accelerator does not use the third model to perform inference tasks for a preset period of time, the execution device releases the memory occupied by the third model on the accelerator.

[0030] Secondly, a memory-sharing device for accelerators is provided, which is connected to an accelerator on which a first model and a second model are deployed. Furthermore, the memory-sharing device can be located within the accelerator or independent of it. The accelerator is used to perform inference tasks based on the first model and training tasks based on the second model. The accelerator memory sharing device includes: a processing module for determining a memory adjustment amount when the inference load of the accelerator changes, the memory adjustment amount indicating the adjustment amount of memory on the accelerator used for performing inference tasks; the processing module is further configured to determine a batch size adjustment amount, a first memory amount, and a second memory amount based on the memory adjustment amount, the batch size adjustment amount indicating the adjustment amount of the batch size of training data used by the accelerator when performing training tasks, the first memory amount indicating the size of memory reallocated on the accelerator to inference tasks, and the second memory amount indicating the size of memory reallocated on the accelerator to training tasks; the processing module is further configured to adjust the batch size of training data used by the accelerator when performing training tasks based on the batch size adjustment amount; and a memory adjustment module for adjusting the amount of memory used by the accelerator when performing inference tasks to the first memory amount and adjusting the amount of memory used by the accelerator when performing training tasks to the second memory amount.

[0031] In one possible implementation, before adjusting the batch size of the training data used by the accelerator when performing a training task, the processing module is further configured to: if the training task currently being performed by the accelerator is a gradient calculation task, control the accelerator to stop performing the gradient calculation task so that the accelerator performs a new training task based on the adjusted batch size.

[0032] In one possible implementation, the gradient computation task includes multiple subtasks, which are maintained by a task management module in the processor. The task management module is used to submit some of the subtasks to the accelerator sequentially based on the progress of the subtasks being processed by the accelerator. The processing module is specifically used to clear all the subtasks maintained by the task management module to stop submitting subtasks of the gradient computation task to the accelerator.

[0033] In one possible implementation, before adjusting the batch size of the training data used by the accelerator when performing the training task, the processing module is further configured to: wait for the accelerator to finish performing the model parameter update task if the training task currently being performed by the accelerator is a model parameter update task.

[0034] In one possible implementation, when the inference load of the accelerator increases, a memory adjustment amount is used to indicate the amount of increase in memory on the accelerator used for performing inference tasks, and a batch size adjustment amount is used to indicate the amount of decrease in the batch size of training data used by the accelerator when performing training tasks; the memory adjustment module is specifically used to: increase the amount of memory used by the accelerator when performing inference tasks to a first memory amount, and decrease the amount of memory used by the accelerator when training a second model to a second memory amount.

[0035] In one possible implementation, when the inference load of the accelerator decreases, a memory adjustment amount is used to indicate the amount by which the memory on the accelerator used for performing inference tasks is reduced, and a batch size adjustment amount is used to indicate the amount by which the batch size of the training data used by the accelerator when performing training tasks is increased; the memory adjustment module is specifically used to: reduce the amount of memory used by the accelerator when performing inference tasks to a first memory amount, and increase the amount of memory used by the accelerator when training a second model to a second memory amount.

[0036] In one possible implementation, the processing module is further configured to: determine the target inference task to be executed by the accelerator based on the acquired user request; determine the target amount of memory required by the accelerator to execute the target inference task; and trigger the determination of a memory adjustment amount when the difference between the target amount of memory and the amount of memory already allocated by the accelerator to the inference task exceeds a first range.

[0037] In one possible implementation, the processing module is further configured to: determine the amount of memory used when the accelerator performs an inference task; and trigger the determination of a memory adjustment amount when the difference between the amount of memory allocated to the inference task and the amount of memory used exceeds a second range.

[0038] In one possible implementation, a third model is also deployed on the accelerator; the processing module is further configured to: release the memory occupied by the third model if the accelerator has not used the third model to perform inference tasks for a preset period of time.

[0039] Thirdly, a shared accelerator memory device is provided, comprising: a processor, an accelerator, and a memory; the memory is used to store computer instructions, which, when executed by the processor, cause the shared accelerator memory device to perform the methods described above.

[0040] Fourthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, cause the computer to perform the methods of any of the above aspects.

[0041] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, enable the computer to perform the methods described above.

[0042] In a sixth aspect, a chip system is provided, the chip system including a processor and a communication interface for communicating with a module other than the chip shown, the processor for running computer programs or instructions such that an apparatus on which the chip system is mounted can perform the methods of any of the above aspects.

[0043] In a seventh aspect, a computing device is provided, the computing device including a shared accelerator memory of the third aspect or a chip system of the sixth aspect, wherein the shared accelerator memory or the chip system in the computing device is used to implement the operational steps of the method of any of the above aspects.

[0044] Eighthly, a computing device cluster is provided, comprising at least one computing device, wherein any one computing device is used to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects. Alternatively, some or all of the computing devices are used together to run a computer program or instructions, such that the computing device cluster can perform the methods of any of the above aspects.

[0045] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0046] Figure 1 is a schematic diagram of a system architecture 100 provided in this application;

[0047] Figure 2 is a structural schematic diagram of an execution device provided in this application;

[0048] Figure 3 is a flowchart illustrating a method for sharing accelerator memory provided in this application;

[0049] Figure 4 is a schematic diagram of a system architecture provided in this application;

[0050] Figure 5 is a schematic diagram of the deployment of a system architecture provided in this application;

[0051] Figure 6 is a schematic diagram of a process provided in this application for reallocating a portion of memory that has been allocated to the training task to the inference task;

[0052] Figure 7 is a schematic diagram of a process provided in this application for reallocating a portion of memory that has been allocated to the inference task to the training task;

[0053] Figure 8 is a schematic diagram of a task management module for issuing tasks to the GPU according to this application;

[0054] Figure 9 is a schematic diagram of how the GPU stops executing training tasks by discarding the kernel, as provided in this application.

[0055] Figure 10 is a schematic diagram of the structure of a shared device for accelerator memory provided in this application;

[0056] Figure 11 is a schematic diagram of the structure of a computing device provided in this application;

[0057] Figure 12 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0058] Figure 13 is a schematic diagram of another computing device cluster provided in this application;

[0059] Figure 14 is a schematic diagram of the structure of a computer-readable storage medium provided in this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some, and not all, of the embodiments of this application. Those skilled in the art will understand that, with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0061] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such descriptions can be used interchangeably where appropriate to allow embodiments to be implemented in a sequence other than that illustrated or described in this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules is not necessarily limited to those explicitly listed, but may include other steps or modules not explicitly listed or inherent to such processes, methods, products, or devices. The naming or numbering of steps appearing in this application does not imply that the steps in the method flow must be performed in the chronological / logical order indicated by the naming or numbering. The execution order of named or numbered process steps can be changed according to the desired technical purpose, as long as the same or similar technical effect is achieved. The division of units in this application is a logical division. In practical applications, there may be other division methods. For example, multiple units may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the shown or discussed mutual coupling, direct coupling, or communication connection may be through some interface, and the indirect coupling or communication connection between units may be electrical or other similar forms, none of which are limited in this application. Furthermore, the units or sub-units described as separate components may or may not be physically separated, may or may not be physical units, or may be distributed among multiple circuit units. Some or all of the units can be selected to achieve the purpose of the solution in this application according to actual needs.

[0062] For ease of understanding, some technical terms involved in the embodiments of this application will be introduced below.

[0063] (1) Neural Network

[0064] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0065] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0066] (2) Loss Function

[0067] During neural network training, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the neural network becomes a process of minimizing this loss.

[0068] (3) Backpropagation algorithm

[0069] Neural networks can employ backpropagation (BP) to correct the parameters of the initial prediction model during training, thereby reducing the error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial prediction model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the prediction model, such as the weight matrix.

[0070] Specifically, during model training, the backpropagation algorithm is typically used to calculate the gradients of each node in the model. This allows for the adjustment of node weights based on the gradients of each node, thereby minimizing the model's loss function value. The gradient represents the rate of change of a function at a given point. Furthermore, the gradient of each node in the model can be determined by calculating its partial derivatives.

[0071] (4) Batch size

[0072] Batch size, also known as batch size, batch processing size, or batch size, refers to the number of training samples input into a neural network model at one time during training. Generally, the larger the batch size, the larger the amount of data that needs to be processed simultaneously during training, and the larger the amount of memory required for training; conversely, the smaller the batch size, the smaller the amount of data that needs to be processed simultaneously during training, and the smaller the amount of memory required for training.

[0073] Currently, to meet the different needs of inference and training tasks while avoiding performance interference between tasks, existing machine learning platforms typically deploy separate server clusters for inference and training tasks—namely, inference server clusters and training server clusters. Since the inference requests required are unpredictable, inference server clusters are usually over-provisioned based on peak inference request volumes to avoid service quality degradation due to surges in inference requests. Therefore, when the number of inference requests is low, the utilization rate of the inference server cluster is low, increasing the operational costs of machine learning services.

[0074] In view of this, this application provides a method for sharing accelerator memory, using the same accelerator to simultaneously execute inference and training tasks, with a portion of the accelerator's memory allocated to inference tasks and another portion allocated to training tasks. When the accelerator's inference load changes, the amount of memory allocated to inference and training tasks on the accelerator changes accordingly, allowing the amount of memory allocated to inference tasks to adaptively meet the inference load requirements, ensuring timely completion of inference tasks while effectively utilizing the accelerator's memory resources. Furthermore, before adjusting the amount of memory allocated to training tasks, the batch size of the training data used by the accelerator when executing training tasks is adjusted, ensuring that the actual amount of memory used by the accelerator during training tasks matches the memory adjustment requirements, guaranteeing successful memory adjustment. In this way, adaptively adjusting the memory allocated to inference and training tasks based on the inference load allows both tasks to run normally on the same accelerator, ensuring fast response to inference requests and improving accelerator utilization.

[0075] Please refer to Figure 1, which is a schematic diagram of a system architecture 100 provided in this application. As shown in Figure 1, in this system architecture 100, the execution device 110 can be implemented by one or more servers, and the execution device 110 is equipped with accelerators for performing inference and training tasks. Exemplarily, the accelerator can specifically be hardware for training or inference of artificial intelligence models, such as a graphics processing unit (GPU), a tensor processing unit (TPU), or a neural network processing unit (NPU).

[0076] Optionally, the execution device 110 can work in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 110 can be deployed on a single physical site or distributed across multiple physical sites. The execution device 110 can use data from the data storage system 120 or call program code from the data storage system 120 to implement the memory sharing method of the accelerator provided in this application, thereby enabling the execution of inference and training tasks on the same accelerator.

[0077] Users can interact with execution device 110 by operating their respective user devices (e.g., local device 101 and local device 102). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0078] Each user's local device can interact with the execution device 110 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0079] In one implementation, local devices 101 and 102 may send user requests to execution device 110 to instruct execution device 110 on the inference tasks to be performed. Thus, based on the received user requests, execution device 110 can determine the inference tasks to be performed and execute the memory-sharing method of the accelerator provided in this application when the inference load changes.

[0080] It should be noted that all the functions of the execution device 110 can also be implemented by a local device. For example, local device 101 implements the functions of the execution device 110 and provides services to its own users, or provides services to the users of local device 102.

[0081] Please refer to Figures 2 and 3. Figure 2 is a schematic diagram of the structure of an execution device provided in this application; Figure 3 is a flowchart of a method for sharing accelerator memory provided in this application.

[0082] As shown in Figure 2, the accelerator memory sharing method provided in this application is applied to an accelerator deployed in an execution device, which has a first model and a second model. The accelerator is used to perform inference tasks based on the first model and training tasks based on the second model. That is, the accelerator performs inference and training tasks simultaneously, so a portion of the accelerator's memory is allocated to the inference task, and another portion is allocated to the training task. Furthermore, the execution device also deploys a processor connected to the accelerator, such as a Central Processing Unit (CPU). This processor can serve as the execution entity for the accelerator memory sharing method provided in this application, thereby adjusting the amount of memory allocated to the tasks executed by the accelerator. Specifically, the processor adjusts the amount of memory allocated to each task in the accelerator by executing the method provided in this application, thereby enabling memory sharing between different tasks in the accelerator. Of course, the execution entity for the accelerator memory sharing method provided in this application can also be other hardware, such as the accelerator itself; this application does not specifically limit this.

[0083] As shown in Figure 3, the execution flow of the accelerator memory sharing method provided in this application includes the following steps 301-304.

[0084] Step 301: When the inference load of the accelerator changes, determine the memory adjustment amount, which is used to indicate the amount of memory adjustment on the accelerator used to perform inference tasks.

[0085] In this application, when the inference load of the accelerator changes, the amount of memory required by the accelerator to execute inference tasks also changes. Therefore, the execution device can determine the amount of memory adjustment based on the changes in inference load to determine how to adjust the memory allocated on the accelerator to the inference tasks. The inference load of the accelerator can, for example, refer to the amount of data that the accelerator needs to process when executing inference tasks. Generally, the inference load of the accelerator tends to change when the number of inference requests that the accelerator needs to process changes; or, the inference load of the accelerator also changes when the amount of data to be processed indicated by the inference requests that the accelerator needs to process changes.

[0086] Specifically, there are two scenarios where the inference load of the accelerator changes.

[0087] Scenario 1: The inference load on the accelerator increases.

[0088] When the inference load on the accelerator increases, the amount of memory allocated to inference tasks on the accelerator also needs to increase. Therefore, the memory adjustment amount mentioned above is used to indicate the amount of memory increase on the accelerator used for executing inference tasks.

[0089] Scenario 2: The inference load on the accelerator decreases.

[0090] When the inference load on the accelerator decreases, the amount of memory allocated to inference tasks on the accelerator also needs to increase. Therefore, the memory adjustment amount mentioned above is used to indicate the amount of reduction in the memory on the accelerator used for executing inference tasks.

[0091] Step 302: Determine the batch size adjustment amount, the first memory amount, and the second memory amount based on the memory adjustment amount. The batch size adjustment amount is used to indicate the adjustment amount of the batch size of the training data used by the accelerator when performing the training task. The first memory amount is used to indicate the size of the memory reallocated to the inference task on the accelerator. The second memory amount is used to indicate the size of the memory reallocated to the training task on the accelerator.

[0092] Since a portion of the accelerator's memory has already been allocated to inference tasks and another portion to training tasks, the first and second memory amounts can be calculated based on the amount of memory already allocated to inference and training tasks, as well as the aforementioned memory adjustment amounts. The first memory amount indicates the amount of memory the accelerator needs to allocate to inference tasks, while the second memory amount indicates the amount of memory the accelerator needs to allocate to training tasks. Generally, the total memory amount of the accelerator is the sum of the first and second memory amounts.

[0093] Furthermore, when the memory allocated to the training task by the accelerator needs to change, in order to ensure that the training task can be executed normally, this application adjusts the batch size of the training data used by the accelerator when executing the training task. This changes the number of training samples processed simultaneously by the accelerator during the execution of the training task, and consequently, the amount of memory required by the accelerator during the execution of the training task. Specifically, the batch size adjustment can be determined based on the batch size of the training data currently used by the accelerator when executing the training task and the aforementioned memory adjustment amount.

[0094] It should be noted that while the memory adjustment amount indicates the increase in memory allocated to inference tasks on the accelerator, the batch size adjustment amount indicates the decrease in the batch size of training data used by the accelerator when performing training tasks. That is, if the amount of memory allocated to inference tasks on the accelerator needs to be increased, then the amount of memory allocated to training tasks on the accelerator needs to be decreased accordingly, and therefore the batch size of training data used by the accelerator when performing training tasks also needs to be decreased.

[0095] While the memory adjustment amount is used to indicate the reduction in memory allocated to inference tasks on the accelerator, the batch size adjustment amount is used to indicate the increase in the batch size of training data used by the accelerator when performing training tasks. That is, if the amount of memory allocated to inference tasks on the accelerator needs to be reduced, then the amount of memory allocated to training tasks can be increased accordingly, and therefore the batch size of training data used by the accelerator when performing training tasks can also be increased.

[0096] It should be noted that since the amount of memory required by the accelerator to process a single training sample during training is usually fixed, the batch size adjustment mentioned above can be well determined based on the memory adjustment amount. That is, the batch size adjustment amount is actually related to the memory adjustment amount and the amount of memory required by the accelerator to process a single training sample, and this application does not make specific limitations on how to determine the batch size adjustment amount.

[0097] Step 303: Adjust the batch size of the training data used by the accelerator when performing the training task based on the batch size adjustment amount.

[0098] Once the batch size adjustment amount is determined, the execution device can trigger an adjustment to the batch size of the training data used by the accelerator when performing training tasks. Furthermore, during one round of iterative training using the training data, the batch size of the training data in the accelerator is often fixed and cannot be changed. Therefore, the execution device typically needs to wait until the accelerator has completed one round of iterative training before adjusting the batch size of the training data used by the accelerator when performing training tasks.

[0099] That is, during the accelerator's iterative training process, the execution device often cannot change the batch size of the training data. The batch size of the training data can only be adjusted when the accelerator performs a new round of iterative training.

[0100] Step 304: Adjust the amount of memory used by the accelerator when performing inference tasks to the first amount of memory, and adjust the amount of memory used by the accelerator when performing training tasks to the second amount of memory.

[0101] Specifically, after adjusting the batch size of the training data used by the accelerator, the amount of memory required by the accelerator to perform training tasks will change. Therefore, the execution device can adjust the amount of memory used by the accelerator to perform inference tasks and training tasks.

[0102] For example, the processor in the execution device can send instructions to the driver in the accelerator responsible for memory management, instructing it to adjust the amount of memory used by the accelerator when performing inference tasks to a first amount of memory, and to adjust the amount of memory used by the accelerator when performing training tasks to a second amount of memory. In this way, when the accelerator performs inference and training tasks, the driver in the accelerator will allocate the first amount of memory for the inference task and the second amount of memory for the training task.

[0103] When the inference load of the accelerator increases, the execution device increases the amount of memory used by the accelerator to perform inference tasks to a first amount of memory, and decreases the amount of memory used by the accelerator to train the second model to a second amount of memory.

[0104] When the inference load of the accelerator is reduced, the execution device reduces the amount of memory used by the accelerator to perform inference tasks to a first amount of memory, and increases the amount of memory used by the accelerator to train the second model to a second amount of memory.

[0105] In this scheme, a single accelerator is used to simultaneously execute inference and training tasks, with a portion of the accelerator's memory allocated to inference tasks and the remainder to training tasks. As the accelerator's inference load changes, the amount of memory allocated to inference and training tasks changes accordingly. This ensures that the memory allocated to inference tasks adaptively meets the inference load requirements, guaranteeing timely completion of inference tasks while effectively utilizing the accelerator's memory resources. Furthermore, before adjusting the memory allocated to training tasks, the batch size of the training data used by the accelerator during training is adjusted to ensure that the actual memory used by the accelerator during training aligns with the memory adjustment requirements, guaranteeing successful memory adjustment.

[0106] Optionally, in some scenarios, a third model is also deployed on the accelerator, which is used to perform inference tasks. When adjusting the amount of memory used by the accelerator for inference tasks, the execution device can release the memory occupied by the third model if the accelerator has not used the third model to perform inference tasks for a preset period of time. The preset period of time can be a value set or adjusted according to the actual application scenario, such as 5 minutes, 30 minutes, or 1 hour.

[0107] In other words, multiple different models may be deployed on an accelerator to perform different inference tasks. If an accelerator has not used a deployed model to perform an inference task for a long time, it means that the model has a low utilization rate. In this case, the memory occupied by the model can be released on the accelerator, thereby freeing up more free memory on the accelerator so that the memory occupied by the inference task can be reallocated to the training task, thus improving the execution efficiency of the training task.

[0108] Understandably, accelerators cannot adjust the batch size of training data during a single training iteration; they can only adjust it when a new iteration begins. Furthermore, the duration of a single training iteration is often significantly longer than the inference iteration. Therefore, if the execution device waits for the accelerator to complete the current iteration before adjusting the batch size, the inference task may experience prolonged memory reallocation waiting times, potentially impacting its responsiveness. To address this, this application proposes controlling whether to halt the training task and initiate a new one based on its progress. This allows for rapid adjustment of the memory used by the inference and training tasks, ensuring a fast response time for the inference task.

[0109] Optionally, before step 303 above, that is, before adjusting the batch size of the training data used by the accelerator when performing the training task, if the training task currently being performed by the accelerator is a gradient calculation task, the device controls the accelerator to stop performing the gradient calculation task so that the accelerator can perform a new training task based on the adjusted batch size.

[0110] In other words, when it is determined that the batch size of the training data needs to be adjusted, if the training task being performed by the accelerator is a gradient calculation task, the execution device will control the accelerator to stop performing the gradient calculation task. That is, all intermediate results obtained by the accelerator in the current round of iterative training will be discarded so that a new round of iterative training can be started.

[0111] Generally, the training tasks performed by an accelerator can be divided into gradient calculation tasks and model parameter update tasks. Gradient calculation includes the process of the accelerator performing forward inference based on the training data to obtain the model output, and the process of the accelerator calculating the gradients of each parameter in the model based on the model output. Model parameter update includes the process of the accelerator updating the parameters in the model based on the gradients of each parameter. In the model parameter update task, since the accelerator has already started updating the model parameters (i.e., some model parameters have been updated), stopping the execution of the model parameter update task at this point may result in some parameters being updated while others are not, leading to inconsistent model parameters. However, in the gradient calculation task, since the accelerator has not yet been involved in updating the model parameters, stopping the gradient calculation task will not affect the model parameters, and it allows for rapid adjustment of the memory used by the inference and training tasks.

[0112] Optionally, when it is determined that the batch size of the training data needs to be adjusted, if the training task being performed by the accelerator is a model parameter update task, the execution device will wait for the accelerator to complete the model parameter update task before triggering the adjustment of the batch size of the training data used by the accelerator during the training task, in order to avoid affecting the normal update of model parameters. Since the model parameter update task only updates the values ​​of each parameter based on the gradient of each parameter in the model, the execution time of the model parameter update task is very short (for example, the execution time of the gradient calculation task may be between one hundred and several hundred milliseconds, while the execution time of the model parameter update task is about 10 milliseconds). Therefore, the execution device does not need to wait too long to adjust the amount of memory used by the inference task and the training task after the accelerator completes the model parameter update task, ensuring that the response of the inference task is not significantly affected.

[0113] Optionally, in this application, to avoid the need for hardware management of the accelerator when stopping training tasks, the execution device can run a task management module on the processor. This task management module is used to maintain the training tasks to be issued to the accelerator. Furthermore, this task management module does not issue all training tasks to the accelerator at once, but divides the training tasks into multiple sub-tasks and issues the sub-tasks according to the accelerator's processing progress, ensuring that the accelerator can quickly stop executing training tasks.

[0114] For example, the gradient computation task to be performed by the accelerator includes multiple subtasks, which are maintained by a task management module in the processor. The task management module is used to sequentially submit some of the subtasks to the accelerator based on the progress of the subtasks being processed.

[0115] During the process of controlling the accelerator to stop executing gradient calculation tasks, the execution device can clear all subtasks maintained by the task management module to stop submitting subtasks in the gradient calculation task to the accelerator.

[0116] In other words, for gradient calculation tasks, this application divides the gradient calculation task into multiple subtasks. These subtasks are not submitted to the accelerator all at once, but are maintained by the task management module on the processor. During the accelerator's operation, the task management module obtains the progress of the accelerator processing the subtasks in real time and periodically sends a certain number of subtasks to the accelerator to ensure that all subtasks acquired by the accelerator can be executed quickly.

[0117] For example, at the beginning of the training task, the task management module can send 64 sub-tasks to the accelerator at once, and the task management module and the accelerator will synchronize the processing status of the sub-tasks. When the task management module finds that there are fewer than 64 sub-tasks to be processed on the accelerator, it can send new sub-tasks to the accelerator, thereby ensuring that the number of sub-tasks to be processed on the accelerator is always kept at around 64.

[0118] Generally, gradient calculation tasks can be divided into thousands or tens of thousands of subtasks, each with an execution time of a few microseconds. Therefore, when an accelerator only acquires a small portion of the subtasks, it can quickly complete the acquired subtasks, allowing it to quickly stop executing the current iteration of the training process.

[0119] In this solution, a task management module is installed on the processor. This module maintains multiple subtasks derived from gradient calculation and periodically distributes some subtasks to the accelerator based on its processing progress. This ensures that the accelerator always receives only a portion of the subtasks, allowing it to quickly complete them and stop training. Furthermore, since the task management module is located on the processor, there is no need for hardware management of the accelerator, avoiding uncontrollable hardware management issues. Running a software module on the processor can trigger the accelerator to quickly stop training, reducing the implementation complexity of the solution.

[0120] In step 301 above, there are several ways to trigger the execution device to determine the memory adjustment amount.

[0121] In implementation method 1, the execution device determines whether memory adjustment is needed by determining the amount of memory required for the accelerator to perform inference tasks and the amount of memory already allocated to inference tasks.

[0122] In other words, the execution device determines whether to adjust the memory allocation by comparing the actual amount of memory required for the inference task with the amount of memory already allocated. For example, if the accelerator requires more memory for the inference task than the allocated memory is, the execution device will determine that the allocated memory needs to be increased. Conversely, if the accelerator requires less memory for the inference task than the allocated memory is larger, the execution device will determine that the allocated memory needs to be decreased.

[0123] For example, the execution device first determines the target inference task to be executed on the accelerator based on the acquired user request. The user request is a request message sent by the user to the execution device through the user's terminal device (such as a smartphone or laptop), indicating the inference task that the execution device needs to complete. Therefore, based on one or more acquired user requests, the execution device can determine the target inference task to be executed on the accelerator. The target inference task may include one or more inference tasks.

[0124] Then, the execution device determines the target amount of memory required by the accelerator to perform the target inference task. Specifically, given that the target inference task to be performed by the accelerator has been determined, the execution device can determine the amount of memory required to perform each inference task included in the target inference task, thereby determining the target amount of memory required by the accelerator to perform the target inference task.

[0125] Secondly, when the difference between the target memory amount and the memory already allocated to the inference task by the accelerator exceeds a first range, the execution device triggers a determination of the memory adjustment amount. The first range can be a range defined by a negative value and a positive value as boundary values, and the specific size of the boundary values ​​of the first range can be determined based on the total memory amount in the accelerator. This application does not specifically limit the specific value of the first range. Therefore, when the difference between the target memory amount and the memory already allocated to the inference task by the accelerator exceeds the first range, it can be considered that the difference between the target memory amount and the memory already allocated to the inference task is large (e.g., the target memory amount is much smaller than the memory already allocated to the inference task or the target memory amount is much larger than the memory already allocated to the inference task), thereby triggering an adjustment of the allocated memory amount in the accelerator.

[0126] In this solution, by comparing the amount of memory already allocated to the accelerator with the amount of memory required by the acquired inference tasks, it is determined whether it is necessary to trigger an adjustment to the accelerator's memory allocation. This enables the adjustment of the accelerator's memory allocation based on its inference load, ensuring that the accelerator can allocate a reasonable amount of memory for subsequent inference tasks.

[0127] In implementation method 2, the execution device determines whether memory adjustment is needed by comparing the actual memory usage when the accelerator executes the inference task with the amount of memory already allocated to the inference task.

[0128] For example, the execution device determines the memory usage of the accelerator when it is currently performing an inference task; and when the difference between the amount of memory allocated to the inference task and the memory usage exceeds a second range, the execution device triggers a determination of a memory adjustment amount. The second range can be a range consisting of two positive values ​​as boundary values, and the size of the boundary values ​​of the second range can be determined based on the total amount of memory in the accelerator. This application does not specifically limit the specific value of the second range.

[0129] For example, assuming the second range is specifically represented as [boundary value 1, boundary value 2], if the difference between the amount of memory allocated to inference tasks and the amount of memory used by the accelerator when performing inference tasks is less than boundary value 1, it means that a large amount of memory has been used on the accelerator, and there is a high possibility of insufficient memory in the future. Therefore, it can trigger an increase in the amount of memory used by the accelerator when performing inference tasks. If the difference between the amount of memory allocated to inference tasks and the amount of memory used by the accelerator when performing inference tasks is greater than boundary value 2, it means that a small amount of memory has been used on the accelerator, resulting in significant memory waste. Therefore, it can trigger a decrease in the amount of memory used by the accelerator when performing inference tasks.

[0130] In this solution, by comparing the amount of memory allocated to the accelerator with the amount of memory actually used by the accelerator, it is determined whether it is necessary to trigger an adjustment to the accelerator's memory allocation. This enables the adjustment of the accelerator's memory allocation based on the accelerator's real-time inference load, ensuring that the accelerator can allocate a reasonable amount of memory for subsequent inference tasks.

[0131] The execution flow of the accelerator memory sharing method provided in this application has been described above. For ease of understanding, the execution process of the accelerator memory sharing method provided in this application will be detailed below with specific examples.

[0132] For example, please refer to Figure 4, which is a schematic diagram of a system architecture provided by this application. As shown in Figure 4, the system architecture provided by this application can be deployed on the processor (e.g., CPU) of an execution device, and the system architecture mainly includes an inference engine module and a training plugin module. The inference engine module is responsible for performing inference tasks based on the model and managing the model's lifecycle, thereby providing inference services to users. Furthermore, the inference engine module can also notify the training plugin module to reallocate memory when the inference load changes.

[0133] The training plugin module is a plugin for the training framework. It is responsible for responding to memory adjustment requests sent by the inference engine module. It changes the memory used during training by reconfiguring the batch size of the training data, thereby adapting to the memory requirements of the inference task.

[0134] For example, please refer to Figure 5, which is a deployment diagram of a system architecture provided in this application. As shown in Figure 5, on an execution device (such as a server) including a CPU and a GPU, a front-end scheduling module, an inference engine module, and a training framework are deployed on the CPU, while the inference process and the training process run on the GPU.

[0135] The front-end scheduling module receives user requests and, based on these requests, distributes inference tasks to the inference engine module. Furthermore, when the inference engine module returns the inference results, the front-end scheduling module also returns those results to the user.

[0136] The inference engine module is used to distribute inference tasks to the inference process on the GPU, enabling the inference process to complete the inference task by running models. The inference model pool within the inference engine module records the models to be loaded onto the GPU. Therefore, the inference engine module is often also used to manage the lifecycle of models deployed on the GPU, thereby clearing out models that have not been run for a long time.

[0137] The training framework executes user-submitted training scripts to generate and distribute corresponding training tasks to the GPU. The training plugin module within the framework receives memory adjustment requests from the inference engine module to adjust the amount of memory used by the training tasks.

[0138] In the above system architecture, the inference engine module and training framework run on the CPU, outside of the accelerator (GPU). In some embodiments, the inference engine module and training framework may also run on the accelerator, for example, the inference engine module, training framework, inference process, and training process all run on the same NPU.

[0139] In the example provided in this application, the inference process and the training process run on the same GPU, and the inference process is used to perform inference tasks, while the training process is used to perform training tasks.

[0140] When the inference load remains unchanged, both the inference and training processes operate according to their normal workflows. When the inference load changes and adjustments to the memory allocation for the inference and training tasks are needed, a memory reallocation process is initiated. This reallocation process includes two scenarios: one where a portion of the memory previously allocated to the training task is reallocated to the inference task; and another where a portion of the memory previously allocated to the inference task is reallocated to the training task.

[0141] For example, please refer to Figure 6, which is a schematic diagram of a process for reallocating a portion of memory already allocated to the training task to the inference task, as provided in this application. As shown in Figure 6, the process for reallocating a portion of memory already allocated to the training task to the inference task specifically includes the following five core steps.

[0142] Step 1: Inference load increases.

[0143] There are several possible reasons for increased inference load. One possibility is that users submit more inference requests, leading to a greater inference load on the model. Another possibility is that user-submitted inference requests become more complex, such as in a question-answering scenario based on a large language model, where users submit longer question texts, thus increasing the model's inference load (i.e., the model needs to process longer input data).

[0144] Step 2: The inference engine module determines whether memory adjustment is needed, and sends a memory adjustment request to the training plugin module when memory adjustment is needed.

[0145] Specifically, the method by which the inference engine module determines whether memory adjustment is needed can be referred to Implementation Method 1 and Implementation Method 2 described in the above embodiments, and will not be repeated here. If it is determined that memory adjustment is needed, the inference engine module sends a memory adjustment request to the training plugin module. This memory adjustment request is used to request the training plugin module to reduce the amount of memory used when performing training tasks, and the memory adjustment request can also indicate the amount of memory to be reduced when performing training tasks.

[0146] Step 3: The training plugin module determines the batch size that needs to be reduced based on the amount of memory required for adjustment.

[0147] Based on the obtained memory adjustment request, the training plugin module can determine the batch size that needs to be reduced (i.e., the batch size adjustment amount) according to the amount of memory to be adjusted indicated in the memory adjustment request. That is, the training plugin module determines how much the batch size of the training data used when performing the training task needs to be reduced to meet the memory requirement indicated in the memory adjustment request.

[0148] In this way, the training plugin module can determine the batch size corresponding to the training data in the new round of iterative training based on the batch size that needs to be reduced, and set the training data to be selected for training according to the new batch size at the beginning of the new round of iterative training.

[0149] Step 4: The inference engine module allocates the freed-up memory for the inference task and loads the inference task to be executed onto the GPU.

[0150] After the training plugin module uses a new batch size to select training data, a portion of the memory originally allocated to the training task on the GPU can be released. Therefore, the inference engine module can allocate the released memory for the inference task and load the inference task to be executed onto the GPU, which then executes the inference task based on the newly allocated memory.

[0151] Step 5: The GPU performs the inference task and outputs the inference results.

[0152] Specifically, the GPU performs the inference task based on all the memory reallocated for the inference task and the model required to execute the inference task, thereby obtaining and outputting the inference results to the CPU. Finally, the CPU returns the obtained inference results to the user.

[0153] Please refer to Figure 7, which is a schematic diagram of a process provided in this application for reallocating a portion of memory already allocated to the inference task to the training task. As shown in Figure 7, the process of reallocating a portion of memory already allocated to the training task to the inference task specifically includes the following 5 core steps.

[0154] Step 1: Reduce inference load.

[0155] There are several possible reasons for a decrease in inference load. One possibility is that fewer inference requests are submitted by users, thus reducing the model's inference load. Another possibility is that the inference requests submitted by users become simpler, such as in a question-answering scenario based on a large language model, where users submit shorter question texts, leading to a decrease in the model's inference load (i.e., the model processes shorter input data).

[0156] Step 2: The inference engine module determines whether memory adjustment is needed, and if so, sends a memory adjustment request to the training plugin module and releases the portion of memory already allocated to the inference task.

[0157] When it is determined that memory adjustment is necessary, the inference engine module sends a memory adjustment request to the training plugin module. This request requests the training plugin module to increase the amount of memory used when performing training tasks, and can specifically indicate the amount of additional memory required for the training tasks. Furthermore, the inference engine module can also release a portion of the memory allocated to inference tasks so that the released memory can be allocated to training tasks. For example, for a model stored in the GPU used for inference tasks, if the model has not been used for a preset period, the inference engine module can notify the GPU to clear the model, thereby releasing the memory occupied by the model in the GPU.

[0158] Step 3: The training plugin module determines the batch size that needs to be increased based on the amount of memory required for adjustment.

[0159] Based on the obtained memory adjustment request, the training plugin module can determine the required increase in batch size (i.e., the batch size adjustment amount) according to the amount of memory to be adjusted indicated in the memory adjustment request. That is, the training plugin module determines how much the batch size of the training data used when performing the training task needs to be increased to make reasonable use of the memory released for the inference task.

[0160] Step four: The training plugin module reconfigures the batch size of the training task and allocates the freed-up memory to the training task.

[0161] Specifically, the training plugin module reconfigures the batch size of the training task in the new round of training iterations and sets it to select training data for training according to the new batch size at the start of the new round of training iterations. Furthermore, since some memory previously allocated to the inference task has been released, the training plugin module can reallocate the released memory for the training task, thus ensuring that the new round of training iterations can execute normally.

[0162] Step 5: The GPU loads the training data based on the reconfigured batch size and executes the training task.

[0163] During a new round of iterative training, the GPU loads training data and executes training tasks based on a reconfigured batch size.

[0164] Please refer to Figure 8, which is a schematic diagram of a task management module for issuing tasks to the GPU according to this application. As shown in Figure 8, in order to ensure that the accelerator can quickly stop executing a round of training tasks, a task management module runs on the CPU. This task management module is used to maintain the kernels (i.e., subtasks used to compose training tasks) issued by the training plugin module. Here, a kernel can be understood as a subtask of the training task, which is essentially a task that calls a function to perform computation.

[0165] During GPU training, the task management module issues new kernels to the GPU based on the kernel processing progress, thus keeping the number of kernels waiting to be processed on the GPU within a certain range. For example, in Figure 8, the task management module initially sends 64 kernels to the GPU, and the GPU synchronizes the kernel processing status with the task management module in real time. In this way, as soon as the GPU completes processing a kernel, the task management module issues a new kernel, ensuring that the number of kernels waiting to be processed on the GPU remains around 64.

[0166] Please refer to Figure 9, which is a schematic diagram illustrating how the GPU can stop executing a training task by discarding kernels, as provided in this application. As shown in Figure 9, after the training plugin module receives a memory adjustment request, it determines that the batch size of the training data needs to be adjusted. Therefore, the training plugin module can first determine whether the kernel currently being executed by the GPU belongs to a gradient calculation task. If the kernel currently being executed by the GPU belongs to a gradient calculation task, the training plugin module can send a kernel discard instruction to the task management module, thereby instructing the task management module to discard any kernels that were not completed during the current iteration of training. In this way, since the task management module has discarded the kernels that were not completed during the current iteration of training, the task management module will no longer issue kernels for the current iteration of training to the GPU, thus ensuring that the GPU can quickly stop executing the current iteration of training.

[0167] After the training plugin module reconfigures the batch size of the training data, it passes the kernel of the new round of iterative training process to the task management module, thereby triggering the GPU to execute the new round of iterative training process based on the new batch size.

[0168] Furthermore, when performing training tasks, GPUs can accumulate gradients calculated in different iterations based on gradient accumulation techniques. This allows for subsequent updates to model parameters based on the accumulated gradients, ensuring that the convergence of model training remains unchanged.

[0169] The method provided in this application has been described in detail above. Next, the device provided in this application for performing the above method will be described.

[0170] Please refer to Figure 10, which is a schematic diagram of the structure of an accelerator memory sharing device provided in this application. As shown in Figure 10, the accelerator memory sharing device is connected to an accelerator on which a first model and a second model are deployed. Furthermore, the accelerator memory sharing device can be located within the accelerator or can be independent of the accelerator. The accelerator is used to perform inference tasks based on the first model and to perform training tasks based on the second model.

[0171] The accelerator memory sharing device includes: a processing module 1001, configured to determine a memory adjustment amount when the inference load of the accelerator changes, the memory adjustment amount indicating the adjustment amount of memory used on the accelerator for performing inference tasks; the processing module 1001 is further configured to determine a batch size adjustment amount, a first memory amount, and a second memory amount based on the memory adjustment amount, the batch size adjustment amount indicating the adjustment amount of the batch size of training data used by the accelerator when performing training tasks, the first memory amount indicating the size of memory reallocated on the accelerator to inference tasks, and the second memory amount indicating the size of memory reallocated on the accelerator to training tasks; the processing module 1001 is further configured to adjust the batch size of training data used by the accelerator when performing training tasks based on the batch size adjustment amount; and a memory adjustment module 1002, configured to adjust the amount of memory used by the accelerator when performing inference tasks to the first memory amount and adjust the amount of memory used by the accelerator when performing training tasks to the second memory amount.

[0172] In one possible implementation, before adjusting the batch size of the training data used by the accelerator when performing a training task, the processing module 1001 is further configured to: control the accelerator to stop performing the gradient calculation task if the training task currently being performed by the accelerator is a gradient calculation task, so that the accelerator performs a new training task based on the adjusted batch size.

[0173] In one possible implementation, the gradient calculation task includes multiple subtasks, which are maintained by a task management module in the processor. The task management module is used to submit some of the subtasks to the accelerator sequentially based on the progress of the subtasks being processed by the accelerator. The processing module 1001 is specifically used to clear all the subtasks maintained by the task management module to stop submitting the subtasks of the gradient calculation task to the accelerator.

[0174] In one possible implementation, before adjusting the batch size of the training data used by the accelerator when performing the training task, the processing module 1001 is further configured to: wait for the accelerator to finish performing the model parameter update task if the training task currently being performed by the accelerator is a model parameter update task.

[0175] In one possible implementation, when the inference load of the accelerator increases, the memory adjustment amount is used to indicate the amount of increase in memory on the accelerator used for performing inference tasks, and the batch size adjustment amount is used to indicate the amount of decrease in the batch size of training data used by the accelerator when performing training tasks; the memory adjustment module 1002 is specifically used to: increase the amount of memory used by the accelerator when performing inference tasks to a first memory amount, and decrease the amount of memory used by the accelerator when training a second model to a second memory amount.

[0176] In one possible implementation, when the inference load of the accelerator decreases, the memory adjustment amount is used to indicate the amount by which the memory used on the accelerator for performing inference tasks is reduced, and the batch size adjustment amount is used to indicate the amount by which the batch size of the training data used by the accelerator when performing training tasks is increased; the memory adjustment module 1002 is specifically used to: reduce the amount of memory used by the accelerator when performing inference tasks to a first amount of memory, and increase the amount of memory used by the accelerator when training the second model to a second amount of memory.

[0177] In one possible implementation, the processing module 1001 is further configured to: determine the target inference task to be executed by the accelerator based on the acquired user request; determine the target memory amount required by the accelerator to execute the target inference task; and trigger the determination of memory adjustment amount when the difference between the target memory amount and the memory amount already allocated by the accelerator to the inference task exceeds a first range.

[0178] In one possible implementation, the processing module 1001 is further configured to: determine the amount of memory used when the accelerator performs an inference task; and trigger the determination of a memory adjustment amount when the difference between the amount of memory allocated to the inference task and the amount of memory used exceeds a second range.

[0179] In one possible implementation, a third model is also deployed on the accelerator; the processing module 1001 is further configured to: release the memory occupied by the third model if the accelerator has not used the third model to perform inference tasks for a preset period of time.

[0180] Both the processing module 1001 and the memory adjustment module 1002 can be implemented in software or in hardware. For example, the implementation of the processing module 1001 will be described below. Similarly, the implementation of the memory adjustment module 1002 can refer to the implementation of the processing module 1001.

[0181] As an example of a software functional unit, the processing module 1001 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the processing module 1001 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0182] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0183] As an example of a hardware functional unit, the processing module 1001 may include at least one computing device, such as a server. Alternatively, the processing module 1001 may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system-on-chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0184] The processing module 1001 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 1001 can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 1001 can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0185] Please refer to Figure 11, which is a schematic diagram of a computing device provided in this application. The computing device 1100 shown in Figure 11 can be used to execute the accelerator memory sharing method provided in this embodiment. As shown in Figure 11, the computing device 1100 includes: a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. The processor 1104, the memory 1106, and the communication interface 1108 communicate with each other via the bus 1102. The computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1100.

[0186] Bus 1102 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 11, but this does not imply that there is only one bus or one type of bus. Bus 1102 can include pathways for transmitting information between various components of computing device 1100 (e.g., memory 1106, processor 1104, communication interface 1108).

[0187] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0188] The memory 1106 may include volatile memory, such as random access memory (RAM). The processor 1104 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0189] The memory 1106 stores executable program code, and the processor 1104 executes this executable program code to implement the functions of the aforementioned acquisition module and processing module, thereby realizing the aforementioned method for sharing accelerator memory. That is, the memory 1106 stores instructions for executing the method for sharing accelerator memory.

[0190] The communication interface 1108 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 1100 and other devices or communication networks.

[0191] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0192] Please refer to Figure 12, which is a schematic diagram of a computing device cluster provided in this application. As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store the same instructions for executing shared methods of accelerator memory.

[0193] In some possible implementations, the memory 1106 of one or more computing devices 1100 in the computing device cluster may also store partial instructions for executing shared methods of accelerator memory. In other words, a combination of one or more computing devices 1100 can jointly execute instructions for executing shared methods of accelerator memory.

[0194] It should be noted that the memory 1106 in different computing devices 1100 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the shared accelerator memory method apparatus. That is, the instructions stored in the memory 1106 of different computing devices 1100 can implement the functions of one or more of the aforementioned acquisition and processing modules.

[0195] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 13 illustrates one possible implementation. Figure 13 is also a schematic diagram of another computing device cluster structure provided in this application. As shown in Figure 13, in computing device cluster 1300, two computing devices 1100A and 1100B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 1106 in computing device 1100A stores instructions for executing the functions of the acquisition module. Simultaneously, the memory 1106 in computing device 1100B stores instructions for executing the functions of the processing module.

[0196] It should be understood that the functions of computing device 1100A shown in Figure 13 can also be performed by multiple computing devices 1100. Similarly, the functions of computing device 1100B can also be performed by multiple computing devices 1100.

[0197] This application also provides a chip comprising a processing unit and a communication unit. The processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the electronic device to perform the methods described in the above embodiments. Optionally, the storage unit may be an in-chip storage unit, such as a register or cache. Alternatively, the storage unit may be an external storage unit located within a wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0198] Referring to Figure 14, which is a schematic diagram of the structure of a computer-readable storage medium provided in this application. This application also provides a computer-readable storage medium in which, in some embodiments, the method disclosed in Figure 3 above can be implemented as computer program instructions encoded in a machine-readable format on a computer-readable storage medium or on other non-transitory media or articles of art.

[0199] Figure 14 schematically illustrates a conceptual partial view of an example computer-readable storage medium arranged according to at least some of the embodiments shown herein, the example computer-readable storage medium including a computer program for executing computer processes on a computing device.

[0200] In one embodiment, the computer-readable storage medium 1400 is provided using a signal bearer medium 1401. The signal bearer medium 1401 may include one or more program instructions 1402, which, when executed by one or more processors, can provide the functions or parts thereof described above with reference to FIG3.

[0201] In some examples, signal carrying medium 1401 may include computer-readable medium 1403, such as, but not limited to, hard disk drive, compact disc (CD), digital video disc (DVD), digital magnetic tape, memory, ROM or RAM, etc.

[0202] In some embodiments, the signal-bearing medium 1401 may include a computer-recordable medium 1404, such as, but not limited to, a memory, a read / write (R / W) CD, a R / W DVD, etc. In some embodiments, the signal-bearing medium 1401 may include a communication medium 1405, such as, but not limited to, digital and / or analog communication media (e.g., fiber optic cables, waveguides, wired communication links, wireless communication links, etc.). Therefore, for example, the signal-bearing medium 1401 may be transmitted by a wireless communication medium 1405 (e.g., a wireless communication medium conforming to the IEEE 1102.X standard or other transmission protocols).

[0203] One or more program instructions 1402 may be, for example, computer-executable instructions or logical implementation instructions. In some examples, the computing device may be configured to provide various operations, functions, or actions in response to one or more program instructions 1402 conveyed to the computing device via a computer-readable medium 1403, a computer-recordable medium 1404, and / or a communication medium 1405.

[0204] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0205] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods of the various embodiments of this application.

[0206] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0207] A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, computer instructions can be transferred from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method for sharing accelerator memory, characterized in that, The method is applied to an accelerator deployed with a first model and a second model, the accelerator being used to perform inference tasks based on the first model and training tasks based on the second model, the method comprising: When the inference load of the accelerator changes, a memory adjustment amount is determined, which is used to indicate the amount of adjustment of the memory on the accelerator used for performing inference tasks; The batch size adjustment, the first memory amount, and the second memory amount are determined based on the memory adjustment amount. The batch size adjustment amount is used to indicate the adjustment amount of the batch size of the training data used by the accelerator when performing the training task. The first memory amount is used to indicate the size of the memory reallocated to the inference task on the accelerator. The second memory amount is used to indicate the size of the memory reallocated to the training task on the accelerator. The batch size of the training data used by the accelerator when performing the training task is adjusted according to the batch size adjustment amount. The amount of memory used by the accelerator when performing inference tasks is adjusted to the first amount of memory, and the amount of memory used by the accelerator when performing training tasks is adjusted to the second amount of memory.

2. The method according to claim 1, characterized in that, Before adjusting the batch size of the training data used by the accelerator when performing the training task, the method further includes: If the training task currently being performed by the accelerator is a gradient calculation task, the accelerator is controlled to stop performing the gradient calculation task so that the accelerator can perform a new training task based on the adjusted batch size.

3. The method according to claim 2, characterized in that, The gradient calculation task includes multiple subtasks, which are maintained by a task management module in the processor. The task management module is used to submit some of the subtasks to the accelerator in sequence based on the progress of the subtasks being processed by the accelerator. The control of the accelerator to stop executing the gradient calculation task includes: Clear all subtasks maintained by the task management module to stop submitting subtasks in the gradient calculation task to the accelerator.

4. The method according to claim 1, characterized in that, Before adjusting the batch size of the training data used by the accelerator when performing the training task, the method further includes: If the training task currently being performed by the accelerator is a model parameter update task, wait for the accelerator to complete the model parameter update task.

5. The method according to any one of claims 1-4, characterized in that, When the inference load of the accelerator increases, the memory adjustment amount is used to indicate the amount of increase in the memory on the accelerator used for performing inference tasks, and the batch size adjustment amount is used to indicate the amount of decrease in the batch size of the training data used by the accelerator when performing training tasks. Adjusting the amount of memory used by the accelerator when performing inference tasks to the first amount of memory, and adjusting the amount of memory used by the accelerator when training the second model to the second amount of memory, includes: The amount of memory used by the accelerator when performing inference tasks is increased to the first amount of memory, and the amount of memory used by the accelerator when training the second model is decreased to the second amount of memory.

6. The method according to any one of claims 1-4, characterized in that, When the inference load of the accelerator decreases, the memory adjustment amount is used to indicate the amount by which the memory on the accelerator used for performing inference tasks decreases, and the batch size adjustment amount is used to indicate the amount by which the batch size of the training data used by the accelerator when performing training tasks increases. Adjusting the amount of memory used by the accelerator when performing inference tasks to the first amount of memory, and adjusting the amount of memory used by the accelerator when training the second model to the second amount of memory, includes: The amount of memory used by the accelerator when performing inference tasks is reduced to the first amount of memory, and the amount of memory used by the accelerator when training the second model is increased to the second amount of memory.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: The target inference task to be executed by the accelerator is determined based on the obtained user request; Determine the target amount of memory required by the accelerator when performing the target inference task; When the difference between the target amount of memory and the amount of memory that the accelerator has allocated to the inference task exceeds a first range, the determination of the memory adjustment amount is triggered.

8. The method according to any one of claims 1-6, characterized in that, The method further includes: Determine the memory usage of the accelerator when performing inference tasks; When the difference between the amount of memory allocated to the inference task by the accelerator and the amount of memory used exceeds a second range, the determination of the memory adjustment amount is triggered.

9. The method according to any one of claims 1-8, characterized in that, A third model is also deployed on the accelerator; Adjusting the amount of memory used by the accelerator when performing inference tasks to the first amount of memory includes: If the accelerator does not use the third model to perform inference tasks for a preset period of time, the memory occupied by the third model will be released.

10. A shared device for accelerator memory, characterized in that, The device is connected to an accelerator on which a first model and a second model are deployed. The accelerator is used to perform inference tasks based on the first model and training tasks based on the second model. The device includes: A processing module is configured to determine a memory adjustment amount when the inference load of the accelerator changes, the memory adjustment amount being used to indicate the amount of adjustment of the memory on the accelerator used for performing inference tasks; The processing module is further configured to determine a batch size adjustment amount, a first memory amount, and a second memory amount based on the memory adjustment amount. The batch size adjustment amount is used to indicate the adjustment amount of the batch size of the training data used by the accelerator when performing the training task. The first memory amount is used to indicate the size of the memory reallocated to the inference task on the accelerator. The second memory amount is used to indicate the size of the memory reallocated to the training task on the accelerator. The processing module is also used to adjust the batch size of the training data used by the accelerator when performing the training task according to the batch size adjustment amount. The memory adjustment module is used to adjust the amount of memory used by the accelerator when performing inference tasks to the first amount of memory, and to adjust the amount of memory used by the accelerator when performing training tasks to the second amount of memory.

11. The apparatus according to claim 10, characterized in that, Before adjusting the batch size of the training data used by the accelerator when performing the training task, the processing module is further configured to: If the training task currently being performed by the accelerator is a gradient calculation task, the accelerator is controlled to stop performing the gradient calculation task so that the accelerator can perform a new training task based on the adjusted batch size.

12. The apparatus according to claim 11, characterized in that, The gradient calculation task includes multiple subtasks, which are maintained by a task management module in the processor. The task management module is used to submit some of the subtasks to the accelerator in sequence based on the progress of the subtasks being processed by the accelerator. The processing module is specifically used for: Clear all subtasks maintained by the task management module to stop submitting subtasks in the gradient calculation task to the accelerator.

13. The apparatus according to claim 10, characterized in that, Before adjusting the batch size of the training data used by the accelerator when performing the training task, the processing module is further configured to: If the training task currently being performed by the accelerator is a model parameter update task, wait for the accelerator to complete the model parameter update task.

14. The apparatus according to any one of claims 10-13, characterized in that, When the inference load of the accelerator increases, the memory adjustment amount is used to indicate the amount of increase in the memory on the accelerator used for performing inference tasks, and the batch size adjustment amount is used to indicate the amount of decrease in the batch size of the training data used by the accelerator when performing training tasks. The memory adjustment module is specifically used for: The amount of memory used by the accelerator when performing inference tasks is increased to the first amount of memory, and the amount of memory used by the accelerator when training the second model is decreased to the second amount of memory.

15. The apparatus according to any one of claims 10-13, characterized in that, When the inference load of the accelerator decreases, the memory adjustment amount is used to indicate the amount by which the memory on the accelerator used for performing inference tasks decreases, and the batch size adjustment amount is used to indicate the amount by which the batch size of the training data used by the accelerator when performing training tasks increases. The memory adjustment module is specifically used for: The amount of memory used by the accelerator when performing inference tasks is reduced to the first amount of memory, and the amount of memory used by the accelerator when training the second model is increased to the second amount of memory.

16. The apparatus according to any one of claims 10-15, characterized in that, The processing module is further configured to: The target inference task to be executed by the accelerator is determined based on the obtained user request; Determine the target amount of memory required by the accelerator when performing the target inference task; When the difference between the target amount of memory and the amount of memory that the accelerator has allocated to the inference task exceeds a first range, the determination of the memory adjustment amount is triggered.

17. The apparatus according to any one of claims 10-15, characterized in that, The processing module is further configured to: Determine the memory usage of the accelerator when performing inference tasks; When the difference between the amount of memory allocated to the inference task by the accelerator and the amount of memory used exceeds a second range, the determination of the memory adjustment amount is triggered.

18. The apparatus according to any one of claims 10-17, characterized in that, A third model is also deployed on the accelerator; The processing module is further configured to: If the accelerator does not use the third model to perform inference tasks for a preset period of time, the memory occupied by the third model will be released.

19. A computing device, characterized in that, The device includes a memory, a processor, and an accelerator; the memory stores code, the processor is configured to execute the code, and when the code is executed, the computing device performs the method as described in any one of claims 1 to 9.

20. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor, an accelerator and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the operational steps of the method as described in any one of claims 1 to 9.

21. A computer storage medium, characterized in that, The computer storage medium stores instructions that, when executed by the computer, cause the computer to perform the method according to any one of claims 1 to 9.

22. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 9.

Images