Distributed task execution method and system under GPU co-location

Abstract

Provided in the embodiments of the specification are a distributed task execution method and system under GPU co-location, which method and system are applied to a platform including a plurality of GPUs, wherein a first-type task and a second-type task are deployed in the platform, and the priority of the second-type task is lower than that of the first-type task. The task execution method comprises: a target application invoking a collective communication library, so as to request the execution of a target communication operation between GPUs, wherein the target application belongs to a second-type task; the collective communication library then determining a subtask list involved in the target communication operation, wherein the subtask list comprises a plurality of subtasks distributed in different GPUs; and by means of an inter-process synchronization operation, causing the plurality of subtasks to synchronously submit a communication kernel to a GPU driver environment; and when determining that synchronous submission has been enabled for the communication kernel, the GPU driver environment immediately submitting the communication kernel to the GPUs.

Description

Distributed task execution methods and systems in GPU-coordinated environments

[0001] This application claims priority to Chinese Patent Application No. 202411548262.1, filed on October 31, 2024, entitled “Distributed Task Execution Method and System under GPU Hybrid Deployment”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This specification relates to hardware computing in the field of computers, and more particularly to distributed task execution in the context of GPU hybrid deployment. Background Technology

[0003] With the rapid development of artificial intelligence (AI) technology, the demand for computing power is growing exponentially, while GPU supply is relatively scarce. At the same time, GPU usage often involves significant waste; for example, GPU inference service traffic exhibits a clear tidal phenomenon, and when a GPU is exclusively used, its utilization rate is low, failing to fully utilize the scarce GPU resources. Therefore, a GPU hybrid deployment solution has recently been proposed, which combines latency-sensitive online inference services and high-throughput offline tasks on a single GPU, sharing GPU resources. By employing certain computing power and memory scheduling mechanisms, a single GPU can serve two purposes, smoothing out peak and valley loads, using limited GPU resources to support more business needs, and simultaneously improving GPU utilization efficiency.

[0004] The rapid development of large-scale models in recent years has led to an increasing demand for distributed deployment of multiple GPUs. When deploying distributed offline tasks, inconsistent execution progress across different GPUs results in long waiting times for collaborative communication, consuming significant amounts of GPU time slices and impacting online service performance.

[0005] Therefore, an improved solution is needed to better facilitate collaborative communication and improve GPU resource utilization in the context of mixed-distribution tasks on GPUs. Summary of the Invention

[0006] This specification describes one or more embodiments of a distributed task execution method and system under GPU co-location, which can avoid communication waiting and resource waste caused by communication operations in distributed applications under normal circumstances, and improve service performance and GPU resource utilization.

[0007] According to a first aspect, a distributed task execution method under GPU co-deployment is provided, applied to a platform containing multiple GPUs, wherein a first type of task and a second type of task are deployed, the second type of task having a lower priority than the first type of task; the method includes:

[0008] The target application sends a call request to the collection communication library to request the execution of target communication operations between GPUs, wherein the target application belongs to the second type of task;

[0009] The collection communication library determines a list of subtasks involved in the target communication operation based on the call request, including multiple subtasks distributed across different GPUs; through inter-process synchronization operations, the multiple subtasks synchronously submit communication kernels to the GPU driver environment.

[0010] If the GPU driver environment determines that the communication kernel is enabled for synchronous submission, it immediately submits the communication kernel to the GPU.

[0011] According to one implementation, before the target application calls the collection communication library, the target application loads the collection communication library and initializes the GPU driver environment; the GPU driver environment obtains the current configuration parameters related to task deployment, and determines whether the target application belongs to a multi-card container deployment distributed across multiple GPUs based on the configuration parameters; in response to the GPU driver environment determining the multi-card container deployment, the collection communication library determines to enable synchronous submission and notifies the GPU driver environment.

[0012] In one embodiment of the above implementation, the GPU driver environment calls the GPU virtualization driver module to obtain the configuration parameters of the current vGPU, including the number of vGPUs and the corresponding physical GPUs.

[0013] According to one implementation, the above execution method further includes: using a collection communication library to determine whether the data size corresponding to the target communication operation exceeds a preset threshold; if it exceeds the preset threshold, dividing the target communication operation into multiple sub-communication operations.

[0014] According to one implementation, the collection communication library determines a list of subtasks involved in the target communication operation, including: if the target communication operation is point-to-point communication, obtaining two subtasks parsed during kernel preprocessing to form the subtask list.

[0015] Alternatively, if the target communication operation is a collection communication operation, all subtasks involved in the target application are added to the subtask list.

[0016] Furthermore, in different examples, the set communication operation can include one of the following: full reduction operation, broadcast operation, reduction operation, and aggregation operation.

[0017] According to one implementation, a set communication library performs inter-process synchronization operations, including: determining an identifier code for the current synchronization based on the cumulative number of times the set of subtasks in the subtask list jointly executes a communication operation of the same type as the target communication operation; and invoking an inter-process synchronization algorithm based on the identifier code.

[0018] In a specific embodiment, determining the identifier code for this synchronization may include: if the target communication operation is point-to-point communication, encoding the two subtask identifiers in the subtask list, and the cumulative number of point-to-point communication kernels submitted by the two subtasks, into the identifier code; if the target communication operation is a first type of set communication operation, encoding the cumulative number of submissions of the first type of communication kernels into the identifier code.

[0019] According to one implementation, the above execution method further includes: collecting communication library records of the time when the communication kernel is synchronously submitted to the GPU driver environment; monitoring the execution status of the communication kernel through a monitoring thread, and terminating the corresponding process if the communication kernel is not completed within a predetermined time threshold.

[0020] According to one implementation, the above execution method further includes: the GPU driver environment receiving a computing kernel submitted by the target application; the GPU driver environment submitting the computing kernel to the GPU within a permitted time slice, wherein the permitted time slice is a time slice that the target application can use for computation, determined according to the performance index requirements of the first type of task.

[0021] According to a second aspect, a distributed task execution system under GPU co-deployment is provided, applied to a platform containing multiple GPUs, wherein a first type of task and a second type of task are deployed, the second type of task having a lower priority than the first type of task; the system includes:

[0022] A collection of communication libraries is used to receive call requests from a target application to execute target communication operations between GPUs, determine a list of subtasks involved in the target communication operation based on the call requests, including multiple subtasks distributed across different GPUs, and, through inter-process synchronization operations, enable the multiple subtasks to synchronously submit communication kernels to the GPU driver environment; wherein, the target application belongs to the second type of task;

[0023] The GPU driver environment is used to immediately submit the communication kernel to the GPU when it is determined that the communication kernel is enabled for synchronous submission.

[0024] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0025] According to a fourth aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the method described in the first aspect.

[0026] According to a fifth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect.

[0027] In the embodiments of this specification, when distributed BE-type applications and GT-type applications are co-deployed on the same GPU, the communication operations of BE applications may generate uncontrollable communication wait times. By coordinating the communication library and the GPU driver environment, the subtasks on each GPU involved in the communication operation synchronously submit communication kernels to the GPU driver environment, greatly reducing the communication wait time on the GPU and thus avoiding the impact on the performance of the GT service. Furthermore, for communication operations with large data volumes, some embodiments reduce the probability of long-running kernels by partitioning data. Moreover, after the communication library submits the kernel, a monitoring thread can be started to detect the kernel execution time in real time and, if necessary, exit the process to avoid the risk of computing power suppression failure. Thus, the solutions in the embodiments of this specification can better guarantee the execution performance of GT tasks and improve GPU resource utilization. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0029] Figure 1 illustrates the execution of a distributed BE task in a GPU-mixed deployment scenario;

[0030] Figure 2 shows a schematic diagram of a system architecture according to one embodiment;

[0031] Figure 3 illustrates a distributed task execution process according to one embodiment;

[0032] Figure 4 shows the detailed operation flow in the NCCL collection communication library during the application runtime phase;

[0033] Figure 5 illustrates the execution and scheduling logic of the kernel in CUDA. Detailed Implementation

[0034] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0035] First, we will introduce and explain some of the technical terms used in this article.

[0036] SLA: Service-level agreement, refers to the level of service that a supplier promises to provide to a customer. In the context of GPUs, it generally refers to the user's requirements for resource quality.

[0037] vGPU: Virtual GPU, uses virtualization technology to virtualize a single GPU into multiple smaller GPUs, allowing multiple tasks to share the same physical GPU.

[0038] GT (Guaranteed) tasks: These are tasks with high SLA requirements, typically with specific requirements for metrics such as request pressure (QPS) or access latency (RT), and have strict requirements for resource quality.

[0039] Best-effort (BE) tasks: These are tasks with lower SLA requirements, typically computationally intensive applications, offering good fault tolerance and retry capabilities, and having relatively relaxed requirements for resource quality. In practice, training tasks for some large AI models, as well as offline inference tasks, fall under the BE category.

[0040] Mixed deployment: Mixed deployment improves hardware utilization by deploying multiple containers on the same physical GPU through vGPU. The applications within these containers include both GT and BE types.

[0041] Kernel: Function code that executes on the GPU.

[0042] Rank: In distributed tasks, each subtask or instance executing in parallel can be identified by a rank. Specifically, when using GPUs for parallel computing, rank is typically used to refer to a unique identifier for each process within a communication group. Each process running on a GPU card can be considered an independent rank.

[0043] It's understandable that GPUs, as acceleration hardware used extensively in AI scenarios, generally have scarce resources and low utilization rates. GPU virtualization technology allows for the mixed deployment of multiple containers and tasks on a single card, improving resource utilization and better reusing GPU resources. Furthermore, the rapid development of large-scale models is increasing the demand for distributed deployment of multiple GPUs.

[0044] It's important to clarify that GPU co-deployment and distributed deployment are concepts from different perspectives. Co-deployment, from the perspective of the GPU card, refers to the mixed deployment of different types or priorities of workloads or tasks on a single GPU card, such as simultaneously deploying ground truth (GT) tasks and background (BE) tasks. Distributed deployment, on the other hand, is a description from the perspective of application tasks; it means that an application task is deployed and executed across multiple GPUs.

[0045] When mixed and distributed deployments coexist, such as when a multi-GPU distributed BE workload is mixed with other GT workloads, it is difficult to guarantee the SLA requirements of the GT workload due to the uncontrollable nature of GPU communication operations. Specifically, the execution progress of the distributed BE workload may be inconsistent across different GPU cards, resulting in the communication kernel on one card waiting for a long time for other cards, thus affecting the execution of the GT workload.

[0046] Figure 1 illustrates the execution of a distributed BE task in a mixed GPU deployment scenario. In Figure 1, it is assumed that a certain BE task is distributed across two physical GPUs on the hardware platform, namely GPU-0 and GPU-1. One subtask of this BE task, denoted as Rank0, is executed on GPU-0, while another subtask, Rank1, is executed on GPU-1. Furthermore, GPU-0 and GPU-1 are mixed-deployment GPUs, each containing other GT tasks in addition to the aforementioned BE task.

[0047] It's understandable that, since ground truth (GT) tasks have a higher execution priority than background (BE) tasks, the GPU's scheduling unit adjusts the time slices or time ratio of GPU computing power used by GT and BE tasks based on the SLA (Solution Level Agreement) of GT tasks. During this process, the execution of BE tasks will be suppressed. The load of GT tasks distributed across different GPUs varies, and the degree of suppression for different BE task ranks also differs.

[0048] As shown in Figure 1, Rank0 on GPU-0 performs the computation first in the computation stream. It then needs to communicate with Rank1 to exchange data before proceeding with subsequent computations. Therefore, Rank0 submits a communication kernel to the GPU at time T0. However, Rank1 on GPU-1, due to significant suppression, may execute the computation later, submitting its communication kernel only at time T1. Consequently, Rank0 has to wait a long time for Rank1 to submit its communication kernel. The waiting time, T1-T0, may even far exceed the time required to execute the communication kernel (T2-T1). Since only one task process context can be running on the GPU at any given time, GPU-0 is occupied by the BE task context for an extended period during this waiting time, severely impacting the performance of the GT service on it and affecting GPU resource utilization.

[0049] In related technologies, research and optimization of GPU co-deployment technology mainly focus on the sharing of individual physical GPUs, emphasizing fine-grained allocation and scheduling strategies for GPU computing power, without considering the differences between distributed load co-deployment and single-card load co-deployment. This means that when distributed BE applications and GT applications are co-deployed on the same GPU, as shown in Figure 1, the communication operations of the BE applications may severely disrupt the finely designed computing power allocation and scheduling strategies, thereby breaking the system's SLA guarantee for GT services.

[0050] In view of this, this specification proposes a solution for scenarios with mixed GPU deployment of distributed tasks. In this solution, the multi-GPU collaborative communication operations of distributed BE tasks are synchronously scheduled to avoid the problem of long waiting times and GPU occupation during communication operations between GPU cards, effectively guaranteeing the SLA of online services, supporting the need for mixed deployment of multi-GPU offline BE tasks and online GT services, and improving GPU resource utilization.

[0051] Figure 2 illustrates a system architecture diagram according to one embodiment. As shown in Figure 2, the lowest layer of this system architecture is the hardware layer, which includes multiple physical GPUs, or multiple GPU cards. Above the GPU hardware layer is the GPU driver environment, which is exemplarily shown as a CUDA driver in Figure 2.

[0052] CUDA is Nvidia's programming environment for GPU hardware. CUDA abstracts the operations and functions of GPU hardware and provides them as a programming interface. All tasks executed by upper-layer applications on the GPU must be implemented through the interfaces provided by CUDA.

[0053] The GPU virtualization module is a functional module based on CUDA. It is used to virtualize a single physical GPU into multiple smaller virtual GPUs (vGPUs) according to the needs of the task container, and to deploy the task container in the corresponding vGPU.

[0054] Above the GPU virtualization module, various services or tasks with different priorities are run and deployed. These services or tasks can specifically include a first category of tasks with higher priority, namely GT services, and a second category of tasks with relatively lower priority, namely BE tasks. BE tasks are distributed across multiple GPUs. Because they are distributed tasks, the execution of the upper-layer BE tasks also relies on a collection communication library, exemplarily shown as the NCCL communication library in Figure 2.

[0055] NCCL is a collective communication library designed specifically for Nvidia GPUs. It is widely used in distributed GPU tasks to implement collective and peer-to-peer communication between GPUs. Peer-to-peer communication can be understood as communication between two specified GPUs (more specifically, their ranks). Collective communication, on the other hand, is applied to joint communication operations between all GPUs involved in a distributed task. Specifically, the NCCL library provides various collective communication operations, such as all-reduce, broadcast, reduce, scatter, and gather.

[0056] For example, a full reduce operation merges (reduces) the dataset from all GPUs into a single value, then broadcasts this value back to all GPUs. A broadcast operation copies data from one GPU to all other GPUs. This may involve kernel-level data copying and transfer. Reduce merges data from all GPUs into a single value; it's typically an aggregation operation such as summation, maximization, or minimization. Distribute data from one GPU to multiple GPUs, with each GPU receiving a portion of the data. Aggregate gathers data fragments from multiple GPUs onto a single GPU.

[0057] Through these NCCL collection communication operations, data parallelism and synchronization between GPUs can be achieved.

[0058] To synchronize and schedule multi-GPU collaborative communication for distributed BE tasks in mixed GPU deployment scenarios, the embodiments in this specification improve upon the existing GPU driver environment and aggregated communication library. These improvements are shown in gray in Figure 2. Specifically, the following functional logic is added to the existing aggregated communication library: For distributed BE tasks, the communication tasks to be submitted on each GPU are coordinated, ensuring that each rank synchronously submits the communication kernel. Furthermore, if the communication data size is too large, it is divided into multiple smaller-scale communications. If the execution time of a submitted communication kernel is too long, the process is triggered to exit, preventing impact on the performance of the GT service.

[0059] The improved GPU driver environment distinguishes between communication kernels and computation kernels for multi-GPU distributed BE tasks. For communication kernels, tasks are quickly submitted to the GPU hardware for expedited execution. For computation kernels, task scheduling is planned based on the SLA requirements of the GT task, and computational power is dynamically managed.

[0060] The following describes the execution of distributed BE tasks using the CUDA driver environment (hereinafter referred to as CUDA driver or CUDA) and NCCL collection communication library corresponding to Nvidia GPU products. However, it should be understood that the corresponding GPU driver environment and collection communication library can be used interchangeably with GPU hardware products from other manufacturers.

[0061] Figure 3 illustrates a distributed task execution process according to one embodiment. As shown in Figure 3, the overall execution process can be divided into an initialization phase and a running phase. These are described below.

[0062] During the initialization phase, when the target application process starts, the NCCL collection communication library is loaded in step S11, and the CUDA driver is initialized in S12. It can be understood that the target application here is a BE-class task, whose execution priority is lower than that of GT-class tasks. The target application initializes the CUDA driver to obtain its own execution environment.

[0063] Next, in step S13, during initialization, the CUDA driver obtains the configuration parameters related to the current task deployment. Based on these parameters, it determines whether the target application is a multi-GPU deployment distributed across multiple GPUs. This is used to determine whether communication-based collaborative submission optimization (i.e., synchronous submission) needs to be enabled for distributed BE tasks. Specifically, the CUDA driver can call the GPU virtualization driver module to obtain the current vGPU configuration parameters, including, for example, the number of vGPUs and the corresponding physical GPUs, thereby determining whether the current target application is a distributed multi-GPU deployment.

[0064] Next, in step S14, the NCCL communication library queries the CUDA driver for information about multi-GPU deployment. The CUDA driver returns response information to the NCCL communication library indicating whether the target application is deployed with multiple GPUs. In step S15, the NCCL determines whether to enable synchronous commit based on the response information and notifies the CUDA driver. Specifically, if the response message indicates that the target application is a distributed multi-GPU deployed BE task, the NCCL communication library determines to enable synchronous commit for the communication kernel for the target application and notifies the CUDA driver. The CUDA driver then marks the target application as having synchronous commit enabled.

[0065] It should be noted that synchronous commits of the communication kernel at the NCCL layer may incur some performance loss and even pose a risk of deadlock. Therefore, during the NCCL initialization phase, determining whether to enable synchronous commit mode based on the specific scenario, and thus selectively enabling synchronous commits of the communication kernel only for distributed BE tasks, can avoid unnecessary performance losses and improve overall GPU performance.

[0066] Next, we enter the application's runtime phase.

[0067] During the runtime phase, in step S21, the target application sends a call request to the collective communication library to request the execution of target communication operations between GPUs. That is, the target application submits a communication task request by calling the collective communication library.

[0068] Optionally, the collection communication library first determines whether the data size of the target communication operation exceeds a preset threshold. If it does not exceed the preset threshold, subsequent processing is performed directly. If it exceeds the preset threshold, it is divided into multiple sub-communication operations. In the case of division, the target communication operation mentioned below refers to each sub-communication operation obtained by dividing the original target communication operation.

[0069] Next, the communication library determines a list of subtasks (rank list) involved in the target communication operation, including multiple subtasks distributed across different GPUs; and through inter-process synchronization, the multiple subtasks synchronously submit communication kernels to the GPU driver environment. This is simplified in Figure 2 as step S22, synchronizing the ranks participating in the communication and submitting the communication kernel to CUDA.

[0070] After the communication kernel is submitted to the CUDA driver, in step S23, CUDA detects the communication kernel and determines whether an execution license is required. For communication kernels with synchronous submission enabled, the communication kernel is executed in the GPU immediately without waiting for an execution license.

[0071] Furthermore, after submitting the communication kernel to CUDA, the collection communication library repeatedly checks in step S24 whether the communication kernel has finished executing. If it has not finished executing within a certain time limit, the corresponding process is forcibly terminated to avoid affecting the execution of the GT task.

[0072] The execution logic within the collection communication library and GPU driver environment is described in detail below.

[0073] Figure 4 illustrates the detailed operation flow of the NCCL communication library during the application runtime phase. As shown in Figure 4, firstly, in step S41, the target application in the upper layer calls the NCCL communication library. It can be understood that the NCCL communication library provides multiple communication-related APIs for the upper-layer application. The upper-layer application can call the corresponding APIs as needed to implement inter-GPU task communication without needing to concern itself with the underlying details such as the implementation of the communication kernel. In one embodiment, the target application can issue a call request to the NCCL communication library, which includes at least the API name and usually also necessary parameters, such as the GPU group participating in the communication, the data buffer to be transmitted, etc. Typically, the API name indicates what kind of communication operation is requested to be performed; for example, API ncclAllReduce indicates a full reduction operation, API ncclBroadcast indicates a broadcast operation, and so on.

[0074] Next, the NCCL communication library can determine the list of subtasks involved in the communication operation to be executed based on the call request of the target application.

[0075] Specifically, in step S42, the NCCL communication library determines whether the communication operation indicated by the call request is point-to-point communication. This can be determined by the API name being called. If it is point-to-point communication, in step S43, during kernel preprocessing, the two ranks (i.e., subtasks) involved in the communication are parsed from the parameters of the call request, and these two ranks are recorded in the list of subtasks to be synchronized. Then, proceeding to step S44, NCCL starts loading the communication kernel corresponding to the target communication operation.

[0076] If the communication operation indicated by the call request is not point-to-point communication, it is assumed to be a set communication operation. In this case, the rank parsing mentioned above is unnecessary, and the rank of all GPUs constitutes the subtask list mentioned above. Proceed directly to step S44. As described above regarding the NCCL set communication library, set communication operations include one of the following: full reduction operation, broadcast operation, reduction operation, and aggregation operation.

[0077] After loading the communication kernel in step S44, step S45 determines whether synchronous submission is enabled for the current application. This can be understood as a query of the judgment result of step S15 in the initialization phase. If not enabled, the communication kernel is directly submitted to CUDA in step S46. If synchronous submission is enabled, then as shown in the right-hand branch, in step S47, the NCCL communication library calls the inter-process synchronization operation, causing each subtask rank to synchronously submit the communication kernel. Regarding inter-process synchronization, the example in Figure 4 uses a barrier mechanism. A barrier is a synchronization mechanism that ensures that no process will continue execution until it reaches a certain point, and is often used in parallel computing to synchronize the execution phases of multiple processes.

[0078] To achieve synchronization between subtasks, we can first count the number of ranks managed by the current thread. If it is greater than 1, then for each managed rank, we can create a new thread to perform the subsequent synchronization operation, thus avoiding blocking existing threads.

[0079] Then, the rank list to be synchronized is obtained, and the identifier code for this synchronization is calculated. This is because multiple communication combinations that need to be synchronized may exist at the same time. To avoid interference between different combinations during synchronization, an identifier code needs to be assigned to each synchronization combination. In one embodiment, the identifier code for this synchronization can be determined based on the cumulative number of times the subtask sets in the rank list jointly execute communication operations of the same type as the target communication operation.

[0080] Specifically, for point-to-point communication operations, the two rank identifiers involved in the communication and the cumulative number of point-to-point communication kernels submitted between them can be encoded into an identifier code. For example, if the target communication operation is a point-to-point communication operation performed between rank0 and rank1, and 10 point-to-point communication operations have been performed between rank0 and rank1, then the identifiers of rank0 and rank1, as well as the information of the number 10, can be encoded into an identifier code, such as (01)-10. In this way, through this identifier code, the 11th point-to-point communication kernel to be submitted between rank0 and rank1 can be accurately and uniquely located.

[0081] For group communication operations, the cumulative submission count of kernels of the same type can be encoded into an identifier. For example, suppose the target communication operation to be executed is a group communication operation of a certain type, such as a reduction operation. If a group reduction operation has already been executed 8 times, meaning the cumulative number of times the communication kernel used for the reduction operation has been executed is 8, then this number can be encoded into an identifier. In this way, the group to be synchronized can be uniquely identified as the 9th communication kernel to be submitted for a reduction operation across all ranks.

[0082] Once the identifier code of the combination to be synchronized is determined, an inter-process synchronization algorithm can be invoked based on this identifier code to synchronize the communication kernels indicated by the corresponding rank. Various inter-process synchronization mechanisms and algorithms already exist, and can be selected as needed to achieve process synchronization. For example, the dissemination algorithm can be used to ensure that all process state copies are consistent through socket message passing, thus achieving process synchronization.

[0083] Inter-process synchronization ensures that each rank of the target application in step S48 synchronously submits the communication kernel to CUDA. Furthermore, as shown in step S49, the collection communication library also records the time of synchronous submission of the communication kernel to the GPU driver environment and monitors the execution status of the aforementioned communication kernel through a monitoring thread. If the communication kernel fails to complete within a predetermined time threshold, the corresponding process is terminated (corresponding to step S24 in Figure 3) to prevent prolonged occupation of GPU resources and potential impact on GT tasks deployed on the same GPU.

[0084] The following describes the execution and scheduling logic in CUDA after any kernel is submitted, as shown in Figure 5.

[0085] As shown in Figure 5, when the CUDA driver receives a kernel to be submitted, it first starts and loads the corresponding kernel in step S51, and then determines the priority of the corresponding task in step S52, i.e., whether it is a GT task or a BE task. If it is a GT task, it directly jumps to step S57 and submits the kernel to the GPU. If it is a BE application task, in step S53, it determines whether the kernel is an NCCL communication kernel by parsing the kernel name.

[0086] If the result of step S53 indicates that the current kernel is not a communication kernel, but a computation kernel, then in step S54, the process waits for execution permission. This is because, for computation kernels, the GPU virtualization module dynamically adjusts the time slice of GPU computing power used by the BE task based on the SLA index of the GT load. Only within the scheduled time slice will the BE task be granted execution permission for its computation kernel, allowing it to submit its computation kernel. Therefore, while waiting for execution permission, the process proceeds to step S57, where the kernel is submitted to the GPU.

[0087] If the result of step S53 indicates that the current kernel is a communication kernel, then step S55 continues to determine whether NCCL collaborative commit is used, i.e., whether the aforementioned synchronous commit function is enabled for the current BE task. Note that in step S15 of the initialization phase in Figure 3, after the NCCL communication library determines that synchronous commit of the communication kernel is enabled for a certain application, it notifies CUDA. CUDA accordingly marks the application as having synchronous commit enabled. In step S55 here, CUDA determines whether synchronous commit is enabled by reading the label corresponding to the current application.

[0088] If synchronous commit is enabled for the current communication kernel, CUDA immediately commits the kernel to the GPU and executes it. If the NCCL layer is not synchronized, this is considered an abnormal situation, and in step S56, the process exits to avoid affecting the performance of the GT service.

[0089] Figure 5 illustrates the execution logic for any kernel. It can be understood that, for the target application belonging to the BE task, when CUDA receives a communication kernel submitted synchronously via the NCCL set communication library, the process executes branches S51, S52, S53, S55 to S57. That is, once it is determined to be a communication kernel for a distributed BE task with synchronous submission enabled, the communication kernel is immediately submitted to the GPU and executed. For the computation kernel submitted by the target application, the process executes branches S51, S52, S53, S54 to S57. CUDA submits the computation kernel to the GPU within the permitted time slice, where the permitted time slice is the time slice that the target application can use for computation, determined by the GPU virtualization module based on the performance requirements of the GT task.

[0090] It should be understood that, for clarity, the foregoing embodiments are described in detail in conjunction with the NCCL collective communication library and CUDA driver environment for Nvidia GPUs. However, those skilled in the art will recognize that GPUs from other manufacturers may have different collective communication libraries and driver environments. The technical concepts in the above embodiments can be equivalently extended to other collective communication libraries and driver environments.

[0091] In summary, when distributed BE (Balanced Entity) and GT (Ground-Track) applications are deployed together on the same GPU, communication operations of the BE applications can severely disrupt the SLA (Service Level Agreement) for the GT service. To address this, the solutions in the embodiments of this specification coordinate the communication library and the GPU driver environment, ensuring synchronous submission of communication kernels on all GPUs. This significantly reduces communication waiting time on the GPUs, thereby avoiding performance impact on the GT service. Furthermore, for communication operations with large data volumes, some embodiments reduce the probability of long-running kernels through data partitioning. Moreover, after the communication library submits the kernel, a monitoring thread can be started to detect kernel execution time in real time and, if necessary, exit the process to avoid the risk of computational power suppression failure. Thus, the solutions in the embodiments of this specification better guarantee the execution performance of GT tasks and improve GPU resource utilization.

[0092] According to another embodiment, a distributed task execution system under GPU co-deployment is provided, applied to a platform containing multiple GPUs, wherein a first type of task and a second type of task are deployed, the second type of task having a lower priority than the first type of task. The system includes:

[0093] A collection of communication libraries is used to determine a list of subtasks involved in the target communication operation when called by a target application to request the execution of a target communication operation between GPUs. This list includes multiple subtasks distributed across different GPUs. Through inter-process synchronization operations, the multiple subtasks are made to synchronously submit communication kernels to the GPU driver environment. The target application belongs to the second type of task.

[0094] The GPU driver environment is used to immediately submit the communication kernel to the GPU when it is determined that the communication kernel is enabled for synchronous submission.

[0095] For examples of specific implementation methods of this system, please refer to the foregoing examples and descriptions of the combined execution method, which will not be repeated here.

[0096] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the aforementioned method.

[0097] According to yet another embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor, implement the aforementioned method.

[0098] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein executable code is stored in the memory, and the processor executes the executable code to implement the aforementioned method.

[0099] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0100] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A distributed task execution method under GPU co-deployment, applied to a platform containing multiple GPUs, wherein a first type of task and a second type of task are deployed, the second type of task having a lower priority than the first type of task; the method includes: The target application sends a call request to the collection communication library to request the execution of target communication operations between GPUs, wherein the target application belongs to the second type of task; The collection communication library determines a list of subtasks involved in the target communication operation based on the call request, including multiple subtasks distributed across different GPUs; through inter-process synchronization operations, the multiple subtasks synchronously submit communication kernels to the GPU driver environment. If the GPU driver environment determines that synchronous commit is enabled for the communication kernel, it immediately commits the communication kernel to the GPU.

2. The method according to claim 1, wherein, Before the target application sends a call request to the collection communication library, it also includes: The target application loads the collection communication library and initializes the GPU driver environment; The GPU driver environment obtains the current configuration parameters related to task deployment and determines whether the target application belongs to a multi-card container deployment distributed across multiple GPUs based on the configuration parameters. In response to the GPU driver environment determining the multi-GPU container deployment, the collection communication library determines to enable synchronous commit and notifies the GPU driver environment.

3. The method according to claim 2, wherein, The GPU driver environment obtains the current configuration parameters related to task deployment, including: The GPU driver environment calls the GPU virtualization driver module to obtain the current vGPU configuration parameters, including the number of vGPUs and the corresponding physical GPUs.

4. The method according to claim 1, further comprising: The collection communication library determines whether the data size corresponding to the target communication operation exceeds a preset threshold; If the preset threshold is exceeded, the target communication operation will be divided into multiple sub-communication operations.

5. The method according to claim 1, wherein, The collection communication library determines a list of subtasks involved in the target communication operation based on the call request, including: If the communication operation indicated by the call request is point-to-point communication, obtain the two subtasks parsed during kernel preprocessing, and form the subtask list.

6. The method according to claim 1, wherein, The collection communication library determines a list of subtasks involved in the target communication operation based on the call request, including: If the communication operation indicated by the call request is a collection communication operation, all subtasks involved in the target application are added to the subtask list.

7. The method according to claim 6, wherein, The aggregated communication operation includes one of the following: full reduction operation, broadcast operation, reduction operation, and aggregation operation.

8. The method according to claim 1, wherein, The inter-process synchronization operations include: The identifier code for this synchronization is determined based on the cumulative number of times the subtask set in the subtask list jointly performs a communication operation of the same type as the target communication operation. Based on the identification code, the inter-process synchronization algorithm is invoked.

9. The method according to claim 8, wherein, The identifier for this synchronization includes: If the target communication operation is point-to-point communication, the two subtask identifiers in the subtask list, and the cumulative number of point-to-point communication kernels submitted by the two subtasks, are encoded into an identifier code; If the target communication operation is a first type of collection communication operation, the cumulative number of submissions of the first type of communication kernel is encoded into the identifier code.

10. The method according to claim 1, further comprising: The collection communication library records the time when the communication kernel is synchronously submitted to the GPU driver environment; The execution status of the communication kernel is monitored by a monitoring thread. If the communication kernel fails to complete within a predetermined time threshold, the corresponding process is terminated.

11. The method according to claim 1, further comprising: The GPU driver environment receives the computing kernel submitted by the target application; The GPU driver environment submits the computing kernel to the GPU within a permitted time slice, wherein the permitted time slice is a time slice that the target application can use for computation, determined according to the performance metric requirements of the first type of task.

12. A distributed task execution system under GPU co-deployment, applied to a platform containing multiple GPUs, wherein a first type of task and a second type of task are deployed, the second type of task having a lower priority than the first type of task; the system includes: The collection communication library is used to receive call requests from the target application to execute target communication operations between GPUs, and to determine a list of subtasks involved in the target communication operation based on the call requests, including multiple subtasks distributed across different GPUs; And through inter-process synchronization operations, the multiple subtasks synchronously submit communication kernels to the GPU driver environment; wherein, the target application belongs to the second type of task; The GPU driver environment is used to immediately submit the communication kernel to the GPU when it is determined that synchronous submission is enabled for the communication kernel.

13. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-11.

14. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-11.

Images