Data transmission methods and electronic apparatus
By implementing data plane communication on the host CPU and utilizing computing device pass-through network card technology and RDMA one-sided atomic operation, the problem of increased GPU computing resource overhead and communication latency in large model training and push clusters by the NCCL communication interface is solved. This achieves efficient cross-network data transmission and memory space reuse, improving the utilization rate of GPU computing resources.
Patent Information
- Application Number
- PCT/CN2026/092672
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-06
- Filing Date
- 2026-04-23
- Publication Date
- 2026-11-12
AI Technical Summary
In existing technologies, the NCCL communication interface performs data copying operations through kernel functions in large model training and push cluster communication, which significantly increases the GPU computing power resource overhead. Furthermore, the copying of video memory data increases communication latency and may be limited by PCIe bus bandwidth.
Data plane communication is achieved through the host CPU. Cross-network data transmission is realized by using the computing device's direct-to-network card technology, avoiding GPU computing power from participating in data transmission. GDR technology is used to send the data generated by the training and push tasks directly to the communication video memory space of the receiver, avoiding internal data copying operations. RDMA one-sided atomic operation is used to detect the status of the receiving video memory, realizing remote read and write synchronization between the sender and receiver.
It significantly improves the efficiency of large model training and induction operations, reduces communication latency, increases the utilization of GPU computing resources, and reduces the computational load of kernel functions.
Smart Images

Figure CN2026092672_12112026_PF_FP_ABST
Abstract
Description
Data transmission method and electronic device
[0001] Cross-reference to related applications
[0002] The present disclosure is based on Chinese Patent Application No. CN202510577022.2 entitled “Data transmission method and electronic device” filed on May 6, 2025, and claiming priority to the same, the disclosure of which is incorporated herein by reference in its entirety. TECHNICAL FIELD
[0003] Embodiments of the present disclosure relate to the field of communication, in particular to a data transmission method and an electronic device. BACKGROUND
[0004] With the expansion of the parameter scale of large models, training clusters need to implement kilo-card, ten-thousand-card parallel networks. The communication overhead of large-scale clusters greatly affects the efficiency of large model training and inference. How to optimize the communication performance between GPUs and reduce the overhead of cross-node data migration has become an important direction for large model training cluster optimization.
[0005] Currently, the communication of large model training clusters mainly uses the NCCL communication library. The collection communication interface represented by NCCL meets the communication needs of graphics processing units (GPUs) in the training and inference process and has become the communication standard for large model clusters. The traditional communication interface represented by NCCL requires the joint participation of GPUs, video memory, RDMA network interface cards (RNICs), and random access memory (RAM) to complete the communication. The GPU kernel function completes the data copy operation, increasing the GPU computing resource overhead. If the communication layer space uses video memory space, the video memory overhead is increased. If the communication memory uses RAM space, the video memory data is copied to the RAM memory, which is limited by the PCIe bus bandwidth, resulting in increased communication delay. SUMMARY
[0006] Embodiments of the present disclosure provide a data transmission method and an electronic device to at least solve the problem in the related art that in the communication of large model training clusters, the NCCL communication interface completes the data copy operation through the kernel function, significantly increasing the GPU computing resource overhead, and the video memory data copy increases the communication delay and may be limited by the PCIe bus bandwidth.
[0007] According to one embodiment of this disclosure, a data transmission method is provided, applied to a first central processing unit (CPU) of a first computing server, comprising: detecting the state of a first video memory and a second video memory, wherein the first video memory is allocated in the first computing server and the second video memory is allocated in the second computing server; in response to the state of the first video memory and the second video memory satisfying a preset condition, directly sending the business computing data written by the first graphics processor (GPU) of the first computing server into the first video memory to the second video memory.
[0008] According to another embodiment of this disclosure, a data transmission method is provided, applied to a second central processing unit (CPU) of a second computing server, comprising: detecting the state of a first video memory and a second video memory, wherein the first video memory is allocated in the first computing server and the second video memory is allocated in the second computing server; in response to the state of the first video memory and the second video memory satisfying a preset condition, directly receiving the business computing data written by the first graphics processing unit (GPU) of the first computing server into the first video memory into the second video memory.
[0009] According to yet another embodiment of this disclosure, a computer-readable storage medium is also provided, in which a computer program is stored, wherein the computer program is configured to perform the steps in any of the above method embodiments when it is run.
[0010] According to yet another embodiment of this disclosure, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0011] According to yet another embodiment of this disclosure, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps in any of the above method embodiments. Attached Figure Description
[0012] Figure 1 is a schematic diagram of the P2P transmission process in the prior art;
[0013] Figure 2 is a hardware structure block diagram of the computer terminal used in the embodiments of the method disclosed herein;
[0014] Figure 3 is a flowchart (a) of a data transmission method according to an embodiment of the present disclosure;
[0015] Figure 4 is a flowchart (II) of a data transmission method according to an embodiment of the present disclosure;
[0016] Figure 5 is a schematic diagram of a pipelined parallel network according to an embodiment of the present disclosure;
[0017] Figure 6 is a schematic diagram of a model parallel network according to an embodiment of the present disclosure;
[0018] Figure 7 is a schematic diagram of memory reuse for a large model training task according to an embodiment of the present disclosure.
[0019] Figure 8 is a schematic diagram of memory reuse for large model inference tasks according to an embodiment of the present disclosure;
[0020] Figure 9 is a flowchart (III) of a data transmission method according to an embodiment of the present disclosure;
[0021] Figure 10 is a schematic diagram of memory status code migration according to an embodiment of the present disclosure. Detailed Implementation
[0022] The embodiments of this disclosure will be described in detail below with reference to the accompanying drawings and examples.
[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.
[0024] Currently, when the large model computing server starts, it reads the configuration file to build the entire training and inference cluster, configuring the specific parallel mode. This involves using one or a combination of several parallel methods, such as data parallelism (DP), model parallelism (TP), and pipelined parallelism (PP). The configuration also specifies how all computing nodes in the cluster are grouped and collaborate. Different GPUs may need to exchange sample data, model parameter data, and calculated gradients. Communication within the large model training and inference cluster primarily uses the NCCL communication library. NCCL, as a representative of aggregated communication interfaces, meets the GPU communication needs during training and inference, becoming the communication standard for large model clusters. Multiple aggregated communication interfaces in the NCCL library are called; for example, a reduce operation is required during gradient aggregation. Most other communication processes only require data migration and do not require computational operations.
[0025] NCCL communication interfaces can be divided into two categories. One category, represented by the Reduce interface, requires simultaneous transmission and computation operations. The other category includes interfaces such as Point-to-Point (P2P), Broadcast, and AllGather, which only require data transmission and do not need to perform computation tasks. Currently, both types of interfaces use the same implementation method. Taking NCCL's P2P communication interfaces Send and Recv as examples, Figure 1 is a schematic diagram of the P2P transmission process in the existing technology. As shown in Figure 1, the training and push task sends the content of the video memory src to the target video memory space dst through the Remote Direct Memory Access (RDMA) network, through the following steps:
[0026] 1. The receiving task calls ncclRecv to notify the communication task of the address dst for receiving data;
[0027] 2. The sending task executes ncclSend to notify the communication task of the address src of the data to be sent;
[0028] 3. The sender triggers the GPU device to execute the kernel function, which copies the src space data to the network send buffer sendMem.
[0029] 4. The communication task transmits data to the receiver's network receive buffer recvMem via the RDMA Write instruction;
[0030] 5. The receiver triggers the GPU device to execute the kernel function, which performs the data copying from the receive buffer to the target address dst.
[0031] 6. After the RDMA communication is completed, notify the training push task.
[0032] If it's a Reduce(sum) operation, the specific kernel function needs to be called during the transmission to implement the sum operation. The send and receive buffers for the communication task can be allocated from GPU memory or RAM space, and both need to be registered with the RDMA network in advance. During the transmission, both the sender and receiver need the GPU to participate, and the sender and receiver each perform a data copy through a kernel function.
[0033] It is evident that traditional communication interfaces, such as NCCL, require the joint participation of the GPU, video memory, RNIC, and RAM. The GPU kernel function performs the data copying operation, increasing the GPU's computing power resource overhead. If the communication layer uses video memory, it increases video memory overhead. If the communication memory uses RAM, the copying of video memory data to RAM is limited by the PCIe bus bandwidth, resulting in increased communication latency.
[0034] To address the aforementioned technical issues, this application provides a data transmission method that optimizes the communication interface of computing devices based on RDMA networks, enabling more efficient cross-network data transmission. The method utilizes the host CPU to implement data plane communication and leverages direct-access network card (NIC) technology for fast transmission, eliminating the need for GPU computing power. It requires the computing device's video memory to be directly accessible via the NIC, such as NVIDIA GPUs or robust BR cards, and supports both Infiniband and RoCE RDMA network types.
[0035] This method addresses the memory access characteristics of training and push tasks by enabling communication and training / pull tasks to share GPU memory space. Using GDR (Geometric Memory Deployment) technology, data generated by the training / pull tasks is directly sent to the receiving party's communication GPU memory space, avoiding internal data copying operations, reducing end-to-end communication latency, bypassing the GPU device to complete the communication process, and improving the utilization of GPU computing resources. To achieve state synchronization of the communication GPU memory, on the one hand, ordered state transitions ensure strict serial access to data, avoiding access conflicts between training / pull tasks and communication tasks; on the other hand, this method utilizes RDMA (Remote Data Memory Deployment) one-sided atomic operations to detect the receiving GPU memory state, achieving remote read / write synchronization between the sender and receiver. Compared with communication mechanisms in related technologies, this method completely offloads communication overhead to the host CPU, freeing up valuable GPU computing resources from the communication process, significantly improving the efficiency of large-scale model training and push operations, and effectively shortening the data transmission path.
[0036] The method embodiments provided in this application can be executed in a mobile terminal, computer terminal, or similar computing device. Taking running on a computer terminal as an example, FIG2 is a hardware structure block diagram of the computer terminal running the method embodiments of this disclosure. As shown in FIG2, the computer terminal may include one or more (only one is shown in FIG2) processors 202 (processor 202 may include, but is not limited to, processing devices such as microprocessors MCU or programmable logic devices FPGA) and a memory 204 for storing data. The computer terminal may also include a transmission device 206 for communication functions and an input / output device 208. Those skilled in the art will understand that the structure shown in FIG2 is only illustrative and does not limit the structure of the computer terminal. For example, the computer terminal may also include more or fewer components than shown in FIG2, or have a different configuration than shown in FIG2.
[0037] The memory 204 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data transmission method in this embodiment. The processor 202 executes various functional applications and data processing by running the computer program stored in the memory 204, thus implementing the aforementioned method. The memory 204 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 204 may further include memory remotely located relative to the processor 202, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0038] The transmission device 206 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 206 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 206 may be a Radio Frequency (RF) module used for wireless communication with the Internet.
[0039] This embodiment provides a data transmission method running on the aforementioned computer terminal, applied to the first central processing unit (CPU) of a first computing server. Figure 3 is a flowchart (I) of the data transmission method according to an embodiment of this disclosure. As shown in Figure 3, the process includes the following steps:
[0040] Step S302: Detect the status of the first video memory and the second video memory, wherein the first video memory is allocated to the first computing server and the second video memory is allocated to the second computing server.
[0041] For example, the first CPU of the first computing server can detect the status of the first video memory and the second video memory, which can be in the form of status code, field combination, bitmap, etc.
[0042] For example, the states of the first and second video memory are represented by status codes. Taking three status codes as an example: 0 indicates the first video memory is idle, 1 indicates a write state, and 2 indicates a send state. Similarly, 0 indicates the first video memory is idle, 1 indicates a processing state, and 2 indicates a receiving state. Specifically, the write state indicates that when the first GPU executes a business computing task, it writes the output business computing data to the first video memory. The processing state indicates that the second GPU uses the business computing data received from the second video memory as input data to execute a business computing task, performing calculations on the received data.
[0043] For example, this service includes training and inference operations for deep learning networks such as large and small models. A computational task refers to the computational function executed on the GPU. The send and receive states refer to the states of the first and second GPU memory when the CPU executes a communication task, which is a communication function executed on the CPU.
[0044] In one exemplary embodiment, prior to step S302, the following is included:
[0045] The size of the first video memory is determined based on the attributes of the business, so that the first GPU can allocate the first video memory according to the size of the first video memory.
[0046] Register the first video memory to the first RDMA network card;
[0047] Allocate the first page locked video memory, where the first page locked video memory is used to store the state of the first video memory and the second video memory;
[0048] Initialize the state of the first video memory to an idle state;
[0049] Send the status and configuration information of the first video memory to the second CPU of the second computing power server;
[0050] In response to the need for the first GPU to write business computing data into the first video memory, the state of the first video memory is changed from idle state to write state.
[0051] For example, the first video memory can be registered as a local read / write mode. Taking the status of the first and second video memory as an example, the status code of the first video memory can be initialized to 0.
[0052] For example, determining the size of the first video memory based on the attributes of the business is to determine the size of the first video memory required when the first GPU performs business computing tasks.
[0053] For example, business computing tasks may include training tasks, inference tasks, etc. Training tasks include forward propagation training tasks and backward propagation training tasks, and inference tasks include forward propagation inference tasks. When the first GPU performs a forward propagation large model training task or an inference task, the size of the first GPU memory can be determined based on the attributes of the large model and the context size of the large model. When the first GPU performs a backward propagation large model training task, the size of the first GPU memory can be determined based on the attributes of the large model.
[0054] For example, when the first GPU performs a large model training or inference task using forward propagation, the size of the first GPU memory can be determined based on factors such as the number of hidden layers and the upper limit of the training character length. When the first GPU performs a large model training task using backward propagation, the size of the first GPU memory can be determined based on factors such as the weight values of the large model.
[0055] It should be noted that the steps before step S302 can be understood as the initialization phase. This initialization only needs to be performed once when the computing server is powered on, and there is no need to perform the initialization operation every time data is sent.
[0056] In one exemplary embodiment, the configuration information includes address information and key-value information.
[0057] For example, configuration information can be initialized based on configuration items.
[0058] In step S304, in response to the state of the first video memory and the second video memory meeting the preset conditions, the business computing data written by the first graphics processor GPU of the first computing server into the first video memory is directly sent to the second video memory.
[0059] For example, by directly sending the business computing data written to the first video memory by the first graphics processor (GPU) of the first computing server to the second video memory, the effect of video memory space reuse can be achieved. The first GPU writes the data output when performing business computing tasks (business computing data) to the first video memory. After the writing is completed, the first video memory can be used as the video memory for the first CPU to perform communication tasks. That is, the video memory for the first GPU to output data overlaps with the video memory for the first CPU to send data, avoiding the waste of video memory space resources and realizing the reuse of video memory space.
[0060] For example, sending business computation data directly to the second GPU memory avoids data copying, simplifies data transmission steps, and improves data transmission efficiency. This step does not require the participation of the first GPU; it is led by the first CPU. Compared with solutions in related technologies, it avoids kernel functions participating in communication and data copying operations, eliminates the need for GPU participation in data transmission, frees up GPU computing resources from the communication process, and significantly improves the efficiency of large model training and push services.
[0061] In one exemplary embodiment, step S304 includes:
[0062] In response to the first GPU performing business calculations and writing business calculation data to the first video memory, the state of the first video memory is changed from the write state to the send state;
[0063] In response to the need to send the business computing data in the first video memory, the state of the first video memory is detected as the sending state and the state of the second video memory is detected as the receiving state.
[0064] The first RDMA network card is directly accessed through the first remote memory and the second RDMA network card to send the business computing data in the first video memory directly to the second video memory. The first RDMA network card is configured in the first computing server and the second RDMA network card is configured in the second computing server. The first RDMA network card and the second RDMA network card are connected to each other.
[0065] For example, taking the state of the first video memory as including idle, write, and send states, when the first GPU performs business calculations and writes business calculation data to the first video memory, the state of the first video memory can be changed from write state to send state. Based on the switching of the state of the first video memory, the space of the output data of the first GPU performing the calculation task can overlap with the space of the first CPU performing the communication task, so as to realize the reuse of the first video memory.
[0066] For example, when the first GPU performs a large model training or inference task using forward propagation, it writes the output large model computation data (e.g., activation tensors, bias parameters, intermediate results of the loss function, etc.) into a first memory space, which can be the same memory space as the memory space that sent the large model computation data. When the first GPU performs a large model training task using backward propagation, it writes the output large model computation data (e.g., gradients, learning rates, regularization parameters, etc.) into the first memory space, which can also be the same memory space as the memory space that sent the large model computation data.
[0067] For example, when the business computing data in the first video memory needs to be sent, the first CPU changes the status code of the first video memory to 2. If the first CPU detects that the status code of the second video memory is 2, it means that the second computing server (receiver) is ready, and the two parties transmit data through GDR technology and RDMA network. If the status code of the second video memory is not 2, it means that the receiver is not ready to receive data, which may be due to uneven service distribution, resulting in slow processing by the receiver.
[0068] In an exemplary embodiment, directly sending service computing data from the first video memory to the second video memory via a first RDMA network card and a second RDMA network card includes:
[0069] The first RDMA network card is instructed to send an RDMA write request to the second RDMA network card so that the first RDMA network card can directly write the service calculation data into the second video memory.
[0070] For example, the first CPU instructs the first RDMA network card to initiate an RDMA Write request to the second RDMA network card of the receiver, so as to directly write the service computing data in the first video memory of the first computing server (sender) into the second video memory space of the receiver.
[0071] In one exemplary embodiment, after step S304, the following is included:
[0072] In response to the first RDMA network card completing the direct writing of service calculation data, the state of the first video memory is changed from the sending state to the idle state.
[0073] For example, taking the status of the first and second video memory as a status code, after the transmission of business computing data is completed, the first CPU can change the status code of the first video memory from 2 to 0.
[0074] By following the steps described above, data communication is completely offloaded to the host CPU, enabling the reuse of GPU memory. This avoids data copying and kernel function involvement in communication, freeing up GPU computing resources from the communication process, improving communication efficiency, and reducing the computational load of kernel functions. This addresses the problems in large-scale model training and push cluster communication where the NCCL communication interface performs data copying operations through kernel functions, significantly increasing GPU computing resource overhead, and the increased communication latency caused by GPU memory data copying, which may be limited by PCIe bus bandwidth.
[0075] This embodiment provides a data transmission method applied to the second central processing unit (CPU) of a second computing server. Figure 4 is a flowchart (II) of the data transmission method according to an embodiment of this disclosure. As shown in Figure 4, the process includes the following steps:
[0076] Step S402: Detect the status of the first video memory and the second video memory, wherein the first video memory is allocated to the first computing server and the second video memory is allocated to the second computing server.
[0077] For example, the second CPU of the second computing server can detect the status of the first and second video memory, which can be in the form of status codes, field combinations, bitmaps, etc.
[0078] For example, the states of the first and second video memory are represented by status codes. Taking three status codes as an example: 0 indicates the first video memory is idle, 1 indicates a write state, and 2 indicates a send state. Similarly, 0 indicates the first video memory is idle, 1 indicates a processing state, and 2 indicates a receiving state. Specifically, the write state indicates that when the first GPU executes a business computing task, it writes the output business computing data to the first video memory. The processing state indicates that the second GPU uses the business computing data received from the second video memory as input data to execute a business computing task, performing calculations on the received data.
[0079] For example, this service includes training and inference operations for deep learning networks such as large and small models. A computational task refers to the computational function executed on the GPU. The send and receive states refer to the states of the first and second GPU memory when the CPU executes a communication task, which is a communication function executed on the CPU.
[0080] In one exemplary embodiment, prior to step S402, the following is included:
[0081] The size of the second video memory is determined based on the attributes of the business, so that the second GPU can allocate the second video memory according to the size of the second video memory.
[0082] Register the second video memory to the second RDMA network card;
[0083] Allocate a second page of locked video memory, where the second page of locked video memory is used to store the states of the first and second video memory;
[0084] Initialize the second video memory to an idle state;
[0085] The status and configuration information of the second video memory are sent to the first CPU of the first computing server.
[0086] For example, the second video memory can be registered as a local read / write or remote write mode. Taking the status of the first and second video memory as an example, the status code of the second video memory can be initialized to 0.
[0087] For example, determining the size of the second video memory based on the attributes of the business is to determine the size of the second video memory required when the second GPU performs business computing tasks.
[0088] For example, business computing tasks may include training tasks, inference tasks, etc. Training tasks include forward propagation training tasks and backward propagation training tasks, and inference tasks include forward propagation inference tasks. When the second GPU performs a forward propagation large model training task or inference task, the size of the first GPU memory can be determined based on the attributes of the large model and the context size of the large model. When the second GPU performs a backward propagation large model training task, the size of the first GPU memory can be determined based on the attributes of the large model.
[0089] For example, when the second GPU performs a large model training or inference task using forward propagation, the size of the second GPU memory can be determined based on factors such as the number of hidden layers and the upper limit of the training character length. When the second GPU performs a large model training task using backward propagation, the size of the second GPU memory can be determined based on factors such as the weight values of the large model.
[0090] It should be noted that the steps before step S402 can be understood as the initialization phase. This initialization only needs to be performed once when the computing server is powered on, and there is no need to perform the initialization operation every time data is sent.
[0091] In one exemplary embodiment, the configuration information includes address information and key-value information.
[0092] For example, configuration information can be initialized based on configuration items.
[0093] In step S404, in response to the state of the first video memory and the second video memory meeting the preset conditions, the business computing data written by the first graphics processor GPU of the first computing server into the first video memory is directly received into the second video memory.
[0094] For example, the second CPU of the second computing server can directly receive the business computing data in the first video memory into the second video memory based on the state of the first and second video memory. Compared with the solutions in related technologies, this avoids the kernel function participating in communication and data copying operations, eliminates the need for the GPU to participate in data transmission, frees up GPU computing resources from the communication process, and significantly improves the efficiency of large model training and push services.
[0095] In one exemplary embodiment, step S404 includes:
[0096] In response to the fact that there is business calculation data in the first video memory that needs to be sent, the state of the second video memory is detected to be receiving and the state of the first video memory is detected to be sending.
[0097] The first RDMA network card and the second RDMA network card are used to directly receive the business computing data in the first video memory into the second video memory. The first RDMA network card is configured in the first computing server and the second RDMA network card is configured in the second computing server. The first RDMA network card and the second RDMA network card are connected to each other.
[0098] For example, taking the status of the first and second video memory as a status code, when there is business calculation data in the first video memory that needs to be sent, the second CPU of the second computing server can change the status code of the second video memory to 2, indicating that the second computing server (receiver) is ready to receive data. If the status code of the first video memory is detected to be 2, it means that the first computing server (sender) is ready to send data. If the status code of the first video memory is not detected to be 2, it means that the first computing server (sender) is not ready to send data, which may be due to being busy with business processing, the data not being ready, a problem with the video memory status update mechanism, resources being competed for by multiple tasks, or system configuration errors or hardware failures.
[0099] In one exemplary embodiment, after step S404, the following is included:
[0100] In response to the second GPU of the second computing server needing to perform business calculations based on the received business calculation data, the state of the second video memory is changed from the receiving state to the processing state, wherein the second video memory overlaps with the video memory of the second GPU performing business calculations;
[0101] In response to the second GPU completing the business calculation, the state of the second video memory is changed from the processing state to the idle state.
[0102] For example, taking the status of the first and second video memory as a status code, after receiving business computing data, the second GPU of the second computing server can execute the corresponding kernel function to process the received data, the second CPU changes the status code of the second video memory from 2 to 1, and after processing, changes the status code of the second video memory from 1 to 0.
[0103] For example, taking the state of the second video memory as including idle, processing, and receiving states, when the second GPU of the second computing server needs to perform business calculations based on the received business calculation data, the second CPU can change the state of the second video memory from the receiving state to the processing state. Based on the switching of the state of the second video memory, the space of the input data for the second GPU to perform the calculation task can overlap with the space of the second CPU to perform the communication task, so as to realize the reuse of the second video memory.
[0104] For example, when the second GPU performs a large model training or inference task via forward propagation, the GPU memory space for the input data can overlap with the space where the second CPU performs the communication task. This input data can be received large model computation data, such as activation tensors, bias parameters, intermediate results of the loss function, etc. Similarly, when the first GPU performs a large model training task via backward propagation, the GPU memory space for the input data can overlap with the space where the second CPU performs the communication task. This input data can be received large model computation data, such as gradients, learning rates, regularization parameters, etc.
[0105] By following the steps described above, data communication is completely offloaded to the host CPU, reusing GPU memory space and avoiding data copying and kernel function involvement in communication. This frees up GPU computing resources from the communication process, improves communication efficiency, and reduces the computational load of kernel functions. This addresses the problems in large-scale model training and push cluster communication where the NCCL communication interface performs data copying operations through kernel functions, significantly increasing GPU computing resource overhead, and the increased communication latency caused by GPU memory data copying, which may be limited by PCIe bus bandwidth.
[0106] In an exemplary embodiment, the business training and inference cluster typically consists of multiple computing servers, each containing multiple GPU computing devices. Only the GPU devices on a computing server that need to send and receive data externally require allocated communication memory; internal communication still uses traditional interfaces and does not require allocated communication memory. Different parallel networks require different distribution patterns. The data transmission method provided in the above embodiment can run on different parallel network architectures, taking pipelined parallel and model parallel network architectures as examples.
[0107] Figure 5 is a schematic diagram of a pipelined parallel network according to an embodiment of the present disclosure. As shown in Figure 5, the communication and memory interaction of the pipeline cluster in the pipelined parallel network is as shown in Figure 5. Both computing server 0 and computing server 1 contain 3 GPUs (taking 3 GPUs as an example, it can also be 1 GPU, 2 GPUs, etc.). Computing server 0 has no upstream node and does not need a receive buffer, so the first memory is a send buffer (SendBuf), and only send memory is allocated. Computing server 1 is a node in the middle of the pipeline and needs to allocate both receive memory and send memory. Therefore, the second memory includes a send buffer (SendBuf) and a receive buffer (RecvBuf). The business computing data in the SendBuf of computing server 0 is directly sent to the RecvBuf of computing server 1 through RDMA. Computing server 1 can directly send the business computing data in the SendBuf to other computing servers (e.g., computing server 2) through RDMA.
[0108] Figure 6 is a schematic diagram of a model parallel network according to an embodiment of the present disclosure. As shown in Figure 6, in the model parallel network, there are communication methods such as allgather and scatter in the model parallel cluster. The communication memory is implemented by gathering. Computing server 0, computing server 1 and computing server 2 each contain 3 GPUs (taking 3 GPUs as an example, it can also be 1 GPU, 2 GPUs, etc.). Computing server 0 and computing server 1 have no upstream nodes and do not need to receive buffers. Therefore, the first memory is a send buffer (SendBuf), and only send memory is allocated. Computing server 2 is a node in the middle of the pipeline and needs to be allocated both receive memory and send memory. Therefore, the second memory includes a send buffer (SendBuf) and a receive buffer (RecvBuf). The business computing data in the SendBuf of computing server 0 and computing server 1 are sent directly to the RecvBuf of computing server 2 via RDMA. Specifically, the business computing data sent by computing server 0 is sent directly to the first RecvBuf of computing server 2, and the business computing data sent by computing server 1 is sent directly to the second RecvBuf of computing server 2. Computing server 2 can use RDMA to send the business computing data in the SendBuf directly to other computing servers (such as computing server 3).
[0109] In an exemplary embodiment, taking a large model training task as an example, the process is divided into forward propagation and backward propagation. During forward propagation, activation tensors are passed between nodes, while during backward propagation, gradient tensors are passed. The activation tensors and the weights of the large model have different sizes and lifecycles. This embodiment designs different reuse strategies for the communication memory (first and second memory) for forward and backward propagation. The activation tensor in the forward propagation process is related to the attributes and context size of the large model; therefore, the communication memory space for forward propagation needs to be determined according to the upper limit of the training character length. The gradient tensor in backward propagation is related to the attributes of the large model (e.g., the weights of the large model); therefore, a fixed memory space can be pre-allocated to accelerate RDMA communication. To avoid idle memory space, this embodiment overlaps the communication memory space with the actual forward and backward propagation spaces, achieving complete reuse of the communication memory.
[0110] Figure 7 is a schematic diagram of memory reuse for a large model training task according to an embodiment of this disclosure. As shown in Figure 7, F_In is the forward propagation input, B_In is the backward propagation input, FR is the forward propagation receive buffer, FS is the forward propagation send buffer, BR is the backward propagation receive buffer, BS is the backward propagation send buffer, Act is the activation tensor, and Grad is the gradient. A computing server (computing node) in a pipelined parallel cluster contains 8 GPUs (taking 8 GPUs as an example, it can also be 1 GPU, 4 GPUs, etc.). The memory space in the gray area of Figure 7 does not need to participate in network transmission and can be dynamically allocated and released. During training, four types of memory need to participate in network communication: forward propagation input F_In, calculated activation tensor value Act, backward propagation input B_In, and calculated gradient Grad.
[0111] As shown in Figure 7, GPU 1 is allocated memory space, namely FR (Forward RecvBuf) and BS (Backward SendBuf). During the forward propagation phase, FR and F_In can reuse the same space. During data transmission, this space serves as FR to receive data. After communication is completed, this space serves as F_In to participate in forward computation. Each GPU needs to complete multiple layers of computation. After F_In completes the first layer of computation, it can be used as communication memory to receive subsequent activation data. Therefore, it can effectively support parallel computation and communication algorithms. The FR space is pre-allocated relatively large. The F_In activation tensor is related to the actual input. When FR > F_In, the current GPU multi-layer activation tensor values can be stored in the FR space, improving memory utilization efficiency. During the backward propagation phase, BS and Grad reuse the same space. During backward computation, this space stores the latest Grad gradient. After the gradient calculation is completed, this space serves as the BS buffer to send the newly generated gradient tensor to the network.
[0112] Similarly, graphics card #8 is allocated video memory space, namely FS (Forward SendBuf) and BR (Backward RecvBuf). During the forward propagation phase, FS and Act (activation tensor) share video memory space, while during the backward propagation phase, BR and Grad (gradient data) share the same memory space.
[0113] Therefore, according to this embodiment, the communication memory neither increases memory usage nor changes the memory reuse strategy for forward and backward propagation. If only a unidirectional pipeline is implemented, the forward propagation process can directly use the backward communication memory to store temporary data, and the backward propagation process can also use the forward communication memory to store temporary data.
[0114] In an exemplary embodiment, taking a large model inference task as an example, this process only involves forward propagation and does not require backward propagation. The inference process only needs to allocate memory for forward propagation, namely FS (Forward SendBuf) and FR (Forward RecvBuf). Similar to the process of a large model training task, the FR space shares the same space as the input activation data, and the FS space shares the same space as the sent activation data. This will not be elaborated further in this embodiment.
[0115] Figure 8 is a schematic diagram of memory reuse for large model inference tasks according to an embodiment of this disclosure. As shown in Figure 8, F_In is the forward propagation input, FR is the forward propagation receive buffer, FS is the forward propagation send buffer, and Act is the activation tensor. In pipelined parallel mode, micro-batch granular pipeline parallelism is usually adopted. In this embodiment, the allocation and management of communication memory are managed at the micro-batch granularity, and different batch requests do not interfere with each other.
[0116] Through the above embodiments, the pre-allocation of fixed-address communication memory (first and second memory) avoids data copying during training and push. The communication memory reuse strategy can solve the problem of low memory resource utilization that may result from pre-allocated memory space, and ensure high utilization of communication memory in different business scenarios.
[0117] In an exemplary embodiment, FIG9 is a flowchart (III) of a data transmission method according to an embodiment of the present disclosure. As shown in FIG9, S represents the sending video memory space (the first video memory of the first computing server), and R represents the receiving cache space (the second video memory of the second computing server). Taking the state of the first and second video memory as represented by status codes, each video memory is allocated a uint64 type status bit. State 0 indicates that the space is idle, state 1 indicates that the space is occupied locally, and state 2 indicates that the space is occupied by the network. The migration of video memory status codes is shown in FIG10. FIG10 is a schematic diagram of the migration of video memory status codes according to an embodiment of the present disclosure. Taking the support of three state changes for each type of video memory as an example, the write state indicates that when the first GPU performs a business computing task, it writes the output business computing data to the first video memory. The processing state indicates that the second GPU uses the business computing data received from the second video memory as input data for performing a business computing task, performs calculation processing on the business computing data received from the second video memory, and performs a business computing task.
[0118] I. Configuration
[0119] For example, the configuration information of the status bits, first and second video memory can be configured according to the configuration file, such as the uint64 type status code, the address information of the first and second video memory (source address src, destination address dst), key value information, etc.
[0120] For example, when initializing the inference cluster based on the configuration file, a new configuration item can be added to specify which types of communication need to use the new optimized interface, such as:
[0121] PP_P2P_usingNewApi:True# The P2P interface in the pp parallel cluster uses an optimized interface.
[0122] When pipeline parallelism is used, a newly designed optimized interface is used. Similarly, a high-performance interface with optimized P2P communication configuration can be used for Sequence Parallel (SP) and Context Parallel (CP) modes.
[0123] II. Initialization
[0124] For example, communication video memory (first and second video memory) and status flag fields can be allocated and registered according to the configuration items in the configuration file.
[0125] For example, when initializing the cluster for a large model training and push task, the size of the communication memory (first and second memory) is determined based on the attributes and context size of the large model. For instance, the size of the activation tensor value transmitted between nodes in a pipelined parallel cluster can be calculated using the following formula: ActivationSize = MaxSequenceLength × HiddenSize × MicroBatchSize × Sizeof(fp16)
[0126] The sequence length is the maximum sequence length supported by the model. HiddenSize is the size of the hidden layer of the model. MicroBatchSize is the number of concurrent requests per batch. Two communication memory segments of size ActivationSize are allocated to each MicroBatch, designated as SendBuf and RecvBuf respectively. In PP parallel scenarios, multiple MicroBatches are usually computed in parallel, so multiple sets of communication memory need to be allocated. Sizeof(type) is the data size (e.g., the context size of a large model). fp16 is a data type, but it can also be fp4, fp8, etc., which are not limited in this embodiment.
[0127] After the first and second GPUs allocate the first and second video memory according to their respective sizes, the first and second CPUs can register the first and second video memory to the first and second RDMA network cards, respectively. Specifically, the first video memory can be registered in local read / write mode for sending data to remote nodes, and the second video memory can be registered in local read / write and remote write mode for receiving data from remote nodes.
[0128] The first and second CPUs can use interfaces such as cudaMallocHost and cuMemMap to allocate page-locked video memory, avoiding physical address remapping and impacting read / write performance. This page-locked video memory is used to store the states of the first and second video memory. The first and second CPUs can initialize the status codes of the first and second video memory to 0, indicating that the video memory is in an idle state. The first and second CPUs can notify each other of the video memory status and configuration information to synchronize the initialization state (i.e., execute the synchronization initialization state in Figure 9). The notification method can be ordinary TCP network notification or through the already initialized NCCL network. After receiving the address information, the receiving communication task thread maintains the RecvBuf address information of all downstream nodes through an address table.
[0129] The first GPU executes the kernel function to perform large model training and inference tasks, and writes the large model calculation data into the first video memory. The first CPU modifies the status code of the first video memory to 1, indicating the write status.
[0130] III. Communication Process
[0131] 1. The receiver executes Recv.
[0132] In this embodiment, the receiver (second computing power server) has pre-allocated recvBuf (second video memory) and notified the sender (first computing power server). The second CPU changes the status code of the second video memory to 2, announcing that the local recvBuf is available.
[0133] 2. The sender executes Send.
[0134] After the business thread of the first CPU executes the Send interface, it sets the status code of the first video memory to 2 and notifies the communication task thread of the receiver's address information, data offset address, data length, and other information.
[0135] 3. Check the status of the second video memory and write data.
[0136] The communication task of the first CPU finds the RecvBuf (second video memory) address information based on the target node, obtains and checks the status code Rstat value of the second video memory through the RDMA Read interface (i.e., executes Fetch&Check in Figure 9). If Rstat is not 2, it means that the receiver is not ready to receive data, which may be due to uneven service distribution, resulting in slow processing by the receiver. If Rstat = 2, it means that the receiver is ready, and then executes the send notification communication task to send data. The two communicating parties realize data transmission through GDR technology and RDMA network.
[0137] The first CPU can instruct the first RDMA network card to send an RDMA Write remote write request to the second RDMA network card, so that the first RDMA network card can directly write the business calculation data into the second video memory. Multi-threading and multi-link concurrent transmission can be used to improve transmission efficiency.
[0138] 4. Communication completed
[0139] The first CPU immediately modifies the first video memory Rstat=0, enters an idle state, and can be used for subsequent computing tasks. The second CPU modifies Rstat=1, executes the corresponding kernel function to process the received data, and uses it for subsequent business calculations. After processing, it modifies Rstat=0.
[0140] Through the above embodiments, communication tasks and computing tasks share data, reducing the calls to data copying functions. Compared with the communication process in related technologies, the GPU is freed from the communication process, improving the utilization of computing resources. By using state transition, the access to the buffer by the local communication task, computing task, and remote communication task is coordinated. The state of the receiving end's video memory is detected by RDMA instructions, realizing the synchronous transition of the send and receive buffer states and avoiding read and write conflicts.
[0141] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of this disclosure, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this disclosure.
[0142] Embodiments of this disclosure also provide a computer-readable storage medium storing a computer program configured to perform the steps in any of the above method embodiments when executed.
[0143] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0144] Embodiments of this disclosure also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.
[0145] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.
[0146] Specific examples in this embodiment can be found in the examples described in the above embodiments and exemplary implementations, and will not be repeated here.
[0147] It is obvious to those skilled in the art that the modules or steps of this disclosure described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this disclosure is not limited to any particular combination of hardware and software.
[0148] The above are merely preferred embodiments of this disclosure and are not intended to limit this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A data transmission method, applied to a first central processing unit (CPU) of a first computing server, comprising: The status of the first video memory and the second video memory is detected, wherein the first video memory is allocated on the first computing server and the second video memory is allocated on the second computing server; In response to the fact that the states of the first video memory and the second video memory meet preset conditions, the business computing data written by the first graphics processor GPU of the first computing server into the first video memory is directly sent to the second video memory.
2. The method according to claim 1, wherein, In response to the conditions being met by the first and second video memory, the business computing data written by the first GPU to the first video memory is directly sent to the second video memory, including: In response to the first GPU performing business calculations and writing the business calculation data into the first video memory, the state of the first video memory is changed from the write state to the send state; In response to the need to send the service calculation data in the first video memory, the state of the first video memory is detected to be in the sending state and the state of the second video memory is detected to be in the receiving state. The first RDMA network card is directly sent to the second video memory by directly accessing the first remote memory RDMA network card and the second RDMA network card. The first RDMA network card is configured in the first computing server and the second RDMA network card is configured in the second computing server. The first RDMA network card and the second RDMA network card are connected to each other.
3. The method according to claim 2, wherein, Sending the service calculation data in the first video memory directly to the second video memory via the first RDMA network card and the second RDMA network card includes: The first RDMA network card is instructed to send an RDMA write request to the second RDMA network card, so that the first RDMA network card can directly write the service calculation data into the second video memory.
4. The method according to claim 2, wherein, After the service calculation data in the first video memory is directly sent to the second video memory via the first RDMA network card and the second RDMA network card, the process includes: In response to the first RDMA network card completing the direct writing of the service calculation data, the state of the first video memory is changed from the sending state to the idle state.
5. The method according to claim 1, wherein, Before detecting the status of the first and second video memory, the following is included: The size of the first video memory is determined according to the attributes of the business, so that the first GPU can allocate the first video memory according to the size of the first video memory. Register the first video memory to the first RDMA network card; Allocate a first page-locked video memory, wherein the first page-locked video memory is used to store the states of the first video memory and the second video memory; Initialize the state of the first video memory to an idle state; Send the status and configuration information of the first video memory to the second CPU of the second computing power server; In response to the first GPU needing to write the business computing data into the first video memory, the state of the first video memory is changed from idle state to write state.
6. The method according to claim 5, wherein, The configuration information includes address information and key-value information.
7. A data transmission method, applied to the second central processing unit (CPU) of a second computing server, comprising: The status of the first video memory and the second video memory is detected, wherein the first video memory is allocated on the first computing server and the second video memory is allocated on the second computing server; In response to the fact that the states of the first video memory and the second video memory meet preset conditions, the business computing data written by the first graphics processor (GPU) of the first computing server into the first video memory is directly received into the second video memory.
8. The method according to claim 7, wherein, In response to the conditions being met by the first and second video memory, the business computing data written by the first graphics processor (GPU) of the first computing server into the first video memory is directly received into the second video memory, including: In response to the need to send the business computing data in the first video memory, the system detects that the second video memory is in the receiving state and the first video memory is in the sending state. The first RDMA network card is directly received into the second video memory by directly accessing the first remote memory RDMA network card and the second RDMA network card. The first RDMA network card is configured in the first computing server, the second RDMA network card is configured in the second computing server, and the first RDMA network card and the second RDMA network card are connected to each other.
9. The method according to claim 7, wherein, After the business computing data written by the first graphics processor (GPU) of the first computing server into the first video memory is directly received into the second video memory in response to the preset conditions of the states of the first video memory and the second video memory, the process includes: In response to the second GPU of the second computing server needing to perform business calculations based on the received business calculation data, the state of the second video memory is changed from the receiving state to the processing state, wherein the second video memory overlaps with the video memory of the second GPU performing business calculations; In response to the second GPU completing the business calculation, the state of the second video memory is changed from the processing state to the idle state.
10. The method according to claim 7, wherein, Before detecting the status of the first and second video memory, the following is included: The size of the second video memory is determined according to the attributes of the business, so that the second GPU can allocate the second video memory according to the size of the second video memory. Register the second video memory to the second RDMA network card; Allocate a second page-locked video memory, wherein the second page-locked video memory is used to store the states of the first video memory and the second video memory; Initialize the state of the second video memory to an idle state; The status and configuration information of the second video memory are sent to the first CPU of the first computing server.
11. The method according to claim 10, wherein, The configuration information includes address information and key-value information.
12. A computer-readable storage medium storing a computer program, wherein, When the computer program is executed by a processor, it implements the steps of the method described in any one of claims 1 to 6, or the steps of the method described in any one of claims 7 to 11.
13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 6, or to implement the steps of the method according to any one of claims 7 to 11.
14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6, or implements the steps of the method according to any one of claims 7 to 11.
