Data transmission method based on remote direct memory access, and related device

Abstract

Disclosed in the present application are an RDMA-based data transmission method and a related device, which can reduce, to a certain extent, the time cost and performance overhead required in the overall data transmission process. The method in the present application comprises: a first processor providing a memory block to a first RDMA network interface card, wherein the memory block comprises a plurality of sub-memories, including a first sub-memory and a second sub-memory; after a second RDMA network interface card sends a packet to the first RDMA network interface card, which packet carries first data generated by a second processor, the first RDMA network interface card generating a first address for the first data; then, the first RDMA network interface card storing the first address in the first sub-memory and storing the first data in the second sub-memory, wherein the first address is used for indicating the location of the first data in the second sub-memory; and subsequently, the first RDMA network interface card notifying the first processor to read the first address from the first sub-memory, so as to read the first data from the second sub-memory on the basis of the first address, thereby realizing data transmission.

Description

A data transfer method and related equipment based on remote direct memory access

[0001] This application claims priority to Chinese Patent Application No. 202411570006.2, filed on November 5, 2024, entitled "A Data Transmission Method Based on Remote Direct Memory Access and Related Devices thereof", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to a data transmission method and related equipment based on remote direct memory access (RDMA). Background Technology

[0003] With the rapid development of computer technology, RDMA technology has been widely used in communication between physical servers. In this communication process, the physical server that initiates the access is called the sender, and the physical server being accessed is called the receiver. The sender can directly access the memory on the receiver's side, thereby directly writing data to the receiver's memory.

[0004] In related technologies, when a sending end needs to transmit data to a receiving end, it can notify the receiving end of the address information of this data. This address information describes the location of the data in the sending end's memory. Next, the receiving end can initiate a data read request to the sending end, carrying this address information. Then, based on the address information carried in the request, the sending end can read this data from its own memory and remotely write it to the receiving end's memory, enabling the receiving end to read this data from its own memory, thereby completing the data transmission.

[0005] In the process described above, both the sending and receiving ends require a large number of operations to achieve data transmission, resulting in excessively high time costs and performance overhead for the entire data transmission process. Summary of the Invention

[0006] This application provides an RDMA-based data transmission method and related equipment, which can reduce the time cost and performance overhead of the entire data transmission process to a certain extent.

[0007] The first aspect of this application provides a data transmission method based on Remote Direct Memory Access (RDMA). This method can be implemented using an RDMA system, which may include a first server and a second server. The first server may include a first RDMA network interface card (NIC) and a first processor, and the second server may include a second RDMA NIC and a second processor. The method includes:

[0008] Before the first server and the second server transmit data, the first processor of the first server can provide a memory block to the first RDMA network card of the first server. This memory block can contain multiple sub-memories, such as a first sub-memory and a second sub-memory.

[0009] When the second processor of the second server needs to send the first data to the first processor, the second processor can provide the first data to the second RDMA network card of the second server. Therefore, the second RDMA network card can generate a first message carrying the first data and send the first message to the first RDMA network card.

[0010] Upon receiving the first message, the first RDMA network card can parse the first message to obtain the first data. Then, the first RDMA network card can generate a first address for the first data, which can be used to describe the location of the first data stored in the second sub-memory. After obtaining the first data and the first address, the first RDMA network card can store the first address in the first sub-memory and store the first data in the second sub-memory.

[0011] After storing the first data and the first address, the first RDMA network card can instruct the first processor to read the first address from the first sub-memory, and then, based on the first address, read the first data from the second sub-memory. At this point, the first data transfer between the first processor and the second processor is successfully completed.

[0012] As can be seen from the above method: the first processor of the first server can provide a memory block for the first RDMA network card of the first server. This memory block includes multiple sub-memories, such as a first sub-memory and a second sub-memory. After the second RDMA network card of the second server sends a first message to the first RDMA network card of the first server, since the first message carries the first data generated by the second processor of the second server, the first RDMA network card of the first server can request the first sub-memory and the second sub-memory from the memory block and generate the first address of the first data. Then, the first RDMA network card of the first server can store the first address in the first sub-memory and store the first data in the second sub-memory. The first address is used to describe the location of the first data in the second sub-memory. Subsequently, the first RDMA network card of the first server can notify the first processor of the first server to read the first address from the first sub-memory, so as to read the first data from the second sub-memory based on the first address, thereby realizing the data transmission between the first server and the second server. In the aforementioned data transmission process, since the first server as the receiving end and the second server as the sending end only need to complete one communication for the message, and the first server completes the message parsing, data storage and data reading on its own, the number of communication between the first server and the second server during the data transmission process and the processing operations of each end can be reduced to a certain extent, thereby reducing the time cost and performance overhead required for the entire data transmission process to a certain extent.

[0013] In one possible implementation, the method further includes: the first RDMA network card dividing the memory block into multiple sub-memories, each sub-memory being the same size as a preset size, and each sub-memory containing a first sub-memory and a second sub-memory. In the aforementioned implementation, after receiving the memory block provided by the first processor, the first RDMA network card can logically divide the memory block into multiple sub-memories (of course, the memory block can also be divided by the first processor to obtain multiple sub-memories, and the processor will notify the first RDMA network card of the division result so that the first RDMA network card can subsequently use the multiple sub-memories obtained from the memory block, etc.). These multiple sub-memories can include a first sub-memory, a second sub-memory, and a third sub-memory, etc., and each sub-memory is the same size as a preset size. Therefore, in the first server acting as the receiving end, the first processor and the first RDMA network card can pre-allocate sub-memories of a certain size for subsequently received first data, second data, etc., to provide sufficient storage space for these data, thereby successfully achieving data transmission.

[0014] In one possible implementation, the size of the first address is smaller than the size of the first sub-memory, and the size of the first data is smaller than or equal to the size of the second sub-memory. In the aforementioned implementation, the size of the first address is typically smaller than the size of the first sub-memory, and the size of the first data is typically smaller than or equal to the size of the second sub-memory. Therefore, the first address can be stored in the first sub-memory, and the first data can be stored in the second sub-memory. Thus, the second processor can control the size of the first data it generates to match the size of the second sub-memory provided by the first processor. This eliminates the need for the first processor to reserve a specific size of sub-memory based on the size of the first data on the second processor side for data transfer. Consequently, data between the second and first processors can be transferred using send / recv semantics, offering greater flexibility and practicality.

[0015] In one possible implementation, the memory block further includes a third sub-memory. The method further includes: a first RDMA network card receiving a second message sent by a second server, the second message carrying second data generated by the second server; the first RDMA network card generating a second address for the second data carried in the second message, the second address indicating the location of the first data in the third sub-memory; the first RDMA network card storing the second address in the first sub-memory and storing the second data in the third sub-memory; the first RDMA network card notifying the first processor to retrieve the second address from the first sub-memory and retrieve the second data from the third sub-memory based on the second address. In the aforementioned implementation, when the second processor of the second server needs to send second data to the first processor, the second processor can provide the second data to the second RDMA network card of the second server. Therefore, the second RDMA network card can generate a second message carrying the second data and send the first message to the first RDMA network card. After receiving the second message, the first RDMA network card can parse the second message to obtain the second data. Then, the first RDMA network card can generate a second address for the second data, which can be used to describe the location of the second data stored in the third sub-memory. After obtaining the second data and the second address, the first RDMA network card can store the second address in the first sub-memory and the second data in the third sub-memory. After storing the second data and the second address, the first RDMA network card can instruct the first processor to read the second address from the first sub-memory, and then, based on the second address, read the second data from the third sub-memory. At this point, the transfer of the second data between the first and second processors is successfully completed.

[0016] In one possible implementation, the positions of the first address and the second address in the first sub-memory are adjacent. In the aforementioned implementation, since the first data and the second data are two data sets sent sequentially by the second processor to the first processor, the positions of the first address and the second address in the first sub-memory are usually adjacent. Therefore, when the first processor reads an address from the first sub-memory, it can read the first address and the second address consecutively, and then read the first data and the second data sequentially based on the first address and the second address. Thus, the order of the data obtained by the first processor is consistent with the order of the data to be sent by the second processor, which is beneficial for achieving data sorting consistency.

[0017] In one possible implementation, the size of the second address is smaller than the size of the first sub-memory, and the size of the second data is smaller than or equal to the size of the third sub-memory. In the aforementioned implementation, the size of the second address is typically smaller than the size of the first sub-memory, and the size of the second data is typically smaller than or equal to the size of the third sub-memory. Therefore, the second address can be stored in the first sub-memory, and the second data can be stored in the third sub-memory. This demonstrates that the second processor can control the size of the second data it generates to match the size of the third sub-memory provided by the first processor. The first processor does not need to reserve a specific size of sub-memory based on the size of the second data on the second processor side for data transfer. This allows data between the second and first processors to be transferred using send / recv semantics, providing greater flexibility and practicality.

[0018] In one possible implementation, the first and second data are obtained by the second server from the division of the third data. The size of the third data is larger than the size of the second sub-memory, and the size of the third data is also larger than the size of the third sub-memory. In the aforementioned implementation, the first and second data can originate from the third data obtained by the second processor from the division of its generated business data. Since the size of the third data is larger than both the size of the second and third sub-memory, the third data cannot be stored in the second or third sub-memory. Therefore, the second processor will divide the third data into the first and second data, such that the sizes of the first and second data are both less than or equal to the sizes of the second and third sub-memory, respectively. Then, it will notify the second RDMA network card to send the first and second data to the first RDMA network card sequentially. In this way, the first RDMA network card can successfully store the first and second data in the second and third sub-memory respectively, which is beneficial for the first processor to successfully read the first and second data subsequently.

[0019] A second aspect of this application provides a server, which serves as a first server. The first server includes a first RDMA network interface card (NIC) and a first processor. The first processor is configured to provide a memory block to the first RDMA NIC, the memory block including a first sub-memory and a second sub-memory. The first RDMA NIC is configured to receive a first message sent by a second server, the first message carrying first data generated by the second server. The first RDMA NIC is further configured to generate a first address of the first data carried by the first message, the first address indicating the location of the first data in the second sub-memory. The first RDMA NIC is further configured to store the first address in the first sub-memory and store the first data in the second sub-memory. The first RDMA NIC is further configured to notify the first processor to obtain the first address from the first sub-memory and obtain the first data from the second sub-memory based on the first address.

[0020] In one possible implementation, the first RDMA network card is also used to divide the memory block into multiple sub-memories, each sub-memory having a size equal to a preset size, and each sub-memory containing a first sub-memory and a second sub-memory.

[0021] In one possible implementation, the size of the first address is less than the size of the first sub-memory, and the size of the first data is less than or equal to the size of the second sub-memory.

[0022] In one possible implementation, the memory block further includes a third sub-memory; a first RDMA network card is also used to receive a second message sent by a second server, the second message carrying second data generated by the second server; the first RDMA network card is also used to generate a second address of the second data carried by the second message, the second address indicating the location of the first data in the third sub-memory; the first RDMA network card is also used to store the second address in the first sub-memory and store the second data in the third sub-memory; the first RDMA network card is also used to notify the first processor to obtain the second address from the first sub-memory and to obtain the second data from the third sub-memory based on the second address.

[0023] In one possible implementation, the location of the first address in the first sub-memory is adjacent to the location of the second address in the second sub-memory.

[0024] In one possible implementation, the size of the second address is less than the size of the first sub-memory, and the size of the second data is less than or equal to the size of the third sub-memory.

[0025] In one possible implementation, the first data and the second data are obtained by the second server dividing the third data, and the size of the third data is greater than the size of the second sub-memory, and the size of the third data is greater than the size of the third sub-memory.

[0026] A third aspect of this application provides an RDMA system, which includes a second server and a first server as described in the second aspect or any possible implementation thereof.

[0027] A fourth aspect of this application provides a computing device cluster, the computing device cluster including at least one computing device, each computing device including a processor and a memory: the memory is used to store instructions; the processor is used to cause the computing device cluster to perform the method described in the first aspect or any possible implementation of the first aspect according to the instructions.

[0028] A fifth aspect of this application provides a computer storage medium storing one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method described in the first aspect or any possible implementation of the first aspect.

[0029] A sixth aspect of this application provides a computer program product storing instructions that, when executed by a computer, cause the computer to perform the method described in the first aspect or any possible implementation of the first aspect.

[0030] In this embodiment, the first processor of the first server can provide a memory block for the first RDMA network card of the first server. This memory block includes multiple sub-memories, such as a first sub-memory and a second sub-memory. After the second RDMA network card of the second server sends a first message to the first RDMA network card of the first server, since the first message carries the first data generated by the second processor of the second server, the first RDMA network card of the first server can request the first sub-memory and the second sub-memory from the memory block and generate a first address for the first data. Then, the first RDMA network card of the first server can store the first address in the first sub-memory and store the first data in the second sub-memory. The first address is used to describe the location of the first data in the second sub-memory. Subsequently, the first RDMA network card of the first server can notify the first processor of the first server to read the first address from the first sub-memory, so as to read the first data from the second sub-memory based on the first address, thereby realizing data transmission between the first server and the second server. In the aforementioned data transmission process, since the first server as the receiving end and the second server as the sending end only need to complete one communication for the message, and the first server completes the message parsing, data storage and data reading on its own, the number of communication between the first server and the second server during the data transmission process and the processing operations of each end can be reduced to a certain extent, thereby reducing the time cost and performance overhead required for the entire data transmission process to a certain extent. Attached Figure Description

[0031] Figure 1 is a schematic diagram of an RDMA system provided in an embodiment of this application;

[0032] Figure 2 is a flowchart illustrating a data transmission method based on RDMA provided in an embodiment of this application;

[0033] Figure 3 is a schematic diagram of an application example of the data transmission method based on RDMA provided in the embodiments of this application;

[0034] Figure 4 is a schematic diagram of a server provided in an embodiment of this application;

[0035] Figure 5 is a schematic diagram of a computing device provided in an embodiment of this application;

[0036] Figure 6 is a schematic diagram of a computing device cluster provided in an embodiment of this application;

[0037] Figure 7 is a schematic diagram of computer devices in a computer cluster connected via a network according to an embodiment of this application. Detailed Implementation

[0038] This application provides an RDMA-based data transmission method and related equipment, which can reduce the time cost and performance overhead of the entire data transmission process to a certain extent.

[0039] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0040] With the rapid development of computer technology, RDMA technology has been widely used in communication between physical servers. In this communication process, the physical server that initiates the access is called the sender, and the physical server being accessed is called the receiver. The sender can directly access the memory on the receiver's side, thereby directly writing data to the receiver's memory.

[0041] In related technologies, when a sending end needs to transmit data to a receiving end, the sending end's processor can notify the receiving end of the address information of this data through its RDMA network card. This address information describes the location of the data in the sending end's processor's memory. Next, the receiving end's processor can initiate a data read request to the sending end through its RDMA network card, carrying this address information. Then, based on the address information carried in the request, the sending end's RDMA network card can read this data from the sending end's processor's memory and send it to the receiving end's RDMA network card. This allows the receiving end's RDMA network card to store this data in the receiving end's processor's memory and notify the receiving end's processor to read this data from memory, thus completing the data transmission.

[0042] In the process described above, both the sending and receiving ends require a large number of operations from their processors and RDMA network cards to achieve data transmission, resulting in excessively high time costs and performance overhead for the entire data transmission process.

[0043] To address the aforementioned issues, this application provides a data transmission method based on RDMA. This method can be implemented using an RDMA system. Figure 1 is a schematic diagram of the RDMA system provided in this application. As shown in Figure 1, the RDMA system includes multiple physical servers, each containing an RDMA network interface card (NIC) and a processor. The RDMA NICs of each physical server can be connected to a switch, thereby enabling communication connections between the RDMA NICs of multiple physical servers. Furthermore, the processors of each physical server can run one or more user-mode processes. These user-mode processes on the processors of different physical servers often need to communicate, and these communication processes often involve data transmission to meet the business processing requirements between these user-mode processes.

[0044] It should be noted that the processors of each physical server can be central processing units (CPUs), data processing units (DPUs), etc., without any specific restrictions here.

[0045] It should also be noted that the aforementioned RDMA system can be applied to various fields. For example, the RDMA system can be applied to the field of cloud technology, that is, the RDMA system can be a cloud service system in the field of cloud technology. Therefore, the physical servers in the RDMA system can serve as computing devices in the cloud service system. These computing devices can provide users with various cloud services (e.g., cloud computing, cloud storage, and cloud networking services). In the process of providing cloud services, these computing devices often need to communicate with each other, so they can also implement the data transmission method provided in the embodiments of this application. As another example, the RDMA system can also be applied to the field of artificial intelligence (AI), that is, the RDMA system can be a distributed training system in the field of AI. Therefore, the physical servers in the RDMA system can serve as training devices in a distributed model training system. These training devices can be used to execute user model training tasks (e.g., training tasks for a specific neural network model, etc.). In the process of task execution, these training devices often need to communicate with each other, so they can also implement the data transmission method provided in the embodiments of this application. Of course, the aforementioned RDMA system can also be applied to other fields, which will not be elaborated here.

[0046] In the aforementioned RDMA system, the two physical servers that need to communicate can be referred to as the sender and receiver, respectively. The receiver's processor provides a memory block for its RDMA network card, which contains multiple sub-memories. After the sender's RDMA network card sends a message to the receiver's RDMA network card, since the message carries data generated by the sender's processor, the receiver's RDMA network card can request two sub-memories from the memory block and generate the address of the data. Then, the receiver's RDMA network card can store this address in one of the two sub-memories and store the data in the other sub-memory, with the address describing the location of the data in the other sub-memory. Subsequently, the receiver's RDMA network card can instruct the receiver's processor to read the address from one sub-memory, and based on that address, read the data from the other sub-memory, thereby realizing data transmission between the sender and receiver. In the aforementioned data transmission process, since the sending end and receiving end only need to complete one communication for the message, and the receiving end handles message parsing, data storage, and data retrieval itself, the number of communications between the sending end and receiving end during data transmission, as well as the processing operations of each end, can be reduced to a certain extent, thereby reducing the time cost and performance overhead required for the entire data transmission process. To further understand the workflow of the aforementioned RDMA system, the following description, in conjunction with Figure 2, further illustrates this workflow. Figure 2 is a flowchart illustrating a data transmission method based on RDMA provided in this application embodiment. As shown in Figure 2, this method is implemented through the RDMA system shown in Figure 1. The RDMA system may include a physical server as the receiving end and a physical server as the sending end. For ease of explanation, the physical server as the receiving end will be referred to as the first server, and the physical server as the sending end will be referred to as the second server. The first server includes a first RDMA network card and a first processor, and the second server includes a second RDMA network card and a second processor. The method includes:

[0047] 201. The first processor provides a memory block to the first RDMA network card. The memory block includes a first sub-memory and a second sub-memory.

[0048] In this embodiment, before the first server and the second server transmit data, the first processor of the first server can provide a memory block to the first RDMA network card of the first server. It should be noted that the memory block is a part of the memory of the first processor, and the memory block can contain multiple sub-memories such as the first sub-memory and the second sub-memory.

[0049] Specifically, after the first processor provides the memory block to the first RDMA network card, the first RDMA network card can perform the following operations on the memory block:

[0050] After receiving the memory block provided by the first processor, the first RDMA network card can logically divide the memory block into multiple sub-memories. The size of each sub-memory is equal to a preset size (e.g., the size of each sub-memory is 4KB, etc.). Among these sub-memories, the sub-memory ranked first can be called the first sub-memory, the sub-memory ranked second can be called the second sub-memory, the sub-memory ranked third can be called the third sub-memory, and so on. They will not be described in detail here.

[0051] It should be noted that the first sub-memory can be used as the sub-memory for storing the address of the first RDMA network card, and the second sub-memory, the third sub-memory, and other sub-memories can be used as the sub-memory for storing data of the first RDMA network card.

[0052] For example, as shown in Figure 3 (Figure 3 is a schematic diagram of an application example of the data transmission method based on RDMA provided in the embodiments of this application), let physical server 1 be the receiving end and physical server 2 be the sending end. Physical server 1 includes RDMA network card 1 and processor 1, and physical server 2 includes RDMA network card 2 and processor 2. After the user-mode process 1 of processor 1 (which can also be called a running application on processor 1) is initialized, user-mode process 1 can entrust a memory block of 2GB (a contiguous physical memory) to RDMA network card 1. Therefore, RDMA network card 1 can divide the memory block into multiple sub-memories in 4KB units, including sub-memory 1, sub-memory 2, sub-memory 3, etc., each with a size of 4KB.

[0053] Among them, sub-memory 1 can be used as the sub-memory where the address of RDMA network card 1 exists, and sub-memory 2 and sub-memory 3 can be used as the sub-memory where the data of RDMA network card 1 exists.

[0054] It should be understood that in this embodiment, the division of the memory block by the first RDMA network card is only an illustrative operation. In actual applications, the first processor can also divide the memory block to obtain multiple sub-memories and inform the first RDMA network card of the division result of the memory block so that the first RDMA network card can subsequently use the multiple sub-memories obtained by dividing the memory block.

[0055] 202. The first RDMA network card receives the first message sent by the second server. The first message carries the first data generated by the second server.

[0056] When the second processor of the second server needs to send the first data generated by the second processor to the first processor, the second processor can provide the first data to the second RDMA network card of the second server. Therefore, the second RDMA network card can generate a first message carrying the first data and send the first message to the first RDMA network card.

[0057] Continuing with the example above, after user-space process 2 of processor 2 generates the entire service data, it can also divide the service data into multiple data segments, including data 1 (256 bytes), data 2 (4 KB), data 3 (4 KB), and so on. User-space process 2 can send each data segment to user-space process 1. First, user-space process 2 can send data 1 to user-space process 1. Therefore, user-space process 2 can provide data 1 to RDMA network card 2, and RDMA network card 2 can send data 1 as message 1 to RDMA network card 1.

[0058] 203. The first RDMA network card generates the first address of the first data carried in the first message. The first address is used to indicate the location of the first data in the second sub-memory.

[0059] Upon receiving the first message, the first RDMA network card can parse the first message to obtain the first data. Since the second sub-memory, third sub-memory, and other sub-memory are memory for storing data, the first RDMA network card can select the second sub-memory as the sub-memory for storing the first data. Therefore, the first RDMA network card can generate the first address of the first data. The first address can contain the starting position of the first data in the second sub-memory and the size (length) of the first data. Thus, the first address can be used to describe the location of the first data stored in the second sub-memory.

[0060] Specifically, the size of the first address is usually smaller than the size of the first sub-memory, and the size of the first data is usually smaller than or equal to the size of the second sub-memory. Therefore, the first address can be stored in the first sub-memory, and the first data can be stored in the second sub-memory.

[0061] As in the example above, after receiving message 1, RDMA network card 1 can parse message 1 to obtain data 1. Since sub-memory 2, sub-memory 3, etc. are used to store data, RDMA network card 1 can select sub-memory 2 as the sub-memory to store data 1, and generate information such as the starting position of data 1 in sub-memory 2 and the size of data 1 (256B). This information is the address 1 of data 1, which is used to indicate the position of data 1 in sub-memory 2.

[0062] It is worth noting that address 1 is typically 12 bytes in size, which is smaller than the size of sub-memory 1 (4KB). Data 1 is 256 bytes in size, which is smaller than the size of sub-memory 2 (4KB). Therefore, address 1 can be stored in sub-memory 1, and data 1 can be stored in sub-memory 2.

[0063] 204. The first RDMA network card stores the first address in the first sub-memory and stores the first data in the second sub-memory.

[0064] After obtaining the first data and the first address, the first RDMA network card can store the first address in the first sub-memory and store the first data in the second sub-memory through direct memory access (DMA).

[0065] As in the example above, after obtaining data 1 and address 1, RDMA network card 1 can store address 1 in sub-memory 1 and data 1 in sub-memory 2 via DMA.

[0066] It should be understood that in this embodiment, the first RDMA network card first generates a first address, and then stores the first address and the first data in the first sub-memory and the second sub-memory respectively for illustrative purposes. In actual applications, the first RDMA network card may also first store the first data in the second sub-memory, then generate the first address based on the position of the first data in the second sub-memory, and then store the first address in the first sub-memory. There are no specific restrictions on the order of these steps here.

[0067] 205. The first RDMA network card notifies the first processor to obtain the first address from the first sub-memory and obtain the first data from the second sub-memory based on the first address.

[0068] After storing the first data and the first address, the first RDMA network card can send a first data retrieval request to the first processor. This request indicates the address of the first address, which is located in the first sub-memory. The first processor can then parse the request, read the first address from the first sub-memory based on the first address, and then read the first data from the second sub-memory based on that address. Thus, the transfer of the first data between the first and second processors is successfully completed.

[0069] Continuing with the example above, after storing address 1 and data 1, RDMA network card 1 can send a data retrieval request to user-space process 1. User-space process 1, based on this request, reads address 1 from sub-memory 1, and then reads data 1 from sub-memory 2 based on address 1. In this way, data 1 is successfully transferred from user-space process 2 to user-space process 1.

[0070] Specifically, the second processor can also continue to send second data to the first processor in the following ways:

[0071] When the second processor of the second server needs to continue sending the second data generated by the second processor to the first processor, the second processor can provide the second data to the second RDMA network card of the second server. Therefore, the second RDMA network card can generate a second message carrying the second data and send the second message to the first RDMA network card.

[0072] Upon receiving the second message, the first RDMA network card can parse the second message to obtain the second data. Since the second sub-memory, third sub-memory, and other sub-memory are memory for storing data, the first RDMA network card can select the third sub-memory as the sub-memory for storing the second data. Therefore, the first RDMA network card can generate a second address for the second data. The second address can contain the starting position of the second data in the third sub-memory and the size of the second data. Thus, the second address can be used to describe the location of the second data stored in the third sub-memory.

[0073] After obtaining the second data and the second address, the first RDMA network card can store the second address in the first sub-memory and store the second data in the third sub-memory via DMA.

[0074] After storing the second data and the second address, the first RDMA network card can send a second data retrieval request to the first processor. This request indicates the address of the second address, which is located in the first sub-memory. The first processor can then parse the request, read the second address from the first sub-memory based on that address, and then read the second data from the third sub-memory based on that address. Thus, the transfer of the second data between the first and second processors is successfully completed.

[0075] It should be understood that the first data acquisition request and the second data acquisition request can be requests sent by the first RDMA network card to the first processor simultaneously, or requests sent by the first RDMA network card to the first processor sequentially; no restriction is imposed here.

[0076] As in the example above, after user-mode process 2 of processor 2 sends data 1 to user-mode process 1, user-mode process 2 can continue to send data 2 of size 4KB to user-mode process 1. Therefore, user-mode process 2 can provide data 2 to RDMA network card 2, so RDMA network card 2 can send data 2 as message 2 to RDMA network card 1.

[0077] After receiving message 2, RDMA network card 1 can parse message 2 to obtain data 2. Since sub-memory 2, sub-memory 3 and other sub-memory are used to store data, RDMA network card 1 can select sub-memory 3 as the sub-memory to store data 2, and generate information such as the starting position of data 2 in sub-memory 3 and the size of data 2 (4KB). This information is the address 2 of data 2, which is used to indicate the position of data 2 in sub-memory 3.

[0078] After obtaining data 2 and address 2, RDMA network card 1 can store address 2 in sub-memory 1 and data 2 in sub-memory 3 via DMA.

[0079] After storing address 2 and data 2, RDMA network card 1 can send a data retrieval request to user-space process 1, so that user-space process 1 can read address 2 from sub-memory 1 based on the data retrieval request, and then read data 2 from sub-memory 3 based on address 2. In this way, data 2 is successfully transferred from user-space process 2 to user-space process 1.

[0080] It should be noted that RDMA network card 1 can notify user-mode process 1 to read address 1 and address 2 from sub-memory 1 simultaneously with a single notification, thereby enabling user-mode process 1 to read data 1 and data 2. RDMA network card 1 can notify user-mode process 1 to read address 1 and address 2 from sub-memory 1 sequentially with multiple notifications, thereby enabling user-mode process 1 to read data 1 and data 2. No restrictions are imposed here.

[0081] More specifically, since the first data and the second data are adjacent data (i.e., the first data and the second data are two data sent to the first processor by the second processor in succession), the position of the first address in the first sub-memory is usually adjacent to the position of the second address in the second sub-memory.

[0082] As in the example above, after obtaining address 2, RDMA network card 1 will store address 2 in sub-memory 1 at a position after address 1, so that address 1 and address 2 are adjacent.

[0083] More specifically, the size of the second address is usually smaller than the size of the first sub-memory, and the size of the second data is usually smaller than or equal to the size of the third sub-memory. Therefore, the second address can be stored in the first sub-memory, and the second data can be stored in the third sub-memory.

[0084] As in the example above, address 2 is typically 12 bytes, which is smaller than the size of sub-memory 1 (4KB). Data 2 is 4KB in size, which is equal to the size of sub-memory 3 (4KB). Therefore, address 2 can be stored in sub-memory 1, and data 2 can be stored in sub-memory 3.

[0085] More specifically, since both the first data and the second data are data partitioned from the second processor's service data, if the first data and the second data are two independent raw data partitioned by the second processor, and the sizes of these two raw data are both less than or equal to the sizes of the second sub-memory and the third sub-memory (i.e., the aforementioned preset sizes), the second processor can instruct the second RDMA network card to send the first data and the second data to the first RDMA network card sequentially. If the first data and the second data originate from a certain raw data partitioned by the second processor (i.e., the third data), and the size of this raw data is greater than the sizes of the second sub-memory and the third sub-memory, even if the second processor directly instructs the second RDMA network card to send the raw data to the first RDMA network card, the first RDMA network card will be unable to store the raw data in the second sub-memory or the third sub-memory. Therefore, the second processor will partition the raw data into the first data and the second data in advance, so that the sizes of the first data and the second data are both less than or equal to the sizes of the second sub-memory and the third sub-memory, and then instruct the second RDMA network card to send the first data and the second data sequentially. In this way, the first RDMA network card can successfully store the first data and the second data in the second sub-memory and the third sub-memory respectively, which is beneficial for the first processor to successfully read the first data and the second data in the future.

[0086] Continuing with the example above, suppose user-space process 2 initially divides its service data into data 0 (4KB + 256B), data 3 (4KB), data 4 (4KB), and so on. Since data 0 is larger than the 4KB sub-memory size provided by user-space process 1 to RDMA network card 1, user-space process 2 can further divide data 0 into data 1 (256B) and data 2 (4KB). Subsequently, user-space process 2 can successively notify RDMA network card 2 to send all data (data 1, data 2, data 3, etc.) sequentially to RDMA network card 1. RDMA network card 1 then stores these data in sub-memory spaces of 4KB each (sub-memory 2, sub-memory 3, sub-memory 4, etc.). When RDMA network card 1 has received and stored the last data, it has essentially received all the service data from user-space process 2 and can notify user-space process 1 to read all data (data 1, data 2, data 3, etc.) from these sub-memory spaces, thus completing the transmission of all service data.

[0087] In this embodiment, the first processor of the first server can provide a memory block for the first RDMA network card of the first server. This memory block includes multiple sub-memories, such as a first sub-memory and a second sub-memory. After the second RDMA network card of the second server sends a first message to the first RDMA network card of the first server, since the first message carries the first data generated by the second processor of the second server, the first RDMA network card of the first server can request the first sub-memory and the second sub-memory from the memory block and generate a first address for the first data. Then, the first RDMA network card of the first server can store the first address in the first sub-memory and store the first data in the second sub-memory. The first address is used to describe the location of the first data in the second sub-memory. Subsequently, the first RDMA network card of the first server can notify the first processor of the first server to read the first address from the first sub-memory, so as to read the first data from the second sub-memory based on the first address, thereby realizing data transmission between the first server and the second server. In the aforementioned data transmission process, since the first server as the receiving end and the second server as the sending end only need to complete one communication for the message, and the first server completes the message parsing, data storage and data reading on its own, the number of communication between the first server and the second server during the data transmission process and the processing operations of each end can be reduced to a certain extent, thereby reducing the time cost and performance overhead required for the entire data transmission process to a certain extent.

[0088] Furthermore, in this embodiment, the second server, acting as the sending end, can segment the business data itself, and the first server, acting as the receiving end, can segment the memory blocks accordingly. This allows the segmented data (e.g., the aforementioned first data, second data, etc.) to be stored separately in the segmented sub-memory (e.g., the aforementioned second sub-memory and third sub-memory, etc.). In this way, business data of any size can be transmitted using this send / recv semantic method, without needing to reserve a specific amount of memory for the business data to achieve data transmission, thus possessing greater flexibility and practicality.

[0089] The above is a detailed description of the RDMA-based data transmission method provided in the embodiments of this application. The server provided in the embodiments of this application will be described below. Figure 4 is a schematic diagram of the structure of a server provided in the embodiments of this application. As shown in Figure 4, this server serves as a first server, and includes a first RDMA network card 402 and a first processor 401.

[0090] The first processor 401 is used to provide a memory block to the first RDMA network card 402. The memory block includes a first sub-memory and a second sub-memory. For example, the first processor 401 is used to implement step 201 in the embodiment shown in FIG2.

[0091] The first RDMA network card 402 is used to receive a first message sent by the second server, the first message carrying first data generated by the second server; for example, the first RDMA network card 402 is used to implement step 202 in the embodiment shown in FIG2.

[0092] The first RDMA network card 402 is also used to generate a first address of the first data carried by the first message, the first address being used to indicate the location of the first data in the second sub-memory; for example, the first RDMA network card 402 is used to implement step 203 in the embodiment shown in FIG2.

[0093] The first RDMA network card 402 is also used to store the first address in the first sub-memory and store the first data in the second sub-memory; for example, the first RDMA network card 402 is used to implement step 204 in the embodiment shown in FIG2.

[0094] The first RDMA network card 402 is also used to notify the first processor 401 to obtain a first address from the first sub-memory and to obtain first data from the second sub-memory based on the first address. For example, the first RDMA network card 402 is used to implement step 205 in the embodiment shown in FIG2.

[0095] In one possible implementation, the first RDMA network card 402 is also used to divide the memory block into multiple sub-memories, each sub-memory having a size equal to a preset size, and each sub-memory including a first sub-memory and a second sub-memory.

[0096] In one possible implementation, the size of the first address is less than the size of the first sub-memory, and the size of the first data is less than or equal to the size of the second sub-memory.

[0097] In one possible implementation, the memory block further includes a third sub-memory; the first RDMA network card 402 is also used to receive a second message sent by the second server, the second message carrying second data generated by the second server; the first RDMA network card 402 is also used to generate a second address of the second data carried by the second message, the second address being used to indicate the location of the first data in the third sub-memory; the first RDMA network card 402 is also used to store the second address in the first sub-memory and store the second data in the third sub-memory; the first RDMA network card 402 is also used to notify the first processor 401 to obtain the second address from the first sub-memory and to obtain the second data from the third sub-memory based on the second address.

[0098] In one possible implementation, the location of the first address in the first sub-memory is adjacent to the location of the second address in the second sub-memory.

[0099] In one possible implementation, the size of the second address is less than the size of the first sub-memory, and the size of the second data is less than or equal to the size of the third sub-memory.

[0100] In one possible implementation, the first data and the second data are obtained by the second server dividing the third data, and the size of the third data is greater than the size of the second sub-memory, and the size of the third data is greater than the size of the third sub-memory.

[0101] It should be noted that the information interaction and implementation process between the modules / units of the above-mentioned device are based on the same concept as the method embodiments of this application, and the resulting technical effects are the same as those of the method embodiments of this application. For details, please refer to the description in the method embodiments shown above in the embodiments of this application, and will not be repeated here.

[0102] Please refer to Figure 5, which is a schematic diagram of a computing device provided in an embodiment of this application. As shown in Figure 5, the computing device 500 (which can be used to present the aforementioned first server or second server) includes: a processor 501, a memory 502, a communication interface 503, and a bus 504. The processor 501, the memory 502, and the communication interface 503 are coupled through the bus (not labeled in the figure). The memory 502 stores instructions. When the execution instructions in the memory 502 are executed, the computing device 500 executes the method performed by the first server or the second server in the above method embodiment.

[0103] The computing device 500 may be one or more integrated circuits configured to implement the methods described above, such as: one or more application-specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs), or a combination of at least two of these forms of integrated circuits. Furthermore, when the units in the device can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. Alternatively, these units may be integrated together to implement a system-on-a-chip (SOC).

[0104] Processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0105] Memory 502 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0106] The memory 502 stores executable program code, which the processor 501 executes to implement the RDMA-based data transfer method described above. That is, the memory 502 stores instructions for executing the RDMA-based data transfer method described above.

[0107] The communication interface 503 (i.e., the aforementioned first RDMA network card or second RDMA network card) uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 500 and other devices or communication networks.

[0108] In addition to the data bus, the 504 bus can also include a power bus, a control bus, and a status signal bus. The bus can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The bus can be divided into address bus, data bus, and control bus.

[0109] Please refer to Figure 6, which is a schematic diagram of a computing device cluster provided in an embodiment of this application. As shown in Figure 6, the computing device cluster 600 includes at least one computing device 500.

[0110] As shown in Figure 6, the computing device cluster 600 includes at least one computing device 500. The memory 502 of one or more computing devices 500 in the computing device cluster 600 may store the same instructions for executing the aforementioned RDMA-based data transfer method.

[0111] In some possible implementations, the memory 502 of one or more computing devices 500 in the computing device cluster 600 may also store partial instructions for executing the aforementioned RDMA-based data transfer method. In other words, a combination of one or more computing devices 500 can jointly execute the aforementioned RDMA-based data transfer method.

[0112] It should be noted that the memory 502 in different computing devices 500 in the computing device cluster 600 can store different instructions, which are used to execute some of the functions of the first server or the second server mentioned above.

[0113] In some possible implementations, one or more computing devices 500 in the computing device cluster 600 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0114] Please refer to Figure 7, which is a schematic diagram of computer devices in a computer cluster provided in an embodiment of this application connected via a network. As shown in Figure 7, two computing devices 500A and 500B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0115] In one possible implementation, the memory in computing device 500A stores instructions for performing the functions of the aforementioned first server. Simultaneously, the memory in computing device 500B stores instructions for performing the functions of the second server.

[0116] It should be understood that the functions of computing device 500A shown in Figure 7 can also be performed by multiple computing devices. Similarly, the functions of computing device 500B can also be performed by multiple computing devices.

[0117] This application also relates to a computer storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform the steps performed by the first server or the second server in the embodiment shown in FIG2.

[0118] This application also relates to a computer program product that stores instructions that, when executed by a computer, cause the computer to perform the steps performed by the first server or the second server in the embodiment shown in FIG2.

[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data transfer method based on Remote Direct Memory Access (RDMA), characterized in that, The method is applied to a first server, the first server including a first RDMA network card and a first processor, the method including: The first processor provides a memory block to the first RDMA network card, the memory block comprising a first sub-memory and a second sub-memory; The first RDMA network card receives a first message sent by the second server, the first message carrying first data generated by the second server; The first RDMA network card generates a first address for the first data carried in the first message, and the first address is used to indicate the location of the first data in the second sub-memory; The first RDMA network card stores the first address in the first sub-memory and stores the first data in the second sub-memory; The first RDMA network card notifies the first processor to obtain the first address from the first sub-memory and to obtain the first data from the second sub-memory based on the first address.

2. The method according to claim 1, characterized in that, The method further includes: The first RDMA network card divides the memory block into multiple sub-memories, each sub-memory having a size equal to a preset size, and each sub-memory including the first sub-memory and the second sub-memory.

3. The method according to claim 1 or 2, characterized in that, The size of the first address is less than the size of the first sub-memory, and the size of the first data is less than or equal to the size of the second sub-memory.

4. The method according to any one of claims 1 to 3, characterized in that, The memory block further includes a third sub-memory, and the method further includes: The first RDMA network card receives a second message sent by the second server, the second message carrying second data generated by the second server; The first RDMA network card generates a second address for the second data carried in the second message, and the second address is used to indicate the location of the first data in the third sub-memory; The first RDMA network card stores the second address in the first sub-memory and stores the second data in the third sub-memory; The first RDMA network card notifies the first processor to obtain the second address from the first sub-memory and to obtain the second data from the third sub-memory based on the second address.

5. The method according to claim 4, characterized in that, The location of the first address in the first sub-memory is adjacent to the location of the second address in the second sub-memory.

6. The method according to claim 4 or 5, characterized in that, The size of the second address is less than the size of the first sub-memory, and the size of the second data is less than or equal to the size of the third sub-memory.

7. The method according to any one of claims 4 to 6, characterized in that, The first data and the second data are obtained by the second server dividing the third data. The size of the third data is greater than the size of the second sub-memory, and the size of the third data is greater than the size of the third sub-memory.

8. A server, characterized in that, The server is designated as a first server, and the first server includes a first RDMA network card and a first processor; The first processor is configured to provide a memory block to the first RDMA network card, the memory block comprising a first sub-memory and a second sub-memory; The first RDMA network card is used to receive a first message sent by the second server, the first message carrying first data generated by the second server; The first RDMA network card is further configured to generate a first address of the first data carried in the first message, wherein the first address is used to indicate the location of the first data in the second sub-memory; The first RDMA network card is further configured to store the first address in the first sub-memory and store the first data in the second sub-memory; The first RDMA network card is further configured to notify the first processor to obtain the first address from the first sub-memory and to obtain the first data from the second sub-memory based on the first address.

9. The server according to claim 8, characterized in that, The first RDMA network card is also used to divide the memory block into multiple sub-memories, the size of each of the multiple sub-memories being equal to a preset size, and the multiple sub-memories including the first sub-memory and the second sub-memory.

10. The server according to claim 8 or 9, characterized in that, The size of the first address is less than the size of the first sub-memory, and the size of the first data is less than or equal to the size of the second sub-memory.

11. The server according to any one of claims 8 to 10, characterized in that, The memory block also includes a third sub-memory; The first RDMA network card is also used to receive a second message sent by the second server, the second message carrying second data generated by the second server; The first RDMA network card is also used to generate a second address of the second data carried in the second message, the second address being used to indicate the location of the first data in the third sub-memory; The first RDMA network card is also used to store the second address in the first sub-memory and store the second data in the third sub-memory; The first RDMA network card is also used to notify the first processor to obtain the second address from the first sub-memory and to obtain the second data from the third sub-memory based on the second address.

12. The server according to claim 11, characterized in that, The location of the first address in the first sub-memory is adjacent to the location of the second address in the second sub-memory.

13. The server according to claim 11 or 12, characterized in that, The size of the second address is less than the size of the first sub-memory, and the size of the second data is less than or equal to the size of the third sub-memory.

14. The server according to any one of claims 11 to 13, characterized in that, The first data and the second data are obtained by the second server dividing the third data. The size of the third data is greater than the size of the second sub-memory, and the size of the third data is greater than the size of the third sub-memory.

15. An RDMA system, characterized in that, The RDMA system includes a second server and a first server as described in any one of claims 8 to 14.

16. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, each computing device including a processor and memory: The memory is used to store instructions; The processor is configured to, according to the instructions, cause the computing device cluster to perform the method of any one of claims 1 to 7.

17. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions that, when executed by one or more computers, cause the one or more computers to perform the method according to any one of claims 1 to 7.

18. A computer program product, characterized in that, The computer program product stores instructions that, when executed by a computer, cause the computer to perform the method described in any one of claims 1 to 7.

Images