Data transmission method and apparatus, DPU, electronic device, medium, and program product

Abstract

Disclosed in the embodiments are a data transmission method and apparatus, a DPU, an electronic device, a medium, and a program product. The DPU is used for carrying out data transmission between a server and a storage device. The data transmission method comprises: acquiring a first identifier when a data transmission error is reported, wherein the first identifier is an identifier of the most recently completed first data transmission operation fed back by a storage device to a DPU, and the first data transmission operation indicates a data transmission operation initiated by a server to the storage device; notifying the server to use a stored packet corresponding to the first identifier as a first stored packet in a stored packet sequence, such that the server sends each stored packet in the stored packet sequence in a preset order, starting with the stored packet corresponding to the first identifier; and receiving the stored packet sequence, and sequentially performing data transmission operations on the basis of the stored packet sequence.

Description

Data transmission methods, DPUs, devices, electronic equipment, media, and software products

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202510072780.9, filed on January 16, 2025, entitled “Data Transmission Method, DPU, Apparatus, Electronic Device, Medium and Program Product”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application belongs to the field of data transmission technology, and specifically relates to a data transmission method, a data processing unit (DPU), an apparatus, an electronic device, a medium, and a program product. Background Technology

[0004] In public cloud scenarios, data center DPUs typically handle storage services such as tenant system disks and data disks. To address storage packet loss or errors during abnormal data transmission, public clouds commonly use Transmission Control Protocol / Internet Protocol (TCP / IP) to retransmit storage packets and ensure normal data transmission. TCP / IP connections consume more system resources, such as memory and Central Processing Unit (CPU) processing power, and the TCP / IP protocol stack configuration is relatively complex, requiring configuration of congestion window size, retransmission intervals, etc. Summary of the Invention

[0005] This application provides a data transmission method, a digital processing unit (DPU), an apparatus, an electronic device, a medium, and a program product.

[0006] This application provides a data transmission method applied to a Data Processing Unit (DPU), wherein the DPU is used to carry data transmission between a server and a storage device; the method includes:

[0007] In the event of a data transmission error, a first identifier is obtained; the first identifier is the identifier of the latest completed first data transmission operation fed back by the storage device to the DPU; the first data transmission operation represents a data transmission operation initiated by the server to the storage device;

[0008] The server is notified to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, so that the server sends each stored message in the stored message sequence in a preset order, starting from the stored message corresponding to the first identifier.

[0009] Receive the stored message sequence and perform data transmission operations sequentially based on the stored message sequence.

[0010] In some embodiments, the first identifier is a used descriptor index (Used idx). Before notifying the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, the method further includes: updating the first identifier to a second identifier, where the second identifier is an available descriptor index (Avail idx) to be sent; and notifying the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence includes: notifying the server to use the stored message corresponding to the second identifier as the first stored message in the stored message sequence.

[0011] It can be seen that updating Used idx to Avail idx helps to quickly determine the first stored message in the stored message sequence, enabling fast data retransmission.

[0012] In some embodiments, the data transmission operation based on the storage message sequence includes: sending the storage message sequence to the storage device; receiving a first data set returned by the storage device based on the storage message sequence; determining whether there is duplicate data in the first data set and the historical data set; if there is duplicate data, deduplicating the data in the historical data set to obtain a second data set; the historical data set is the data set received by the DPU before receiving the first data set; and returning the second data set to the server.

[0013] As can be seen, the deduplication method given in this embodiment helps the server obtain accurate data transmission results and improves the correctness of data transmission.

[0014] In some embodiments, before sending the storage message sequence to the storage device, the method further includes: initiating a second data transmission operation to the storage device based on a second storage message in the DPU; the second storage message represents a storage message received by the DPU but not sent to the storage device before a data transmission error occurs; and receiving a set of historical data returned by the storage device based on the second storage message.

[0015] As can be seen, the method provided in this embodiment is beneficial for achieving orderly processing of messages stored in the DPU; the deduplication method provided in this embodiment can obtain accurate data transmission results and improve the correctness of data transmission when data retransmission is restored.

[0016] In some embodiments, before obtaining the first identifier, the method further includes: stopping receiving storage messages from the server; after obtaining the first identifier, the method further includes: resuming receiving storage messages from the server.

[0017] It can be seen that by stopping the reception of stored messages from the server before obtaining the first identifier, it is beneficial to quickly restore the DPU's ability to process stored messages and to quickly restore the current data transmission in the event of a data transmission error.

[0018] This application embodiment also provides a DPU, which is used to carry data transmission between a server and a storage device; the DPU includes a Storage Performance Development Kit (SPDK) and a Field Programmable Gate Array (FPGA), wherein,

[0019] The SPDK is used to restart in the event of a data transmission error, and after restarting, obtain a first identifier; send the first identifier to the FPGA; the first identifier is the identifier of the latest completed first data transmission operation fed back by the storage device to the DPU; the first data transmission operation represents a data transmission operation initiated by the server to the storage device;

[0020] The FPGA is configured to receive the first identifier, notify the server to use the storage message corresponding to the first identifier as the first storage message in the storage message sequence, and enable the server to send each storage message in the storage message sequence in a preset order, starting from the storage message corresponding to the first identifier; receive the storage message sequence, and perform data transmission operations sequentially based on the storage message sequence.

[0021] This application embodiment also provides a data transmission device applied to a DPU, wherein the DPU is used to carry data transmission between a server and a storage device; the device includes:

[0022] The acquisition module is configured to acquire a first identifier in the event of a data transmission error; the first identifier is an identifier of the latest completed first data transmission operation fed back by the storage device to the DPU; the first data transmission operation represents a data transmission operation initiated by the server to the storage device;

[0023] The notification module is configured to notify the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, so that the server sends each stored message in the stored message sequence in a preset order, starting from the stored message corresponding to the first identifier.

[0024] The processing module is configured to receive the stored message sequence and perform data transmission operations sequentially based on the stored message sequence.

[0025] This application provides an electronic device, which includes a processor and a memory for storing computer programs capable of running on the processor; wherein,

[0026] The processor is used to run the computer program to implement any of the above-described data transmission methods.

[0027] This application provides a computer storage medium storing a computer program, which, when executed by a processor, implements any of the above-described data transmission methods.

[0028] This application provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described data transmission methods.

[0029] This application provides a data transmission method, a data processing unit (DPU), an apparatus, an electronic device, a medium, and a program product. In the event of data transmission errors, the DPU handles and recovers the data transmission, which helps reduce CPU and memory resource consumption during data transmission. The method provided in this application allows for rapid location of the stored message that needs to be sent by the server in the event of a data transmission error, without requiring configuration of congestion window size, retransmission intervals, etc., thus avoiding complex protocol stack configurations and improving the efficiency of data transmission error handling and overall data transmission efficiency. Attached Figure Description

[0030] Figure 1 is a schematic diagram of a timeout retransmission mechanism provided in an embodiment of this application;

[0031] Figure 2 is a schematic diagram of a fast retransmission mechanism provided in an embodiment of this application;

[0032] Figure 3 is a flowchart of a data transmission method provided in an embodiment of this application;

[0033] Figure 4 is a schematic diagram of a DPU structure provided in an embodiment of this application;

[0034] Figure 5 is a flowchart of a DPU-based lossless data retransmission process provided in an embodiment of this application;

[0035] Figure 6 is a flowchart of a data deduplication process provided in an embodiment of this application;

[0036] Figure 7 is a schematic diagram of a data transmission device provided in an embodiment of this application;

[0037] Figure 8 is a schematic diagram of the composition structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] In public cloud scenarios, data center DPUs typically handle storage services such as tenant system disks and data disks. To address the loss or error of storage packets during abnormal data transmission, public clouds commonly use TCP / IP to retransmit storage packets, ensuring normal data transmission. TCP / IP retransmission mechanisms primarily include timeout retransmission and fast retransmission.

[0039] Figure 1 illustrates a timeout retransmission mechanism. In this mechanism, if the sender does not receive a response from the receiver within a specified time, it will retransmit the data. For this mechanism, if the timeout period is set too long, message retransmission will be slow, resulting in poor overall data transmission performance. Conversely, if the timeout period is too short, frequent retransmissions will occur, easily causing network congestion and poor performance.

[0040] Figure 2 illustrates a fast retransmission mechanism, in which a retransmission is triggered when three identical acknowledgments (ACKs) are received consecutively. For the fast retransmission mechanism, if the stored packet is not lost, the ACK will also retransmit the packet if link congestion occurs.

[0041] Therefore, when data transmission is based on TCP / IP, TCP / IP connections often consume more system resources, such as memory and CPU processing power. In addition, the TCP / IP protocol stack configuration is relatively complex, requiring configuration of congestion window size, retransmission interval, etc.

[0042] To overcome the problems existing in related technologies, embodiments of this application provide a data transmission method, a data processing unit (DPU), an apparatus, an electronic device, a medium, and a program product. The data transmission method proposed in this application uses a DPU to monitor data transmission. When a data transmission error occurs, the stored message for data transmission is relocated based on the currently transmitted data. This eliminates the need to consume system resources such as CPU and memory, allowing a single node to complete lossless data retransmission.

[0043] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the embodiments provided herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application. Furthermore, the embodiments provided below are some embodiments for implementing this application, and not all embodiments for implementing this application. Unless otherwise specified, the technical solutions described in the embodiments of this application can be implemented in any combination.

[0044] It should be noted that, in the embodiments of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a method or apparatus that includes a list of elements includes not only the elements expressly described, but also other elements not expressly listed, or elements inherent to implementing the method or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of other related elements in the method or apparatus that includes that element (e.g., steps in the method or units in the apparatus; for example, a unit in the apparatus may be a portion of circuitry, a portion of a processor, a portion of a program or software, etc.).

[0045] The data transmission method provided in this application includes a series of steps, but the data transmission method provided in this application is not limited to the steps described. Similarly, the data transmission device provided in this application includes a series of modules, but the device provided in this application is not limited to the modules explicitly described, and may also include modules that need to be set up for obtaining relevant information or processing based on information.

[0046] This application provides a data transmission method applied to a Data Processing Unit (DPU), which carries data transmission between a server and a storage device. A DPU is a novel computing architecture designed specifically for computationally intensive tasks in data centers. It combines the general-purpose computing power of a CPU with the parallel computing power of a Graphics Processing Unit (GPU), further optimizing data throughput and latency performance. Figure 3 shows a flowchart of a data transmission method, which includes:

[0047] Step 301: In the event of a data transmission error, obtain the first identifier; the first identifier is the identifier of the latest completed first data transmission operation fed back by the storage device to the DPU; the first data transmission operation represents the data transmission operation initiated by the server to the storage device.

[0048] In the event of a data transmission error, the identifier of the latest completed first data transmission operation, fed back from the storage device to the DPU, is obtained. During data transmission, various situations can lead to errors, such as hardware failure, software problems, and communication protocol incompatibility. For example, a disconnected or poorly connected data transmission line, or network congestion, may cause data packet loss, resulting in an error. The DPU involved in this embodiment is designed to promptly report and restart upon detecting a data transmission error.

[0049] Specifically, the DPU described in this embodiment can implement a data transmission error reporting mechanism using the open-source Storage High Performance Development Kit (SPDK). SPDK is a high-performance, scalable, open-source storage software development kit. SPDK's data transmission error reporting mechanism includes error codes and logging. During data transmission, if an error is encountered, SPDK generates a corresponding error code and records it through a logging system. These error codes and logs are crucial for troubleshooting. In practical applications, SPDK can be configured to attempt to recover data transmission by restarting relevant components or processes when an error occurs. This restart mechanism improves system stability and reliability, reducing system downtime caused by data transmission errors.

[0050] In the event of a data transmission error, the first identifier can be determined through the logs recorded in the DPU, for example, by obtaining the first identifier from the SPDK logs. When a data transmission error occurs, i.e., packet loss occurs during data transmission, it is assumed that the server failed to receive response data from the storage device for the latest data transmission. This response data includes one or more of the data to be transmitted and related control information, such as one or more of the following: source address, destination address, data length, and checksum. The response data can specifically be a response message. In this case, the DPU obtains the first identifier of the latest completed first data transmission operation received from the storage device. This first identifier is used to determine the first storage message sent by the server when resuming data transmission.

[0051] Step 302: Instruct the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, so that the server sends each stored message in the stored message sequence in a preset order, starting from the stored message corresponding to the first identifier.

[0052] In this embodiment, the identifier of the data transmission operation corresponds one-to-one with the stored message, or the identifier of the data transmission operation can be stored in the message header as a unique identifier of the stored message. The identifier of the data transmission operation may include a completed identifier for responding to a completed data transmission operation, or a pending identifier for indicating a data transmission operation to be processed. The first identifier obtained in the above steps is a completed identifier.

[0053] After the DPU determines the first identifier, since the first identifier is the identifier of the latest completed data transmission operation in the DPU, there may be a situation where the server already has the first identifier before a data transmission error occurs, or there may be a situation where the server has not received the first identifier. Therefore, after a data transmission error occurs, the server can be notified to re-acquire the stored message corresponding to the first identifier based on the first identifier.

[0054] Here, the first identifier can be updated from a completed identifier to a pending identifier. This updated identifier is sent to the server, which then determines the corresponding stored message based on it. The server designates the stored message corresponding to the first identifier as the first stored message to be sent, and sends each message in the stored message sequence in a preset order. This preset order can be determined based on the design rules of the pending identifiers. For example, if the pending identifiers are designed to increment sequentially, the pending identifier in the message is incremented by one each time the server sends a stored message. Based on this preset rule, when the first identifier is determined to be "10", the stored message corresponding to "10" is used as the first stored message, and the server sequentially sends the stored messages corresponding to pending identifiers "10", "11", and "12" to the DPU.

[0055] The storage messages involved in this embodiment are used to realize data transmission from the server to the storage device. The data transmission includes, but is not limited to, data read and write operations.

[0056] Step 303: Receive the stored message sequence and perform data transmission operations sequentially based on the stored message sequence.

[0057] After receiving the notification from the DPU, the server reissues the storage message sequence to the DPU based on the first identifier. The DPU then sequentially sends each storage message in the received sequence to the storage device. The storage device, based on each received storage message, sequentially returns response data to the DPU. The DPU then returns the response data from the storage device to the server. This response data may specifically include the first identifier.

[0058] This embodiment provides a data transmission method that uses a Data Processing Unit (DPU) to carry out data transmission between the server and the storage device. When a data transmission error occurs, the DPU promptly reports the error and determines the data that needs to be retransmitted. This bypasses the TCP / IP retransmission mechanism, reduces the resource consumption of CPU and memory, and improves the efficiency of data retransmission in the event of a data transmission error.

[0059] In practical applications, steps 301 to 303 can be implemented based on a processor, which can be at least one of the following: Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), CPU, controller, microcontroller, and microprocessor.

[0060] In some embodiments, the first identifier is a used descriptor index Used idx. Before the notification server uses the storage message corresponding to the first identifier as the first storage message in the storage message sequence, the method further includes: updating the first identifier to a second identifier, where the second identifier is a descriptor index Available idx to be sent; the notification server using the storage message corresponding to the first identifier as the first storage message in the storage message sequence includes: the notification server using the storage message corresponding to the second identifier as the first storage message in the storage message sequence.

[0061] In a virtualized transport environment, such as when accessing cloud / virtual machine (guest) data via a host, the semi-virtualized Virtio data transfer mechanism can provide an efficient input / output (I / O) solution. Here, the cloud / virtual machine refers to the storage device in the above embodiments, and the host refers to the server in the above embodiments. In the Virtio data transfer mechanism, Available idx and Used idx are two key index values, which are used to track the descriptor status in the Available Ring and Used Ring, respectively.

[0062] `Avail idx` is an index value used to record the location where the next available descriptor should be stored. When data needs to be transmitted, it populates the available ring table with descriptors and updates `Avail idx` to indicate the location of the next free descriptor. This allows the host to retrieve descriptors from the available ring table based on `Avail idx` and use them to receive data. `Used idx` is an index value used to record used descriptors. When a host retrieves descriptors from the available ring table and uses them to store data, it updates the used ring table and increments `Used idx` to indicate the location of the next used descriptor.

[0063] In a virtualized transmission environment, using the method described in this embodiment, in the event of a data transmission error, the latest first Used idx in the DPU can be obtained, and the latest first Used idx can be updated to the first Available idx. That is, the first identifier is changed from a completed identifier to a second identifier representing a pending identifier, where the second identifier is the first Available idx. The second identifier is reported to the host, which then obtains the corresponding storage packet containing the second identifier based on it. Following the method described in step 302 above, the host sequentially sends each storage packet to the DPU based on the second identifier. Here, the header of each storage packet includes the Available idx.

[0064] Based on the data transmission method given in the above embodiments, this embodiment provides a method for data retransmission in a virtualized transmission environment by combining the used descriptor index and the descriptor index to be sent.

[0065] In some embodiments, the above-mentioned data transmission operation based on the storage message sequence includes: sending the storage message sequence to the storage device; receiving a first data set returned by the storage device based on the storage message sequence; determining whether there is duplicate data in the first data set and the historical data set; if there is duplicate data, deduplicating the data in the historical data set to obtain a second data set; the historical data set is the data set received by the DPU before receiving the first data set; and returning the second data set to the server.

[0066] Before a data transmission error occurs, the DPU may still contain a set of historical data that has already been transmitted, or it may contain stored messages that have not yet been sent to the storage device. In this case, when the data transmission error occurs and the DPU re-initiates the data transmission, it will continue to execute the unsent stored messages, and at this time, it will also receive a set of historical data returned from the storage device. Here, the historical data set, as well as the first data set, includes the storage device's response data to different stored messages.

[0067] After a data transmission error occurs, the DPU re-receives the sequence of storage messages sent by the server and sends the sequence of storage messages to the storage device in order. The storage device then sends a first set of data containing response data to the DPU based on the received storage messages.

[0068] Once the DPU receives the first data set, it compares the response data in the first data set with the response data in the historical data set, removing duplicate data from the historical data set. For example, it can compare the completed flag in the response message in the first data set with the completed flag in the response message in the historical data set. If they are the same, it removes the identical completed flag and corresponding response message from the historical data set.

[0069] In a virtualized transport environment, the used idx in the first data set can be compared with the used idx in the historical data set to see if there is a duplicate used idx. If there is a duplicate used idx, the duplicate used idx in the historical data set and the corresponding response data are deleted.

[0070] In some embodiments, before sending the storage message sequence to the storage device, the method further includes: initiating a second data transmission operation to the storage device based on the second storage message in the DPU; the second storage message represents a storage message received by the DPU but not sent to the storage device before the data transmission error occurred; and receiving a set of historical data returned by the storage device based on the second storage message.

[0071] In the event of a data transmission error, such as before an SPDK restart, the DPU might receive a second storage packet from the server, but fail to send it to the storage device upon SPDK restart. In this case, after SPDK completes its restart or the DPU re-initiates data transmission, the DPU will prioritize sending the second storage packet to the storage device, followed by a sequence of storage packets. Here, the second storage packet may consist of multiple storage packets.

[0072] For example, the server continuously sends storage messages to the DPU, with the identifiers corresponding to these messages being "10", "11", and "12" sequentially. When a data transmission error occurs, suppose the DPU receives a processed identifier of "10" from the storage device, but the storage messages corresponding to identifiers "11" and "12" in the DPU have not yet been sent to the storage device. When the DPU initiates data retransmission and notifies the server that the first identifier "10" is a pending identifier, the DPU still contains storage messages corresponding to identifiers "11" and "12". In this case, based on the method described in the above embodiment, the server retransmits storage messages to the DPU with identifiers "10", "11", "12", and "13" for each storage message in the retransmission sequence. At this time, the identifiers of the storage messages executed internally by the DPU are "11", "12", "10", "11", "12", and "13" sequentially. It can be seen that there are repeatedly executed storage files in the DPU, and the corresponding storage device returns repeated response data to the DPU. That is, the historical data set includes response data corresponding to the identifiers "11" and "12", and the first data set includes response data corresponding to "10", "11", "12", and "13".

[0073] To avoid the above situation and ensure that the server ultimately receives the correct response data, the DPU will receive data from the historical data set, compare it with the data in the first data set to remove duplicates, and delete duplicate data from the historical data set to ensure the consistency of the response data received by the server and the corresponding completed flags.

[0074] In some embodiments, before obtaining the first identifier, the method further includes: stopping the reception of storage messages from the server; after obtaining the first identifier, the method further includes: resuming the reception of storage messages from the server.

[0075] As can be seen, stopping the reception of stored messages from the server before obtaining the first identifier after a data transmission error helps reduce the number of pending stored messages in the DPU, prevents the loss of pending stored messages, and facilitates data transmission recovery as quickly as possible. For example, in a virtualized transmission environment, relevant statements can be written to block the DPU's receive path and stop receiving stored messages sent by the server.

[0076] Once the first identifier is obtained, the DPU's receiving path can be restored through relevant statements, allowing it to continue receiving storage messages from the server and continuing to carry out data transmission between the server and the storage device.

[0077] Based on the data processing method given in the above embodiments, Figure 4 shows a schematic diagram of a DPU structure. As shown in Figure 4, the DPU 40 in this embodiment carries data transmission between the server 50 and the storage device 60. The DPU 40 includes an SPDK 401 and an FPGA 402.

[0078] SPDK 401 is used to restart in case of data transmission error, and after restarting, it obtains the first identifier and sends the first identifier to FPGA 402. The first identifier is the identifier of the latest completed first data transmission operation fed back by storage device 60 to DPU 40. The first data transmission operation represents the data transmission operation initiated by server 50 to storage device 60.

[0079] FPGA 402 is used to receive the first identifier and notify server 50 to use the storage message corresponding to the first identifier as the first storage message in the storage message sequence, so that server 50 sends each storage message in the storage message sequence in a preset order, starting from the storage message corresponding to the first identifier; and receives the storage message sequence and performs data transmission operations sequentially based on the storage message sequence.

[0080] The specific data transmission method involved in this embodiment is the same as that in the above embodiments. The SPDK 401 can be deployed in a System on Chip (SOC). In the virtualized transmission environment, Avail idx is a descriptor index maintained by the server 50 driver. The Avail idx value is incremented by one each time the server 50 sends a storage message. Used idx is a descriptor index maintained by the FPGA 402. The Used idx value is incremented by one each time the FPGA 402 transmits a storage message read / write response to the server 50.

[0081] As shown in Figure 4, FPGA402 also includes a virtualized memory direct access technology module (HOST Virtual I / O Device Direct Memory Access, HOST Virtio DMA) 4021 on the server 50 side and a virtualized memory direct access technology module (System on Chip Virtual I / O Device Direct Memory Access, SOC Virtio DMA) 4022 on the SOC side.

[0082] Based on the method given in the above embodiments and the schematic diagram of the DPU 40 structure shown in Figure 4, when the SPDK 401 process hangs or a storage message error occurs on the transmission path of FPGA 402, after SPDK 401 restarts, it reads the value of Used idx on the local end and writes it into the register Hw avail idx of the HOST Virtio DMA of FPGA 402. The HOST Virtio DMA 4021 in FPGA 402 then rereads the data from the memory pointed to by Avail idx of the block device driver Virtio blk in server 50 and sends it to the storage device 60.

[0083] Based on the structural diagram of DPU 40 shown in Figure 4, Figure 5 shows a flowchart of a lossless data retransmission process based on DPU, including:

[0084] Step 501: SPDK sends a notification to the FPGA to initiate a blocking data link.

[0085] After SPDK 401 restarts, it sends a notification to FPGA 402 to write 1 to the Blk pause en register in FPGA 402, thus blocking the data flow of FPGA 402. The specific interaction corresponds to the arrow corresponding to number 1 in Figure 4. SPDK 401 then sends a notification to Host Virtio DMA 4021 to write 1 to the Blk pause en register in Host Virtio DMA 4021.

[0086] Step 502: SPDK writes the Used idx to the Avail idx in the FPGA.

[0087] Corresponding to the arrow with number 1 in Figure 4, SPDK 401 indicates that after the data link is blocked, the first Used idx of the last normally transmitted storage packet in SPDK 401 should be written back to the first Available idx of HOST Virtio DMA4021.

[0088] Step 503: SPDK sends a notification to the FPGA to restore the data link.

[0089] Corresponding to the arrow with number 1 in Figure 4, after writing the first Avail idx to the HOST Virtio DMA 4021, the SPDK 401 writes 0 to the storage link blocking register Blk pause en, restoring the storage data path.

[0090] At this time, the HOST Virtio DMA 4021 notifies the server 50 of the first Avail idx written, so that the server 50 sends the storage message sequence in sequence according to the method given in the above embodiment, that is, the arrow corresponding to number 2 in Figure 4.

[0091] Step 504: The FPGA notifies the SPDK to re-read and write data according to Avail idx.

[0092] After receiving the stored message sequence, the HOST Virtio DMA 4021 sends the stored message sequence and the corresponding Avail idx to the SOC Virtio DMA 4022 according to the process indicated by the arrow corresponding to number 3 in Figure 4. The SOC Virtio DMA 4022 then sends the stored message sequence and the corresponding Avail idx to the SPDK 401.

[0093] Step 505: SPDK sends a request to the storage device.

[0094] Corresponding to the arrow with number 5 in Figure 4, SPDK 401 sends the received storage message sequence and the corresponding Avail idx to storage device 60 for data read / write, i.e., sends a request.

[0095] Step 506: The storage device responds to the SPDK request.

[0096] Storage device 60 responds to the request from SPDK 401, returning the historical data set and the first data set. This corresponds to the arrow associated with number 6 in Figure 4.

[0097] Step 507: SPDK responds to the FPGA request.

[0098] Corresponding to arrow number 7 in Figure 4, SPDK 401 responds to the request from SOC Virtio DMA 4022 in FPGA 402, returning the historical data set and the first data set. Corresponding to arrow number 8 in Figure 4, SOC Virtio DMA 4022 returns a response request to HOST Virtio DMA 4021, returning the historical data set and the first data set. Finally, corresponding to arrow number 9 in Figure 4, HOST Virtio DMA 4021 returns a response request to server 50, returning the historical data set and the first data set, completing the data transmission.

[0099] Figure 6 illustrates a data deduplication flowchart. As shown in Figure 6, due to data transmission anomalies, residual stored packets m to m+z remain in the HOST Virtio DMA 4021. In this case, the residual stored packets in the HOST Virtio DMA 4021 need to be processed, and then the data is re-forwarded from the first Avail idx back to stored packet y. Stored packets y to y+n contain stored packets m to m+z. Based on this, storage device 60 receives two identical stored packets, i.e., stored packets m to m+z. Therefore, storage device 60 returns the same response data twice for stored packets m to m+z. At this time, the HOST Virtio DMA 4021 receiver, based on the method given in the above embodiment, deduplicates the response data of storage device 60, i.e., the historical data set and the first data set returned by storage device 60, and obtains the response data fed back to server 50 after deduplication.

[0100] This application provides a data transmission method and a Data Processing Unit (DPU) for data retransmission in abnormal situations using SPDK, presenting a novel lossless retransmission mechanism for DPU storage. By introducing a lossless storage mechanism into the DPU and combining it with SPDK for retransmission, a storage data retransmission solution for abnormal storage path situations is implemented through hardware and software collaboration. Compared with related technologies that achieve lossless retransmission via TCP / IP, the lossless storage retransmission mechanism implemented in this application does not consume system resources such as CPU and memory, and a single node can complete the lossless retransmission of stored data. The lossless retransmission speed provided in this application is fast and imperceptible to the user. By using SPDK to write back the FPGA's Available IDx, the FPGA retrieves the incorrectly transmitted data from the server and sends it to the storage device, thus achieving lossless data retransmission. This application proposes a design method and apparatus for lossless DPU storage, utilizing the DPU for data write-back, overcoming the shortcomings of related technologies, and can be used to implement low-cost, feasible, and easily promoted cloud computing and big data platforms, demonstrating high practicality.

[0101] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0102] Based on the data transmission method proposed in the foregoing embodiments, this application also provides a data transmission device applied to a DPU, which is used to carry data transmission between a server and a storage device; as shown in Figure 7, the data transmission device includes:

[0103] The acquisition module 701 is configured to acquire a first identifier in the event of a data transmission error; the first identifier is the identifier of the latest completed first data transmission operation fed back by the storage device to the DPU; the first data transmission operation represents the data transmission operation initiated by the server to the storage device.

[0104] The notification module 702 is configured to notify the server to use the storage message corresponding to the first identifier as the first storage message in the storage message sequence, so that the server sends each storage message in the storage message sequence in a preset order, starting from the storage message corresponding to the first identifier.

[0105] The processing module 703 is configured to receive a sequence of stored messages and perform data transmission operations sequentially based on the sequence of stored messages.

[0106] In practical applications, the acquisition module 701, notification module 702, and processing module 703 can be implemented based on a processor and a communication device.

[0107] In some embodiments, the first identifier is Used idx. Before the server notifies the storage message corresponding to the first identifier to be the first storage message in the storage message sequence, the notification module 702 is further configured to update the first identifier to a second identifier, the second identifier being Avail idx; the notification module 702 is further configured to notify the server to use the storage message corresponding to the second identifier as the first storage message in the storage message sequence.

[0108] In some embodiments, the processing module 703 is further configured to: send a storage message sequence to the storage device; receive a first data set returned by the storage device based on the storage message sequence; determine whether there is duplicate data in the first data set and the historical data set; if there is duplicate data, deduplicate the data in the historical data set to obtain a second data set; the historical data set is the data set received by the DPU before receiving the first data set; and return the second data set to the server.

[0109] In some embodiments, before sending the storage message sequence to the storage device, the processing module 703 is further configured to initiate a second data transmission operation to the storage device based on the second storage message in the DPU; the second storage message represents a storage message that the DPU received but did not send to the storage device before the data transmission error occurred; and to receive the historical data set returned by the storage device based on the second storage message.

[0110] In some embodiments, before obtaining the first identifier, the processing module 703 is further configured to stop receiving storage messages from the server; after obtaining the first identifier, the processing module 703 is further configured to resume receiving storage messages from the server.

[0111] It should be noted that the descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0112] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a terminal, server, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0113] This application also provides an electronic device. Figure 8 is a schematic diagram of the composition structure of an electronic device provided in this application. As shown in Figure 8, the electronic device 80 may include:

[0114] Memory 801 is used to store executable instructions.

[0115] The processor 802 is used to implement any of the above-described data transmission methods when executing executable instructions stored in the memory 801.

[0116] The processor 802 mentioned above can be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor.

[0117] The aforementioned computer-readable storage medium or memory 801 may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it may also be various terminals that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0118] This application embodiment further provides a computer storage medium storing computer-executable instructions, which are used to implement any of the data transmission methods provided in the above embodiments.

[0119] Correspondingly, this application embodiment further provides a computer program product, the computer program product including computer executable instructions, which are used to implement any of the data transmission methods provided in the above embodiments.

[0120] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0121] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0122] The methods disclosed in the various method embodiments provided in this application can be arbitrarily combined to obtain new method embodiments without conflict.

[0123] The features disclosed in the various product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0124] The features disclosed in the various method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0125] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0126] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of protection of this application, and these forms are all within the protection scope of this application.

Claims

1. A data transmission method applied to a data processing unit (DPU), wherein the DPU is used to carry data transmission between a server and a storage device; the method includes: In the event of a data transmission error, obtain the first identifier; The first identifier is the identifier of the latest completed first data transfer operation fed back by the storage device to the DPU; The first data transmission operation represents a data transmission operation initiated by the server to the storage device; The server is notified to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, so that the server sends each stored message in the stored message sequence in a preset order, starting from the stored message corresponding to the first identifier. Receive the stored message sequence and perform data transmission operations sequentially based on the stored message sequence.

2. The method according to claim 1, wherein, The first identifier is a used descriptor index (Used idx). Before notifying the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, the method further includes: Update the first identifier to the second identifier, where the second identifier is the descriptor index Avail idx to be sent; The notification to the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence includes: The server is notified to use the storage message corresponding to the second identifier as the first storage message in the storage message sequence.

3. The method according to claim 1, wherein, The data transmission operation based on the stored message sequence includes: Send the storage message sequence to the storage device; Receive the first data set returned by the storage device based on the storage message sequence; Determine whether there is duplicate data in the first data set and the historical data set; if there is duplicate data, remove duplicate data from the historical data set to obtain the second data set; the historical data set is the data set received by the DPU before receiving the first data set; The second data set is returned to the server.

4. The method according to claim 3, wherein, Before sending the storage message sequence to the storage device, the method further includes: Based on the second stored message in the DPU, a second data transmission operation is initiated to the storage device; the second stored message represents a stored message that the DPU received but did not send to the storage device before the data transmission error occurred. Receive the historical data set returned by the storage device based on the second storage message.

5. The method according to claim 1, wherein, Before obtaining the first identifier, the method further includes: Stop receiving stored messages from the server; After obtaining the first identifier, the method further includes: Resume receiving stored messages from the server.

6. A data processing unit (DPU) for carrying data transmission between a server and a storage device; the DPU includes a storage high-performance development kit (SPDK) and a field-programmable gate array (FPGA), wherein, The SPDK is used to restart in the event of a data transmission error, and after restarting, it obtains a first identifier and sends the first identifier to the FPGA. The first identifier is the identifier of the latest completed first data transmission operation fed back by the storage device to the DPU. The first data transmission operation represents a data transmission operation initiated by the server to the storage device; The FPGA is used to receive the first identifier and notify the server to use the storage message corresponding to the first identifier as the first storage message in the storage message sequence, so that the server sends each storage message in the storage message sequence in a preset order, starting from the storage message corresponding to the first identifier. Receive the stored message sequence and perform data transmission operations sequentially based on the stored message sequence.

7. A data transmission apparatus applied to a data processing unit (DPU), the DPU being used to carry data transmission between a server and a storage device; the apparatus comprising: The acquisition module is configured to acquire the first identifier in the event of a data transmission error. The first identifier is the identifier of the latest completed first data transfer operation fed back by the storage device to the DPU; The first data transmission operation represents a data transmission operation initiated by the server to the storage device; The notification module is configured to notify the server to use the stored message corresponding to the first identifier as the first stored message in the stored message sequence, so that the server sends each stored message in the stored message sequence in a preset order, starting from the stored message corresponding to the first identifier. The processing module is configured to receive the stored message sequence and perform data transmission operations sequentially based on the stored message sequence.

8. An electronic device comprising a processor and a memory for storing a computer program capable of running on the processor; wherein, The processor is used to run the computer program to perform the method according to any one of claims 1 to 5.

9. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the method of any one of claims 1 to 5.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 5.

Images