Data processing method, device, storage medium and computer program product

Abstract

Disclosed in the present application are a data processing method, a device, a storage medium and a computer program product. The method comprises: at a computing layer, a first device receiving a first processing request sent by a second device, wherein the first processing request is used to request to write first data into a storage layer of the first device, the first data is stored in a first buffer, and the first processing request comprises address information of the first buffer and feature information of the first data; and using the address information of the first buffer and the feature information of the first data to write the first data into the storage layer. In the technical solution provided in the present application, data write processing or data read processing is implemented between devices by means of transmitting address information of a data buffer, thereby achieving zero copy of data. In this way, the present application can reduce resource occupation, overheads and processing delay, thereby greatly improving device performance.

Description

Data processing methods, equipment, storage media and computer program products

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202411907270.0, filed on December 23, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of network transmission technology, and in particular to a data processing method, device, storage medium, and computer program product. Background Technology

[0004] Remote Direct Memory Access (RDMA) is a high-performance network transmission technology that enables efficient or high-performance data transfer between two electronic devices, reducing direct intervention from the Central Processing Unit (CPU). The core of RDMA lies in the capabilities of the RDMA network interface card (NIC), allowing electronic devices to directly access memory for data transfer. This is highly useful for applications requiring low latency and high throughput, such as high-performance computing, large-scale data transfer, and cloud computing environments.

[0005] However, in the process of data transmission between electronic devices based on RDMA, there are problems such as high resource consumption and high overhead, which will affect the performance of electronic devices. Summary of the Invention

[0006] To address the related technical problems, embodiments of this application provide a data processing method, apparatus, storage medium, and computer program product.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides a data processing method applied to a first device, including:

[0009] In the computing layer, a first processing request sent by the second device is received. The first processing request is used to request that first data be written into the storage layer of the first device. The first data is stored in the first buffer. The first processing request includes the address information of the first buffer and the feature information of the first data.

[0010] Using the address information of the first buffer and the feature information of the first data, the first data is written into the storage layer.

[0011] In the above scheme, receiving the first processing request sent by the second device includes:

[0012] The first thread receives the first processing request, and the first thread is used at least for information transmission between the first device and the second device.

[0013] In the above scheme, writing the first data into the storage layer using the address-related information of the first buffer and the feature information of the first data includes:

[0014] The second thread reads the first data based on the address information of the first buffer and the feature information of the first data. The second thread is used at least for information transmission between the computing layer and the storage layer.

[0015] The second thread writes the first data read into the storage layer.

[0016] The method in the above scheme further includes:

[0017] A third thread generates second data, which is used to describe the first buffer. The third thread is used at least to perform business-related processing on processing requests associated with the data.

[0018] The second thread writes the second data into the storage layer.

[0019] In the above scheme, the first data is stored in the first buffer in the form of SGE, and each SGE is used to describe the size and / or location of a data block associated with the first data.

[0020] This application also provides a data processing method applied to a first device, including:

[0021] At the computing layer, a second processing request is received from the second device. The second processing request is used to request the reading of third data from the storage layer of the first device. The second processing request contains feature-related information of the third data, and the third data is stored in one or more second buffers.

[0022] Using the feature-related information of the third data, the address information of the one or more second buffers is obtained from the storage layer;

[0023] Send the address information of the one or more second buffers to the second device.

[0024] In the above scheme, receiving the second processing request sent by the second device includes:

[0025] The first thread receives the second processing request, and the first thread is used at least for information transmission between the first device and the second device.

[0026] In the above scheme, obtaining the address information of the one or more second buffers from the storage layer using the feature-related information of the third data includes:

[0027] The second thread sends the second processing request to the storage layer, and the second thread is used at least for information transmission between the computing layer and the storage layer.

[0028] The second thread receives address information of one or more second buffers sent by the storage layer.

[0029] In the above scheme, sending the second processing request to the storage layer via the second thread includes:

[0030] The second processing request is processed by a third thread to obtain query results. The third thread is used at least to perform business-related processing on processing requests associated with data. The query results represent the number and / or location of the second buffer associated with the third data.

[0031] The second processing request is segmented using the query results through the third thread to obtain one or more third processing requests, each of which contains feature-related information of the third data.

[0032] The second thread sends one or more third processing requests to the storage layer.

[0033] The method in the above scheme further includes:

[0034] The third thread uses the logical address information of the third data to sort the address information of one or more second buffers to obtain the sorted address information of one or more second buffers. The third thread is used at least to perform business-related processing on processing requests associated with the data.

[0035] The first thread sends the address information of one or more second buffers after sorting to the second device. The first thread is used at least for information transmission between the first device and the second device.

[0036] In the above scheme, the third data is stored in the one or more second buffers in the form of SGE, and each SGE is used to describe the size and / or location of a data block associated with the third data.

[0037] This application also provides an apparatus, including: a processor and a memory for storing a computer program capable of running on the processor.

[0038] When the processor runs the computer program, it executes the steps of any of the data processing methods described above.

[0039] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of any of the data processing methods described above.

[0040] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the data processing methods described above.

[0041] The data processing method, apparatus, storage medium, and computer program product provided in this application embodiment, in the computing layer, a first device receives a first processing request sent by a second device. The first processing request is used to request that first data be written into the storage layer of the first device. The first data is stored in a first buffer. The first processing request includes address information of the first buffer and feature information of the first data. Using the address information of the first buffer and the feature information of the first data, the first data is written into the storage layer. In the computing layer, the first device receives a second processing request sent by the second device. The second processing request is used to request that third data be read from the storage layer of the first device. The second processing request includes feature-related information of the third data. The third data is stored in one or more second buffers. Using the feature-related information of the third data, the address information of the one or more second buffers is obtained from the storage layer. The address information of the one or more second buffers is sent to the second device. The technical solution provided in this application adopts a memory address sharing mechanism during data transmission between the client (i.e., the second device) and the network device (i.e., the first device). By transmitting the address information of the data buffer, the write or read processing of data is realized, thereby avoiding data copying operations. In other words, the zero-copy technology can avoid multiple data copies during the data transmission process, thereby greatly reducing the occupation, overhead and processing latency of resources (such as memory or bandwidth), and improving the system performance of the network device. Attached Figure Description

[0042] Figure 1 is a schematic flowchart of the first data processing method according to an embodiment of this application;

[0043] Figure 2 is a schematic diagram of a data writing structure according to an embodiment of this application;

[0044] Figure 3 is a flowchart illustrating the second data processing method according to an embodiment of this application;

[0045] Figure 4 is a schematic diagram of a data reading structure according to an embodiment of this application;

[0046] Figure 5 is a schematic diagram of the structure of the first data processing device according to an embodiment of this application;

[0047] Figure 6 is a schematic diagram of the structure of a second type of data processing device according to an embodiment of this application;

[0048] Figure 7 is a schematic diagram of the device structure according to an embodiment of this application. Detailed Implementation

[0049] The present application will now be described in further detail with reference to the accompanying drawings and embodiments.

[0050] In related technologies, when using traditional network interface cards (NICs), the data transmission process between the client and the network device includes the following steps:

[0051] Step 1: The application puts the data to be sent into the application buffer (which can also be understood as user space), and the application is deployed on the client.

[0052] Step 2: The application passes the data to the kernel through a system call; during this process, the data is copied from user space to kernel space, which is the first data copy.

[0053] Step 3: The kernel's Transmission Control Protocol (TCP) / Internet Protocol (IP) stack performs protocol-related processing, such as adding TCP header information and determining routes.

[0054] Step 4: The protocol stack passes the data to the network device driver; during this process, the data needs to be copied from the protocol stack's kernel buffer to the network device's buffer, which is the second data copy.

[0055] Step 5: Once the data arrives at the network device's buffer, a hardware interrupt is usually triggered, requiring the CPU to process the data.

[0056] Step 6: The CPU begins processing the interrupt and transmits the data to the network device; during this process, the data is transferred from the network device's buffer to the network device's transmit queue, which is the third data copy.

[0057] Step 7: The network device sends the data to the target address.

[0058] As described above, the absence of Direct Memory Access (DMA) introduces more data copying operations (or actions) and CPU intervention. This can lead to higher CPU overhead, especially in high-speed networks or high-throughput scenarios. DMA, on the other hand, reduces direct CPU involvement, thereby improving data transfer efficiency and performance.

[0059] Furthermore, when using an RDMA network card, for write operations, the client transmits the data to be written to the network device (such as an Enterprise Solid-State Drive (ESSD)) via Remote Procedure Call (RPC). This data is represented in the form of Scatter-Gather Entries (SGEs) in RDMA. At the compute layer, considering the system availability and data integrity of the network device, some metadata information needs to be generated and written to the network device along with the data. Specifically, the network device requests a contiguous memory buffer from the operating system through a memory allocation function at the compute layer, and copies the metadata information and the data in the SGE buffer to the operating system's memory buffer. Then, the operating system's memory buffer is copied to the RDMA SGEs for transmission to the storage layer for disk writing. This process involves two data copy operations.

[0060] For read operations, since the data being written may be distributed across different instances of the storage layer, when a client needs to read a contiguous block of data, the read request will be split into multiple sub-requests by the network device's compute layer. These sub-requests will be sent to the storage layer to retrieve the relevant data buffers. The compute layer then integrates the retrieved data buffers into a contiguous block and returns the integrated buffer to the client. Specifically, after the compute layer obtains all the non-contiguous data buffers, it requests a certain number of non-contiguous memory buffers from the operating system. Next, the compute layer copies the multiple non-contiguous data buffers returned from the storage layer into these memory buffers. Then, the compute layer requests a contiguous memory buffer from the operating system and copies these non-contiguous data buffers sequentially into the contiguous memory buffer. Finally, the compute layer copies this contiguous memory buffer to the SGE buffer used for RDMA and returns the SGE buffer to the client. This process involves three data copy operations.

[0061] As can be seen from the above description, both write and read operations involve multiple data copying operations, which can lead to the following problems:

[0062] 1. High CPU overhead: Data copying requires direct CPU intervention. Each time data is copied from one memory location to another, the CPU needs to perform data transfer operations. Therefore, the CPU consumes time and computing resources to perform these copy operations, potentially leading to a decrease in processor performance. Particularly under high load, high CPU overhead can become a bottleneck for the operating system, limiting overall performance.

[0063] 2. High Resource Consumption: Because data needs to be transferred between different memory locations, it consumes memory bandwidth resources, putting additional pressure on the memory subsystem, especially in high-throughput applications. This memory bandwidth consumption may limit other tasks' access to memory resources, further impacting system performance. Furthermore, data copying requires the operating system's involvement in memory allocation and management, involving operations such as requesting memory buffers from the operating system, releasing memory, and managing memory pages. These processes consume time and computing resources and may lead to memory fragmentation, resulting in inefficient memory usage.

[0064] 3. Data redundancy: Multiple data copies (also known as multiple data copies) mean that the same data may be stored in memory multiple times. Each data copy will lead to redundant storage of data, which not only wastes memory space, but may also lead to data inconsistency problems. That is, if the data in one copy is modified, the other copies may not be updated in time.

[0065] 4. High processing latency: Because data must be copied from one memory location to another, a certain amount of latency is introduced. For applications requiring low-latency responses, this additional latency can become a performance bottleneck, thereby reducing the system's real-time performance.

[0066] 5. High memory management complexity: Because different memory buffers need to be tracked during the data copying process to ensure the integrity and correctness of the data, the complexity of memory management is increased, making memory difficult to maintain and debug.

[0067] In summary, under an RDMA-based distributed storage architecture, multiple data copies introduce several problems into the data transmission process. These problems need to be carefully considered during system design and optimization to improve system performance, efficiency, and maintainability.

[0068] Based on this, in various embodiments of this application, during data transmission between the client and network device, the data buffer is located using its address information to enable data writing and reading without requiring multiple copies of the data in the buffer. This minimizes the number of data copies during write and read operations, thereby reducing overhead and resource consumption. Thus, even under high load, the network device can efficiently process data requests and provide better system performance and response time.

[0069] This application provides a data processing method, as shown in FIG1, applied to a first device, the method comprising:

[0070] Step 101: In the computing layer, a first processing request is received from the second device. The first processing request is used to request that first data be written into the storage layer of the first device. The first data is stored in the first buffer. The first processing request includes the address information of the first buffer and the feature information of the first data.

[0071] Step 102: Using the address information of the first buffer and the feature information of the first data, write the first data into the storage layer.

[0072] In practical applications, the first device can be referred to as a network device, hardware device, or electronic device, etc. The first device may include a disk or hard disk, etc. This application embodiment does not limit the type of the first device, as long as its function is implemented. Furthermore, the first device can support distributed storage; that is, the storage layer of the first device can contain multiple instances (also called storage services or storage engines), and can interact with multiple instances (also called computing services or computing engines) contained in the computing layer and store data in multiple buffers to achieve distributed data storage.

[0073] In practical applications, the second device can be called a client, user device, etc. In this application embodiment, the name of the second device is not limited, as long as its function is implemented.

[0074] In practical applications, in step 101, at the computing layer, the first device can receive the first processing request through a thread used to communicate with the second device; wherein, the first device can communicate with the second device based on RDMA.

[0075] Specifically, in one embodiment, receiving the first processing request sent by the second device includes:

[0076] The first thread receives the first processing request, and the first thread is used at least for information transmission between the first device and the second device.

[0077] The first thread can be understood as a thread of the computing layer, which can realize communication between the computing layer and the second device. The first thread can be called a request receiving thread. The number of the first threads can be one or more, and this application embodiment does not limit this.

[0078] In practical applications, when it is necessary to write the first data, the second device can request the first buffer for the first data and store the first data in the first buffer in a specific data format.

[0079] In one embodiment, the first data is stored in the first buffer in the form of SGEs, where each SGE is used to describe the size and / or location of a data block associated with the first data.

[0080] In practical applications, when the first buffer stores data in the form of SGE, the first buffer can be called the first SGE buffer, the first SGE data buffer, etc.; where SGE can be understood as a single element (also called an item) in SGL, which can describe the size (also understood as length) and / or location of a data block associated with the first data, such as the starting address (also understood as memory address) or length of the data block in the buffer.

[0081] It's important to note that SGL is a data structure, typically composed of multiple SGEs, that describes the location and size of multiple data blocks that a device needs to transfer. This allows the device to know how to read and transfer data. SGL at least supports the device reading data from multiple locations in a scattered manner and writing data to multiple locations in a gatherd manner. Therefore, SGL and SGE can be understood as key elements of RDMA-based data operations, enabling devices to efficiently transfer data while simultaneously reading and writing data from multiple locations. This ability to scatter and gather is crucial for handling large-scale data transfers and high-performance computing. By using SGL and SGE, devices can manage and transfer data in a highly flexible manner, while reducing CPU intervention and data copying.

[0082] In practical applications, after storing the first data, the second device can send the first processing request to the first device, enabling the first device to receive the first processing request through the first thread. The characteristic information of the first data may include its size and offset information, etc. The characteristic information of the first data and the address information of the first buffer can be sent in the form of SGEs. That is, the first processing request can contain multiple SGEs, and each SGE can contain the address information of the first buffer corresponding to a data block associated with the first data, as well as the characteristic information corresponding to the data block. This allows for the transmission of multiple non-contiguous data blocks.

[0083] In practical applications, in step 102, at the computing layer, the first device can write the first data through a thread used for information transmission between the computing layer and the storage layer.

[0084] Specifically, in one embodiment, writing the first data into the storage layer using the address-related information of the first buffer and the feature information of the first data includes:

[0085] The second thread reads the first data based on the address information of the first buffer and the feature information of the first data. The second thread is used at least for information transmission between the computing layer and the storage layer.

[0086] The second thread writes the first data read into the storage layer.

[0087] The second thread can be understood as a thread of the computing layer, which can realize read and write interaction between the computing layer and the storage layer. The second thread can be called an interaction thread, and the number of the second thread can be one or more, which is not limited in this embodiment.

[0088] In practical applications, to write the first data, the first device can use a third thread to reference the address information of the first buffer from the first processing request and pass the address information of the first buffer to the second thread. This allows the second thread to locate the physical address of the first buffer based on the address information, and then read the first data and write it by combining the characteristic information of the first data. The third thread can be understood as a thread in the computing layer, and there can be one or more third threads. These threads can perform business-related processing (such as verification or query processing) on ​​the processing request. The third thread can be called a task processing thread, which, together with the first and second threads, forms a three-layer thread model in the computing layer. This allows the data writing and reading processes to be executed in an orderly manner based on the three-layer thread model. In other words, using different threads to perform different functions ensures the correctness of data processing, prevents data corruption or loss, helps maintain data integrity and consistency, and improves the maintainability and scalability of the system.

[0089] In practical applications, during the process of writing the first data, the first device can also generate metadata associated with the first data to ensure the correctness of the data.

[0090] Based on this, in one embodiment, the method may further include:

[0091] A third thread generates second data, which is used to describe the first buffer. The third thread is used at least to perform business-related processing on processing requests associated with the data.

[0092] The second thread writes the second data into the storage layer.

[0093] The second data can be understood as metadata, and the second data can at least indicate the start address or end address of the first data in the first buffer.

[0094] Here, through the third thread, the first device can construct the second data, and combine the first data and the second data as a whole, and write the whole data into the first buffer through the second thread.

[0095] In practical applications, after the first data and the second data are written, the first device can send a first request response to the second device through the first thread, so that the second device can know that the first data has been successfully written.

[0096] For example, as shown in Figure 2, in the architecture of the three-layer thread model, after the first device (also known as a block server) receives multiple SGEs (data) contained in the first processing request through the first thread (which can be described as rev_worker), the third thread (which can be described as rpc_worker) directly references the address information (which can be described as ref) of the first buffer in the SGE and passes the address information of the first buffer to the second thread (which can be described as task_worker). The second thread locates the physical address of the first buffer and forwards the first data to the storage engine in the storage layer (worker memory) to write the first data.

[0097] In addition, through the third thread, the first device can also record the mapping relationship between the physical address information of the first buffer and the logical address information of the first data, so as to perform data querying later.

[0098] In this embodiment, the data writing scheme based on RDMA memory sharing technology can avoid copying data from the first buffer to memory, thereby minimizing the number of data copies between the computing layer and the storage layer. This reduces redundant data storage in memory, lowers CPU overhead, and reduces memory bandwidth consumption. It not only improves the efficiency of write operations but also helps reduce system latency, thus bringing significant performance and efficiency advantages to the system.

[0099] Accordingly, this application also provides a data processing method applied to a first device, as shown in FIG3, comprising the following steps:

[0100] Step 301: In the computing layer, a second processing request is received from the second device. The second processing request is used to request the reading of third data from the storage layer of the first device. The second processing request contains feature-related information of the third data, and the third data is stored in one or more second buffers.

[0101] Step 302: Using the feature-related information of the third data, obtain the address information of the one or more second buffers from the storage layer;

[0102] Step 303: Send the address information of the one or more second buffers to the second device.

[0103] In practical applications, in the reading scenario of the computing layer, the second processing request can be obtained through the first thread.

[0104] Specifically, in one embodiment, receiving the second processing request sent by the second device includes:

[0105] The first thread receives the second processing request, and the first thread is used at least for information transmission between the first device and the second device.

[0106] In practical applications, the third data may be stored in multiple second buffers in a distributed manner or in one second buffer in an aggregated manner; wherein, the third data may be stored in one or more second buffers in a specific data format.

[0107] Specifically, in one embodiment, the third data may be stored in the one or more second buffers in the form of SGEs, each SGE being used to describe the size and / or location of a data block associated with the third data.

[0108] In the case where the second buffer stores data in the form of SGE, the second buffer may be called the second SGE buffer, the second SGE data buffer, etc., and this application embodiment does not limit it.

[0109] In practical applications, in step 302, the first device can request a data segment associated with the third data from the storage layer through the second thread.

[0110] Specifically, in one embodiment, obtaining the address information of the one or more second buffers from the storage layer using the feature-related information of the third data includes:

[0111] The second thread sends the second processing request to the storage layer, and the second thread is used at least for information transmission between the computing layer and the storage layer.

[0112] The second thread receives address information of one or more second buffers sent by the storage layer.

[0113] In practical applications, since the third data may be distributed and stored in multiple second buffers, the first device can send the second processing request according to the storage status of the third data.

[0114] Specifically, in one embodiment, sending the second processing request to the storage layer via a second thread includes:

[0115] The second processing request is processed by a third thread to obtain query results. The third thread is used at least to perform business-related processing on processing requests associated with data. The query results represent the number and / or location of the second buffer associated with the third data.

[0116] The second processing request is segmented using the query results through the third thread to obtain one or more third processing requests, each of which contains feature-related information of the third data.

[0117] The second thread sends one or more third processing requests to the storage layer.

[0118] In practical applications, the third processing request can be understood as a sub-request split from the second processing request.

[0119] Here, through the third thread, the first device can perform verification processing on the second processing request; if the verification passes, it performs query-related processing on the second processing request to obtain the query result, thereby determining the storage distribution of the third data. Once the storage distribution of the third data is determined, through the third thread, the first device can divide the second processing request into one or more third processing requests, and transmit the one or more third processing requests in parallel to one or more second threads, so that the one or more second threads can request the third data from one or more instances of the storage layer; wherein, the number of third processing requests is related to the distribution of the third data, for example, if the third data is distributed and stored in 5 different second buffers, then the number of third processing requests is 5.

[0120] In practical applications, through the second thread, the first device can obtain the address information of the one or more second buffers from the storage layer; wherein, the address information of the one or more second buffers can be transmitted in the form of SGE; that is, the first device can receive one or more SGEs returned by the storage layer, and each SGE can contain the address information of the second buffer corresponding to a data block associated with the third data. During the transmission of the third data from the storage layer to the computing layer, no redundant copies are generated in the memory of the computing layer.

[0121] In this embodiment, a three-layer thread model is used to divide the read request into multiple sub-requests for parallel processing, which improves the processing efficiency of read operations and enables the system to better handle high-concurrency read requests. Furthermore, parallel processing of multiple sub-requests fully utilizes multi-core CPU resources, reducing resource waste and improving system resource utilization. The aforementioned data processing scheme has potential applications in many fields (especially those requiring high performance and low latency), including cloud computing, big data processing, high-performance storage, and scientific computing.

[0122] In practical applications, since the actual storage order of the third data in the second buffer may be different from the logical order of the third data, the first device can adjust the order of the second buffer to ensure the orderliness of the data.

[0123] Based on this, in one embodiment, the method may further include:

[0124] The third thread uses the logical address information of the third data to sort the address information of one or more second buffers to obtain the sorted address information of one or more second buffers. The third thread is used at least to perform business-related processing on processing requests associated with the data.

[0125] The first thread sends the address information of one or more second buffers after sorting to the second device. The first thread is used at least for information transmission between the first device and the second device.

[0126] In practical applications, using the second processing request, the first device can determine the logical address information of the third data through the third thread. This logical address information includes the Logical Block Address (LBA). Then, using the logical address information of the third data, the first device can sort the address information of one or more second buffers in ascending order. Finally, the first thread sends the sorted address information of one or more second buffers back to the second device. In this process, zero-copy operation efficiently organizes, transmits, and returns data, avoiding multiple copies of data in system memory, reducing CPU overhead and memory bandwidth consumption, thereby significantly improving the efficiency and performance of read operations.

[0127] For example, as shown in Figure 4, in the architecture of the three-layer thread model, for the second processing request, the first device can obtain multiple SGE buffers (which can be understood as data fragments of third data returned in the form of SGE) from the storage layer through the second thread; through the third thread, the address information of the SGE buffers is sorted according to the size of the LBA address to generate an SGE list; through the first thread, the SGE list is added to the SGL and the SGL is sent to the second device.

[0128] The data processing method provided in this application embodiment includes the following steps: At the computing layer, a first device receives a first processing request sent by a second device. The first processing request requests that first data be written into the storage layer of the first device. The first data is stored in a first buffer. The first processing request includes address information of the first buffer and feature information of the first data. Using the address information of the first buffer and the feature information of the first data, the first data is written into the storage layer. At the computing layer, the first device receives a second processing request sent by the second device. The second processing request requests that third data be read from the storage layer of the first device. The second processing request includes feature-related information of the third data. The third data is stored in one or more second buffers. Using the feature-related information of the third data, the address information of the one or more second buffers is obtained from the storage layer. The address information of the one or more second buffers is then sent to the second device. The technical solution provided in this application adopts a memory address sharing mechanism during data transmission between the client (i.e., the second device) and the network device (i.e., the first device). By transmitting the address information of the data buffer, the write or read processing of data is realized, thereby avoiding data copying operations. In other words, the zero-copy technology can avoid multiple data copies during the data transmission process, thereby greatly reducing the occupation, overhead and processing latency of resources (such as memory or bandwidth), and improving the system performance of the network device.

[0129] To implement the method of the embodiments of this application, the embodiments of this application also provide a data processing apparatus, disposed on a first device, as shown in FIG5, the apparatus including:

[0130] The first receiving unit 501 is configured to receive a first processing request sent by the second device at the computing layer. The first processing request is used to request that first data be written into the storage layer of the first device. The first data is stored in a first buffer. The first processing request includes address information of the first buffer and feature information of the first data.

[0131] The writing unit 502 is configured to write the first data into the storage layer using the address information of the first buffer and the feature information of the first data.

[0132] In one embodiment, the first receiving unit 501 is configured to receive the first processing request via a first thread, wherein the first thread is used at least for information transmission between the first device and the second device.

[0133] In one embodiment, the writing unit 502 is configured as follows:

[0134] The second thread reads the first data based on the address information of the first buffer and the feature information of the first data. The second thread is used at least for information transmission between the computing layer and the storage layer.

[0135] The second thread writes the first data read into the storage layer.

[0136] In one embodiment, the writing unit 502 is further configured as follows:

[0137] A third thread generates second data, which is used to describe the first buffer. The third thread is used at least to perform business-related processing on processing requests associated with the data.

[0138] The second thread writes the second data into the storage layer.

[0139] In practical applications, the first receiving unit 501 can be implemented by the communication interface in the data processing device; the writing unit 502 can be implemented by the processor in the data processing device.

[0140] To implement the method of the embodiments of this application, the embodiments of this application also provide a data processing apparatus, disposed on a first device, as shown in FIG6, the apparatus including:

[0141] The second receiving unit 601 is configured to receive a second processing request sent by a second device at the computing layer. The second processing request is used to request the reading of third data from the storage layer of the first device. The second processing request includes feature-related information of the third data. The third data is stored in one or more second buffers.

[0142] The acquisition unit 602 is configured to acquire the address information of the one or more second buffers from the storage layer using the feature-related information of the third data;

[0143] The sending unit 603 is configured to send the address information of the one or more second buffers to the second device.

[0144] In one embodiment, the second receiving unit 601 is configured to receive the second processing request via a first thread, wherein the first thread is used at least for information transmission between the first device and the second device.

[0145] In one embodiment, the acquisition unit 602 is configured as follows:

[0146] The second thread sends the second processing request to the storage layer, and the second thread is used at least for information transmission between the computing layer and the storage layer.

[0147] The second thread receives address information of one or more second buffers sent by the storage layer.

[0148] In one embodiment, the acquisition unit 602 is configured to perform query-related processing on the second processing request through a third thread to obtain query results. The third thread is at least used to perform business-related processing on processing requests associated with data. The query results represent the number and / or location of the second buffer associated with the third data. The second processing request is segmented using the query results through the third thread to obtain one or more third processing requests. Each third processing request contains feature-related information of the third data.

[0149] The sending unit 603 is configured to send one or more third processing requests to the storage layer via the second thread.

[0150] In one embodiment, the acquisition unit 602 is further configured to use a third thread to sort the address information of one or more second buffers using the logical address information of the third data, so as to obtain the sorted address information of one or more second buffers. The third thread is at least used to perform business-related processing on the processing requests associated with the data.

[0151] The sending unit 603 is further configured to send the address information of one or more second buffers after sorting to the second device through a first thread, wherein the first thread is used at least for information transmission between the first device and the second device.

[0152] In practical applications, the second receiving unit 601 and the sending unit 603 can be implemented by the communication interface in the data processing device; the acquiring unit 602 can be implemented by the processor in the data processing device in combination with the communication interface.

[0153] It should be noted that the data processing apparatus provided in the above embodiments is only illustrated by the division of the above program modules. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the apparatus can be divided into different program modules to complete all or part of the processing described above. In addition, the data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0154] Based on the hardware implementation of the above program modules, and in order to implement the method on the first device side of the embodiments of this application, the embodiments of this application also provide a device (i.e., the first device mentioned above), as shown in FIG7, the first device 700 includes:

[0155] The first communication interface 701 is capable of interacting with the second device;

[0156] The first processor 702 is connected to the first communication interface 701 to enable interaction with the second device, and is configured to execute the methods provided by one or more technical solutions on the first device side when running a computer program;

[0157] The computer program is stored in the first memory 703.

[0158] Specifically, in the data writing scheme, the first communication interface 701 is configured to receive a first processing request sent by the second device at the computing layer. The first processing request is used to request that first data be written to the storage layer of the first device. The first data is stored in the first buffer. The first processing request includes the address information of the first buffer and the feature information of the first data.

[0159] The first processor 702 is configured to write the first data into the storage layer using the address information of the first buffer and the feature information of the first data.

[0160] In one embodiment, the first communication interface 701 is configured to receive the first processing request via a first thread, wherein the first thread is used at least for information transmission between the first device and the second device.

[0161] In one embodiment, the first processor 702 is configured as follows:

[0162] The second thread reads the first data based on the address information of the first buffer and the feature information of the first data. The second thread is used at least for information transmission between the computing layer and the storage layer.

[0163] The second thread writes the first data read into the storage layer.

[0164] In one embodiment, the first processor 702 is further configured to:

[0165] A third thread generates second data, which is used to describe the first buffer. The third thread is used at least to perform business-related processing on processing requests associated with the data.

[0166] The second thread writes the second data into the storage layer.

[0167] In the data reading scheme, the first communication interface 701 is configured to receive a second processing request sent by the second device at the computing layer. The second processing request is used to request the reading of third data from the storage layer of the first device. The second processing request includes feature-related information of the third data, and the third data is stored in one or more second buffers.

[0168] The first processor 702 is configured to use the feature-related information of the third data to obtain the address information of the one or more second buffers from the storage layer; and to send the address information of the one or more second buffers to the second device through the first communication interface 701.

[0169] In one embodiment, the first communication interface 701 is configured to receive the second processing request via a first thread, wherein the first thread is used at least for information transmission between the first device and the second device.

[0170] In one embodiment, the first communication interface 701 is configured as follows:

[0171] The second thread sends the second processing request to the storage layer, and the second thread is used at least for information transmission between the computing layer and the storage layer.

[0172] The second thread receives address information of one or more second buffers sent by the storage layer.

[0173] In one embodiment, the first processor 702 is configured to perform query-related processing on the second processing request through a third thread to obtain query results. The third thread is at least used to perform business-related processing on processing requests associated with data. The query results represent the number and / or location of the second buffer associated with the third data. The second processing request is segmented using the query results through the third thread to obtain one or more third processing requests. Each third processing request contains feature-related information of the third data.

[0174] The first communication interface 701 is configured to send one or more third processing requests to the storage layer via the second thread.

[0175] In one embodiment, the first processor 702 is further configured to use a third thread to sort the address information of one or more second buffers using the logical address information of the third data, so as to obtain the sorted address information of one or more second buffers. The third thread is at least used to perform business-related processing on processing requests associated with data.

[0176] The first communication interface 701 is further configured to send the address information of one or more second buffers after sorting to the second device via a first thread, wherein the first thread is used at least for information transmission between the first device and the second device.

[0177] It should be noted that the specific processing procedures of the processor 702 and the communication interface 701 can be understood by referring to the above method.

[0178] Of course, in practical applications, the various components in device 700 are coupled together through bus system 704. It can be understood that bus system 704 is configured to enable communication between these components. In addition to the data bus, bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 704 in Figure 7.

[0179] The memory 703 in this embodiment is configured to store various types of data to support the operation of the device 700. Examples of such data include any computer program used to operate on the device 700.

[0180] The methods disclosed in the embodiments of this application can be applied to, or implemented by, the processor 702. The processor 702 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuitry of the hardware in the processor 702 or by instructions in software form. The processor 702 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 702 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, specifically in memory 703. The processor 702 reads information from memory 703 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0181] In an exemplary embodiment, device 700 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0182] It is understood that the memory (memory 703) in the embodiments of this application can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable types of memories.

[0183] In an exemplary embodiment, this application also provides a storage medium, namely a computer storage medium, specifically a computer-readable storage medium, such as a memory 703 storing a computer program, which can be executed by the processor 702 of the device 700 to complete the steps described in the aforementioned first device-side method. The computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disc, or CD-ROM.

[0184] In an exemplary embodiment, this application also provides a computer program product, including a computer program that can be executed by a processor 702 of a device 700 to perform the steps described in the aforementioned first device-side method.

[0185] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0186] Furthermore, the technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0187] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. A data processing method, applied to a first device, comprising: In the computing layer, a first processing request sent by the second device is received. The first processing request is used to request that first data be written into the storage layer of the first device. The first data is stored in the first buffer. The first processing request includes the address information of the first buffer and the feature information of the first data. Using the address information of the first buffer and the feature information of the first data, the first data is written into the storage layer.

2. The method of claim 1, wherein, The receiving of the first processing request sent by the second device includes: The first thread receives the first processing request, and the first thread is used at least for information transmission between the first device and the second device.

3. The method of claim 1, wherein, The step of writing the first data into the storage layer using the address-related information of the first buffer and the feature information of the first data includes: The second thread reads the first data based on the address information of the first buffer and the feature information of the first data. The second thread is used at least for information transmission between the computing layer and the storage layer. The second thread writes the first data read into the storage layer.

4. The method of claim 3, wherein, The method further includes: A third thread generates second data, which is used to describe the first buffer. The third thread is used at least to perform business-related processing on processing requests associated with the data. The second thread writes the second data into the storage layer.

5. The method according to any one of claims 1 to 4, wherein, The first data is stored in the first buffer in the form of SGEs, each SGE being used to describe the size and / or location of a data block associated with the first data.

6. A data processing method, applied to a first device, comprising: At the computing layer, a second processing request is received from the second device. The second processing request is used to request the reading of third data from the storage layer of the first device. The second processing request contains feature-related information of the third data, and the third data is stored in one or more second buffers. Using the feature-related information of the third data, the address information of the one or more second buffers is obtained from the storage layer; Send the address information of the one or more second buffers to the second device.

7. The method of claim 6, wherein, The receiving of the second processing request sent by the second device includes: The first thread receives the second processing request, and the first thread is used at least for information transmission between the first device and the second device.

8. The method of claim 6, wherein, The step of obtaining the address information of the one or more second buffers from the storage layer using the feature-related information of the third data includes: The second thread sends the second processing request to the storage layer, and the second thread is used at least for information transmission between the computing layer and the storage layer. The second thread receives address information of one or more second buffers sent by the storage layer.

9. The method of claim 8, wherein, The step of sending the second processing request to the storage layer via the second thread includes: The second processing request is processed by a third thread to obtain query results. The third thread is used at least to perform business-related processing on processing requests associated with data. The query results represent the number and / or location of the second buffer associated with the third data. The second processing request is segmented using the query results through the third thread to obtain one or more third processing requests, each of which contains feature-related information of the third data. The second thread sends one or more third processing requests to the storage layer.

10. The method of claim 6, wherein, The method further includes: The third thread uses the logical address information of the third data to sort the address information of one or more second buffers to obtain the sorted address information of one or more second buffers. The third thread is used at least to perform business-related processing on processing requests associated with the data. The first thread sends the address information of one or more second buffers after sorting to the second device. The first thread is used at least for information transmission between the first device and the second device.

11. The method according to any one of claims 6 to 10, wherein, The third data is stored in the one or more second buffers in the form of SGEs, each SGE describing the size and / or location of a data block associated with the third data.

12. An apparatus comprising: The processor and the memory used to store computer programs that can run on the processor. Wherein, when the processor is configured to run the computer program, it performs the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 11.

13. A storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 11.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 11.

Images