Detection method for data reading and writing process, and related apparatus

Abstract

A detection method for a data reading and writing process, which is applied to a data storage scenario. A data packet transmitted between nodes carries a specific latency tag, such that each processing module responsible for executing data reading and writing in the two nodes can record, on the basis of a target identifier, a time point at which the data packet is acquired and a time point at which the processing of the data packet is completed. In addition, each processing module reports the recorded time points to a management node, and the management node performs unified analysis, so as to determine which processing module causes high latency in data reading and writing, thereby accurately positioning a link at which a problem occurs in a data reading and writing process, and thus avoiding erroneous isolation of a storage node.

Description

Method for detecting data read-write process and related device

[0001] The present application claims priority to the Chinese patent application No. 202510213851.2, filed on February 24, 2025, and entitled "Method for detecting data read-write process and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of data storage, and in particular to a method for detecting a data read-write process and related device. BACKGROUND

[0003] Distributed storage refers to storing data on different devices to improve data storage performance. In the implementation process of distributed storage, the same data is usually divided into multiple data segments and sent to different storage nodes for storage. In this way, the time length spent by the last storage node to complete data storage among multiple storage nodes is the completion time length of the entire data storage process. That is, in the distributed storage scenario, for data divided into multiple storage paths, the data storage time length is actually determined by the slowest storage path.

[0004] In the distributed storage scenario, if a storage node has a problem, it will cause the data throughput to decrease, which seriously affects the normal storage or reading of data. Therefore, in the related art, it is usually necessary to detect data storage or data reading, so as to isolate the storage node with a longer data storage time or data reading time, to avoid affecting the data read-write performance.

[0005] However, in the related art, it is usually difficult to accurately locate the link where the problem occurs in the data read-write process, and normal storage nodes are easily mistaken for abnormal storage nodes, thereby easily causing storage node isolation errors. SUMMARY

[0006] The present application provides a method for detecting a data read-write process, which is used to accurately locate the link where the problem occurs in the data read-write process.

[0007] In a first aspect, a method for detecting a data read-write process is provided, which includes: a first processing module acquires a first data packet, the first data packet being a data packet transmitted from a first node to a second node in a first data read-write. The first node or the second node is a storage node, and the first node or the second node includes a plurality of processing modules for performing data processing in the data read-write, and the first processing module is any one of the plurality of processing modules. That is, the first processing module is one processing module in the node participating in the data read-write process, and the first data packet is one data packet to be processed in the data read-write process.

[0008] Then, in response to the target identifier being included in the first data packet, the first processing module records a first time point at which the first data packet is acquired, and the first mark in the target identifier is used to indicate the recording of the acquisition time point and the processing completion time point of the first data packet. Next, the first processing module processes the first data packet and records a second time point at which the first data packet is processed.

[0009] Finally, the first processing module sends a statistical message to the management node, and the statistical message includes the first time point and the second time point corresponding to the first data read-write at the first processing module. The management node is configured to detect data read-write based on the statistical message.

[0010] In the scheme, the data packets transmitted between nodes carry specific target identifiers, so that each processing module responsible for executing data read-write in the two nodes records the time points at which the data packets are acquired and processed based on the target identifiers. Moreover, each processing module reports the recorded time points to the management node, and the management node performs unified analysis to determine which processing module causes the data read-write delay to be long, thereby accurately locating the link in which the data read-write process fails and avoiding error isolation of the storage node.

[0011] In a possible implementation, the first processing module statistics traffic information of the plurality of data read-writes at the first processing module in the first period, and the statistical message further includes the traffic information. The traffic information includes one or more of the following information of the plurality of data read-writes at the first processing module in the first period: the number of received data packets, the number of bytes of received data packets, the number of sent data packets, the number of bytes of sent data packets, the number of lost data packets, and the number of bytes of lost data packets.

[0012] In the scheme, the processing module on the node can analyze and determine the link in which the packet loss phenomenon occurs in the data read-write process by reporting the traffic information such as the packet reception information or the packet loss information in a period to the management node.

[0013] In a possible implementation, the second mark included in the target identifier is used to indicate a period to which the corresponding data packet belongs. The first processing module statistics information of the data packet whose second mark in the target identifier is a first value from the plurality of data packets, to obtain the traffic information. The plurality of data packets are the data packets including the target identifier received or sent by the first processing module, and the first value of the second mark is used to indicate the first period.

[0014] In the scheme, the period to which the data packet belongs is indicated by setting the mark in the target identifier, so that each processing module in the node can count the sending and receiving conditions of the data packet in the same period, so as to realize the counting of the sending and loss conditions of the data packet in each period by each processing module.

[0015] In a possible implementation, the data packet in the plurality of data packets and having the second value of the second mark belongs to a second period, the second period is different from the first period, and the second value is different from the first value. Moreover, the first period and the second period can be adjacent periods.

[0016] That is, different values of the same mark are used to distinguish different periods, so that the processing module can determine the period to which each data packet belongs in the plurality of continuously received data packets, and the counting of the packet receiving condition and the packet loss condition of the data packet in each period is facilitated.

[0017] In a possible implementation, the third mark in the target identifier is used to indicate a flow identifier, the flow identifier is used to indicate a data flow to which the corresponding data packet belongs, and the statistical message further includes the flow identifier.

[0018] In the scheme, the flow identifier is carried in the data packet, so that the processing module can determine the data flow to which the counted data packet corresponds, so as to report the statistical information corresponding to the data flow to the management node, and realize accurate statistics and analysis of the data reading and writing process.

[0019] In a possible implementation, the flow identifier includes a node identifier and a path identifier, the node identifier is used to identify the first node, and the path identifier is used to identify the transmission path of the corresponding data packet. That is, the flow identifier is represented based on the first node of the data reading and writing and the transmission path of the data packet.

[0020] In a possible implementation, one data flow includes a plurality of sub-flows, the target identifier further includes a sub-flow identifier, and the sub-flow identifier is used to indicate a sub-flow to which the corresponding data packet belongs. The flow identifier of the first data packet is used to indicate that the first data packet belongs to the target data flow, and the sub-flow identifier of the first data packet is used to indicate that the first data packet belongs to a first sub-flow in the plurality of sub-flows of the target data flow.

[0021] In the case that the first processing module is located at the first node, the first processing module sends the indication information of the target data flow to the management node, and the indication information of the target data flow is used to indicate the sub-flows included in the target data flow.

[0022] In the scheme, the sub-flow identifier is carried in the data packet, so that the sub-flow to which the data packet belongs can be identified, the scene requirement that the data flow is divided into a plurality of sub-flows for transmission in the data reading and writing scene is met, and the management node can count and analyze the transmission conditions of each sub-flow.

[0023] In a possible implementation, the data packets in the multiple sub-streams of one data stream carry the same node identifier and different path identifiers. That is, the first nodes of different sub-streams are the same, but the transmission paths are different.

[0024] In a possible implementation, the first processing module runs on a network card, a storage medium or a processor of the first node, or the first processing module runs on a network card, a storage medium or a processor of the second node.

[0025] In a possible implementation, when the first processing module runs on the network card, the first processing module detects the target identifier in the protocol header or the payload of the first data packet. Alternatively, when the first processing module runs on the storage medium or the processor, the first processing module detects the target identifier in the payload of the first data packet.

[0026] In a possible implementation, the protocol header is a protocol header of an InfiniBand (IB) protocol or a protocol header of an Internet Protocol Version 6 (IPv6) protocol.

[0027] In a possible implementation, when the first processing module runs on the first node, the first processing module obtains target data to be transmitted to the second node, and generates the first data packet, the first data packet including the target data and the target identifier.

[0028] In a possible implementation, the first processing module receives a notification message from the management node, and the notification message includes any one of the following information: an instruction for the first processing module to stop performing data read and write, an instruction for the first processing module to restart, or an instruction for the first processing module to stop transmitting data through a target path, the target path being a path through which the first processing module transmits the first data packet.

[0029] In a second aspect, a method for detecting a data read-write process is provided, including: receiving, by a management node, a first statistical message and a second statistical message sent by a first processing module and a second processing module in a plurality of processing modules, respectively, the plurality of processing modules being modules running on a first node or a second node and used for performing data processing in the data read-write process, the first node or the second node being a storage node, the first statistical message including a first time point and a second time point corresponding to the first processing module in a first data read-write, the second statistical message including a third time point and a fourth time point corresponding to the second processing module in the first data read-write, the first time point being used to indicate a time point at which the first processing module acquires a first data packet, the second time point being used to indicate a time point at which the first processing module finishes processing the first data packet, the third time point being used to indicate a time point at which the second processing module acquires the first data packet, and the fourth time point being used to indicate a time point at which the second processing module finishes processing the first data packet, the first data packet being a data packet that needs to be transmitted from the first node to the second node in the first data read-write.

[0030] Then, based on the first statistical message and the second statistical message, the management node sends a first notification message to the first processing module in a case where it is determined that the first processing module in the plurality of processing modules is in a sub-healthy state, the first notification message being used to instruct the first processing module to stop performing the data read-write or to instruct the first processing module to restart.

[0031] In this solution, the management node determines the time points at which each processing module acquires a data packet and finishes processing the data packet by acquiring the statistical message reported by each processing module participating in the data read-write process, and then analyzes whether the data read-write delay is caused by which processing module, so as to accurately locate the link where the problem occurs in the data read-write process and avoid isolating the storage node by mistake.

[0032] In a possible implementation, based on the first statistical message, the management node determines that the first processing module in the plurality of processing modules is in a sub-healthy state, including: based on the first statistical message, in a case where it is determined that a time delay of the first processing module in processing the data packet satisfies a preset condition, the management node determines that the first processing module is in a sub-healthy state, the preset condition including that an average time delay of the first processing module in processing a plurality of data packets exceeds a first time delay value.

[0033] In this solution, after the management node locates the plurality of data streams with abnormal transmission time delays from the perspective of clustering and analyzing the data stream transmission time delays, the management node locates the processing module with abnormal time delays in processing data packets in the plurality of data streams, and then regards the processing module as being in a sub-healthy state, so as to accurately locate the abnormal processing module and avoid isolating the processing module without abnormality by mistake.

[0034] In a possible implementation, the second processing module is a next processing module of the first processing module in the plurality of processing modules, and the management node determines a target transmission time delay of the first processing module transmitting the first data packet to the second processing module based on the second time point and the third time point; in a case where the target transmission time delay exceeds the second time delay value, the management node sends a second notification message to the first processing module, and the second notification message is used to instruct the first processing module to stop transmitting data through a target path, and the target path is a path of the first processing module transmitting the first data packet to the second processing module.

[0035] In this scheme, the management node determines the time delay of the transmission path between the two adjacent processing modules based on the time points at which the two adjacent processing modules respectively transmit and receive the data packet, can accurately locate the transmission path with the abnormal transmission time delay, and helps to improve the subsequent data read-write efficiency.

[0036] In a possible implementation, the first statistical message further includes first traffic information of a plurality of data read-writes at the first processing module in the first period, and the second statistical message further includes second traffic information of the plurality of data read-writes at the second processing module in the first period, the first traffic information includes a quantity of data packets sent by the first processing module after processing the data packets belonging to the first period, and the second traffic information includes a quantity of data packets belonging to the first period and obtained by the second processing module from the first processing module.

[0037] Based on the first traffic information and the second traffic information, the management node determines a first packet loss quantity, and the first packet loss quantity is used to indicate a packet loss quantity of the transmission path between the first processing module and the second processing module; in a case where the first packet loss quantity is greater than a preset packet loss value, the management node sends a second notification message to the first processing module, and the second notification message is used to instruct the first processing module to stop transmitting data through a target path, and the target path is a path of the first processing module transmitting the data packets belonging to the first period to the second processing module.

[0038] In a possible implementation, the first statistical message further includes a flow identifier, and the flow identifier is used to indicate a data flow to which the data packet corresponding to the first statistical message belongs. The management node determines a data flow corresponding to the first data packet based on the flow identifier, so as to analyze the difference between the transmission conditions of different data flows.

[0039] In a possible implementation, the flow identifier includes a node identifier and a path identifier, the node identifier is used to identify the first node, and the path identifier is used to identify the transmission path of the first data packet. The management node determines the transmission path of the first data packet based on the path identifier, so as to analyze the difference between the transmission time delays of different transmission paths.

[0040] In a possible implementation, the first statistical message further includes a sub-flow identifier, the sub-flow identifier being used to indicate that the first data packet corresponding to the first statistical message belongs to a target sub-flow in a target data flow, and the target data flow includes a plurality of sub-flows that depart from the first node. The management node determines transmission delays of the plurality of sub-flows of the target data flow, and in a case where a transmission delay of a target sub-flow in the plurality of sub-flows is greater than a third delay value, the management node sends a sub-flow adjustment message to the first node, the sub-flow adjustment message being used to instruct the first node to adjust a transmission path of the target sub-flow.

[0041] In a possible implementation, the first processing module runs on a network card, a storage medium, or a processor of the first node, or the first processing module runs on a network card, a storage medium, or a processor of the second node.

[0042] In a third aspect, a data read-write device is provided, including a processor, a network card, and a memory, the memory being configured to store program code, and the processor or the network card being configured to invoke the program code in the memory to cause the data read-write device to perform the method according to any of the implementations of the first aspect.

[0043] In a fourth aspect, a management device is provided, including a processor and a memory, the memory being configured to store program code, and the processor being configured to invoke the program code in the memory to cause the management device to perform the method according to any of the implementations of the second aspect.

[0044] In a fifth aspect, a data read-write system is provided, including the data read-write device according to the third aspect and the management device according to the fourth aspect.

[0045] In a sixth aspect, a computer-readable storage medium is provided, which stores instructions, when the instructions are executed on a computer, causing the computer to perform the method according to any of the implementations of the first aspect or the second aspect.

[0046] In a seventh aspect, a computer program product is provided, when the computer program product is executed on a computer, causing the computer to perform the method according to any of the implementations of the first aspect or the second aspect.

[0047] In an eighth aspect, a chip is provided, including one or more processors. Part or all of the processors are configured to read and execute computer instructions stored in a memory to perform the method in any of the possible implementations of any of the aspects. Optionally, the chip further includes the memory. Optionally, the chip further includes a communication interface, and the processor is connected with the communication interface. The communication interface is configured to receive data and / or information to be processed, and the processor obtains the data and / or information from the communication interface, processes the data and / or information, and outputs the processing result through the communication interface. Optionally, the communication interface is an input-output interface or a bus interface. The method provided in the present application is implemented by one chip or by multiple chips in cooperation.

[0048] The solutions provided by the second aspect to the eighth aspect are used to implement or assist in implementing the method provided by the first aspect or the second aspect, and thus can achieve the same or corresponding beneficial effects as the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0049] FIG. 1 is a schematic diagram of a system architecture provided by the present application;

[0050] FIG. 2A is a schematic diagram of an architecture of an application scenario provided by the present application;

[0051] FIG. 2B is a schematic diagram of an architecture of another application scenario provided by the present application;

[0052] FIG. 3 is a schematic diagram of a flow of a detection method of a data read-write process provided by the present application;

[0053] FIG. 4 is a schematic diagram of a format of a data packet provided by the present application;

[0054] FIG. 5 is a schematic diagram of a flow of detection data storage in a distributed storage scenario provided by the present application;

[0055] FIG. 6 is a schematic diagram of a format of an INT header added in a data packet provided by the present application;

[0056] FIG. 7 is a schematic diagram of data packet related information reported by a processing module to a management node provided by the present application;

[0057] FIG. 8 is a schematic diagram of data interaction between a computing node and a storage node based on a connection session provided by the present application;

[0058] FIG. 9 is a schematic diagram of a format change of a data packet provided by the present application;

[0059] FIG. 10 is a schematic diagram of a principle of packet loss detection provided by the present application;

[0060] FIG. 11 is a schematic diagram of a principle of latency detection provided by the present application;

[0061] FIG. 12 is a schematic diagram of a state machine of a transmission path provided by the present application;

[0062] FIG. 13 is a schematic diagram of a state machine of a processing module in a node provided by the present application;

[0063] FIG. 14 is a schematic diagram of a structure of a network device provided by the present application. DETAILED DESCRIPTION

[0064] In order to make the purposes, technical solutions and advantages of the present application clearer, the embodiments of the present application are described below with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art can know that, as new application scenarios appear, the technical solutions provided by the present application are also applicable to similar technical problems.

[0065] The terms "first", "second", and the like in the description and claims of the present application and the above drawings are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. It should be understood that the descriptions used in this way can be interchanged under appropriate circumstances, so that the embodiments can be implemented in an order other than that illustrated or described in the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or modules does not have to be limited to those steps or modules clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The naming or numbering of the steps appearing in the present application does not mean that the steps in the method flow must be performed in the time / logical order indicated by the naming or numbering. The flow steps that have been named or numbered can change the order of execution according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units appearing in the present application is a logical division, which can have another division method in actual application, for example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the units shown or discussed can be through some interface, the indirect coupling or communication connection between the units can be electrical or other similar forms, which are not limited in the present application. In addition, the units or sub-units described as separate components can or can not be physically separate, can or can not be physical units, or can be distributed to multiple circuit units, and some or all of the units can be selected according to actual needs to achieve the purpose of the present application scheme.

[0066] At present, in the related art, whether a storage node has a problem is determined by testing the data read-write duration of an end-to-end (i.e., the data read-write duration of a computing node to a storage node), which is difficult to accurately locate whether the problem is caused by the storage node, and thus the problem of isolating the storage node error is prone to occur.

[0067] For example, when performing data storage, if the data storage duration of computing node 1 to storage node 1 and the data storage duration of computing node 2 to storage node 1 are both long, it can be considered in the related art that storage node 1 has a problem, and thus storage node 1 is isolated.

[0068] However, through the research of the applicant, it is found that in actual data storage scenarios, the data storage process between the computing node and the storage node usually involves multiple links. Specifically, the client on the computing node generates a data packet based on the data to be stored, and then delivers the data packet to the network card on the computing node, which sends the data packet to the network card on the storage node through the transmission network. Then, the network card on the storage node continues to deliver the received data packet to the Input / Output (IO) service process on the storage node responsible for implementing data read and write; the IO service process continues to deliver the data packet to the storage medium (such as a hard disk) on the storage node, so that the storage medium implements the persistent storage of data. Therefore, in the entire data storage process, if any link (such as the client on the computing node, the network card on the computing node or the network card on the storage node, the IO service process, or the storage medium, or the transmission network) has a problem, it may cause the data storage time to be lengthened. Similarly, the data reading process between the storage node and the computing node also involves the above-mentioned multiple links, and any link with a problem will also cause the data reading time to be lengthened.

[0069] Based on the above analysis, it can be seen that in the related art, only the data read and write time between the end-to-end is tested, it is difficult to accurately locate the link where the problem occurs in the data read and write process, and the storage node without a problem is easily mis-isolated, thereby affecting the performance of the data read and write service.

[0070] Therefore, the present application provides a data read and write process detection method, which carries a specific target identifier in the data packet transmitted between nodes, so that each processing module responsible for executing data read and write in the two nodes records the time point of obtaining the data packet and the processed data packet based on the target identifier. Moreover, each processing module reports the recorded time point to the management node, which performs unified analysis to determine which processing module caused the data read and write delay, accurately locate the link where the problem occurs in the data read and write process, and avoid mis-isolating the storage node.

[0071] Please refer to FIG. 1, which is a schematic diagram of a system architecture provided by the present application. As shown in FIG. 1, the data read and write process detection method provided by the present application can be applied to the system architecture shown in FIG. 1. The system architecture includes a management node, a node 1 and a node 2 for executing a data read and write process. Among them, the node 1 and the node 2 are communicatively connected through a transmission network.

[0072] Node 1 includes a software module for implementing data read and write (e.g., the software module can be a certain application as a client when node 1 is a computing node) and one or more network cards (e.g., network cards 1-N in FIG. 1). Node 2 is a storage node for persisting data, and node 2 includes one or more network cards (e.g., network cards 1-M in FIG. 1), an IO service process for implementing data read and write, and a storage medium. During data read and write, each software module or hardware module on node 1 and node 2 records a time point at which the software module or hardware module obtains a data packet and a time point at which the software module or hardware module finishes processing the data packet, and uploads the recorded time points to the management node for unified analysis by the management node.

[0073] When the management node analyzes and finds that a certain processing module on node 1 or node 2 has a long time delay in processing a data packet, the management node can consider that the processing module has a problem, and then notify the processing module to stop processing data or restart, so as to avoid affecting the normal read and write of data.

[0074] It should be noted that the management node can be a separate node from node 1 and node 2, and is specially used for detecting the data read and write process between nodes. The management node can also be a certain node in node 1 and node 2, that is, the management node participates in the data read and write process and the detection of the data read and write process.

[0075] Exemplarily, referring to FIGS. 2A and 2B, FIG. 2A is an architecture schematic diagram of an application scenario provided by the present application, and FIG. 2B is an architecture schematic diagram of another application scenario provided by the present application. As shown in FIG. 2A, in a possible application scenario, a plurality of computing nodes (e.g., computing nodes 1-1000 in FIG. 2A) in a computing cluster are connected to a plurality of storage nodes (e.g., storage nodes 1-256 in FIG. 2A) in a storage cluster through a transmission network. Any one of the plurality of computing nodes can send data to any one of the plurality of storage nodes for persistent storage, or read data from any one of the plurality of storage nodes. For example, node 1 shown in FIG. 1 can be any one of the computing nodes shown in FIG. 2A, and node 2 shown in FIG. 1 can be any one of the storage nodes shown in FIG. 2A.

[0076] As shown in FIG. 2B, in another possible application scenario, a plurality of storage nodes (e.g., storage nodes 1-256 in FIG. 2B) in a storage cluster are connected to each other through a transmission network. Any one of the plurality of storage nodes can send data to another one of the plurality of storage nodes for persistent storage, or read data from another one of the plurality of storage nodes. For example, node 1 and node 2 shown in FIG. 1 can be any two of the storage nodes shown in FIG. 2B.

[0077] Referring to FIG. 3, FIG. 3 is a flowchart of a method for detecting a data read-write process provided by the present application. As shown in FIG. 3, the method for detecting the data read-write process includes the following steps 301-306.

[0078] In step 301, a first processing module acquires a first data packet. The first data packet is a data packet transmitted from a first node to a second node in a first data read-write. The first node or the second node is a storage node. The first node or the second node includes a plurality of processing modules for performing data processing in the data read-write. The first processing module is any one of the plurality of processing modules.

[0079] Specifically, the first data packet can be a data packet in a process of writing data from the first node to the second node (i.e., a data storage process). The first data packet can also be a data packet in a process of reading data from the first node by the second node (i.e., a data read process). Regardless of the data storage process or the data read process, the first data packet needs to be transmitted from the first node to the second node. The first data read-write is a data read-write process for the first data packet.

[0080] In addition, in the data storage process or the data read process, the first node and the second node each include a plurality of processing modules for performing data read-write. The first processing module can be one of the plurality of processing modules included in the first node or the second node.

[0081] For example, the first processing module can be a software module running on a network card, a storage medium, or a processor. For example, in the scenario of storing data from the first node to the second node, the first node can be a computing node or a storage node, and the second node can be a storage node. The first processing module can be an application program running on a processor of the first node for requesting to store data, or a microcode program running on a network card for transmitting data through a transmission network. Alternatively, the first processing module can be a microcode program running on a network card of the second node, an IO service process running on a processor for processing data read-write requests, or a program running on a storage medium for persistently storing data. The processor can be a general-purpose processor such as a central processing unit (CPU) or a field programmable gate array (FPGA), or a special-purpose processor. The type of the processor is not limited in the present application.

[0082] For example, in a scenario where the second node reads data in the first node, the first node can be a storage node, the second node can be a computing node or a storage node, and the first processing module can be a microcode program of a network card running on the first node, an IO service process running on a processor and responsible for processing data read / write requests, or a program running on a storage medium responsible for persistently storing data. Alternatively, the first processing module can be an application running on a processor in the second node for reading data or a microcode program running on a network card responsible for receiving data through a transmission network.

[0083] When the first processing module is an application running on a processor in the first node for requesting storage data, the first processing module can generate the first data packet based on the data to be stored. Alternatively, when the first processing module is an IO service process running on the first node and responsible for processing data read / write requests, the first processing module can generate the first data packet based on the data stored in the storage medium. That is, in the case where the first processing module is a software module in the first node, the first processing module generates the first data packet carrying the target identifier according to the data to be transmitted to the second node. Specifically, as the module in the first node that generates the first data packet, the first processing module generates the target identifier in the first data packet. This can be understood as the first processing module coloring the first data packet, so that other processing modules that subsequently receive the first data packet can perform corresponding processing based on the target identifier in the first data packet. The timing at which the first processing module triggers coloring of the data packet can be the time point at which the data read / write link between the first node and the second node is established. Moreover, after the first processing module triggers coloring of the data packet, the first processing module can start a timer to continuously update the coloring period of the data packet.

[0084] When the first processing module is another processing module in the first node or the second node, the first processing module can obtain the first data packet from the previous processing module.

[0085] In step 302, in response to the target identifier included in the first data packet, the first processing module records the first time point at which the first data packet is obtained, and the first flag in the target identifier is used to indicate the recording of the acquisition time point of the first data packet and the processing completion time point.

[0086] In the present application, the target identifier carried in the first data packet includes a first mark. The first mark can be understood as a time delay mark, which is used to instruct the processing module that obtains the data packet to record the time point at which the first data packet is obtained (i.e., the time point at which the first data packet is obtained) and the time point at which the processing is completed (i.e., the time point at which the first data packet is processed). Therefore, in the case where the first mark is detected in the first data packet, the first processing module records the first time point at which the first data packet is obtained by the module.

[0087] In step 303, the first processing module processes the first data packet and records a second time point at which the processing of the first data packet is completed.

[0088] After obtaining the first data packet, the first processing module needs to process the first data packet according to the requirements of the data read-write process. For example, taking data storage as an example, when the first processing module is an application program on the first node that requests to store data, the first processing module can determine the transmission path of the first data and deliver the first data packet to a certain network card on the first node based on the transmission path, so as to send the first data packet to the second node. When the first processing module is a microcode program of the network card on the first node, the first processing module can send the first data packet to the second node through the transmission network according to the transmission path of the first data packet. For another example, when the first processing module is an IO service process on the second node responsible for processing data read-write requests, the first processing module can parse the first data packet, so as to write the effective data carried by the first data to a specified address of a storage medium.

[0089] In general, when the first processing module is not the last processing module in the data storage path or the data read path, the first processing module will continue to send the first data packet to the next processing module after processing the first data packet. Therefore, the second time point at which the first processing module processes the first data packet can be understood as the time point at which the first processing module sends the first data packet to the next processing module.

[0090] In step 304, the first processing module sends a first statistical message to the management node, and the first statistical message includes the first time point and the second time point corresponding to the first processing module at the data read-write, and the management node is used to detect the data read-write based on the statistical message.

[0091] After recording the first time point at which the first data packet is obtained and the second time point at which the first data packet is processed, the first processing module can send a first statistical message to the management node, and report the first time point and the second time point, so that the management node can detect the entire data read-write process.

[0092] It should be noted that the first processing module is taken as an example to introduce the process of the first processing module processing the first data packet in this embodiment, and the first processing module can be any one of the processing modules in the first node or the second node. In actual application, each processing module in the first node and the second node for performing data reading and writing can perform the steps 301-304, so that the management node can obtain the first time point at which each processing module obtains the data packet carrying the time delay mark and the second time point at which the data packet carrying the time delay mark is processed.

[0093] The management node can be a node independent of the first node and the second node, or can be one of the first node or the second node. In addition, each processing module in the first node and the second node can independently send the recorded first time point and second time point to the management node, or can first report the recorded first time point and second time point to a certain processing module in the node and send the first time point and second time point to the management node by the processing module. For example, each processing module in the first node can report the recorded first time point and second time point to an application program in the first node, and send the first time point and second time point to the management node by the application program.

[0094] In step 305, the management node receives the statistical messages respectively sent by the plurality of processing modules.

[0095] Specifically, each processing module on the first node and the second node will perform the process introduced in the steps 301-304, so as to send the statistical message to the management node, so that the management node can obtain the time point at which each processing module obtains the first data packet and the time point at which the first data packet is processed.

[0096] Exemplarily, taking the first processing module and the second processing module as an example, the management node receives the first statistical message and the second statistical message respectively sent by the first processing module and the second processing module in the plurality of processing modules. The plurality of processing modules are modules running on the first node or the second node for performing data processing in the data reading and writing process, and the first node or the second node is a storage node. The first statistical message includes a first time point and a second time point corresponding to the first processing module in the first data reading and writing, and the second statistical message includes a third time point and a fourth time point corresponding to the second processing module in the first data reading and writing. The first time point is used to indicate the time point at which the first processing module obtains the first data packet, the second time point is used to indicate the time point at which the first processing module processes the first data packet, the third time point is used to indicate the time point at which the second processing module obtains the first data packet, and the fourth time point is used to indicate the time point at which the second processing module processes the first data packet. The first data packet is a data packet that needs to be transmitted from the first node to the second node in the first data reading and writing.

[0097] In step 306, the management node sends a first notification message to the first processing module in a case where it is determined that the first processing module in the plurality of processing modules is in a sub-healthy state, the first notification message being used to instruct the first processing module to stop performing data reading and writing or to instruct the first processing module to restart.

[0098] Specifically, based on the first statistical message, in a case where it is determined that the time delay of the first processing module in processing data packets meets a preset condition, the management node determines that the first processing module is in a sub-healthy state. The preset condition includes that the average time delay of the first processing module in processing a plurality of data packets exceeds a first time delay value.

[0099] That is, the management node can determine the transmission time delay of each data flow from the first node to the second node by receiving the statistical message reported by each processing module for the data packet in each data flow. If the average time delay of the data packet of each data flow from the first node to the second node exceeds a second time delay value, it means that there is a problem in the transmission process from the first node to the second node. At this time, the management node further determines the time delay of each processing module in processing the data packet in the plurality of data flows based on the time point reported by each processing module. If the time delay of a certain processing module (such as the first processing module) in processing the data packet in the plurality of data flows exceeds the first time delay value, it means that the first processing module is not in a normal state, but may be in a sub-healthy state. At this time, in order to ensure the performance of data reading and writing, the management node can send a first notification message to the first processing module, the first notification message being used to instruct the first processing module to stop performing data reading and writing or to instruct the first processing module to restart. That is, in a case where the first processing module is in a sub-healthy state, the management node instructs the first processing module to stop performing data reading and writing so as not to affect the subsequent data reading and writing process; or the management node can instruct the first processing module to restart in order to try to get rid of the sub-healthy state.

[0100] In the present scheme, by carrying a specific target identifier in the data packet transmitted between nodes, each processing module responsible for performing data reading and writing in the two nodes can record the time point of obtaining the data packet and the time point of processing the data packet based on the target identifier. Moreover, each processing module will report the recorded time point to the management node for unified fine-grained analysis by the management node, so as to determine which processing module causes the long data reading and writing time delay, accurately locate the link where the problem occurs in the data reading and writing process, and avoid error isolation of the storage node.

[0101] It can be understood that the above introduction is that the management node determines whether each processing module is in a sub-healthy state based on the time point at which each processing module obtains the data packet and the time point at which the data packet is processed, so as to accurately locate the processing module with problems in the node and realize isolation. However, since the first node and the second node transmit data through the transmission network, the transmission network may also have problems such as congestion, which may cause the data read-write delay to be too high. Therefore, the management node can further determine whether the transmission network between the first node and the second node has a problem based on the time points reported by different processing modules on the first node and the second node.

[0102] Exemplarily, it is assumed that the second processing module is the next processing module of the first processing module in the plurality of processing modules. Then, based on the second time point at which the first processing module processes the first data packet and the third time point at which the second processing module obtains the first data packet, the management node determines a target transmission delay of the first processing module in transmitting the first data packet to the second processing module.

[0103] In the case where the target transmission delay exceeds the second delay value, the management node sends a second notification message to the first processing module, and the second notification message is used to instruct the first processing module to stop transmitting data through the target path, and the target path is the path of the first processing module in transmitting the first data packet to the second processing module. That is, the second time point sent by the first processing module represents the time point at which the network card in the first node sends the first data packet to the second node, and the third time point sent by the second processing module represents the time point at which the network card in the second node receives the first data packet, so that the target transmission delay of the first processing module in transmitting the first data packet to the second processing module can be obtained by subtracting the third time point sent by the second processing module from the second time point sent by the first processing module.

[0104] In the case where the first processing module runs on the network card in the first node and the second processing module runs on the network card in the second node, the target transmission delay of the first processing module in transmitting the first data packet to the second processing module represents the transmission delay of the transmission network between the first node and the second node (i.e., the transmission delay of the IP layer).

[0105] Further, assuming that the first processing module is a processing module in the first node that generates the first data packet, and the second processing module is a processing module in the second node that is the last processing module to obtain the first data packet. Then, based on the first time point at which the first processing module generates the first data packet and the fourth time point at which the second processing module finishes processing the first data packet, the management node can determine the transmission delay of the first data packet in the entire data read-write process (i.e., the transmission delay of the IO layer). If the transmission delay of the IO layer exceeds the normal transmission delay, the management node can consider that the transmission process of the first data packet is abnormal, thereby triggering the positioning of which link between the first node and the second node is abnormal.

[0106] For the first processing module, after receiving the second notification message, the first processing module can feed back the second notification message to the application program in the first node, so as to make the application program switch the transmission path of the data packet when subsequently sending the data packet, i.e., no longer selecting the target path to transmit the data packet, thereby ensuring the normal execution of the data read-write process.

[0107] That is, based on the time point at which the data packet is sent from the first node and the time point at which the data packet is received by the second node, the management node can determine the transmission delay of the transmission path between the first node and the second node, and further determine whether the root cause of the long data read-write process delay is the transmission path between the two nodes. In this way, in the case that the transmission delay of the data packet is long, the management node can locate the root cause of the long data read-write process delay to the specific transmission path or transmission network, and further notify the node not to transmit through the current transmission path of the data packet, so as to avoid affecting the subsequent data read-write process. Of course, if the transmission delay of the transmission path between the first node and the second node returns to normal, the management node can send a recovery notification message to the first node, thereby instructing the first node to continue to use the transmission path to transmit new IO data.

[0108] In the present scheme, the management node analyzes the transmission delay of the transmission network between nodes based on the time points reported by each processing module in the node, without the need for additional measurement of the transmission delay of the transmission network, i.e., the network transmission process in the data read-write process can be accurately identified to determine whether there is a problem, and further the transmission path is switched to ensure the normal execution of the subsequent data read-write process.

[0109] Further, in addition to the data read-write delay, the phenomenon of packet loss in the data read-write process will also affect the data read-write performance. Therefore, the present application proposes that each processing module on the node records the number of received data packets to count the packet loss in the data read-write process, so as to locate the link where the packet loss is more serious.

[0110] For example, taking the first processing module as an example, during the data read / write process, the first processing module statistically analyzes the traffic information of multiple data read / write operations within the first period at the first processing module. The statistical message also includes traffic information. The traffic information includes one or more of the following information for multiple data read / write operations at the first processing module within the first period: the number of received data packets, the number of bytes in the received data packets, the number of sent data packets, the number of bytes in the sent data packets, the number of lost data packets, and the number of bytes in the lost data packets.

[0111] Of course, the first processing module can also collect traffic information of the node where it is located (for example, collect traffic information collected by various processing modules on the same node), and send the collected node traffic information to the management node through statistical messages. The node traffic information collected by the first processing module includes one or more of the following: the number of data packets received by the node at the IO layer, the number of data packets received by the node at the IP layer, the number of bytes of data packets received by the node at the IO layer, the number of bytes of data packets received by the node at the IP layer, the number of data packets sent by the node at the IO layer, the number of data packets sent by the node at the IP layer, the number of bytes of data packets sent by the node at the IO layer, the number of bytes of data packets sent by the node at the IP layer, the number of data packets lost by the node at the IO layer, the number of data packets lost by the node at the IP layer, the number of bytes of data packets lost by the node at the IO layer, and the number of bytes of data packets lost by the node at the IP layer.

[0112] For example, in order for the first processing module to identify the period to which the data packet belongs, the second tag included in the target identifier is used to indicate the period to which the corresponding data packet belongs.

[0113] So, when counting the traffic information of multiple data reads and writes in the first processing module within the first cycle, the first processing module can specifically count the information of the data packets whose target identifier has a second tag value of the first value from multiple data packets to obtain the traffic information. The multiple data packets are data packets including target identifiers that are received or sent by the first processing module, and the first value is used to indicate the first cycle.

[0114] Specifically, the first processing module can continuously count the number of data packets with the second tag value set to the first value within a preset time period, in order to count the number of data packets it receives belonging to the first period. Specifically, the data packet generating processing module in the first node adds a second tag to all data packets within the first period and sets the value of the second tag to the first value, so that subsequent processing modules can identify all data packets belonging to the first period based on the second tag.

[0115] It should be noted that the above is introduced by taking the first processing module as an example. In actual applications, each processing module in the first node and the second node can count the traffic information of multiple data reads and writes in the first period, and send the counted traffic information to the management node. Specifically, in addition to counting the first period, each processing module in the first node and the second node can count the traffic information in each period according to the period number in the target identifier, and send the counted traffic information in each period to the management node.

[0116] For example, the first statistical report sent by the first processing module also includes the first traffic information of multiple data reads and writes at the first processing module in the first period. The second statistical report sent by the second processing module also includes the second traffic information of multiple data reads and writes at the second processing module in the first period. The first traffic information includes the number of data packets sent by the first processing module after processing the data packets belonging to the first period (i.e., the packet sending quantity of the first processing module), and the second traffic information includes the number of data packets belonging to the first period obtained by the second processing module from the first processing module (i.e., the packet receiving quantity of the second processing module).

[0117] At this time, based on the first traffic information and the second traffic information, the management node determines the first packet loss quantity, and the first packet loss quantity is used to indicate the packet loss quantity of the transmission path between the first processing module and the second processing module.

[0118] When the first packet loss quantity is greater than the preset packet loss value, the management node sends a second notification message to the first processing module, and the second notification message is used to instruct the first processing module to stop transmitting data through the target path, and the target path is the path for the first processing module to transmit the data packets belonging to the first period to the second processing module.

[0119] That is to say, if the management node determines that the packet loss of the transmission path between two certain processing modules is serious by analyzing the packet receiving quantity and the packet sending quantity sent by each processing module, then the management node can send a notification message to the processing module to instruct the processing module to switch the transmission path to avoid continuous packet loss during the data read and write process and ensure the normal execution of the data read and write process.

[0120] Furthermore, in some embodiments, when processing data packets within the same cycle, the first processing module may perform operations such as merging data packets, thereby reducing the number of data packets. However, the data content carried by the reduced number of data packets remains unchanged. Therefore, in this case, the first processing module can also proactively report the number of packet losses for each cycle it processes, so that the management node can determine the actual number of packet losses by the first processing module and avoid the management node calculating an incorrect number of packet losses based on the number of received and sent packets by the first processing module.

[0121] Optionally, in order to facilitate the statistical analysis of stable and accurate packet loss, the processing modules in the first and second nodes can continuously analyze the reception of data packets over multiple periods, so that the management node can determine the packet loss situation over a long period of time.

[0122] For example, taking the first processing module as an example, in addition to collecting traffic information of multiple data reads and writes in the first processing module within the first period, the first processing module can also collect traffic information of multiple data reads and writes in the second period.

[0123] Specifically, during the generation of data packets by the first processing module in the first node, a corresponding value can be set for the second marker in the data packet according to the period to which the data packet belongs. For example, the second marker in data packets belonging to the first period is set to the first value, and the second marker in data packets belonging to the second period is set to the second value, so that subsequent processing modules can distinguish the period to which the data packets belong when receiving consecutive data packets. Therefore, for the first processing module, the second marker in multiple consecutive data packets received by the first processing module may have different values, and the first processing module can determine the period to which each data packet belongs based on the value of the second marker.

[0124] The first and second cycles mentioned above can be adjacent cycles with the same duration. Furthermore, the first processing module continuously counts data packets within a cycle whose preset duration is longer than the duration of the first cycle. The duration of the first and second cycles can be any duration between 0.5 seconds and 10 seconds, and can be set according to actual conditions. For example, assuming both the first and second cycles are 3 seconds long, the first processing module continuously counts the number of data packets with the second tag set to the first value for the next 5 seconds after the start of the first cycle, thus obtaining the number of packets received in the first cycle; the first processing module also continuously counts the number of data packets with the second tag set to the second value for the next 5 seconds after the start of the first cycle, thus obtaining the number of packets received in the second cycle.

[0125] In other words, the time taken by the first processing module to count data packets within a certain period is actually longer than the actual duration of that period. This ensures that the first processing module can completely count the data packets within the entire period. Specifically, data packets may be out of order during transmission, causing some data packets belonging to a later period to arrive earlier than some data packets belonging to a previous period. As a result, for data packets within a period sent by the previous processing module, the subsequent processing module may have difficulty receiving all the data packets within a single period. Therefore, extending the time taken by the processing module to count the received data packets belonging to a certain period ensures the accuracy of the statistical data.

[0126] It should be noted that, in this application, the second flag can specifically be a 1-bit flag bit, and the value of the second flag can be 0 (such as the first value mentioned above) or 1 (such as the second value mentioned above). In this case, the two different values ​​can represent the different periods to which the data packet belongs. Alternatively, the second flag can specifically be a flag bit of multiple bits, and the value of the second flag can be used to indicate the specific period number or timestamp, so as to indicate the specific period to which the data packet carrying the second flag belongs. Or, the second flag can be implemented using two flag bits: one is a 1-bit flag bit used to distinguish the period to which the data packet belongs, and the other is a multiple-bit flag bit used to indicate the specific period number or timestamp.

[0127] The above describes how the processing module analyzes the data read / write latency and packet loss of each processing module or transmission path by statistically analyzing the relevant time points of data packet acquisition and processing, as well as the number of data packets received or sent. The following will detail how to include information in data packets to instruct the processing module to perform various statistical analyses.

[0128] In distributed storage scenarios, a management node may manage a large number of storage nodes, resulting in numerous read and write data streams among them. To facilitate effective differentiation between the data streams reported by various processing modules, the data packets processed by each module can carry stream identifiers, and each processing module can synchronously report the stream identifiers corresponding to the information it has collected to the management node. A data stream typically refers to an ordered sequence of byte data with a start and an end point; essentially, it's a sequence of data packets traveling from a specific start point to a specific end point. Generally, when a data stream has no sub-streams, different data packets belonging to the same data stream share the same start point, end point, and transmission path. When a data stream has multiple sub-streams, it corresponds to one start point and multiple end points. Thus, different data packets within the same data stream often correspond to the same start point; however, data packets belonging to different sub-streams will correspond to different end points.

[0129] For example, based on the above embodiments, the third marker in the target identifier of the first data packet received by the first processing module is used to indicate the flow identifier, which indicates the data flow to which the corresponding data packet belongs. The statistical message sent by the first processing module to the management node also includes the flow identifier and the period number. Generally, data packets belonging to the same data flow will carry the same flow identifier. Furthermore, different data flows will correspond to different flow identifiers. For example, data flows sent by the same compute node or the same storage node can carry the same flow identifier.

[0130] In this way, the management node determines the data stream corresponding to the first data packet based on the flow identifier, enabling it to analyze the differences in transmission between different data streams. Specifically, based on the flow identifier in the statistical messages reported by the processing modules, the management node can determine the data stream to which the latency, packet reception, and packet loss information statistically recorded in the current statistical message belongs, and then analyze the specific transmission status of each data stream. Furthermore, the management node can also unify the time points and period numbers sent by different processing modules processing the same data stream based on the flow identifier, facilitating a unified analysis of the data read / write activities of each processing module handling the same data stream.

[0131] Specifically, the flow identifier includes a node identifier and a path identifier. The node identifier identifies the first node, and the path identifier identifies the transmission path of the corresponding data packet. Optionally, the flow identifier may also include a service identifier, which identifies the IO service process on the second node that handles the data read and write process.

[0132] At this point, for the first statistical message reported by the first processing module, the management node determines the transmission path of the first data packet based on the path identifier in the first statistical message, so that the management node can analyze the differences in transmission latency between different transmission paths. That is, each statistical message reported by the processing module includes a flow identifier. Based on the path identifier in the flow identifier, the management node can determine the transmission path to which the data packet statistically reported by each processing module belongs, thereby enabling the analysis of the transmission latency of the data stream on different transmission paths. In this way, by comparing the transmission latency of the data stream on different transmission paths, the management node can analyze the differences in transmission latency between different transmission paths, so as to identify transmission paths with abnormal transmission latency (such as transmission latency that is particularly long).

[0133] Specifically, the period number sent by the first processing module to the management node can be used to indicate the period to which the content in the statistical message reported by the first processing module belongs, so that the management node can determine the period to which the latency information, packet reception information and packet loss information statistically collected by the first processing module and other processing modules belong.

[0134] By designing period numbers, multiple processing modules, including the first processing module, can continuously collect traffic information for multiple periods and report this information to the management node. This allows the management node to determine whether the first processing module is in a sub-healthy state based on the traffic information from multiple periods (e.g., judging whether the first processing module has excessively long packet processing latency based on the packet processing latency of multiple periods), ensuring the correctness of identifying abnormal processing modules. Furthermore, the management node uses period numbers to perform packet loss statistics and comparisons for each period, facilitating the identification and display of packet loss conditions over each time period, which is beneficial for packet loss statistics implementation. Moreover, based on period numbers, processing modules on the first and second nodes can collect traffic information for the same period without time synchronization, allowing the management node to clearly identify the traffic information reported by each processing module for the same period, ensuring the alignment of traffic information collected by different processing modules.

[0135] Furthermore, with a period number, each processing module can count the delay information only once in a single period, thereby reducing the load on the processing module and improving the processing efficiency of data packets.

[0136] Furthermore, since the same computing node may send the same data to different storage nodes for storage, a data stream may be split into multiple sub-streams, and different sub-streams may go to different destination addresses.

[0137] For example, a data stream may include multiple substreams, and the target identifier mentioned above also includes a substream identifier, which is used to indicate the substream to which the corresponding data packet belongs. That is, when the data stream to which the data packet belongs includes multiple substreams, the target identifier in the data packet also includes the substream identifier corresponding to the substream to which the data packet actually belongs.

[0138] For example, the flow identifier of the first data packet indicates that the first data packet belongs to the target data stream, and the sub-flow identifier of the first data packet indicates that the first data packet belongs to the first sub-flow among multiple sub-flows of the target data stream. In this case, when the first data packet has a sub-flow identifier, the path identifier in the first data packet can represent the specific transmission path of the first sub-flow to which the first data packet belongs.

[0139] When the first processing module is located at the first node, it can send indication information of the target data stream to the management node. This indication information indicates the sub-streams included in the target data stream. That is, since the target data stream is divided into multiple sub-streams from the first node and destined for different nodes, the first node can report the multiple sub-streams included in the target data stream to the management node. This allows the management node to determine whether the data packets processed by these processing modules belong to different sub-streams of the same target data stream when it subsequently receives statistical messages from processing modules at different nodes.

[0140] In addition, each processing module in the first and second nodes can also report the sub-stream to which the statistical data packets belong when sending statistical messages to the management node.

[0141] For example, the first statistical message sent by the first processing module further includes a sub-flow identifier, which indicates that the first data packet corresponding to the first statistical message belongs to a sub-flow in the target data flow. The target data flow includes multiple sub-flows originating from the first node. Furthermore, data packets in multiple sub-flows of a data flow carry the same node identifier and different path identifiers.

[0142] The management node determines the transmission latency of multiple sub-streams of the target data stream based on statistical messages sent by multiple processing modules on different nodes. Specifically, each processing module's statistical message carries a corresponding flow identifier and sub-stream identifier. The management node can use these identifiers to determine the sub-stream corresponding to the data packets processed by each module, and thus determine the transmission latency of the multiple sub-streams of the target data stream.

[0143] If the transmission delay of the target sub-stream is greater than the third delay value among multiple sub-streams, the management node sends a sub-stream adjustment message to the first node. The sub-stream adjustment message is used to notify the first node to adjust the transmission path of the target sub-stream.

[0144] In other words, if the management node finds that the transmission delay of a certain sub-stream under the same target data stream is large, the management node can notify the first node to change the transmission path of that sub-stream, thereby improving the transmission efficiency of the entire target data stream.

[0145] Optionally, in order to ensure that each processing module in the node can obtain the target identifier mentioned above, this application may add extended content at different positions of the data packet to carry the above information.

[0146] When the first processing module runs on the network interface card (NIC), it detects the target identifier in the protocol header or payload of the first data packet. For example, when the first processing module runs on the NIC of the first node, it receives the first data packet to be sent to the second node from the software module of the first node, and detects the target identifier from the payload of the first data packet. When the first processing module runs on the NIC of the second node, it receives the first data packet to be sent to the second node from the NIC of the first node, and detects the target identifier from the protocol header of the first data packet. The protocol header of the first data packet can be, for example, the header of a network protocol, such as Internet Protocol Version 6 (IPv6) or the InfiniBand (IB) protocol header. The IB protocol is a communication protocol used between compute nodes and storage nodes for data read and write operations, specifically a communication protocol used by Remote Direct Memory Access (RDMA).

[0147] Alternatively, if the first processing module is running on a storage medium or a processor, the first processing module detects the target identifier in the payload of the first data packet.

[0148] For example, this application can define an in-band network telemetry (INT) header as a target identifier to carry the aforementioned extended content, and add the INT header at different positions in the data packet so that different processing modules can obtain the extended content. Please refer to Figure 4, which is a schematic diagram of a data packet format provided by this application. As shown in Figure 4(a), the data packet includes, from the outside to the inside, an Internet Protocol Version 4 (IPv4) header, an IB header, an INT header, and a payload, wherein the payload also carries an INT header. Specifically, carrying an INT header in the payload allows the application and network card in the data sending node, as well as the IO service process and storage medium in the data receiving node, to read the INT header and thus obtain the aforementioned extended content. Encapsulating the INT header after the IB header allows the network card in the data receiving node to read the INT header after the IB header and thus obtain the extended content.

[0149] As shown in Figure 4(b), because the IPv6 header has fields that support extensions, when a data packet is encapsulated using IPv6, the packet can include, from the outside in, an IPv6 header, an IB header, and a payload. The IPv6 header can carry an INT header via a Next Hop Header or other optional fields. Furthermore, the payload can also carry an INT header.

[0150] To facilitate understanding, the following will provide a detailed explanation of the execution process of the data read / write detection method provided in this application in practical applications, using specific examples.

[0151] For example, please refer to Figure 5, which is a schematic diagram of the data storage process for detection in a distributed storage scenario provided by this application. As shown in Figure 5, taking the storage of data from a compute node to a storage node as an example, the client in the compute node is responsible for generating the data packets to be sent to the storage node. The data packets pass sequentially through the client and network card in the compute node, the transmission network, and the network card, IO service process, and storage medium of the storage node, and are finally stored in the storage medium. Therefore, for each processing module in the compute node and storage node (i.e., the client and network card in the compute node, the network card, IO service process, and storage medium in the storage node), the time points when the data packets are acquired and processed can be recorded.

[0152] Specifically, the process of detecting whether there are problems in the data storage process mainly includes the following four stages.

[0153] Phase 1: The client that generates the data packets colors the data packets to add coloring tags to the data packets.

[0154] Specifically, when a client determines that it needs to monitor the data read / write process, it can add extended content, including coloring markers, to the data packet during data packet generation. This extended content is used to monitor the data packet transmission process. Specifically, the client can add the extended content, including the coloring markers, to the data packet payload by adding an INT header. The client can determine the need to monitor the data read / write process by receiving a monitoring notification command from the management node; or, the client can periodically trigger monitoring of the data storage process, thus determining that monitoring is required when the periodic monitoring time is reached. Of course, the client can also determine whether monitoring is needed based on other methods, which are not specifically limited here.

[0155] Please refer to Figure 6, which is a schematic diagram of the format of the INT header added to the data packet according to this application. As shown in Figure 6, the INT header, as an extension, includes the following fields: Flow Node ID, Flow Path ID, Loss Flag (L), Delay Flag (D), Period No, and Period. Optionally, the INT header, as an extension, also includes the following optional fields: Type, Flow Server ID, End to End Flag (E), Fra Flag, Sub-Flow Flag (S), and Reserved (R).

[0156] The type field indicates the type of the current INT header, meaning that the information carried by the INT header is for detecting data read and write processes.

[0157] The Flow Node ID, Flow Server ID, and path identifier constitute a flow identifier, used to uniquely identify a data flow. The Flow Node ID needs to be unique across the entire network within the detection domain so that each node can identify the data flow based on the Flow Node ID. Generally, the Flow Node ID can be a unique identifier for a node sending data.

[0158] The Flow Server ID is used to uniquely identify a specific service within a node, indicating which service is handling the data flow.

[0159] Flow Path ID is used to identify the path of a data flow, and can be represented by a 5-tuple. The 5-tuple includes the source IP address, source port address, destination IP address, destination port, and transport layer protocol.

[0160] L is used to indicate the coloring mark for packet loss measurement (i.e., the second mark mentioned above).

[0161] D is used to represent the coloring mark for delay measurement (i.e., the first mark mentioned above). When the value of D is 1, it indicates that delay measurement needs to be performed; when the value of D is 0, it indicates that delay measurement does not need to be performed.

[0162] E is used to indicate hop-by-hop or end-to-end mode. When E is 0, it indicates that the current data stream is in hop-by-hop mode; when E is 1, it indicates that the current data stream is in end-to-end mode.

[0163] When Fra is 1, it means that subsequent fragments of the data packet are not counted.

[0164] F is used to indicate whether the data stream is a forward flow. When the value of F is 1, it indicates that the data stream is a forward flow (that is, the data stream belongs to the data storage).

[0165] S is used to indicate whether the data stream is a substream. When the value of S is 0, it means that the data stream is not a substream; when the value of S is 1, it means that the data stream is a substream.

[0166] Period No. is used to represent the period number, indicating the period to which the current data packet belongs (such as the first period or the second period mentioned above).

[0167] It should be noted that in some embodiments, the INT header may include only the color marker L for indicating packet loss measurement, or only the Period No for indicating period number, or both L and Period No.

[0168] Period is used to represent the duration of a single cycle, in seconds. Specifically, the duration of a single cycle can be, for example, 0.1-10 seconds.

[0169] In Phase 2, each processing module along the data packet records relevant information about the data packet based on coloring tags, thus completing the check-in process.

[0170] For each processing module that acquires a data packet, these modules can decide whether to record packet loss information or latency information based on the contents of the INT header carried by the data packet (such as coloring markers indicating packet loss measurement and latency measurement). Simultaneously, while recording packet loss or latency information, the processing module also records other information in the INT header of the data packet to achieve simultaneous recording of the data flow information corresponding to the packet loss or latency information. Specifically, the processing module can record the latency and packet loss information corresponding to the current Flow ID and period number using Flow ID and period number as the recording units, ensuring that all latency and packet loss information has a unique corresponding Flow ID and period number.

[0171] Phase 3: Each processing module along the data packet's path reports the recorded data packet information to the management node.

[0172] Specifically, after each processing module completes the recording of data packet information, each processing module can periodically report the recorded data packet information to the management node. Alternatively, each processing module can also trigger the reporting of the recorded data packet information to the management node only after receiving a reporting instruction from the management node.

[0173] For example, please refer to Figure 7, which is a schematic diagram of data packet information reported by a processing module to a management node according to this application. As shown in Figure 7, the data packet information reported by the processing module to the management node includes: Flow ID, period number, in timestamp, out timestamp, in statistics bytes / packet, out statistics bytes / packet, and discarded bytes / packet. Wherein, if the data packet is not a sub-flow, the Flow ID reported by the processing module can specifically be a flow identifier composed of the aforementioned Flow Node ID, Flow Server ID, and path identifier. Furthermore, the in timestamp refers to the time point at which the processing module acquires the data packet that needs to be used for latency measurement (i.e., the value of D is 1) (i.e., the first time point mentioned above); the out timestamp refers to the time point at which the processing module finishes processing the data packet that needs to be used for latency measurement (i.e., the second time point mentioned above); the in statistics bytes / packets refers to the total number of bytes of data carried by the data packet belonging to period i and the number of data packets acquired by the processing module; the out statistics bytes / packets refers to the total number of bytes of data carried by the data packet belonging to period i and the number of data packets output by the processing module; and the discard statistics bytes / packets refers to the total number of bytes of data carried by the data packet belonging to period i and the number of data packets discarded by the processing module.

[0174] It should be noted that when the processing module runs on the network interface card (NIC), since the NIC has an RDMA unit that performs I / O layer processing on the data packets and an IP unit that performs IP layer processing on the data packets, the processing module running on the NIC can further report the in timestamp, out timestamp, in statistical bytes / packet, and out statistical bytes / packet of the data packets in the RDMA unit, as well as the in timestamp, out timestamp, in statistical bytes / packet, and out statistical bytes / packet of the IP unit. This allows the management node to more clearly understand the latency and packet loss of the data packets within the NIC.

[0175] Phase 4: The management node analyzes the data packet information reported by each processing module to identify and address any issues that arise during data storage.

[0176] Based on the data packet information reported by each processing module, the management node can analyze the data read / write process to determine the transmission latency and packet loss in the transmission links related to each processing module. Specifically, by analyzing the data read / write process, the management node can achieve I / O awareness between the compute node and the storage node, as well as IP link awareness between the compute node and the storage node, in order to determine which link is causing excessive I / O latency or severe I / O packet loss.

[0177] For example, please refer to Figure 8, which is a schematic diagram of data interaction between a compute node and a storage node based on a connection session provided in this application. As shown in Figure 8, the client in the compute node establishes a connection session with the IO service process in the storage node, and then realizes data interaction (such as storing data from the compute node to the storage node) based on the connection session. Since both the compute node and the storage node are equipped with multiple network interface cards (NICs), and each NIC can include multiple ports, any two NIC ports between the compute node and the storage node can form a transmission path. Therefore, there can be multiple transmission paths between the compute node and the storage node. A transmission path is identified by a 5-tuple, such as the source IP address being the IP address of a NIC on the compute node, the source port being the port number of a NIC port on the compute node, the destination IP address being the IP address of a NIC on the storage node, and the destination port being the port number of a NIC port on the storage node.

[0178] For the management node, a transmission path formed by any two network interface card (NIC) ports between the compute node and the storage node can be understood as an IP link. Therefore, when the management node performs IP link awareness, it can detect the transmission latency and packet loss of each transmission path. Furthermore, the management node's I / O awareness between the compute and storage nodes can be used to detect the overall I / O latency or packet loss from the compute node to the storage node, or it can detect the I / O latency or packet loss of specific processing modules within the compute and storage nodes.

[0179] It should be noted that because data packets need to pass through the processing modules inside the compute and storage nodes, as well as the transmission network between the compute and storage nodes, and because data often needs to be encapsulated with specific message headers when transmitted in the transmission network, the format of the data packets will change during transmission.

[0180] For example, please refer to Figure 9, which is a schematic diagram of a data packet format change provided in this application. As shown in Figure 9, the data packet generated by the client in the compute node at the application layer includes a payload, and the payload includes an INT header. To facilitate the identification of the INT header by various processing modules, the INT header can specifically be carried in the application header of the payload. The application header is a header defined by the application program and is essentially also part of the payload. The application program in the compute node, the IO service process in the storage node, and the storage medium will all detect the INT header in the payload, and then measure latency and packet loss based on the extended content in the INT header, thereby obtaining the measurement of IO latency and packet loss.

[0181] After the client in the compute node sends the generated data packet to the network interface card (NIC), the NIC typically segments the data packet to meet the size requirements for transmission over the network, as the size of a single data packet is limited. Furthermore, since the data packet needs to be transmitted over the network according to a transport protocol, the NIC encapsulates an IB header at the IB layer and an IP header (e.g., an IPv4 header) at the IP layer. If the IP header (e.g., IPv4) does not support extended fields, the NIC can refer to the INT header in the payload and encapsulate a new INT header after the IB header. If the IP header (e.g., IPv6) supports extended fields, the NIC can refer to the INT header in the payload and encapsulate a new INT header within the IPv6 header. Based on this new INT header outside the payload, IP link latency and packet loss can be measured.

[0182] When a network interface card (NIC) segments a data packet into multiple packets (i.e., sends packets from the I / O layer to the IB layer), to avoid duplicate data collection, the NIC, when encapsulating a new INT header, can copy the D Flag value from the INT header in the payload only to the first segmented data packet, while copying the remaining fields of the INT header to all segmented data packets. In this way, when the NIC in the storage node receives multiple segmented data packets, it only performs latency and packet loss measurements once, avoiding duplicate measurements.

[0183] Furthermore, when the network interface card (NIC) of the compute node performs packet segmentation, it needs to ensure that the Maximum Transmission Unit (MTU) of the IP layer is synchronized with the MTU of the Interconnect Block (IB) layer; that is, the IB layer MTU = IP layer MTU - IB layer header. In this way, the number of packet losses at the IB layer matches the number of packet losses at the IP layer, achieving the goal of packet loss detection at the smallest granularity of the network. At the same time, this design also avoids performance issues caused by fragmentation from the IB layer to the IP layer.

[0184] The following section will continue to explain how to detect packet loss and latency based on extended content in data packets.

[0185] For example, please refer to Figure 10, which is a schematic diagram of the principle of packet loss detection provided by this application. As shown in Figure 10, for any two processing modules (i.e., processing module 1 and processing module 2 in Figure 10) that have data packet interaction during data read and write, processing module 1 processes the acquired data packets and then outputs the data packets to processing module 2; processing module 2 processes the acquired data packets and then continues to output them to the next processing module or ends the processing of the data packets. The number of data packets acquired by each processing module can be simply referred to as the number of in packets, and the number of data packets output by each processing module after processing can be simply referred to as the number of out packets.

[0186] For the processing module responsible for generating data packets, it performs packet loss coloring on the generated data packets in each cycle, that is, sets the L Flag in the INT header of the data packet to either 0 or 1. Furthermore, the value of L Flag is toggled once in each cycle to ensure that adjacent cycles can be represented by different values. For example, the processing module sets the L Flag value to 0 for all data packets belonging to cycle i, sets the L Flag value to 1 for all data packets belonging to cycle i+1, sets the L Flag value to 0 for all data packets belonging to cycle i+2, and so on.

[0187] For the data packet receiving processing module, when receiving and counting data packets in each cycle, the processing module can extend the statistical time range by 2 / 3 of a cycle to mitigate interference caused by out-of-order data packets. For example, assuming the duration of a cycle is T, the statistical time range for a cycle by the processing module would be T + 2 / 3T. In this case, the processing module will consider data packets with the same L Flag value counted within a single statistical time range as data packets of the same cycle.

[0188] As shown in Figure 10, the content ultimately reported by processing module 1 and processing module 2 may include Flow ID, period number, number of in packets, number of out packets, number of packet losses for this module, and module information (i.e., which module in the node this module belongs to). Therefore, when counting packet losses, the management node can obtain the number of packet losses along the transmission path between these two processing modules by subtracting the number of in packets from the number of out packets of the previous processing module. For example, in Figure 10, the number of packet losses along the transmission path between processing module 1 and processing module 2 in period i is: number of out packets of processing module 1 in period i - number of in packets of processing module 2 in period i = 5 - 5 = 0; the number of packet losses along the transmission path between processing module 1 and processing module 2 in period i+1 is: number of out packets of processing module 1 in period i+1 - number of in packets of processing module 2 in period i+1 = 5 - 4 = 1.

[0189] Furthermore, since each processing module may merge data packets during processing, each module can also report its packet loss count to indicate the actual number of packets lost during data packet processing. This helps the management node determine the actual packet loss count for each processing module (i.e., the number of packets lost due to problems with the processing module). For example, regarding the content reported by processing module 1, processing module 1 has 6 in packets and 5 out packets in period i, and its packet loss count is 0. Therefore, the management node can determine that the actual packet loss count for processing module 1 is 0. Regarding the content reported by processing module 2, processing module 2 has 4 in packets and 3 out packets in period i+1, and its packet loss count is 0. Therefore, the management node can determine that the actual packet loss count for processing module 2 is 0.

[0190] For example, please refer to Figure 11, which is a schematic diagram of the principle of latency detection provided by this application. As shown in Figure 11, for the processing module responsible for generating data packets, the processing module will perform latency coloring on a certain data packet (such as the first or last data packet) within each cycle, that is, set the value of D Flag in the INT header of the data packet to 1.

[0191] For the data packet receiving processing module, when receiving data packets in each cycle, the processing module can record the timestamp of the data packet with D Flag value of 1 received in each cycle (i.e., the in timestamp), and the timestamp of the data packet with D Flag value of 1 sent out (i.e., the out timestamp).

[0192] As shown in Figure 10, the content ultimately reported by processing module 1 and processing module 2 may include Flow ID, period number, in timestamp, out timestamp, and module information. In this way, when calculating latency, the management node can subtract the in timestamp from the out timestamp of the same processing module within the same period to obtain the latency of that processing module in processing data packets. The management node can also subtract the in timestamp of the previous processing module within the same period from the in timestamp of the subsequent processing module within a certain period to obtain the transmission latency of the transmission path between the two processing modules. For example, if the out timestamp of processing module 1 in period 1 is 688086335 nanoseconds (ns), and the in timestamp of processing module 2 in period 1 is 688721531 ns, then the transmission latency of the transmission path between processing module 1 and processing module 2 is: 688721531 - 688086335 = 635196 ns.

[0193] In summary, by analyzing the latency of the transmission paths between processing modules and the latency of the processing modules themselves in processing data packets, the management node can determine whether there are problems with the transmission paths and processing modules, and then take appropriate measures.

[0194] Taking the management node's awareness of sub-health in IP links (i.e., the transmission path between network cards in two nodes) as an example, the following will introduce how the management node can implement the handling of sub-health in IP links based on a state machine.

[0195] For example, please refer to Figure 12, which is a schematic diagram of a state machine for a transmission path provided in this application. As shown in Figure 12, the transmission path as an IP link can include three states: normal, sub-healthy, and faulty. Furthermore, switching between the three states can be achieved based on various events. Specifically, the switching process between the three states can be referred to Table 1 below.

[0196] Table 1

[0197] In this system, transmission paths in normal condition are considered available links, and load balancing for data read and write operations is performed. Transmission paths in a sub-optimal state are considered degraded and will no longer be used to transmit new data, but data already queued for transmission will continue to be transmitted. Transmission paths in a faulty state will no longer be used to transmit data.

[0198] If, under normal conditions, the management node detects a single I / O timeout (i.e., a timeout for a single data storage or retrieval operation) and the average latency of the transmission path also times out, the node can switch the transmission path to a sub-healthy state, thereby instructing the processing module not to use this transmission path for data transmission in the future. The average latency of the transmission path can be obtained by calculating the average of the transmission latency over multiple cycles.

[0199] If the management node detects a fault in the transmission path when the transmission path is in a normal state, it can switch the status of the transmission path to fault, thereby instructing the processing module to stop using the transmission path to transmit data.

[0200] If the management node detects that the average latency of the transmission path has not timed out when the transmission path is in a sub-healthy state, it can switch the transmission path status to a normal state, instructing the processing module to continue using the transmission path to transmit data. It should be noted that when the transmission path is in a sub-healthy state and there is no business data reading or writing, the management node can detect the average latency of the transmission path through out-of-band detection.

[0201] Furthermore, based on the latency values ​​reported by each processing module, the management node can determine the average IO latency corresponding to each processing module (i.e., the average latency required for the processing module to process a single IO), thereby determining whether each processing module has encountered a problem.

[0202] For example, please refer to Figure 13, which is a schematic diagram of the state machine of a processing module in a node according to this application. As shown in Figure 13, the processing module in the node can include four states: normal, overload, sub-health, and fault. Furthermore, the four states can be switched based on various events. To achieve the switching between states, the management node needs to detect the average IO latency of each processing module, whether the IO is overloaded, and whether the processing module exhibits excessive redundancy. Excessive redundancy refers to a state where the processing module enters an infinite loop due to some anomalies.

[0203] When the processing module is in a normal state, if the average IO latency of the processing module times out, the processing module can be switched to a sub-healthy state; if the IO of the processing module is overloaded but the average IO latency has not timed out, the processing module can be switched to an overloaded state; if the processing module has excessive redundancy, the processing module can be switched to a fault state.

[0204] When the processing module is in a sub-healthy state, if the average I / O latency of the processing module has not timed out, the processing module can be switched to a normal state; if the processing module is over-redundant, the processing module can be switched to a fault state.

[0205] When the processing module is overloaded, if the average I / O latency of the processing module times out, the processing module can be switched to a sub-healthy state; if the I / O of the processing module is not overloaded, the processing module can be switched to a normal state; if the processing module has excessive redundancy, the processing module can be switched to a fault state.

[0206] When the processing module is in a faulty state, if the processing module recovers its redundancy (i.e., no longer gets stuck in an infinite loop), the processing module can be switched to a normal state.

[0207] Furthermore, when the processing module is in a normal state, it can execute data read and write processes normally. When the processing module is in a fault state, it cannot execute data read and write processes normally, so the management node isolates the processing module, meaning it is no longer used to perform data read and write processes. When the processing module is in an overload state, the management node can provide feedback to the client that generates the data packets, instructing the client to control the rate at which data traffic is sent.

[0208] When a processing module is in a sub-optimal state, the management node can further determine whether its I / O latency is worse than that of similar processing modules on other nodes. If the I / O latency is worse than that of similar processing modules on other nodes, the management node can instruct the processing module to stop executing data read / write processes or restart it. If the I / O latency is not worse than that of similar processing modules on other nodes, the management node can assume that the current situation may be due to a special type of transmitted data packet causing a short-term increase in I / O latency, and therefore will not perform any special processing on the processing module.

[0209] The above describes the detection method for the data reading and writing process provided in this application. The following will introduce the device for implementing the detection method for the data reading and writing process.

[0210] This application provides a data read / write device, including: a processor, a network card, and a memory. The memory is used to store program code, and the processor or network card is used to call the program code in the memory to cause the data read / write device to perform the steps performed by the first processing module or the second processing module as described above.

[0211] This application also provides a management device, including: a processor and a memory, the memory for storing program code, and the processor for calling the program code in the memory to cause the management device to perform the steps performed by the management node as described above.

[0212] Please refer to Figure 14, which is a schematic diagram of the structure of a network device provided in this application. The network device may be equipped with the aforementioned data read / write device and / or management device. That is, the first node, the second node, and the management node mentioned above can all be implemented by the network device.

[0213] The network device includes at least one processor 1401, a communication bus 1402, a memory 1403, and at least one network interface card 1404.

[0214] Optionally, the processor 1401 is a general-purpose CPU, NP, microprocessor, or one or more integrated circuits for implementing the solutions of this application, such as application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0215] The communication bus 1402 is used to transmit information between the aforementioned components. The communication bus 1402 includes an address bus, a data bus, and a control bus. For ease of representation, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus.

[0216] Optionally, memory 1403 is read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions. Alternatively, memory 1403 is random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions. Alternatively, memory 1403 is electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. Optionally, memory 1403 exists independently and is connected to processor 1401 via communication bus 1402. Optionally, memory 1403 and processor 1401 are integrated together.

[0217] Network interface card 1404 uses any transceiver-like device for communicating with other devices or communication networks. Network interface card 1404 includes wired network interface cards or wireless network interface cards. Wired network interface cards, for example, use an Ethernet interface. The Ethernet interface is an optical interface, an electrical interface, or a combination thereof. Wireless network interface cards use wireless local area network (WLAN) interfaces, cellular network interfaces, or combinations thereof.

[0218] In a specific implementation, as one embodiment, the processor 1401 includes one or more CPUs, such as CPU0 and CPU1 as shown in FIG14.

[0219] In a specific implementation, as one example, the network device includes multiple processors, such as processor 1401 and processor 1405 as shown in FIG14. Each of these processors is a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor refers to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0220] In some embodiments, memory 1403 is used to store program code 1406 that executes the scheme of this application, and processor 1401 executes the program code 1406 stored in memory 1403. That is, the network device implements the above-described method embodiments through processor 1401 and program code 1406 in memory 1403.

[0221] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. Wherein, "A refers to B" means that A is the same as B or A is a simple variation of B.

[0222] The terms "first" and "second," etc., used in the specification and claims of this application are used to distinguish different objects, not to describe a specific order of objects, and should not be construed as indicating or implying relative importance. For example, "first speed limit lane" and "second speed limit lane" are used to distinguish different speed limit lanes, not to describe a specific order of speed limit lanes, and should not be construed as the first speed limit lane being more important than the second speed limit lane.

[0223] In this application embodiment, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more.

[0224] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, they can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0225] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for detecting data read / write processes, characterized in that, include: The first processing module acquires the first data packet, which is a data packet transmitted from the first node to the second node in the first data read / write operation. The first node or the second node is a storage node. The first node or the second node includes multiple processing modules for performing data processing in the data read / write operation. The first processing module is any one of the multiple processing modules. In response to the inclusion of a target identifier in the first data packet, the first processing module records the first time point at which the first data packet was acquired, wherein the first marker in the target identifier is used to indicate the time point at which the first data packet was acquired and the time point at which the processing was completed. The first processing module processes the first data packet and records the second time point at which the processing of the first data packet is completed; The first processing module sends a statistical message to the management node. The statistical message includes the first time point and the second time point corresponding to the first data read / write at the first processing module. The management node is used to detect data read / write based on the statistical message.

2. The method according to claim 1, characterized in that, The method further includes: The first processing module statistically analyzes the traffic information of multiple data reads and writes within the first period at the first processing module, and the statistical message also includes the traffic information; The traffic information includes one or more of the following information read and written to the first processing module during the first period: the number of data packets received, the number of bytes of data packets received, the number of data packets sent, the number of bytes of data packets sent, the number of data packets lost, and the number of bytes of data packets lost.

3. The method according to claim 2, characterized in that, The second marker included in the target identifier is used to indicate the period to which the corresponding data packet belongs; The first processing module statistically analyzes the traffic information of multiple data reads and writes within the first period, including: The first processing module counts the data packets whose second marker value in the target identifier is a first value from multiple data packets to obtain the traffic information. The multiple data packets are data packets including the target identifier that are received or sent by the first processing module, and the first value is used to indicate the first period.

4. The method according to claim 3, characterized in that, Among the plurality of data packets, the data packet with the second tag value being the second value belongs to the second period, which is different from the first period, and the second value is different from the first value.

5. The method according to any one of claims 1-4, characterized in that, The third marker in the target identifier is used to indicate the flow identifier, which is used to indicate the data flow to which the corresponding data packet belongs. The statistical message also includes the flow identifier.

6. The method according to claim 5, characterized in that, The flow identifier includes a node identifier and a path identifier. The node identifier is used to identify the first node, and the path identifier is used to identify the transmission path of the corresponding data packet.

7. The method according to claim 5 or 6, characterized in that, A data stream includes multiple sub-streams, and the target identifier further includes a sub-stream identifier, which is used to indicate the sub-stream to which the corresponding data packet belongs; The flow identifier of the first data packet is used to indicate that the first data packet belongs to the target data stream, and the sub-flow identifier of the first data packet is used to indicate that the first data packet belongs to the first sub-flow among multiple sub-flows of the target data stream; The method further includes: When the first processing module is located at the first node, the first processing module sends the indication information of the target data stream to the management node. The indication information of the target data stream is used to indicate the sub-streams included in the target data stream.

8. The method according to claim 7, characterized in that, Data packets in multiple substreams of a data stream carry the same node identifier, but different path identifiers.

9. The method according to any one of claims 1-8, characterized in that, The first processing module runs on the network interface card, storage medium, or processor of the first node, or the first processing module runs on the network interface card, storage medium, or processor of the second node.

10. The method according to claim 9, characterized in that, When the first processing module is running on the network card, the first processing module detects the target identifier in the protocol header or payload of the first data packet; Alternatively, when the first processing module is running on the storage medium or the processor, the first processing module detects the target identifier in the payload of the first data packet.

11. The method according to claim 10, characterized in that, The header is either the header of the Infinite Bandwidth IB protocol or the header of the Internet Protocol version 6.

12. The method according to any one of claims 9-11, characterized in that, When the first processing module is running on the first node, the first processing module acquires the first data packet, including: The first processing module acquires the target data to be transmitted to the second node and generates the first data packet, which includes the target data and the target identifier.

13. The method according to any one of claims 1-12, characterized in that, The method further includes: The first processing module receives a notification message from the management node. The notification message includes any one of the following: instructing the first processing module to stop performing data read / write, instructing the first processing module to restart, or instructing the first processing module to stop transmitting data through the target path, where the target path is the path through which the first processing module transmits the first data packet.

14. A method for detecting data read / write processes, characterized in that, include: The management node receives a first statistical message and a second statistical message respectively sent by a first processing module and a second processing module from multiple processing modules. The multiple processing modules are modules running on the first node or the second node for performing data processing during data read and write. The first node or the second node is a storage node. The first statistical message includes a first time point and a second time point corresponding to the first processing module during the first data read and write. The second statistical message includes a third time point and a fourth time point corresponding to the first data read and write at the second processing module. The first time point is used to indicate the time point when the first processing module acquires the first data packet. The second time point is used to indicate the time point when the first processing module finishes processing the first data packet. The third time point is used to indicate the time point when the second processing module acquires the first data packet. The fourth time point is used to indicate the time point when the second processing module finishes processing the first data packet. The first data packet is the data packet that needs to be transmitted from the first node to the second node during the first data read and write. Based on the first statistical message and the second statistical message, if the management node determines that the first processing module among the plurality of processing modules is in a sub-healthy state, it sends a first notification message to the first processing module. The first notification message is used to instruct the first processing module to stop performing data read and write or to instruct the first processing module to restart.

15. The method according to claim 14, characterized in that, Based on the first statistical message, the management node determines that the first processing module among the plurality of processing modules is in a sub-healthy state, including: Based on the first statistical message, if the delay of the first processing module in processing data packets meets a preset condition, the management node determines that the first processing module is in a sub-healthy state. The preset condition includes that the average delay of the first processing module in processing multiple data packets exceeds a first delay value.

16. The method according to claim 14 or 15, characterized in that, The second processing module is the next processing module after the first processing module among the plurality of processing modules, and the method further includes: Based on the second time point and the third time point, the management node determines the target transmission delay for the first processing module to transmit the first data packet to the second processing module; If the target transmission delay exceeds a second delay value, the management node sends a second notification message to the first processing module. The second notification message is used to instruct the first processing module to stop transmitting data through the target path, where the target path is the path from the first processing module to the second processing module for transmitting the first data packet.

17. The method according to any one of claims 14-16, characterized in that, The first statistical message also includes first traffic information of multiple data reads and writes at the first processing module within the first period, and the second statistical message also includes second traffic information of multiple data reads and writes at the second processing module within the first period. The first traffic information includes the number of data packets sent by the first processing module after processing the data packets belonging to the first period, and the second traffic information includes the number of data packets belonging to the first period obtained by the second processing module from the first processing module. The method further includes: Based on the first traffic information and the second traffic information, the management node determines a first packet loss number, which is used to indicate the number of packets lost in the transmission path between the first processing module and the second processing module. If the first number of lost packets exceeds a preset value, the management node sends a second notification message to the first processing module. The second notification message is used to instruct the first processing module to stop transmitting data through the target path, where the target path is the path through which the first processing module transmits data packets belonging to the first period to the second processing module.

18. The method according to any one of claims 14-17, characterized in that, The first statistical message also includes a flow identifier, which is used to indicate the data flow to which the data packet corresponding to the first statistical message belongs; The method further includes: the management node determining the data stream corresponding to the first data packet based on the flow identifier.

19. The method according to claim 18, characterized in that, The flow identifier includes a node identifier and a path identifier, wherein the node identifier is used to identify the first node, and the path identifier is used to identify the transmission path of the first data packet; the method further includes: The management node determines the transmission path of the first data packet based on the path identifier.

20. The method according to claim 19, characterized in that, The first statistical message also includes a sub-flow identifier, which indicates that the first data packet corresponding to the first statistical message belongs to a sub-flow in the target data flow, and the target data flow includes multiple sub-flows originating from the first node; The method further includes: The management node determines the transmission delay of multiple sub-streams of the target data stream; If the transmission delay of the target sub-stream in the plurality of sub-streams is greater than a third delay value, the management node sends a sub-stream adjustment message to the first node. The sub-stream adjustment message is used to notify the first node to adjust the transmission path of the target sub-stream.

21. The method according to any one of claims 14-20, characterized in that, The first processing module runs on the network interface card, storage medium, or processor of the first node, or the first processing module runs on the network interface card, storage medium, or processor of the second node.

22. A data read / write device, characterized in that, include: A processor, a network interface card (NIC), and a memory, wherein the memory is used to store program code, and the processor or NIC is used to invoke the program code in the memory to cause the device to perform the method as described in any one of claims 1-13.

23. A management device, characterized in that, include: A processor and a memory, the memory being used to store program code, the processor being used to invoke the program code in the memory to cause the apparatus to perform the method as described in any one of claims 14-21.

24. A data read / write system, characterized in that, It includes the data read / write device as described in claim 22 and the management device as described in claim 23.

25. A computer-readable storage medium storing instructions, characterized in that, When the instructions are executed on a computer, the computer causes the computer to perform the method as described in any one of claims 1-21.

26. A computer program product, characterized in that, Includes program code that, when a computer runs the computer program product, causes the computer to perform the method as described in any one of claims 1-21.

Images