Network controller

Abstract

The invention relates to the field of computing, and more particularly to network controllers for the transmission of data. A network controller comprises a traffic processor capable of receiving packets and classifying traffic by queues; a packet validation module capable of checking the validity and integrity of data, sending packets to a response processor, and distributing operations by type; a memory capable of receiving RDMA operations from the packet validation module; and a task receiving module for receiving SEND operations and saving same to an external memory. The network controller further comprises a completed receive task module, and a semaphore synchronization and management module capable of storing, modifying and synchronizing semaphores, and receiving requests to modify and read local semaphores. The device also includes an ACK / NAK response processor capable of receiving packets; a task processor and queue arbiter; a task sending module; and a completed send task module. The network controller still further comprises a packet assembler capable of forming an RoCEv2 packet and transmitting said packet in order for it to be sent into the network. The invention makes it possible to reduce the load on a processor during data transmission.

Description

NETWORK CONTROLLER AREA OF TECHNOLOGY

[0001] The claimed technical solution generally relates to the field of computing technology, and in particular to network controllers for data transmission. LEVEL OF TECHNOLOGY

[0002] Modern high-performance computing (HPC) data centers are built on multiple network protocols. Historically, each of the three main services in a data center operates on its own network: HPC - InfiniBand, data storage - Fibre Channel, and the management and service network - Ethernet. Maintaining three types of equipment, three support teams, and transferring data between networks via interface converters is impractical. The idea arose to build the entire network on a single protocol and hardware. This technology is called Converged Ethernet, which combines data center communications into a single network protocol.

[0003] InfiniBand is one of the most widely used protocols for high-performance data centers. However, it has several drawbacks: high CPU load for setting up and synchronizing exchanges, leading to increased RDMA exchange start latency, and increased network load from service traffic between remote CPUs.

[0004] Data exchange in data centers places a number of demands on communications: high exchange speed, low latency, no loss, low CPU load, and network scalability.

[0005] Ethernet was designed for general-purpose networks with varying equipment and link quality. Ethernet is standardized by the IEEE, is widely deployed worldwide, is open to general use, is supported by many manufacturers, and has more affordable network equipment. TCP / IP is the primary protocol stack used for data transmission. It supports segment recovery and guaranteed data delivery. However, such communications are extremely slow, with high processing latencies, require triple copying of data in memory, have a high CPU load, and lack RDMA support. .i

[0006] Fibre Channel is a proprietary protocol currently used as a standard for data storage systems (DSS). It has been underdeveloped and supported recently, significantly lagging behind its competitors in speed, and has a limited number of manufacturers on the market.

[0007] iWARP was one of the first attempts to implement RDMA on Ethernet. iWARP encapsulates RDMA (RDMAP) over TCP / IP, adding two additional intermediate traffic processing layers: Direct Data Placement (DDP) and Marker PDU Alignment (MPA). Its only advantages are RDMA and loss recovery; all other requirements (high throughput, low latency, low CPU load) are not met. This protocol was not supported by manufacturers and did not achieve widespread adoption.

[0008] The disadvantages of known solutions in this area of ​​technology include increased latency during data transmission and increased load on the processor. DISCLOSURE OF THE INVENTION

[0009] The declared technical solution proposes a new approach to organizing the architecture of a network controller for data transmission.

[0010] This solves the technical problem of increased processor load, increased data transfer delays, and increased network load from service traffic. [OOP] The technical result achieved by solving this problem is a reduction in the load on the processor during data transfer.

[0012] The specified technical result is achieved by implementing a network controller containing: o a traffic processor capable of receiving L2 Ethernet packets from external Ethernet PCS / PMA and Ethernet MAC cores, extracting headers of incoming packets, and classifying traffic according to its membership in configured queues; o a packet validation module configured to check the correctness and integrity of data, send ACK / NAK packets to the response handler, select three types of operations: RDMA, SEND, Atomic and send RDMA operations to the registered memory area, SEND operations to the task receiving module, Atomic operations to the synchronization and semaphore control module; o memory configured to receive RDMA operations from the packet validation module; o a task receiving module, configured to receive SEND operations from the validation module and save the received data in external memory; o a module for receiving completed tasks, configured to store records of the status of completed SEND, RDMA WRITE tasks; o a module for synchronization and semaphore management, configured to receive Atomic operations from a module for validation, storage, modification and synchronization of semaphores, receiving requests for modification and reading local semaphores from external modules of high-performance computers; o an ACK / NAK response handler configured to receive ACK / NAK packets from the validation module, generate confirmation of the execution of sending tasks, request resending of packets in the event of an error, and record the status of the completed tasks in the completed tasks module; o a task processor and queue arbiter configured to read work tasks for sending from the task sending module, generate messages from the tasks, split the message into packets, arbitrate the priority of queues, modify local semaphores in the synchronization and semaphore control module, and transfer tasks for sending a packet to the packet assembler; o a task submission module capable of submitting tasks to a task processor and a queue arbiter; o a completed job send module configured to store records of the status of completed jobs from the job send module; o a packet assembler configured to receive a packet send job from a job handler and a queue arbiter, request data from external memory, generate a RoCEv2 packet, and transmit the packet to external cores for sending to the network. BRIEF DESCRIPTION OF DRAWINGS

[0013] Fig. 1 illustrates an example of the RDMA RoCEv2 protocol concept.

[0014] Fig. 2 illustrates an example of network congestion prevention.

[0015] Fig. 3 illustrates an example of a typical network exchange architecture.

[0016] Fig. 4 illustrates an example of a network controller architecture.

[0017] Fig. 5 illustrates an example of the operation of the network controller.

[0018] Fig. 6 illustrates an example of RDMA exchange synchronization. IMPLEMENTATION OF THE INVENTION

[0019] The RoCEv2 protocol is an implementation of RDMA over Ethernet. RoCE stands for RDMA over Converged Ethernet. RoCEv2 encapsulates InfiniBand transport layer data in UDP packets, allowing full use of the IP layer for network addressing and flow prioritization.

[0020] The basic concept of RDMA is a job queue (Queue). The host processor places jobs into a Work Queue Element (WQE), which is analogous to a descriptor. Queues can be Send, Receive, or Completion Queue (Fig. 1).

[0021] Queue Pair (QP) - a pair of queues in the controller, consisting of a Send and Receive Queue for one of the data transfer directions.

[0022] Queue Pairs is a virtual point-to-point interface that provides isolation and protection of data exchange from other Queue Pairs using the R Key. A one-to-one correspondence between the Queue Pairs of two network nodes is established.

[0023] Memory Region (MR) is a virtual, contiguous region of memory allocated for RDMA exchange via QPs. It has the following parameters: Virtual Address, Length, and R Key. The R Key is a simple number that is compared to the one specified during QPs configuration upon packet reception.

[0024] Protection Domain - QPs, MR association mechanism. Access is protected by an R Key authorization mechanism.

[0025] Transport Functions are a set of operations for forwarding data. One transport function fits into one WQE task. One WQE is one message. The message is split into packets of equal PMTU length, a power of 2: 64B, ..., 2K, 8KB.

[0026] SEND is a message sending operation that does not specify a memory address, similar to PUSH. Incoming data utilizes the WQE of the receiving queue, and the data is stored in a pre-allocated buffer. SEND is more often used for service exchanges than for data.

[0027] RDMA READ, RDMA WRITE - data transfer operations specifying a virtual address in remote memory and the protection key R_Key.

[0028] The InfiniBand transport layer defines connection types: Reliable with delivery confirmation and Unreliable without confirmation (DMA is not built on it), point-to-point connections, and multipoint Datagramm. Reliable Connection is most widely used for RDMA—a type of reliable virtual connection called QPs with guaranteed delivery of data packets in strict order. This places requirements on network equipment to avoid losses and channel disruptions.

[0029] If the sequence of psn packet numbers or msn messages is out of order, or if there are losses, a NAK response with an error condition is sent to the transmitter. At the transmitter, the queue is rolled back to the point of failure and the data is retransmitted. Missing segments are not rebuilt, as is the nature of InfiniBand.

[0030] To minimize traffic loss, Ethernet Priority Flow Control (PFC) and Pause Frame protocols are supported for slowing down transmissions. For more granular data integrity, Explicit Congestion Notification (ECN) bandwidth overshoot alerts and Congestion Notification Packets (CNP) congestion avoidance notifications are used.

[0031] The operation of the overload warning mechanism is described in Fig. 2.

[0032] Traffic from several subscribers and queues QP1-3 is transmitted toward network node NIC2. The traffic rate on the switch toward NIC2 exceeds the line's throughput, and the switch's output buffer begins to overflow. Upon reaching the threshold, the switch adds ECN bits to the IP header of outgoing packets, signaling impending overflow and packet discard. In response to the ECN bits, NIC2 sends packets to the CNP transmitter requesting a reduction in the traffic rate. This mechanism proactively prevents losses while maintaining the integrity of the data exchange. According to Huawei, losses of up to 2% in an Ethernet network lead to a drop in the RDMA transmission rate to zero.

[0033] The RDMA protocol describes the data transfer mechanism, but the document does not address queuing between nodes, task coordination, or exchange synchronization. The user implements this through higher-level software protocols and libraries, such as UCX. In this case, the entire management burden falls on the host.

[0034] The typical architecture of network exchange. Fig. 3 assumes that the processors of two nodes (101) and (104) pre-arrange the upcoming exchange, the readiness of data for sending, and the available space for receiving. The host initiating the data exchange (101) generates a task for sending WQE and places it in the sending queue SQ (105) of the network controller (102).

[0035] The controller (102) sends a SEND (106) message with a task to the client host (104). The internal exchange format between the two hosts (101) and (104) is not defined in InfiniBand and is determined by the higher-level exchange protocols. The receiving controller (103) stores the received message in the receiving queue RQ (107) and sends back an ACK (109).

[0036] Host (101) receives a CQE acknowledgment from controller (102) in CQ (110) to publish a task to remote host (104) and is ready for RDMA exchange. In general, several SEND messages may be required to coordinate RDMA exchange.

[0037] The host (104) processes the received SEND (106) message from the RQ (107) and publishes a new WQE task in the SQ (108) for RDMA data transfer. The controller (103) generates an RDMA WRITE (111) message and transfers data from the client memory (112) to the initiator memory (IM). After receiving all ACK confirmations (114), the controller (103) records the CQE write execution status in the CQ (115). The host (104) processes the CQE element from the queue (115) and generates a WQE send task (116). The SEND (117) message contains information about the completion of the RDMA exchange and the readiness of the data in the memory (113).

[0038] Thus, setting up and synchronizing RDMA forwarding requires at least two additional SEND-type exchanges initiated by software on the host processors. This increases data transfer latency by the time required for the processor to process commands. Additional overhead exchanges between hosts are also required, which loads processor cores and increases the network load.

[0039] The declared architecture of the network controller eliminates the above problems.

[0040] As shown in Fig. 4, the network controller (201) comprises: a traffic handler (205) configured to receive L2 Ethernet packets from external Ethernet PCS / PMA and Ethernet MAC cores, extract headers of incoming packets and classify traffic according to membership in configured queues, a packet validation module (206) configured to check the correctness and integrity of data, send ACK / NAK packets to the response handler, select three types of operations: RDMA, SEND, Atomic and send RDMA operations to a registered memory area, SEND operations to a task reception module, Atomic operations to a synchronization and semaphore control module, memory (208) configured to receive RDMA operations from the packet validation module, a task reception module (209) configured to receive SEND operations from the validation module, store the received data in external memory, a completed task reception module (207),configured to store records of the status of completed SEND and RDMA WRITE tasks, a semaphore synchronization and management module (210) configured to receive Atomic operations from the validation module, store, modify, and synchronize semaphores, receive modification requests, and read local semaphores from external high-performance computing modules, an ACK / NAK response handler (213) configured to receive ACK / NAK packets from the validation module, generate confirmation of the execution of sending tasks, request resending of packets in case of an error, record the status of the execution of tasks in the completed tasks module, a task processor and queue arbiter (214) configured to read work tasks for sending from the task sending module, generate a message from the tasks, split the message into packets, arbitrate the priority of queues, modify local semaphores in the synchronization and semaphore management module,transmitting tasks for sending a packet to a packet collector, a task sending module (212) configured to send tasks to a task processor and a queue arbiter, a module of completed tasks for sending (217) configured to store records of the status of completed tasks from the task sending module, a packet collector (215) configured to receive a task for sending packets from the task processor and the queue arbiter, request data from external memory, form a RoCEv2 packet and transmit this packet to external cores for sending to the network.

[0041] The declared network controller processes RDMA RoCEv2 protocol traffic. The controller's operation is shown in detail in Fig. 5.

[0042] The external Ethernet PCS / PMA (203) and Ethernet MAC (204) cores enable the reception and transmission of L2 Ethernet frames.

[0043] The IP, UDP, IBTH (InfiniBand Transport Header) traffic handler (205) extracts the headers of incoming packets and classifies the traffic into the configured QP queues.

[0044] The Packet Validation Module (206) verifies the correctness and integrity of data. It also identifies function types: RDM A operations are sent to the registered Memory Region (MR) (208), SEND operations to the Receive Queue (RQ) Buffer (209), and Atomic operations to the Synchronization and Semaphore Management Module (210). Upon completion of reception or an error, the status of the received SEND message is written to the Receive Completion Queue (RCQ) (207).

[0045] The semaphore synchronization and management module (210) stores, modifies, and synchronizes semaphores. Requests to modify remote semaphores come from module (206). Requests to modify and read local semaphores come from external modules of the HPC high-performance computing system (211).

[0046] The QP job processor and queue arbiter (214) reads WQE work orders from the Send Queue (212), forms a message from the order, splits it into packets, arbitrates the priority of QP queues, waits for semaphore conditions to be met, and modifies local semaphores in the module (210). Upon completion of message sending, the module (214) writes the completion status to the Send Completion Queue (217).

[0047] The packet collector (215) receives a task to send a packet from (214), requests data from Memory (216), forms the Infiniband, Ethernet packet header and transmits it to the external cores (204), (203) for sending to the network.

[0048] In neural network training tasks, all computational stages are well-determined and predictable. The host processor knows the exchange stages at which it must wait for remote nodes to synchronize. The host processor initially schedules the SQ task queue taking into account the required wait stages for Sync semaphores and RDMA data exchanges.

[0049] The RDMA exchange synchronization mechanism is shown in detail in Fig. 6. The local node's SQ (300) sending queue contains RDMA exchange tasks (302) awaiting execution by the Sync (303) synchronization task. The Sync (303) task waits for local (306) and remote (307) semaphores to fire. Local semaphores are modified by internal system processes (completion of task computation (309), readiness of data in internal memory, freeing up memory (310)). Remote semaphores are sent over the network and report the readiness of processes in remote nodes.

[0050] The remote node (301) executes the Sync task (304) and waits for the local semaphore (308) to become ready. After the semaphore (308) is fired, the task queue moves on to the Atomic operation (305) – transferring and modifying the local semaphore (308) to the remote semaphore module (307). The Sync task (303) in the local node completes after all local (306) and remote (307) semaphores are fired. The SQ send queue (300) moves on to the RDMA data transfer task (302) when both nodes are ready.

[0051] Thus, the declared network controller allows solving a number of technical problems.

[0052] When connecting eight or more RDMA RoCEv2 network devices to a single host CPU, its load will be close to its maximum. The server is unable to perform any tasks other than configuring and synchronizing RDMA exchanges over the network.

[0053] The purpose of the proposed network controller is to relieve the load on the central processor. High CPU load leads to increased program execution and network communication latency, limiting the expansion of the number of network interfaces and the throughput of data channels.

[0054] The proposed architecture of the network controller allows solving the technical problem by hardware implementation of functions for synchronizing interactions between nodes.

[0055] The technical effect of applying the proposed solution is: Reduced latency in response to synchronization events. The network card waits for data to be ready and immediately begins network communication upon receiving the flag. No need to access the processor via the PC1e bus, which can take up to 1.5 µs. Reducing network overhead traffic between processors on remote nodes. Processors constantly exchange buffer status and data transfer readiness. The network card waits for memory readiness and sends a single packet to the remote node, allowing transmissions to begin. Reduced CPU load. The software polls the controller less frequently and does not initiate network communications.

[0056] The submitted application materials disclose preferred examples of the implementation of the technical solution and should not be interpreted as limiting other, particular examples of its implementation that do not go beyond the scope of the requested legal protection, which are obvious to specialists in the relevant field of technology.

Claims

FORMULA 1. Network controller, contains: o a traffic processor capable of receiving L2 Ethernet packets from external Ethernet PCS / PMA and Ethernet MAC cores, extracting headers of incoming packets, and classifying traffic according to its membership in configured queues; o a packet validation module, designed with the ability to check the correctness and integrity of data, send ACK / NAK packets to the response handler, identify three types of operations: RDM A, SEND, Atomic and send RDMA operations to the registered memory area, SEND operations to the task reception module, Atomic operations to the synchronization and semaphore control module; o memory configured to receive RDMA operations from the packet validator module; o a task receiving module, configured to receive SEND operations from the validation module, and to save the received data in external memory; o a module for receiving completed tasks, configured to store records of the status of completed SEND, RDMA WRITE tasks; o a module for synchronization and semaphore management, configured to receive Atomic operations from a module for validation, storage, modification and synchronization of semaphores, receiving requests for modification and reading local semaphores from external modules of high-performance computers; o an ACK / NAK response handler capable of receiving ACK / NAK packets from the validation module, generating confirmation of the completion of sending tasks, requesting resending of packets in the event of an error, and recording the status of the completed tasks in the completed tasks module; o a task processor and queue arbiter capable of reading work tasks for sending from the task sending module, generating messages from the tasks, splitting messages into packets, arbitrating queue priority, modifying local semaphores in the synchronization and semaphore management module, and transmitting tasks for sending packets to the packet assembler; o a task submission module capable of submitting tasks to a task processor and a queue arbiter; o a completed job send module configured to store records of the status of completed jobs from the job send module; o a packet assembler configured to receive a packet send job from a job handler and a queue arbiter, request data from external memory, generate a RoCEv2 packet, and transmit the packet to external cores for sending to the network.

Images