RDMA Server Global TCE Tables for Out-of-Order Packet Delivery
Solution Overview
Problem
Existing RDMA technologies are limited by their reliance on reliable transport and in-order packet delivery, leading to bandwidth constraints, complex adapter design, and inability to efficiently implement features like striping, broadcasting, and conditional operations due to the 'send and forget' model and assumption of reliable network transport.
Innovation Solution
The solution involves a mechanism for RDMA data transfer that allows packets to arrive out of order, enabling efficient transfer over unreliable networks, supporting multiple paths, and allowing for features like broadcasting, multicasting, and conditional transfers by using a self-describing packet structure and RDMA context (rCxt) to ensure accurate delivery and handling of packets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reliable transport and in-order packet delivery are assumed, then data integrity is maintained, but bandwidth utilization is limited and adapter design becomes complex
Solution Approach 1:
The patent segments data into packets that can be transmitted independently across multiple paths. Each packet contains sufficient information (sequence numbers, checksums, routing headers) to be correctly reassembled at the destination even if received out of order. This segmentation enables parallel transmission over multiple paths without requiring strict in-order delivery, thereby increasing bandwidth utilization while maintaining data integrity through proper packet reassembly protocols.
Solution Approach 2:
The patent introduces multiple transmission paths as an additional dimension beyond the traditional single-path reliable transport model. By allowing packets to take different routes simultaneously and receiving them in any order, the system transforms a one-dimensional sequential transmission problem into a multi-dimensional parallel transmission problem, effectively increasing throughput without sacrificing integrity through sophisticated packet management.
2Reliability
If packets must be received in order, then data integrity is ensured, but adapter complexity increases due to buffering requirements
Solution Approach 1:
The patent extracts the ordering and reliability functions from the hardware adapter and relocates them to the software protocol layer. The adapter simply handles raw packet transmission without needing to buffer or reorder packets. The complex tasks of sequence tracking, checksum validation, and packet reassembly are performed by software protocols at the destination, thereby simplifying the hardware adapter design while maintaining data integrity through software-based reliability mechanisms.
Solution Approach 2:
The patent replaces the mechanical buffering and reordering mechanism traditionally required in hardware adapters with a software-based packet management system. Instead of using hardware queues and buffers to ensure in-order delivery, the system uses software protocols that can handle out-of-order packets by tracking sequence numbers and reassembling data in the correct order at the destination, thereby eliminating complex hardware buffering requirements.
3Device complexity
If single-path transmission is used, then packet ordering is simple, but bandwidth is limited and striping cannot be implemented
Solution Approach 1:
The patent creates a universal packet structure that can be transmitted over multiple paths simultaneously while maintaining compatibility with existing reliable transport protocols. The standardized packet format includes fields for sequence numbers, checksums, and routing information, allowing the same packet structure to be used whether packets take one path or multiple paths. This universality enables the system to implement striping and load balancing across multiple paths without complicating packet ordering or requiring path-specific packet formats.
Data Source
AI summary
In remote direct memory access (RDMA) transfers in a multinode data processing system in which the nodes communicate with one another through communication adapters coupled to a switch or network, there is a need for the system to ensure efficient memory protection mechanisms across jobs. A method is thus desired for addressing virtual memory on local and remote servers that is independent of the process ID on the local and/or remote node. The use of global Translation Control Entry (TCE) tables that are accessed/owned by RDMA jobs and are managed by a device driver in conjunction with a Protocol Virtual Offset (PVO) address format solves this problem.