NUMA-Aware Overlay Load Balancing with Tunnel Endpoint Groups
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Modern processors with non-uniform memory access (NUMA) nodes experience network performance degradation due to data message flows traversing through physical network interface cards not associated with the running workload's NUMA node, which traditional link aggregation groups cannot address.
Innovation Solution
A method for forwarding data messages by identifying a tunnel endpoint group (TEPG) associated with a specific NUMA node and selecting a particular tunnel endpoint (TEP) for message transmission, using virtual switches to perform load balancing and encapsulation to ensure messages are routed through the correct NUMA node.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional link aggregation groups (LAGs) are used for load balancing, then network throughput is improved, but NUMA node association is lost causing performance degradation
Solution Approach 1:
The patent segments the LAG into multiple TEPGs, where each TEPG is associated with a specific NUMA node. This segmentation allows the system to maintain both load balancing capabilities across multiple network interfaces and strict NUMA node association, preventing the performance degradation that occurs with traditional LAGs.
Solution Approach 2:
The patent applies local quality by creating TEPGs with specific NUMA node affinity, where each TEPG is optimized for its associated NUMA node. This ensures that data messages are processed by the correct NUMA node, maintaining local quality and preventing cross-NUMA node transfers that would degrade performance.
2Productivity
If data message flows are sent through any available network interface, then load balancing is achieved, but cross-NUMA node transfers occur increasing latency
Solution Approach 1:
The patent performs preliminary action by pre-establishing TEPGs with specific NUMA node associations before data message flows are routed. This preliminary configuration ensures that when load balancing is performed, the selection of network interfaces already respects NUMA node boundaries, avoiding cross-NUMA node transfers and the associated latency.
3Adaptability or versatility
If tunnel endpoints are distributed across multiple NUMA nodes, then network versatility is improved, but message routing complexity increases
Solution Approach 1:
The patent introduces TEPGs as an intermediary layer between the virtual switch and the underlying NUMA node architecture. This intermediary abstraction simplifies the routing logic by providing a clear mapping between TEPGs and NUMA nodes, reducing the complexity of determining which tunnel endpoints should receive data messages while maintaining versatility across multiple network interfaces.
Data Source
AI summary
Some embodiments provide a novel method for forwarding data messages between first and second host computers. To send, to a first machine executing on the first host computer, a flow from a second machine executing on the second host computer, the method identifies a destination network address of the flow. The method uses the identified destination network address to identify a particular tunnel endpoint group (TEPG) including a particular set of one or more tunnel endpoints (TEPs) associated with a particular non-uniform memory access (NUMA) node of a set of NUMA nodes of the first host computer. The particular NUMA node executes the first machine. The method selects, from the particular TEPG, a particular TEP as a destination TEP of the flow. The method sends the flow to the particular TEP of the particular NUMA node of the first host computer to send the flow to the first machine.