Smart NIC NUMA Segmentation for Virtual Router Isolation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In virtualized computing infrastructure, deploying containers on a single compute node with multiple smart network interface cards (NICs) poses challenges, such as routing packets between separate Non-Uniform Memory Access (NUMA) nodes and ensuring isolation among pods, while existing solutions face limitations with multiple smart NICs on a single node, leading to configuration failures and performance issues.
Innovation Solution
The implementation of a configuration that includes instances of datapath components, like virtual routers and virtual router agents, instantiated for each NUMA node, each with a single smart NIC, allowing for separate virtual tunnel endpoints and improved performance by avoiding NUMA boundary crossings, and running these components as Docker containers to overcome restrictions on running multiple DPDK applications on a single host.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple smart NICs are deployed on a single compute node to handle different NUMA nodes, then network throughput and packet routing capability are improved, but system complexity and configuration reliability deteriorate due to existing solutions facing limitations with multiple smart NICs
Solution Approach 1:
The system segments the compute node into separate NUMA node instances, each with its own smart NIC and virtual router. This segmentation isolates the configuration management of each NIC to its corresponding NUMA node, preventing configuration conflicts and improving overall reliability while maintaining high throughput through parallel processing across multiple NUMA nodes
Solution Approach 2:
A virtual router is introduced as an intermediary component between the control plane and each smart NIC. The virtual router receives packets from the control plane, determines the appropriate NUMA node and smart NIC for packet forwarding, and routes packets to the correct destination. This intermediary layer simplifies the control plane's interaction with multiple smart NICs, ensuring reliable configuration and operation
2Speed
If multiple DPDK applications are run on a single host to manage different NUMA nodes, then packet processing performance is improved, but system stability deteriorates due to restrictions on running multiple DPDK applications on a single host
Solution Approach 1:
The system creates separate NUMA node instances, each running its own DPDK-based virtual router application. This segmentation allows multiple DPDK applications to run in isolation on different NUMA nodes without interfering with each other, maintaining system stability while achieving high packet processing performance through parallel processing
Solution Approach 2:
The system transitions from a single-host DPDK application model to a multi-NUMA-node model where each NUMA node runs its own DPDK application. This dimensional change from one host to multiple independent NUMA nodes enables concurrent packet processing across different memory access domains while avoiding the stability issues of running multiple DPDK apps on a single host
3Reliability
If packets are routed between separate NUMA nodes, then network connectivity and pod isolation are improved, but processing latency increases due to NUMA boundary crossings
Solution Approach 1:
The virtual router acts as an intermediary that receives packets from the control plane, determines the appropriate NUMA node and smart NIC for packet forwarding, and routes packets to the correct destination. By centralizing the routing decision logic in the virtual router, the system minimizes latency in packet routing between NUMA nodes while maintaining reliable pod isolation through proper virtual network configuration
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Example techniques and computing devices are disclosed. An example computing device includes a first non-uniform memory access (NUMA) node and a second NUMA nod. The first NUMA node includes a first network interface card, a first virtual router for one or more virtual networks, the first virtual router comprising first processing circuitry and configured with a first virtual host interface having a first Internet Protocol (IP) address, and a first workload executing on the first NUMA node. The second NUMA node includes a second network interface card, a second virtual router for the one or more virtual networks, the second virtual router comprising second processing circuitry and configured with a second virtual host interface having a second IP address, and a second workload executing on the second NUMA node.