Virtual NIC Routing for Cross-Subnet AI Training Communication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In AI training platforms, ROCE network cards in different sub-networks cannot communicate directly, leading to potential failure of training tasks due to broadcast storms and the need for unified network management, especially when using containers.
Innovation Solution
A network card communication method that builds a spine-leaf network topology, virtualizes physical network cards into virtual ones, allocates them to job-containers with specific sub-networks, and adds sub-network communication policy routing rules to ensure data transmission across different sub-networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple ROCE network cards are divided into different sub-networks to avoid broadcast storms and comply with unified network management, then network management is improved, but cross sub-network communication fails
Solution Approach 1:
The patent introduces a routing rule mechanism as an intermediary that enables communication between different sub-networks. The routing rule includes a source sub-network identifier and destination sub-network identifier, allowing the system to route data packets between sub-networks through a mediator mechanism, thus resolving the communication failure issue while maintaining the benefits of sub-network division
Solution Approach 2:
The patent segments the network into multiple sub-networks for better management, and then introduces a routing mechanism that segments the communication path between sub-networks. Each sub-network maintains its independence for management purposes while the routing rules create controlled communication paths between them, resolving the contradiction between segmentation benefits and communication capability
2Object-affected harmful factors
If NCCL_CROSS_NIC=0 is set to avoid cross sub-network communication, then broadcast storms are avoided, but training task reliability deteriorates
Solution Approach 1:
The routing rule acts as an intermediary mechanism that selectively enables cross sub-network communication only when needed for training tasks, while maintaining the default isolation that prevents broadcast storms. The routing rule includes source and destination sub-network identifiers that control when communication is permitted, thus preventing harmful broadcast storms while enabling reliable training task completion
3Productivity
If physical network cards are virtualized and allocated to different sub-networks, then resource utilization is improved, but communication reliability deteriorates
Solution Approach 1:
The routing rule mechanism serves as an intermediary that ensures reliable communication between virtual network cards allocated to different sub-networks. The routing rule includes source sub-network identifier and destination sub-network identifier, enabling the system to route data packets correctly between different sub-networks, thus maintaining communication reliability while achieving improved resource utilization through virtualization
Data Source
AI summary
Disclosed are a network card communication method and apparatus for an AI training platform, a device, and a medium. The method includes: building a switch network topology on the basis of a spine-leaf network, and configuring a preset number of virtual local area networks for each leaf switch in the switch network topology; virtualizing a physical network card to obtain virtual network cards, allocating the virtual network cards to corresponding job-containers according to a preset allocation rule, and allocating, to each virtual network card in the job-containers, different sub-networks corresponding to the virtual local area networks; and adding a corresponding sub-network communication policy routing rule to a pod where each job-container is located, whereby a virtual network card in the job-container sends training data to the remaining virtual network cards on a basis of the sub-network communication policy routing rule.