Overlay Fabric Scheduling for AI/ML Tail-Latency Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional networks struggle to handle high-speed, high-volume AI/ML workloads due to their blocking nature, leading to network congestion and increased tail latency, especially in web scale environments where multiple concurrent jobs exacerbate synchronization issues.
Innovation Solution
Implementing an on-demand overlay protocol-based fabric network using Locator ID Separation Protocol (LISP) with Map Resolvers (MRs) and Border Gateway Protocol-Ethernet Virtual Private Network (BGP-EVPN) with Route Reflector (RR) to create a distributed service control plane, which separates forwarding from the control plane and utilizes publication-subscription mechanisms to optimize ingress Virtual Output Queue (VOQ) scheduling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional underlay networks are used, then network infrastructure is simple and cost-effective, but the networks are blocking and cannot handle high-speed, high-volume AI/ML workloads
Solution Approach 1:
The patent implements an overlay network built upon the existing underlay network, where the overlay layer encapsulates complex scheduling and routing functions within the simpler underlay infrastructure. This nested structure allows the system to leverage the cost-effective underlay while adding the necessary intelligence to handle AI/ML workloads without requiring complete network replacement.
Solution Approach 2:
The patent introduces a service control plane as an intermediary layer between the underlay network and the workload traffic. This control plane mediates by centralizing scheduling decisions and distributing them to ingress virtual output queues, enabling non-blocking performance while keeping the underlying data path simple and unchanged.
2Adaptability or versatility
If overlay networks are built on top of blocking underlay networks, then network flexibility is improved, but the networks remain blocking and experience increased tail latency
Solution Approach 1:
The patent implements preliminary scheduling actions by having the service control plane pre-compute optimal routing paths and allocate resources before traffic arrives. Ingress virtual output queues maintain scheduled transmission opportunities, ensuring that packets are forwarded without waiting for congestion to resolve, thereby reducing tail latency while preserving network flexibility.
3Ease of operation
If centralized control plane is used, then scheduling is simplified, but single point of failure risk increases
Solution Approach 1:
The patent segments the centralized control plane functionality into distributed components by implementing multiple independent service control planes that operate autonomously. Each control plane manages scheduling for its associated ingress virtual output queues without requiring coordination with others, eliminating the single point of failure while maintaining scheduling simplicity through local decision-making.
Data Source
AI summary
Techniques and architecture are described for a method, implemented within a Clos configured backend network of a web scale network includes registering, with a distributed service control plane, a plurality of egress endpoints and status of the egress endpoints, and distributing, by the service control plane to a plurality of ingress virtual output queues (VOQs), registration information relating to the plurality of egress endpoints. The method also includes based at least in part on the distributing, scheduling packets for transmission from the plurality of ingress VOQs to the plurality of egress endpoints and forwarding, by the plurality of ingress VOQs to the plurality of egress endpoints, packets. The method may also include updating the registration information.