Disaggregated Switch Control Path With Direct Data Bypass
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing server Host-Accelerator systems face increased latency and reduced bandwidth due to tree topologies in PCIe and CXL protocols, which require traffic to traverse through a switch, leading to bottlenecks and inefficient resource usage.
Innovation Solution
Implementing a computing system with separate data and configuration paths, where data packets bypass the switch using a direct path and configuration packets use an indirect path through the switch, thereby reducing latency and optimizing bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If control logic is distributed to network elements ( disaggregated control ), then control flexibility and adaptability are improved, but control path complexity and communication overhead increase
Solution Approach 1:
The control plane is segmented into distributed control logic within network elements (ingress switches, egress switches) while maintaining a centralized controller for high-level management. This segmentation allows control flexibility at the network element level while the centralized controller simplifies the overall control path by providing unified orchestration.
Solution Approach 2:
A centralized controller acts as an intermediary between distributed network elements, providing a simplified control path for policy management and coordination. The controller mediates communication between ingress and egress switches, reducing direct peer-to-peer complexity while maintaining distributed control benefits.
2Speed
If control logic is distributed to network elements, then response time to local events is improved, but synchronization and consistency across the network become more difficult
Solution Approach 1:
The centralized controller implements feedback mechanisms to monitor and coordinate distributed control decisions across network elements. This feedback loop ensures synchronization and consistency by allowing the controller to observe network state and adjust distributed control actions to maintain global coherence.
Solution Approach 2:
The centralized controller performs preliminary actions by pre-configuring policies and parameters in distributed network elements before actual data plane operations. This preliminary setup ensures consistency is established in advance, allowing fast local response times while maintaining synchronized state across the network.
3Ease of manufacture
If traditional control paths are used in disaggregated switches, then implementation simplicity is maintained, but control latency and processing overhead increase
Solution Approach 1:
The control logic is extracted from the traditional centralized control path and placed directly into network elements (ingress and egress switches). This extraction eliminates intermediate processing steps in the control path, reducing control latency while maintaining implementation simplicity through standardized switch components.
Solution Approach 2:
Control logic is copied from the centralized controller to distributed network elements, creating local copies of control functions. This copying enables fast local decision-making without requiring continuous communication with the central controller, reducing latency while keeping the overall system architecture simple and manageable.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments herein describe techniques for separating data transmitted between I/O functions in an integrated component and a host into separate data paths. In one embodiment, data packets are transmitted using a direct data path that bypasses a switch in the integrated component. In contrast, configuration packets (e.g., hot-swap, hot-add, hot-remove data, some types of descriptors, etc.) are transmitted to the switch which then forwards the configuration packets to their destination. The direct path for the data packets does not rely on switch connectivity (and its accompanying latency) to transport bandwidth sensitive traffic between the host and the I/O functions, and instead avoids (e.g., bypasses) the bandwidth, resource, store/forward, and latency properties of the switch. Meanwhile, the software compatibility attributes, such as hot plug attributes (which are not latency or bandwidth sensitive), continue to be supported by using the switch to provide a configuration data path.