Neural Network Acceleration via Switch Fabric Routing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for accelerating deep learning with hardware accelerators face challenges such as memory bottlenecks and increased power consumption when handling large neural networks, as they often require splitting networks across multiple accelerators, which can lead to inefficient data transfer and latency issues due to reliance on the host processor for intermediate results routing.
Innovation Solution
The approach involves splitting a large neural network into multiple AI inference accelerators, allowing direct communication between them without involving the host processor, and implementing a scheduling policy to manage data and control paths across multiple devices, supporting various topologies, to optimize power consumption and performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a large neural network is split across multiple hardware accelerators, then the processing capacity is improved, but the data transfer latency and power consumption increase due to host processor involvement
Solution Approach 1:
The patent introduces a switch fabric as an intermediary communication infrastructure that enables direct peer-to-peer data transfer between inference accelerators, eliminating the host processor as the mandatory intermediary. This allows accelerators to communicate directly through a dedicated switching network, reducing latency while maintaining the benefits of distributed processing across multiple accelerators.
Solution Approach 2:
The patent transitions from a hierarchical communication topology (accelerators → host processor → accelerators) to a mesh-like direct connectivity topology using switch fabric. This dimensional change in the communication architecture allows any accelerator to directly reach any other accelerator without going through the host processor, fundamentally changing the data flow path and reducing latency.
2Productivity
If a large neural network is split across multiple hardware accelerators, then the processing capacity is improved, but the power consumption increases due to host processor involvement in routing
Solution Approach 1:
The switch fabric serves as a specialized intermediary that handles data routing between accelerators more efficiently than the general-purpose host processor. By offloading the routing function to the switch fabric, the system reduces the computational burden on the host processor and enables more energy-efficient direct communication paths between accelerators.
Solution Approach 2:
The inference accelerators are empowered to communicate directly with each other through the switch fabric without requiring continuous host processor intervention. This self-service capability allows accelerators to autonomously exchange intermediate results and coordinate their operations, reducing the overall system power consumption by eliminating unnecessary host processor involvement in routine data transfer operations.
3Productivity
If neural network inference is distributed across multiple accelerators, then the throughput is improved, but the system complexity increases due to scheduling and coordination requirements
Solution Approach 1:
The switch fabric provides a universal communication infrastructure that handles all types of data transfers between accelerators (inputs, intermediate results, outputs) through a single standardized interface. This multi-functional communication backbone simplifies the scheduling complexity by providing a unified mechanism for coordinating data flow across different accelerators, regardless of the specific operation being performed.
Data Source
AI summary
A method for accelerating machine learning on a computing device is described. The method includes accessing a neural network. The method also includes splitting the neural network into N sub-neural networks. The method further includes hosting the N sub-neural networks in M inference accelerators. The method also includes scheduling the N sub-neural networks in the M inference accelerators. The method further includes executing the N sub-neural networks in the M inference accelerators.