Distributed AI Pipeline Partitioning Across GPUs With RDMA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Single GPUs often lack sufficient processing throughput to complete complex AI tasks, such as real-time processing of high-resolution video streams, within the required timeframe.
Innovation Solution
A processing pipeline is partitioned across multiple computational accelerators, such as GPUs, connected by a packet communication network, with each partition executed on a corresponding accelerator, utilizing RDMA transactions for data transfer between stages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a single GPU is used to process complex AI tasks, then the system structure is simple, but the processing throughput is insufficient
Solution Approach 1:
The processing pipeline is divided into multiple partitions, with each partition containing one or more processing stages. Each partition is executed on a separate computational accelerator, allowing the system to distribute the processing workload across multiple devices. This segmentation enables higher throughput by utilizing multiple GPUs simultaneously while maintaining manageable system complexity through modular partitioning.
2Productivity
If multiple computational accelerators are used to increase processing throughput, then the processing speed improves, but the data transfer overhead increases
Solution Approach 1:
A network interface controller (NIC) is introduced as an intermediary between computational accelerators to manage data transfer. The NIC executes remote direct memory access (RDMA) transactions, enabling efficient data transfer between accelerators without burdening the CPU. This intermediary approach minimizes data transfer overhead and allows parallel processing to proceed with reduced communication delays.
Solution Approach 2:
The patent replaces traditional CPU-mediated data transfer with RDMA technology, which allows direct memory-to-memory transfers between accelerators over the network. This substitution eliminates the mechanical bottleneck of CPU involvement in data transfer, significantly reducing transfer time and enabling faster inter-accelerator communication.
3Productivity
If processing stages are distributed across multiple accelerators, then the processing capacity increases, but the system complexity increases
Solution Approach 1:
The patent employs a universal interface and communication protocol (RDMA) that works across all computational accelerators in the distributed system. This universality allows different accelerators to be interconnected through a standardized mechanism, reducing the complexity of managing heterogeneous devices. The same NIC and RDMA protocol can handle data transfer between any pair of accelerators, simplifying the overall system architecture.
Data Source
AI summary
A method for processing includes receiving a definition of a processing pipeline including multiple sequential processing stages. The processing pipeline is partitioned into a plurality of partitions. The first partition of the processing pipeline is executed on a first computational accelerator, whereby the first computational accelerator writes output data from a final stage of the first partition to an output buffer in a first memory. The output data are copied over a packet communication network to an input buffer in a second memory. The second partition of the processing pipeline is executed on a second computational accelerator using the copied output data in the second memory as input data to a first stage of the second partition.