Ring All-Reduce Partitioning Across Reconfigurable Dataflow Processors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Optimizing compute graphs for the configurable units of coarse-grained reconfigurable architectures (CGRAs) remains a challenge, particularly in efficiently executing massively parallel computations required for machine learning and artificial intelligence tasks.
Innovation Solution
A method involving multiple interconnected reconfigurable dataflow processors (RDPs) with a ring network is used to shard and distribute tensor slices across RDPs, inserting nodes for partial and total reduction operations, optimizing runtime performance by managing inter-chip latency and parallel processing of large neural network models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If tensor data is distributed across multiple RDPs in a ring network, then parallel processing capability and runtime performance are improved, but inter-chip communication latency increases
Solution Approach 1:
The compute graph is partitioned into subgraphs that are distributed across multiple RDPs in the ring network. Each RDP processes a specific portion of the computation independently, enabling parallel processing. The ring network topology segments the communication paths to minimize latency by allowing adjacent RDPs to exchange data efficiently.
Solution Approach 2:
The system introduces a ring network topology dimension to connect multiple RDPs, transforming a single-chip computation into a distributed multi-chip system. This dimensional expansion allows data to flow through multiple paths and enables parallel processing across spatially distributed units while managing communication latency through optimized routing.
2Speed
If compute graphs are partitioned across multiple RDPs, then execution speed for large models is improved, but system complexity increases
Solution Approach 1:
The compute graph is divided into manageable subgraphs that can be independently assigned to different RDPs. This segmentation allows complex models to be processed in smaller units, reducing the complexity burden on any single device while maintaining high execution speed through parallel processing.
Solution Approach 2:
The ring network acts as an intermediary structure that simplifies the coordination between multiple RDPs. It provides standardized communication paths and data flow management, reducing the overall system complexity despite the distributed architecture.
Data Source
AI summary
A method for a reconfigurable computing system includes receiving a compute graph for execution on multiple RDPs interconnected with a ring network having R interconnected RDPs. A compute graph with a node specifying a reduction operation for a first and second tensor is detected. The detected compute graph node is partitioned into a compute subgraph corresponding to an RDP of the R interconnected RDPs. A first node is inserted into the compute subgraph that specifies a partial reduction operation for producing a partial reduction result corresponding to a shard of the first tensor and a shard of the second tensor. A second node is inserted for communicating the partial reduction result to an adjacent RDP. A third node is inserted that specifies a reduction operation for producing a total reduction result. A fourth node is inserted for communicating the total reduction result to at least one other RDP.