Compute Graph Sharding Across Ring-Linked RDPs for Faster Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Optimizing compute graphs for coarse-grained reconfigurable processors (CGRAs) remains a challenge, particularly in efficiently executing massively parallel computations required for machine learning and artificial intelligence tasks.
Innovation Solution
A method is introduced for executing compute graphs on multiple reconfigurable dataflow processors (RDPs) interconnected via a ring network, involving sharding tensors, inserting nodes for partial and total reduction operations, and optimizing inter-chip latency to enhance runtime performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If compute graphs are executed on single RDP, then device complexity is low, but productivity is insufficient for large-scale ML workloads
Solution Approach 1:
The compute graph is partitioned into multiple subgraphs that can be executed on separate RDPs. Each RDP processes a portion of the compute graph independently, and results are aggregated through reduction operations. This segmentation enables parallel processing of large-scale ML workloads while maintaining manageable complexity at each individual RDP.
Solution Approach 2:
Multiple RDPs are interconnected via a ring network to create a distributed computing system. The system transitions from single-node execution to multi-node distributed execution, adding a network dimension to the architecture. This enables scaling of compute capacity without proportionally increasing individual device complexity.
2Productivity
If tensors are distributed across multiple RDPs, then productivity increases through parallel processing, but loss of time increases due to inter-chip communication latency
Solution Approach 1:
The system performs preliminary actions by pre-computing and distributing sharded tensor data to appropriate RDPs before execution begins. The compute graph is analyzed in advance to determine optimal data distribution, and tensors are pre-partitioned and allocated to minimize communication overhead during actual computation.
Solution Approach 2:
The ring network serves as an intermediary communication mechanism between RDPs. It facilitates efficient data exchange and coordination between distributed computing nodes, enabling optimized data flow patterns that reduce overall communication latency while maintaining parallel processing benefits.
3Productivity
If compute graph nodes are partitioned across RDPs, then productivity improves through parallel execution, but device complexity increases due to partitioning overhead
Solution Approach 1:
The system employs universal reduction operations that can be executed across multiple RDPs using standardized protocols. These multi-functional operations handle various computational patterns (sums, means, min, max) through a unified approach, simplifying the partitioning logic while enabling flexible parallel execution of different compute graph topologies.
Solution Approach 2:
The system dynamically adjusts partitioning parameters such as the number of partitions, data distribution strategy, and communication patterns based on the specific compute graph characteristics and hardware configuration. This parameter optimization enables efficient parallel execution without excessive partitioning complexity.
Data Source
AI summary
A method for a reconfigurable computing system includes receiving a compute graph for execution on multiple RDPs interconnected with a ring network having R interconnected RDPs. A compute graph with a node specifying a reduction operation for a first and second tensor is detected. Executing the compute graph on the multiple RDPs.