Hardware Accelerators Ring Communication Matrix Multiplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer systems, particularly heterogeneous systems, are not optimized for General Matrix to Matrix Multiplication (GEMM) operations used in neural networks, leading to inefficiencies in inferencing processes due to unsuitable matrix shapes and sizes.
Innovation Solution
A system comprising multiple hardware accelerators interconnected via an accelerator interconnect, where matrices are split into partitions and processed in parallel, utilizing a ring communication pattern to facilitate efficient matrix multiplication across accelerators, either within accelerator memory or between accelerators and host memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional BLAS implementation is used on heterogeneous systems, then general-purpose computation can be performed, but GEMM operations are not optimized for neural network shapes and sizes leading to inefficiency
Solution Approach 1:
The patent divides the GEMM computation into multiple independent tiles or blocks that can be processed in parallel across multiple hardware accelerators. Each accelerator handles a specific partition of the matrix multiplication, enabling simultaneous execution of multiple GEMM operations with different shapes and sizes, thus improving both efficiency and adaptability to neural network requirements
Solution Approach 2:
The system dynamically configures the number and size of parallel GEMM tiles based on the specific dimensions and characteristics of the input matrices. This dynamic adaptation allows the heterogeneous system to optimize performance for various neural network layer configurations without requiring fixed-structure implementations
2Speed
If matrices are stored and processed in traditional memory hierarchies, then data access is simplified, but data transfer bottlenecks occur when moving data between host memory and accelerator memory
Solution Approach 1:
The patent implements a nested memory hierarchy where accelerator memory is embedded within each hardware accelerator, and multiple accelerators are organized in a hierarchical structure with shared host memory. This nesting reduces data transfer distances and enables faster access to frequently used matrices while maintaining manageable complexity through hierarchical organization
Solution Approach 2:
The system performs preliminary data placement and pre-computation of matrix tiles before the actual GEMM operation. Matrices are pre-partitioned and loaded into accelerator memory in advance, and intermediate results are pre-computed where possible, reducing the amount of data that needs to be transferred during the critical computation phase
3Productivity
If single accelerator is used for GEMM operations, then system complexity is reduced, but computation time increases due to sequential processing
Solution Approach 1:
The GEMM computation is segmented into independent tile operations that can be executed in parallel across multiple accelerators. Each accelerator processes a specific subset of the computation without requiring complex inter-accelerator communication during the computation phase, thereby increasing throughput while keeping coordination complexity manageable
Solution Approach 2:
Multiple hardware accelerators are merged into a coordinated parallel computing system where each accelerator contributes to the overall GEMM result. The system combines the computational power of multiple accelerators while using a unified memory management and task distribution strategy to simplify coordination overhead
Data Source
AI summary
A plurality of hardware accelerators are interconnected and include a special processing unit and accelerator memory. At least one host computer is coupled to each of the plurality of hardware accelerators and includes a general processing unit and host memory. The plurality of hardware accelerators exchange data in a ring communication pattern in computing a linear layer of a neural network.