Matrix-Vector Multiply Tiles for Low-Latency Neural Network Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network deployments using general-purpose CPUs or GPUs are expensive and suffer from high latency, particularly due to inefficient use of resources and poor latency performance even with GPUs.
Innovation Solution
A system utilizing Field Programmable Gate Arrays (FPGAs) with a distributed architecture, where neural network models are pinned to on-chip memories, allowing for parallel processing and low-latency operations by exploiting unconventional data representations and leveraging hundreds to thousands of interconnected nodes for parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general-purpose CPUs are used for neural network deployment, then versatility is maintained, but productivity and latency are poor
Solution Approach 1:
The system segments the neural network processing into multiple independent processing tiles (e.g., 16x16 or 32x32 matrix-vector multiply tiles) that can operate in parallel. Each tile handles a specific portion of the computation, enabling the system to achieve high throughput while maintaining the ability to process different neural network models by reconfiguring the tiles.
2Productivity
If GPUs are used for neural network deployment, then throughput is increased, but latency remains poor
Solution Approach 1:
The system pre-loads neural network weights and parameters into on-chip block RAM memory before processing begins. This preliminary action eliminates the need for repeated memory accesses during computation, significantly reducing latency while maintaining high throughput through parallel processing.
3Loss of time
If FPGAs are used for neural network processing, then latency is reduced through parallelism, but device complexity increases
Solution Approach 1:
The processing tiles are designed with universal functionality to handle various neural network operations (matrix-vector multiplication, convolution, fully connected layers) using the same hardware structure. This multi-functionality reduces device complexity by avoiding the need for specialized hardware for each operation type, while still achieving low latency through parallel execution.
4Speed
If neural network models are pinned to on-chip memories, then speed is improved, but memory capacity requirements increase
Solution Approach 1:
The system processes neural network data in smaller dimensional chunks using multiple processing tiles that operate in parallel. Instead of loading entire large matrices into memory at once, the 16x16 or 32x32 tile structure divides the computation into smaller blocks, reducing the memory capacity requirement while maintaining high processing speed through parallelism.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Hardware and methods for neural network processing are provided. A method in a system comprising a plurality of nodes, where each node comprises a plurality of tiles, is provided. The method includes receiving an N by M matrix of coefficients configured to control a neural network model. The method includes storing a first row and a second row of the N by M matrix of coefficients in a first and a second on-chip memories incorporated within a first and a second of the plurality of tiles. The method includes processing the first row of the coefficients and a first set of input vectors using a first compute unit incorporated within the first of the plurality of tiles. The method includes processing the second row of the coefficients and a second set of input vectors using a second compute unit incorporated within the second of the plurality of tiles.