FPGA Neural Processing Unit With Multi-Function Pipeline Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network deployments using general-purpose CPUs or GPUs are expensive and suffer from high latency, particularly in performing complex tasks like language translation and speech recognition.
Innovation Solution
A system utilizing Field Programmable Gate Arrays (FPGAs) with a neural functional unit that performs matrix-vector multiplications and vector operations in parallel, leveraging on-chip memories to reduce latency and increase throughput by preloading neural network models directly into FPGA resources, allowing for distributed processing across multiple nodes without relying on CPU resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general purpose CPUs or GPUs are used for neural network deployment, then the system can perform complex tasks, but the deployment cost is expensive and latency is high
Solution Approach 1:
The patent segments the neural network processing into distinct functional units (matrix-vector multiplication units, vector addition units, activation function units) that are spatially distributed and operate in parallel. Each unit handles specific operations independently, eliminating sequential bottlenecks present in CPU/GPU architectures.
Solution Approach 2:
The patent transitions from the time-sequential processing of CPUs/GPUs to a spatially-parallel architecture where multiple processing units operate simultaneously across different dimensions of the computational graph. This dimensional shift enables concurrent execution of multiple neural network operations.
2Ease of operation
If neural network models are loaded from main memory, then the system can access model parameters, but the data transfer time increases latency
Solution Approach 1:
The patent pre-loads neural network model parameters into on-chip block memories before inference begins. This preliminary action ensures that all model weights and biases are already resident in fast on-chip storage, eliminating runtime data transfer delays from main memory during actual processing.
Solution Approach 2:
The patent introduces on-chip block memories as intermediary storage between main memory and processing units. These block memories act as buffers that hold frequently accessed model parameters, reducing the need for repeated main memory accesses and thereby reducing latency.
3Productivity
If distributed processing is implemented across multiple nodes, then the system can scale processing capacity, but the system complexity increases
Solution Approach 1:
The patent designs universal processing tiles that can perform multiple neural network operations (matrix-vector multiplication, vector addition, activation functions) within a single hardware unit. This multi-functionality reduces the number of different component types needed in distributed systems, simplifying inter-node communication and coordination.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processors and methods for neural network processing are provided. A method in a processor including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the MVU, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes decoding instructions including a first type of instruction for processing by only the MVU and a second type of instruction for processing by only one of the multifunction units. The method includes mapping a first instruction for processing by the matrix vector unit or to any one of the first multifunction unit, the second multifunction unit, or the third multifunction unit depending on whether the first instruction is the first type of instruction or the second type of instruction.