Neural Network Instruction Chaining for Low-Latency Hardware Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network deployments using general-purpose CPUs or GPUs are expensive and suffer from high latency, particularly with GPUs providing increased throughput but poor latency performance.
Innovation Solution
A hardware node architecture featuring a control processor, global register, and pipeline with a matrix vector unit, multifunction units, and control queues, optimized for performing matrix and vector instructions, allowing for efficient processing of neural network operations by leveraging FPGAs and distributing neural network models across multiple nodes for parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general purpose GPUs are used for neural network processing, then throughput is increased, but latency becomes poor
Solution Approach 1:
The system segments neural network processing into distinct instruction types (matrix instructions for MVU, vector/scalar instructions for MFUs) and routes them through specialized functional units. This segmentation allows simultaneous processing of different operation types, reducing overall latency while maintaining high throughput through parallel execution paths.
Solution Approach 2:
The hardware node employs dynamic instruction routing where the control processor decodes incoming instructions and directs them to appropriate functional units based on instruction type. The system dynamically adapts its processing path for each instruction, enabling optimal execution speed for different operation types while maintaining pipeline efficiency.
2Adaptability or versatility
If general purpose CPUs are used for neural network processing, then deployment flexibility is maintained, but processing cost increases and performance is limited
Solution Approach 1:
The hardware node implements multifunctional units (MFUs) that can execute multiple types of operations (vector operations, scalar operations, pass-through, memory management) within a single functional unit. This multi-functionality maintains deployment flexibility while achieving specialized processing performance, as the same hardware can adapt to different neural network operation types without requiring separate dedicated hardware for each function.
Solution Approach 2:
The system uses lookup tables stored in on-chip memory that can be configured to copy and transform data according to neural network requirements. These lookup tables enable flexible deployment of different neural network models by loading appropriate weight matrices and bias vectors into the hardware node's memory, maintaining adaptability while achieving high-speed processing.
3Ease of operation
If intermediate results are stored in global register, then data accessibility is improved, but processing speed decreases due to memory access overhead
Solution Approach 1:
The system extracts frequently accessed intermediate results from global register storage and keeps them in faster on-chip memory structures within the functional units. By taking out only the necessary data from global memory and retaining it in local high-speed storage, the system maintains data accessibility for subsequent operations while eliminating the speed penalty of repeated global memory accesses.
Solution Approach 2:
The patent introduces local on-chip memory structures as intermediary storage between the functional units and global register. This intermediary layer provides fast access to intermediate results for chained operations while maintaining the ability to access global memory when needed, thus resolving the conflict between data accessibility and processing speed.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Hardware and methods for neural network processing are provided. A method in a hardware node including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the matrix vector unit, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes performing using the MVU a first type of instruction that can only be performed by the MVU to generate a first result. The method further includes performing a second type of instruction that can only be performed by one of the multifunction units and generating a second result and without storing the any of the two results in a global register, passing the second result to the second multifunction and the third multifunction unit.