Position-Dependent Neural Processor Pipeline for Low-Latency Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks are expensive to deploy using general-purpose CPUs or GPUs, and GPUs offer increased throughput but poor latency for complex tasks like reading comprehension and speech recognition.
Innovation Solution
A processor pipeline with a matrix vector unit and multifunction units is used to process instructions efficiently, where instructions are decoded and mapped for processing across different units based on their position in a sequence, allowing for parallel processing and low latency operations using distributed programmable hardware nodes like FPGAs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If general purpose CPUs are used to deploy neural networks, then the system is flexible and easy to deploy, but the deployment cost is high and performance is poor
Solution Approach 1:
The system segments neural network operations into distinct functional units: matrix vector units for parallel computation, multifunction units for various operations (activation functions, pooling, normalization), and dedicated memory units. This segmentation allows each unit to be optimized for its specific function while maintaining overall system flexibility through programmable control.
Solution Approach 2:
The patent replaces general-purpose mechanical computing (CPUs) with specialized hardware architecture designed specifically for neural network operations. This substitution provides both the performance benefits of dedicated hardware and the flexibility of programmable logic through FPGA implementation.
2Productivity
If GPUs are used to increase throughput, then processing throughput is improved, but latency becomes poor
Solution Approach 1:
The system pre-loads weight matrices and bias vectors into dedicated memory units before processing begins. This preliminary action allows the computation units to operate continuously without waiting for memory access during the forward propagation phase, thereby reducing latency while maintaining high throughput.
Solution Approach 2:
The patent introduces a pipelined architecture where multiple operations occur simultaneously at different stages. Data flows through multiple functional units in parallel, with each unit processing different portions of the computation at the same time, effectively adding a temporal dimension to the processing and reducing overall latency.
3Loss of time
If distributed programmable hardware nodes are used, then training time is reduced and online learning is enabled, but device complexity increases
Solution Approach 1:
The multifunction units are designed to perform multiple operations including activation functions (sigmoid, tanh, ReLU), pooling operations (max pooling, average pooling), and normalization. This multi-functionality reduces the need for separate dedicated hardware for each operation, simplifying the overall architecture while enabling diverse neural network operations.
Solution Approach 2:
The patent introduces dedicated memory units that act as intermediaries between the computation units and external storage. These memory units buffer weight matrices, bias vectors, and intermediate results, allowing computation units to operate independently and in parallel, thereby reducing training time while managing complexity through clear separation of concerns.
4Productivity
If position-dependent instruction mapping is implemented, then processing efficiency is improved, but instruction decoding complexity increases
Solution Approach 1:
The system changes the parameter of instruction encoding by incorporating position information directly into the instruction format. Each instruction includes a position field that indicates which functional unit should execute it, allowing the decoder to efficiently route instructions without complex analysis of instruction dependencies or data flow patterns.
Data Source
AI summary
Processors and methods for neural network processing are provided. A method in a processor including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the matrix vector unit, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes decoding a chain of instructions received via an input queue, where the chain of instructions comprises a first instruction that can only be processed by the matrix vector unit and a sequence of instructions that can only be processed by a multifunction unit. The method includes processing the first instruction using the MVU and processing each of instructions in the sequence of instructions depending upon a position of the each of instructions in the sequence of instructions.