Neural Network Instruction Chaining for Low-Latency Hardware Pipelines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks are expensive to deploy using general-purpose CPUs or GPUs, and GPUs provide increased throughput but suffer from poor latency in performing complex tasks like reading comprehension, language translation, and speech recognition.
Innovation Solution
A hardware node with a global register and pipeline, including a matrix vector unit, multifunction units, and a configuration that allows for efficient processing of instructions by distributing neural network operations across multiple nodes, such as FPGAs, to reduce latency and improve throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If general-purpose CPUs are used to deploy neural networks, then ease of operation is maintained, but productivity is poor and training times are expensive
Solution Approach 1:
The patent replaces general-purpose CPU instruction execution with a dedicated neural network instruction set architecture (ISA) that includes specialized instructions for matrix-vector multiplication, activation functions, and other neural network operations. This substitution of the processing mechanism enables high-throughput neural network processing while maintaining ease of operation through a standardized instruction interface.
Solution Approach 2:
The patent changes the operational parameters of the processing system by introducing a new instruction set with specific opcodes for neural network operations, such as MVUL (matrix-vector multiply-accumulate), VUL (vector multiply-accumulate), and activation function instructions. These parameter changes in the instruction format and execution semantics enable optimized neural network processing.
2Productivity
If GPUs are used to increase throughput, then productivity is improved, but latency is poor
Solution Approach 1:
The patent segments the neural network processing pipeline into distinct functional units that can operate in parallel: matrix-vector multiplication units, activation function units, and output processing units. This segmentation allows different stages of neural network computation to proceed simultaneously, reducing overall latency while maintaining high throughput through parallel execution of multiple operations.
Solution Approach 2:
The patent implements preliminary action by pre-computing and caching activation function results, and by preparing data in optimal formats before processing. The instruction set includes instructions that can pre-load data into specialized registers and pre-compute common operations, reducing the time required during actual inference or training execution.
3Productivity
If distributed hardware nodes are used to reduce latency and improve throughput, then productivity is improved, but device complexity increases
Solution Approach 1:
The patent creates a universal hardware node design that can execute any neural network operation through a standardized instruction set. Each hardware node contains a programmable control unit that can interpret and execute the same set of neural network instructions, making the hardware versatile and adaptable to different neural network architectures and operations without requiring custom hardware design for each application.
Solution Approach 2:
The patent introduces an intermediary instruction set architecture layer that sits between the high-level neural network algorithms and the underlying hardware implementation. This ISA acts as a mediator, translating diverse neural network operations into a standardized set of hardware instructions that can be executed efficiently on the distributed hardware nodes, simplifying the overall system architecture.
Data Source
AI summary
Hardware and methods for neural network processing are provided. A method in a hardware node including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the matrix vector unit, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes performing using the MVU a first type of instruction that can only be performed by the MVU to generate a first result. The method further includes performing a second type of instruction that can only be performed by one of the multifunction units and generating a second result and without storing the any of the two results in a global register, passing the second result to the second multifunction and the third multifunction unit.