Chained Neural Network Instructions Without Global Register Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network processing is expensive when deployed using general-purpose CPUs or GPUs, and GPUs have poor latency despite providing increased throughput.
Innovation Solution
A hardware node with a global register and a pipeline that includes a matrix vector unit, multifunction units, and a chain architecture to process instructions efficiently, allowing for parallel execution of matrix operations and vector functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If general purpose CPUs are used for neural network processing, then deployment flexibility is maintained, but processing cost and time increase significantly
Solution Approach 1:
The system segments the neural network processing pipeline into distinct functional units: matrix vector units for parallel matrix operations, multifunction units for vector operations and activation functions, and dedicated pipelines for each operation type. This segmentation enables simultaneous execution of multiple operations that would otherwise be sequential on a general-purpose CPU, dramatically improving training throughput while reducing overall training time.
Solution Approach 2:
The patent replaces the mechanical sequential execution model of general-purpose CPUs with a specialized hardware architecture that uses parallel processing pipelines. The matrix vector units and multifunction units operate concurrently through dedicated data paths, substituting the von Neumann architecture's sequential bottlenecks with parallel hardware acceleration, thereby increasing productivity without proportionally increasing time loss.
2Productivity
If GPUs are used for neural network processing, then throughput is increased, but latency remains poor
Solution Approach 1:
The system pre-loads weight matrices and activation functions into dedicated on-chip memory buffers within the matrix vector units and multifunction units. This preliminary action eliminates the need for repeated memory accesses during processing, reducing latency for each inference operation while maintaining high throughput through parallel pipeline execution. The pre-configured hardware units can immediately process incoming data without waiting for memory fetches.
Solution Approach 2:
The patent introduces dedicated on-chip memory buffers as intermediaries between the input data and the processing units. These buffers hold weight matrices and intermediate results locally, acting as mediators that eliminate the high-latency memory access bottleneck present in GPU architectures. The intermediaries enable fast data exchange between processing stages without requiring external memory access, thereby reducing overall processing latency.
3Ease of operation
If results are stored in global register after each operation, then data accessibility is improved, but memory access time increases
Solution Approach 1:
The architecture implements a nested memory hierarchy where local registers within each functional unit provide immediate data access, embedded within the unit itself. This nested structure allows processing units to access frequently used data (weights, intermediate results) in their local registers without accessing the global register file, thereby maintaining ease of operation for common operations while avoiding the time penalty of global memory access. The global register serves as a higher-level cache for less frequently accessed data.
Data Source
AI summary
Hardware and methods for neural network processing are provided. A method in a hardware node including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the matrix vector unit, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes performing using the MVU a first type of instruction that can only be performed by the MVU to generate a first result. The method further includes performing a second type of instruction that can only be performed by one of the multifunction units and generating a second result and without storing the any of the two results in a global register, passing the second result to the second multifunction and the third multifunction unit.