Position-Dependent Memory Pipeline for Neural Network Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep Neural Networks (DNNs) deployed on general-purpose CPUs or GPUs are expensive and suffer from high latency, especially in tasks like reading comprehension, language translation, and speech recognition, due to inefficient hardware utilization.
Innovation Solution
Implementing DNNs using a distributed platform of Field Programmable Gate Arrays (FPGAs) connected via a low-latency network, where neural network models are pinned to on-chip memories for direct processing, enabling parallelization and reducing reliance on DRAM access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If DNNs are deployed on general-purpose CPUs, then flexibility and ease of deployment are improved, but processing speed and latency are worsened
Solution Approach 1:
The patent replaces general-purpose CPU processing with specialized FPGA hardware implementations. The FPGA-based neural network processor uses dedicated logic circuits, parallel compute units, and on-chip memory structures to perform neural network operations, substituting software-based mechanical processing with hardware-based parallel processing that achieves both flexibility and high speed
Solution Approach 2:
The patent changes the operational parameters by implementing neural network layers directly in hardware logic with configurable parallelism degrees. The system uses multiple compute units that can be configured to process different neural network operations simultaneously, changing the processing paradigm from sequential CPU execution to parallel hardware acceleration while maintaining deployment flexibility through FPGA reconfigurability
2Productivity
If DNNs are deployed on GPUs, then throughput is improved, but latency and memory access efficiency are worsened
Solution Approach 1:
The patent implements on-chip on-board memory structures that are nested within the FPGA device, with multiple levels of memory hierarchy (block RAM, distributed memory, and register files) organized to provide fast access to neural network weights and activations. This nested memory architecture eliminates the need for frequent external memory accesses, reducing latency while maintaining high throughput through parallel data retrieval from multiple memory levels
Solution Approach 2:
The patent introduces on-chip buffer memory structures as intermediaries between the compute units and external memory. These buffers pre-fetch and cache frequently accessed neural network data, acting as a mediator that reduces the latency of memory access while allowing the compute units to maintain high throughput by processing data from the local buffer rather than waiting for external memory operations
3Quantity of substance
If neural network models are stored in external DRAM, then memory capacity is improved, but access speed and energy efficiency are worsened
Solution Approach 1:
The patent segments the neural network model data across multiple on-chip memory blocks, with each block storing specific layers or portions of the network. This segmentation allows parallel access to different memory blocks by multiple compute units simultaneously, achieving both high capacity (by aggregating multiple blocks) and high speed (through parallel access operations), while reducing energy consumption by eliminating external memory transactions
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Processors and methods for neural network processing are provided. A method in a processor including a pipeline having a matrix vector unit (MVU), a first multifunction unit connected to receive an input from the matrix vector unit, a second multifunction unit connected to receive an output from the first multifunction unit, and a third multifunction unit connected to receive an output from the second multifunction unit is provided. The method includes decoding a chain of instructions received via an input queue, where the chain of instructions comprises a first instruction that can only be processed by the matrix vector unit and a sequence of instructions that can only be processed by a multifunction unit. The method includes processing the first instruction using the MVU and processing each of instructions in the sequence of instructions depending upon a position of the each of instructions in the sequence of instructions.