Neural Network Model Pinning in On-Chip Memory for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks are expensive to deploy using general-purpose CPUs or GPUs, and while GPUs provide increased throughput, they suffer from poor latency.
Innovation Solution
A method for evaluating neural network models using a system with interconnected nodes, each equipped with on-chip memory blocks and compute units, where the neural network model is partitioned into separate layers and loaded into on-chip memory blocks for processing, maintaining the model until service interruption or modification, leveraging distributed programmable hardware nodes like FPGAs for efficient processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural network models are deployed using general-purpose CPUs, then deployment flexibility is maintained, but processing cost and time consumption increase significantly
Solution Approach 1:
The system segments the neural network model into separate layers and distributes them across multiple FPGAs. Each FPGA processes specific layers in parallel, dividing the computational workload to achieve faster training times while maintaining deployment flexibility through programmable hardware nodes.
Solution Approach 2:
The patent replaces general-purpose CPU/GPU systems with specialized FPGA-based hardware accelerators. This substitution provides dedicated neural network processing capabilities that significantly reduce training time and improve processing speed compared to general-purpose processors.
2Productivity
If GPUs are used to increase throughput, then processing capacity improves, but latency performance deteriorates
Solution Approach 1:
The system assigns specific neural network layers to specific FPGAs, creating localized processing units optimized for their designated tasks. This local specialization allows each FPGA to process its assigned layers with minimal latency while the collective system achieves high throughput through parallel operation of multiple specialized units.
Solution Approach 2:
The patent implements on-chip memory blocks within each FPGA that pre-store weight matrices and intermediate results. This preliminary storage eliminates repeated memory access delays during processing, reducing latency while maintaining high throughput through efficient data availability for parallel compute units.
3Speed
If model coefficients are maintained in on-chip memory blocks, then processing latency is reduced, but memory resource utilization efficiency decreases
Solution Approach 1:
The system divides the neural network model coefficients into segments that are distributed across multiple FPGAs' on-chip memory blocks. Each FPGA stores only the coefficients needed for its assigned layers, reducing the memory burden on individual devices while maintaining low-latency access. The segmentation enables parallel processing without requiring any single FPGA to hold the entire model in memory.
Data Source
AI summary
Systems and methods for neural network processing are provided. A method in a system comprising a plurality of nodes interconnected via a network, where each node includes a plurality of on-chip memory blocks and a plurality of compute units, is provided. The method includes upon service activation receiving an N by M matrix of coefficients corresponding to the neural network model. The method includes loading the coefficients corresponding to the neural network model into the plurality of the on-chip memory blocks for processing by the plurality of compute units. The method includes regardless of a utilization of the plurality of the on-chip memory blocks as part of an evaluation of the neural network model, maintaining the coefficients corresponding to the neural network model in the plurality of the on-chip memory blocks until the service is interrupted or the neural network model is modified or replaced.