Neural Network Model Pinning in On-Chip Memory for Low Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network deployments on general-purpose CPUs or GPUs are expensive and suffer from high latency, particularly due to inefficient use of resources and suboptimal processing architectures.
Innovation Solution
A system comprising nodes with on-chip memory blocks and compute units, where neural network models are partitioned and loaded into on-chip memory for processing, maintaining the model in memory until service interruption or model modification, leveraging FPGAs for parallel processing and low latency operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If neural network models are deployed on general purpose CPUs, then deployment flexibility is maintained, but processing speed and latency are poor
Solution Approach 1:
The system segments the neural network model into discrete coefficient matrices (N by M matrices where N>=8 and M>=8) that can be independently loaded into on-chip memory blocks. This segmentation allows the model to be distributed across multiple compute units while maintaining the ability to deploy on different hardware platforms through standardized matrix operations.
Solution Approach 2:
The patent introduces on-chip memory blocks as an intermediary between the neural network model coefficients and the compute units. This intermediary layer enables fast access to coefficient matrices during inference, significantly improving processing speed while the standardized memory interface maintains deployment flexibility across different FPGA configurations.
2Productivity
If neural network models are deployed on general purpose GPUs, then throughput is increased, but latency remains poor
Solution Approach 1:
The system performs preliminary action by pre-loading the entire N by M coefficient matrix into on-chip memory blocks before inference begins. This preliminary loading eliminates the need for repeated memory accesses during throughput-critical operations, reducing latency while maintaining high throughput through parallel compute unit operations.
Solution Approach 2:
The patent transitions from traditional CPU/GPU memory hierarchies to a dimension where coefficient matrices reside directly in on-chip memory at the same speed level as compute units. This dimensional change in memory architecture allows simultaneous high throughput processing across multiple compute units without the latency penalty of off-chip memory access.
3Speed
If on-chip memory blocks are maintained for neural network model coefficients, then processing speed improves, but memory resource utilization efficiency decreases
Solution Approach 1:
The system applies local quality by maintaining coefficient matrices in on-chip memory specifically at the locations where they are needed for compute unit operations. Rather than uniformly distributing data across all memory, the N by M coefficient matrices are pinned to specific on-chip memory blocks that are locally coupled to the compute units requiring them, improving speed while minimizing overall memory usage.
Solution Approach 2:
The patent changes the parameter of memory retention by maintaining coefficient matrices in on-chip memory for the duration of service activation until the model is modified or replaced. This parameter change from temporary to persistent memory retention eliminates repeated loading operations, improving processing speed while the total memory footprint remains fixed and efficient.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for neural network processing are provided. A method in a system comprising a plurality of nodes interconnected via a network, where each node includes a plurality of on-chip memory blocks and a plurality of compute units, is provided. The method includes upon service activation receiving an N by M matrix of coefficients corresponding to the neural network model. The method includes loading the coefficients corresponding to the neural network model into the plurality of the on-chip memory blocks for processing by the plurality of compute units. The method includes regardless of a utilization of the plurality of the on-chip memory blocks as part of an evaluation of the neural network model, maintaining the coefficients corresponding to the neural network model in the plurality of the on-chip memory blocks until the service is interrupted or the neural network model is modified or replaced.