Neural Network Layer Grouping for On-Chip Memory Pre-Fetch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require significant memory bandwidth for reading and writing data and weights, which can be inefficient due to repeated access to off-chip memory, leading to high latency and power consumption.
Innovation Solution
The method involves grouping neural network layers into layer groups and tile groups, with pre-fetching input data into on-chip memory to reduce off-chip memory access and optimizing data processing by subdividing input data into blocks and using techniques like Y-splits, X-splits, and F-splits for efficient memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is read from off-chip memory multiple times, then the neural network can be executed, but memory bandwidth usage increases and power consumption increases
Solution Approach 1:
The patent applies preliminary action by pre-fetching input data from off-chip memory into on-chip memory buffers before the neural network execution requires it. This allows the data to be readily available during computation, avoiding repeated reads from off-chip memory and reducing power consumption while maintaining execution productivity
2Productivity
If data is read from off-chip memory multiple times, then the neural network can be executed, but memory bandwidth usage increases
Solution Approach 1:
The patent pre-fetches input data into on-chip memory buffers before execution, so that during the neural network computation, data is read from the faster on-chip memory rather than repeatedly accessing off-chip memory. This reduces the quantity of memory bandwidth consumed while maintaining execution productivity
3Productivity
If data is read from off-chip memory multiple times, then the neural network can be executed, but latency increases
Solution Approach 1:
The patent pre-fetches input data into on-chip memory buffers before the neural network execution requires it. Since on-chip memory access is significantly faster than off-chip memory access, this preliminary action reduces the latency experienced during data retrieval while maintaining execution productivity
4Productivity
If input data is subdivided into blocks and pre-fetched, then memory access efficiency improves, but device complexity increases
Solution Approach 1:
The patent subdivides input data into blocks and pre-fetches them into on-chip memory buffers. This segmentation allows for more efficient memory access patterns and better utilization of on-chip resources, improving memory access efficiency while the added complexity is managed through systematic data organization
Data Source
AI summary
Methods of implementing a neural network in hardware, the neural network including a plurality of layers and the layers being grouped into a plurality of layer groups, each layer group comprising one or more layers of the neural network that are processed in a single pass through the hardware. The layer groups are grouped into a plurality of tile groups, each tile group comprising a set of layer groups that are evaluated when executing the neural network. The method comprises pre-fetching a portion of the input data for a first layer group in a tile group into a buffer slot in on-chip memory; and subsequently releasing the buffer slot after output data for the first layer group has been written to memory.