Neural Network Layer Grouping for Memory Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require significant memory bandwidth for reading and writing data and weights, leading to inefficiencies in hardware performance due to large memory requirements and repeated access to off-chip memory.
Innovation Solution
The method involves defining layer groups and grouping them into tile groups, with pre-fetching of input data from off-chip to on-chip memory when accessed multiple times, and assessing criteria for merging layer groups based on input pre-fetch ratios and output data storage capacity to optimize memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If input data is read from off-chip memory multiple times during neural network execution, then computational accuracy is maintained, but memory bandwidth consumption increases significantly
Solution Approach 1:
The patent implements pre-fetching of input data from off-chip memory to on-chip memory before it is needed by the neural network layers. By determining an input pre-fetch ratio and pre-loading data when the ratio exceeds a threshold, the system reduces repeated memory accesses during execution, thereby lowering memory bandwidth consumption while maintaining computational accuracy.
Solution Approach 2:
The patent introduces on-chip memory as an intermediary between off-chip memory and the neural network processing units. This intermediary buffer allows data to be cached locally, reducing the frequency of high-latency off-chip memory accesses and decreasing overall memory bandwidth requirements while preserving data availability for accurate computation.
2Loss of energy
If layer groups are merged into tile groups to reduce memory accesses, then memory bandwidth usage decreases, but on-chip memory space requirements increase
Solution Approach 1:
The patent dynamically adjusts the pre-fetch buffer size parameter based on the input pre-fetch ratio and output data characteristics. By changing the buffer size parameter adaptively rather than using a fixed large buffer, the system reduces on-chip memory space requirements while still achieving sufficient data caching to lower memory bandwidth usage.
Solution Approach 2:
The patent applies partial pre-fetching by loading only the necessary portion of input data into on-chip memory based on the calculated pre-fetch ratio, rather than loading all possible data. This partial action approach reduces the memory space required while still achieving the primary goal of reducing repeated memory accesses.
3Productivity
If pre-fetching is performed for all layer groups, then repeated memory accesses are minimized, but hardware complexity increases due to additional memory management overhead
Solution Approach 1:
The patent applies different memory management strategies to different layer groups based on their specific characteristics. By calculating the input pre-fetch ratio for each layer group and only applying pre-fetching when the ratio exceeds a threshold, the system avoids unnecessary memory management overhead for layers that do not benefit from pre-fetching, thus reducing overall hardware complexity while maintaining processing efficiency where needed.
Data Source
AI summary
A neural network is mapped to hardware by defining a plurality of layer groups, each layer group comprising one or more layers of the neural network that are processed in a single pass through the hardware. The layer groups are grouped into tile groups, each tile group comprising a set of layers groups that are evaluated when executing the neural network. Grouping the layer groups into a tile group comprises selecting a layer group that precedes a first layer group in the tile group and determining a number of times that input data to the layer group is read from memory. In response to this number exceeding a threshold, it is determined whether to merger the layer group into the tile group by determining an amount of space in on-chip memory required for storing pre-fetched input data and assessing one or more criteria relating to output data of the layer group.