Neural Network Inference Memory Allocation via Input Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant memory for storing weights and intermediate activation values, limiting the capacity for nonzero weights due to the large memory footprint of intermediate activation values, especially in convolutional neural networks where the maximum activation memory usage occurs early in the network.
Innovation Solution
The method involves dividing the input data into blocks and propagating each block separately through a portion of the network, reducing the maximum amount of intermediate activation values stored at any time, thereby allocating more memory to weight values by combining blocks of intermediate data and propagating them through the remaining network layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the network uses more nonzero weights to improve prediction accuracy, then the network becomes less sparse and more predictive, but the memory required for storing weights increases
Solution Approach 1:
The input data is divided into multiple blocks that are propagated separately through the network. This segmentation allows intermediate activation values from different blocks to be processed and stored sequentially rather than simultaneously, reducing the peak memory requirement for activation values and freeing up memory for storing more nonzero weights.
2Productivity
If the network stores more intermediate activation values to maintain processing efficiency, then the network can process data faster, but the memory available for weight storage decreases
Solution Approach 1:
By segmenting the input into blocks and processing them separately through portions of the network, the patent reduces the maximum number of intermediate activation values that need to be stored simultaneously. This maintains processing efficiency while reducing activation memory usage.
Solution Approach 2:
The patent combines blocks of intermediate data before propagating them through the remaining network layers. This preliminary combining action reduces the total number of activation values that need to be stored at any given time, as combined blocks can be processed together rather than individually.
Data Source
AI summary
Some embodiments provide a method for generating neural network program instructions for a neural network inference circuit to execute a neural network. The neural network inference circuit includes a particular amount of available memory. The method receives a specification of the neural network including multiple layers. The method determines (i) a required amount of weight memory for the neural network and (ii) required amounts of activation memory for each of a set of layers of the neural network. When the required amount of weight memory and the required amount of activation memory for at least one layer is greater than the particular amount of available memory, the method generates the program instructions for the neural network inference circuit to execute a first set of the layers of the neural network multiple times for different blocks of input data and execute a second set of the layers in a single pass.