Neural Processing Unit Weight Reuse for Faster Batch Convolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Artificial neural networks face inefficiencies in processing speed and energy consumption due to frequent memory access for weight values in convolutional neural networks, particularly in reading kernels from main memory, which is costly in terms of time and power.
Innovation Solution
A method for optimizing neural processing units by strategically storing and reusing weights in on-chip or internal memory, reducing the number of main memory read operations and maintaining weights until completion of feature map calculations across batch channels, thereby enhancing processing speed and reducing energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weights are frequently read from main memory for convolution operations, then the neural network can process multiple batch channels, but the processing speed decreases and energy consumption increases
Solution Approach 1:
The patent applies preliminary action by pre-loading weight values from main memory into on-chip memory before they are needed for convolution operations. The neural processing unit reads weight values from main memory once and stores them in on-chip memory, where they can be reused multiple times without repeated main memory access. This advance preparation eliminates the need for frequent main memory reads during processing, thereby improving processing speed and reducing energy consumption.
Solution Approach 2:
The patent introduces on-chip memory as an intermediary between main memory and the convolution computation units. This intermediate storage layer acts as a buffer that holds weight values temporarily, allowing fast access during multiple convolution operations across different batch channels. The on-chip memory mediates the data flow, preventing direct repeated access to the slower main memory and thus resolving the speed-energy tradeoff.
2Productivity
If on-chip memory is used to store weight values for reuse, then processing speed improves, but the device complexity increases
Solution Approach 1:
The patent applies universality by designing the on-chip memory to serve multiple functions: it stores weight values for convolution operations, maintains them across different batch channel processing, and supports reuse by multiple computation units. This multi-functional design justifies the added complexity by providing broad benefits across the entire neural network processing pipeline, rather than adding dedicated storage for each specific function.
Data Source
AI summary
A method performs a plurality of operations on an artificial neural network (ANN). The plurality of operations includes storing in at least one memory a set of weights, at least a portion of a first batch channel of a plurality of batch channels, and at least a portion of a second batch channel of the plurality of batch channels; and calculating the at least a portion of the first batch channel and the at least a portion of the second batch channel by the set of weights. A batch mode, configured to process a plurality of input channels, can determine the operation sequence in which the on-chip memory and/or internal memory stores and computes the parameters of the ANN. Even if the number of input channels increases, processing may be performed with one neural processing unit including a memory configured in consideration of a plurality of input channels.