Neural Processing Unit Batch Memory Layout for Weight Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing neural processing units (NPUs) face challenges in reducing the number of main memory read operations and minimizing power consumption during the inference operations of artificial neural networks, particularly in processing multiple batch channels.
Innovation Solution
The proposed method involves optimizing the storage and calculation order of parameters in an artificial neural network by utilizing on-chip memory or NPU internal memory, allowing for the reuse of kernel weights across multiple batch channels, thereby reducing the frequency of main memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by stationary object
If kernel weights are stored in main memory and accessed for each batch channel, then memory capacity is sufficient, but the number of read operations increases and power consumption increases
Solution Approach 1:
The patent segments the memory system into multiple levels: main memory for storing the complete kernel weight dataset, on-chip memory for storing subsets of kernel weights, and NPU internal memory for storing currently active weights. This segmentation allows the system to balance between memory capacity and access efficiency, reducing power consumption by keeping frequently accessed weights in faster, lower-power memory segments.
Solution Approach 2:
The system performs preliminary action by pre-loading kernel weights from main memory into on-chip memory and NPU internal memory before processing batch channels. This advance preparation reduces the need for frequent main memory access during inference, thereby lowering power consumption while maintaining sufficient storage capacity through the hierarchical structure.
2Speed
If kernel weights are loaded from main memory for each batch channel, then data accuracy is maintained, but processing speed decreases due to latency
Solution Approach 1:
The patent extracts subsets of kernel weights from main memory and places them in on-chip memory and NPU internal memory. This extraction allows the system to maintain data accuracy by preserving the complete dataset in main memory while enabling faster processing by having readily available copies in faster memory, thus improving speed without sacrificing data integrity.
Solution Approach 2:
The memory system implements a nested structure where NPU internal memory is nested within on-chip memory, which is nested within the broader main memory system. This nested doll approach allows weights to exist at multiple levels simultaneously, with the complete dataset nested in main memory and progressively smaller subsets nested in faster memories, enabling both high speed and adequate capacity.
3Loss of time
If on-chip memory is used to store batch channels and kernel weights, then access speed increases, but memory capacity is limited
Solution Approach 1:
The patent implements dynamic memory management where the allocation of on-chip memory and NPU internal memory is adjusted based on the processing requirements of different batch channels. This dynamic approach allows the system to optimize access speed for active computations while maintaining the ability to store complete datasets in main memory, effectively balancing access time and capacity through adaptive resource allocation.
Data Source
AI summary
A method performs a plurality of operations on an artificial neural network (ANN). The plurality of operations includes storing in at least one memory a set of weights, at least a portion of a first batch channel of a plurality of batch channels, and at least a portion of a second batch channel of the plurality of batch channels; and calculating the at least a portion of the first batch channel and the at least a portion of the second batch channel by the set of weights. A batch mode, configured to process a plurality of input channels, can determine the operation sequence in which the on-chip memory and/or internal memory stores and computes the parameters of the ANN. Even if the number of input channels increases, processing may be performed with one neural processing unit including a memory configured in consideration of a plurality of input channels.