Neural Processing Unit Scalable Bandwidth Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional neural processing units (NPUs) face inefficiencies due to limited memory capacity, leading to insufficient storage for feature maps and weights, increased power consumption, and reduced processing speed, especially when using conventional single-domain memory structures that cannot efficiently provide data to processing elements in artificial neural networks (ANNs).
Innovation Solution
A neural processing unit with a multi-domain memory structure that allows for variable memory control and capacity allocation based on data domains for each layer of the ANN, enabling simultaneous provision of feature maps and weights, and utilizing a time-division operation among sub-memory units to increase bandwidth and reduce data transfer from main memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional single-domain memory structure is used, then device complexity is reduced, but memory capacity and data provision efficiency deteriorate
Solution Approach 1:
The memory structure is divided into multiple independent sub-memory units (first sub-memory unit, second sub-memory unit, third sub-memory unit) that can be independently controlled. Each sub-memory unit can store different types of data (input feature map, weight, output feature map) and be accessed independently, thereby increasing the effective memory capacity without proportionally increasing overall structural complexity.
Solution Approach 2:
The patent introduces a time-division dimension by enabling simultaneous read operations from multiple sub-memory units. The first sub-memory unit reads input feature map during a first time period while the second sub-memory unit reads weight during a second time period, effectively increasing data provision capacity by utilizing temporal dimension.
2Device complexity
If conventional single-domain memory structure is used, then device complexity is reduced, but processing speed deteriorates
Solution Approach 1:
The patent implements continuous data provision by overlapping read operations across multiple sub-memory units. While the first sub-memory unit provides input feature map in the first time period, the second sub-memory unit simultaneously provides weight in the second time period, ensuring continuous data supply to processing elements without idle waiting periods.
Solution Approach 2:
By utilizing time-division multiplexing across multiple sub-memory units, the system increases the rate of data provision. Multiple read operations occur in parallel across different time periods, effectively increasing the bandwidth and processing speed without requiring a more complex hierarchical memory structure.
3Ease of operation
If conventional single-domain memory structure is used, then ease of operation is maintained, but power consumption increases
Solution Approach 1:
The memory system is segmented into multiple sub-memory units that can be independently activated. The controller can selectively enable only the sub-memory units needed for current operations, reducing unnecessary power consumption. For example, only the first and second sub-memory units need to be active during convolution operations, while the third sub-memory unit can remain inactive.
Solution Approach 2:
The patent implements dynamic control of memory resources through a controller that adjusts the operational state of each sub-memory unit based on real-time requirements. The controller receives read commands and dynamically activates specific sub-memory units and their corresponding read buffers, optimizing power consumption by keeping memory units in low-power states when not in use.
Data Source
AI summary
A neural processing unit (NPU) and a method of operating the same are provided. The NPU may include an artificial intelligence (AI) calculation unit configured to process artificial neural network calculation of at least one artificial neural network model; and an internal memory including at least one memory unit configured to store data of at least one domain among first to third domain data of the at least one artificial neural network model. The at least one memory unit may include a plurality of sub-memory units configured to perform time-division operation. A bandwidth of the at least one memory unit is based on a number of the plurality of sub-memory units.