NPU Multi-Domain Memory Allocation for Faster AI Data Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional AI-dedicated ASICs face challenges with limited memory capacity, inefficient data storage, and increased power consumption due to frequent data transmission between the ASIC and main memory, leading to reduced processing speed and productivity.
Innovation Solution
A neural processing unit (NPU) with a multi-domain memory system that includes separate domains for feature maps and weights, allowing simultaneous data access and optimized memory allocation based on the artificial neural network model, reducing redundant data communication and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If separate memory domains for feature maps and weights are implemented, then data access efficiency is improved, but memory complexity increases
Solution Approach 1:
The memory system is divided into separate domains: a first memory domain for storing feature maps and a second memory domain for storing weights. This segmentation allows simultaneous independent access to different data types, improving data access efficiency for neural network operations while organizing complexity into manageable functional units.
Solution Approach 2:
The memory controller is designed to manage multiple memory domains and perform various functions including reading feature maps, reading weights, writing output feature maps, and allocating memory resources dynamically. This multi-functionality allows a single controller to handle diverse memory operations, improving efficiency while containing complexity within a unified control structure.
2Quantity of substance
If internal memory capacity is increased, then data storage capability is improved, but manufacturing cost increases
Solution Approach 1:
The memory allocation is made dynamic through the memory controller, which can adjust the capacity分配 between the first and second memory domains based on the specific neural network model being executed. This dynamic allocation allows the system to adapt memory capacity to actual needs rather than providing fixed over-provisioned memory, reducing manufacturing costs while maintaining sufficient storage capability.
Solution Approach 2:
The system changes the operational parameters of the memory subsystem by configuring different capacity allocations for feature map storage and weight storage based on the computational requirements of different neural network layers and operations, optimizing the balance between storage capacity and manufacturing cost.
3Use of energy by moving object
If data transmission between ASIC and main memory is reduced, then power consumption is reduced, but processing speed may be affected
Solution Approach 1:
The system performs preliminary actions by pre-loading and caching frequently accessed data such as weights and feature maps into the internal memory domains before they are needed for computation. This preliminary caching reduces the need for repeated data transmission between the ASIC and main memory during processing, lowering power consumption while ensuring data is readily available to maintain processing speed.
Solution Approach 2:
The internal memory system acts as an intermediary buffer between the main memory and the computational units. By storing working data locally in the first and second memory domains, the system reduces direct transmission traffic between the ASIC and main memory, thereby reducing power consumption while maintaining fast access speeds through the intermediate memory layers.
Data Source
AI summary
A neural processing unit includes an internal memory including a plurality of memory units; a controller configured to control read and write operations of data in at least one of an input feature map domain, a weight domain, and an output feature map domain with respect to each of the plurality of memory units based on an operation schedule in a machine code in which a plurality of operation steps of an artificial neural network model are set.