NPU NDMA Core Memory Bandwidth Optimization via Transaction Merging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep convolutional neural networks (DCNNs) face significant memory bandwidth utilization challenges due to the processing of large weight and activation tensors, leading to inefficient resource utilization and energy consumption in neural processing units (NPUs).
Innovation Solution
A neural processing unit (NPU) with a direct memory access (NDMA) core that transparently combines transaction requests for data stripes, optimizing memory bandwidth by streaming data blocks between the NPU and external memory, reducing the number of transactions and increasing throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If input feature padding is used to preserve input feature size during computations, then the input feature size is preserved, but memory bandwidth utilization increases
Solution Approach 1:
The patent extracts and processes only the necessary non-padded portions of feature maps, separating them from the padded regions. By using region of interest (ROI) masking and selective processing, the system processes only valid data areas, eliminating the need to process padded values while preserving the original feature size for subsequent layers.
Solution Approach 2:
The feature maps are segmented into valid regions and padded regions. The patent applies ROI masking to divide the feature map into meaningful segments, processing only the valid data portions through the computational graph while excluding padded areas, thereby reducing memory bandwidth requirements.
2Quantity of substance
If the number of NDMA transaction requests is increased to process large weight and activation tensors, then more data can be processed, but resource utilization becomes inefficient
Solution Approach 1:
The patent merges multiple small NDMA transaction requests into fewer, larger transactions by buffering data in local memory and coalescing access patterns. This consolidation reduces the total number of transactions to external memory while maintaining the processing of large tensors, thereby improving resource utilization.
Solution Approach 2:
Data is pre-loaded and buffered in local NPU memory before processing. By anticipating data access patterns and pre-fetching tensors into fast local memory, the system reduces the frequency of external memory transactions, improving resource utilization while handling large data volumes.
3Quantity of substance
If the number of NDMA transaction requests is increased to process large weight and activation tensors, then more data can be processed, but energy consumption increases
Solution Approach 1:
Multiple NDMA transactions are merged into consolidated transfer operations, reducing the total number of memory access events. This merging strategy decreases the energy overhead associated with transaction management and memory interface operations while maintaining comprehensive data processing.
Solution Approach 2:
The system maintains continuous data flow through buffered processing, eliminating idle cycles and repeated memory access setup overhead. By keeping data in local memory and processing it continuously without frequent external memory round-trips, energy consumption is reduced while processing large tensors.
Data Source
AI summary
A neural processing unit (NPU) is described. The NPU includes an NPU direct memory access (NDMA) core. The NDMA core includes a read engine having a read buffer. The NDMA core also includes a write engine having a write buffer. The NPU also includes a controller. The controller is configured to direct the NDMA core to perform hardware memory bandwidth optimization for reading/writing NDMA data in the read buffer and/or NDMA data in the write buffer. The NDMA core is also configured to transparently combine NDMA transaction requests for a data stripe to increase local access to available tensors in artificial neural networks.