NPU NDMA Hardware Pre-Processing for Memory Bandwidth Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks tend to shrink input features during computations, leading to increased memory bandwidth utilization due to the need for input feature padding, which further increases processing demands.
Innovation Solution
A neural processing unit (NPU) with direct memory access (NDMA) hardware pre-processing and post-processing capabilities, which includes a read engine, write engine, and controller to manage data blocks in buffers, reducing memory bandwidth pressure and optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If input feature padding is used to preserve input feature size during computations, then the original size of input features is preserved, but memory bandwidth utilization increases
Solution Approach 1:
The NDMA core performs pre-processing of input features (including padding operations) before the main computation begins. By preparing padded input features in advance during the data transfer phase rather than during computation, the system preserves input feature sizes without adding computational overhead that would increase memory bandwidth utilization during the critical computation phase.
Solution Approach 2:
The NDMA core acts as an intermediary between external memory and the NPU computation units. It handles padding and other pre-processing operations in its internal buffers, serving as a mediator that prepares data before it reaches the computation pipeline. This intermediary role allows padding to be performed without burdening the main memory bandwidth during computation.
2Manufacturing precision
If input feature padding is used to preserve input feature size during computations, then the original size of input features is preserved, but processing demands increase
Solution Approach 1:
Padding and other pre-processing operations are performed in advance by the NDMA core before data enters the main computation pipeline. This preliminary action ensures that when computation begins, all preprocessing is already complete, avoiding additional processing demands during the critical computation phase and maintaining productivity.
Solution Approach 2:
The NDMA core performs pre-processing operations autonomously as part of its data transfer and buffering functions. Rather than requiring separate processing units or additional computational resources, the NDMA core self-services by handling padding, cropping, and other pre-processing tasks within its existing data flow architecture, thus not increasing overall processing demands.
3Quantity of substance
If hardware pre-processing and post-processing are implemented in the NDMA core, then memory bandwidth pressure is reduced, but device complexity increases
Solution Approach 1:
The NDMA core is designed with multi-functionality, serving both as a data transfer mechanism and as a pre-processing/post-processing unit. By integrating padding, cropping, and other processing capabilities into the existing NDMA infrastructure rather than adding separate dedicated hardware blocks, the patent reduces memory bandwidth pressure while minimizing the increase in device complexity through versatile, multi-purpose components.
Solution Approach 2:
The patent merges pre-processing and post-processing functions with the NDMA data transfer operations. Instead of implementing separate processing pipelines, the NDMA core combines data movement and data preparation/finishing operations into a unified architecture. This merging approach reduces memory bandwidth pressure by handling processing locally while avoiding the complexity of multiple independent processing units.
Data Source
AI summary
A neural processing unit (NPU) is described. The NPU includes an NPU direct memory access (NDMA) core. The NDMA core includes a read engine having a read buffer. The NDMA core also includes a write engine having a write buffer. The NPU also includes a controller. The controller is configured to direct the NDMA core to perform hardware pre-processing of NDMA data in the read buffer and post-processing of NDMA data in the write buffer on blocks of a data stripe to process tensors in artificial neural networks.