Neural Processor Tensor Block Loading for Flexible NPU Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware architectures struggle to efficiently execute diverse neural networks due to the need to support a wide range of operations and minimize external memory transactions, leading to inefficiencies in processing and resource utilization.
Innovation Solution
A processor architecture with a neural processing unit (NPU) that includes a local storage, handling unit, storage access controller, and execution sub-units, which efficiently loads and processes tensor blocks by transforming operation spaces into local sections, allowing for simplified execution and easy addition of new operations without significant hardware modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the processor supports a wide range of neural network operations, then the adaptability is improved, but the device complexity increases
Solution Approach 1:
The processor uses a universal tensor block processing architecture where the same execution units can handle different neural network operations by varying the invocation parameters. The handling unit generates invocation data that specifies different tensor blocks, dimensions, and operations, allowing a single hardware design to support diverse operations including convolution, pooling, and fully connected layers without requiring separate dedicated circuits for each operation type.
Solution Approach 2:
The system achieves operational diversity by changing parameters such as tensor dimensions, block sizes, and invocation data configurations rather than changing the hardware structure. The storage access controller and execution units operate with configurable parameters that can be adjusted through software to support different neural network architectures and operations, maintaining hardware simplicity while achieving high adaptability.
2Productivity
If the processor minimizes external memory transactions, then the productivity is improved, but the local storage requirements increase
Solution Approach 1:
The processor divides tensors into smaller blocks that can be loaded into local storage in manageable chunks. The handling unit identifies and extracts specific tensor blocks based on invocation data, loading only the necessary portions into local storage rather than loading entire tensors. This segmentation allows efficient processing with reduced local storage requirements while maintaining high productivity through minimized external memory transactions.
Solution Approach 2:
The system performs preliminary identification of required tensor blocks before execution, using the handling unit to determine which blocks need to be loaded based on the operation parameters. This preliminary action allows the storage access controller to pre-load necessary data into local storage before the execution units need it, reducing the frequency of external memory transactions during actual processing while keeping local storage usage optimized.
3Manufacturing precision
If the processor loads complete tensor blocks, then the manufacturing precision of data processing is improved, but the loss of time increases due to memory transactions
Solution Approach 1:
The processor loads exactly the required portion of tensor data (partial action) rather than complete tensors, based on the specific operation parameters in the invocation data. The handling unit calculates the precise block boundaries and dimensions needed for each operation, loading only those specific blocks into local storage. This partial loading approach maintains data processing accuracy by ensuring complete and correct tensor blocks are loaded while minimizing memory transaction time by avoiding unnecessary data transfers.
Data Source
AI summary
A processor comprising a neural processing unit is provided. The neural processing unit comprises a local storage and a handling unit configured to generate invocation data to cause loading of a block of a tensor into the local storage from a storage of the processor. The tensor has a first predetermined number of dimensions, and the block of the tensor has a size of one in one or more of the first predetermined number of dimensions such that the block consists of tensor elements arrayed in a second predetermined number of dimensions that is fewer than the first predetermined number of dimensions. A storage access controller configured to receive the invocation data and load data of the identified block into the local storage.