DNN Accelerator Weight Layout Rearrangement
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
DNN accelerators face challenges in energy efficiency due to high computation demands and suboptimal weight data transfer, particularly in heterogeneous computation architectures where sparsity-aware and dense weight layouts lead to performance bottlenecks and increased inference latency.
Innovation Solution
A DMA engine is implemented to rearrange weight layouts for convolutional operations, converting a 3D weight tensor into a linear layout and writing it into a PE array, allowing for optimized data transfer and reducing the need for compilation-time weight layout rearrangement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If weight layout rearrangement is performed at compilation time, then optimal weight layout for specific XPU is achieved, but inference latency increases
Solution Approach 1:
The system performs weight layout rearrangement in advance during model loading or idle periods, preparing optimized weight layouts for different XPU types before inference begins. This preliminary action ensures that when inference is executed, the weights are already in the optimal layout for the target XPU, eliminating latency during actual inference operations.
Solution Approach 2:
The system creates multiple copies of weight tensors in different layouts (dense and sparse) and stores them in memory. When inference is needed, the appropriate pre-prepared copy is selected based on the target XPU type, avoiding the need to perform layout rearrangement during inference and thus minimizing latency.
2Productivity
If separate weight copies are created for each XPU type, then optimal performance for each XPU is achieved, but memory usage and data transfer overhead increase
Solution Approach 1:
The system stores weight tensors in a unified format in main memory and performs layout rearrangement locally in the memory buffer of each XPU when needed. This approach maintains a single master copy of weights while allowing each XPU to have its own optimized view, reducing overall memory usage while still providing XPU-specific optimization.
Solution Approach 2:
The system introduces a layout conversion buffer or intermediate storage area that can hold weights in different layouts. This intermediary space allows efficient conversion between dense and sparse layouts without requiring separate full copies for each XPU type, reducing memory overhead while maintaining performance benefits.
3Adaptability or versatility
If dense weight layout is used, then compatibility across different XPUs is improved, but sparsity-aware XPUs achieve suboptimal performance
Solution Approach 1:
The system dynamically selects and switches between dense and sparse weight layouts based on the type of XPU being used for inference. When a sparsity-aware XPU is detected, the system automatically uses the sparse layout; when a dense XPU is used, it switches to the dense layout. This dynamic adaptation maintains compatibility across different XPU types while optimizing performance for each specific type.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An DNN accelerator includes a DMA engine that can rearrange weight data layout. The DMA engine may read a weight tensor from a memory (e.g., DRAM). The weight tensor includes weights arranged in a 3D matrix. The DMA engine may partition the weight tensor into a plurality of virtual banks based on a structure of a PE array, e.g., based on the number of activated PE columns in the PE array. Then the DMA engine may partition a virtual bank into a plurality of virtual sub-banks. The DMA engine may also identify data blocks from different ones of the plurality of virtual sub-banks. A data block may include a plurality of input channels and may have a predetermined spatial size and storage size. The DMA engine form a linear data structure by interleaving the data blocks. The DMA engine can write the linear data structure into another memory (e.g., SRAM).