DMA Weight Layout Transposition for DNN Accelerator Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
DNNs require significant computation and memory resources due to their high computing demands and large weight data, leading to inefficiencies in existing weight storage formats that are not optimized for heterogeneous processing units, resulting in performance degradation.
Innovation Solution
A DMA engine transposes and compresses weight memory layouts from a KCF format to a KFC format, optimizing storage for DNN accelerators by removing sparse weights and aligning with the preferred formats of different processing units, reducing the need for multiple copies and software reformating.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If weights are stored in KCF format for CPU/GPU compatibility, then software reformating is simplified, but DNN accelerator performance degrades due to non-optimal memory access patterns
Solution Approach 1:
The patent segments the weight storage system into multiple format copies (KCF for CPU/GPU, KFC for DNN accelerator) and introduces a format conversion mechanism that selectively provides appropriate formats to different processing units. This segmentation allows each processor type to access weights in its optimal format without compromising others.
Solution Approach 2:
The patent introduces a format conversion layer (software or hardware) that acts as an intermediary between the unified KCF storage and various processing units. This intermediary converts weights to the appropriate format (KCF or KFC) based on the target processor, eliminating the need for all processors to directly access suboptimal formats.
2Productivity
If multiple copies of weights in different formats are maintained, then processing efficiency for different units is optimized, but storage requirements increase
Solution Approach 1:
The patent extracts only the essential format variations needed for different processors. Instead of maintaining all possible format combinations, it identifies and maintains only the critical formats (KCF and KFC), reducing storage overhead while still optimizing for both CPU/GPU and DNN accelerator workloads.
Solution Approach 2:
The patent implements dynamic format conversion where weights are converted from the primary KCF format to KFC format on-demand or in advance based on the processing unit's needs. This dynamic approach allows a single primary storage to serve multiple format requirements, reducing the need for permanent multiple copies.
3Adaptability or versatility
If software reformating is used to convert weight formats, then storage flexibility is maintained, but compute resources are wasted due to reformating overhead
Solution Approach 1:
The patent performs format conversion in advance during model loading or preprocessing stages, rather than during actual inference computations. By converting weights to the appropriate format before processing begins, it eliminates runtime conversion overhead and allows the DNN accelerator to operate directly on optimally-formatted weights without additional compute resource consumption during inference.
Data Source
AI summary
A compute block includes a DMA engine that reads data from an external memory and write the data into a local memory of the compute block. An MAC array in the compute block may use the data to perform convolutions. The external memory may store weights of one or more filters in a memory layout that comprises a sequence of sections for each filter. Each section may correspond to a channel of the filter and may store all the weights in the channel. The DMA engine may convert the memory layout to a different memory layout, which includes a sequence of new sections for each filter. Each new section may include a weight vector that includes a sequence of weights, each of which is from a different channel. The DMA engine may also compress the weights, e.g., by removing zero valued weights, before the conversion of the memory layout.