Sparse Tensor Packing Circuit for On-Chip Memory Limits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale neural networks face performance issues due to the limited on-chip memory in neural network accelerators, leading to data spilling to off-chip memory during tensor processing, which increases latency and reduces computational efficiency.
Innovation Solution
A method and circuit arrangement that compresses non-zero data elements of sparse tensors by packing them into packed words and writing them to RAM within a compressed address range, allowing for efficient storage and retrieval of only non-zero values, thereby reducing the need for off-chip memory access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If tensors are stored in on-chip memory to increase computational efficiency, then bandwidth and latency are improved, but the limited on-chip memory capacity causes data spilling to off-chip memory
Solution Approach 1:
The patent extracts only the non-zero elements from sparse tensors for storage in on-chip memory, while excluding zero elements entirely. This selective extraction allows the system to store only the essential data needed for computation, maximizing the utilization of limited on-chip memory capacity without requiring full tensor storage.
Solution Approach 2:
The patent changes the storage format parameter from dense tensor storage to sparse tensor storage, where data is represented in a compressed format that stores only non-zero elements. This parameter change fundamentally alters how data occupies memory space, enabling efficient use of limited on-chip memory while maintaining computational efficiency.
2Quantity of substance
If data is spilled to off-chip memory when on-chip memory is full, then memory capacity is effectively increased, but performance is reduced due to increased latency
Solution Approach 1:
By extracting and storing only non-zero elements in on-chip memory, the patent ensures that the most frequently accessed data resides in fast memory. This extraction strategy minimizes the need to spill data to off-chip memory, thereby reducing memory access latency and maintaining high computational performance.
Solution Approach 2:
The patent applies partial action by storing only a portion of the tensor data (specifically, only the non-zero elements) in on-chip memory rather than the entire tensor. This partial storage approach is sufficient to maintain performance because zero elements do not contribute to computational operations, thus avoiding the latency penalty of off-chip memory access.
3Quantity of substance
If compression circuits pack non-zero data elements into packed words, then on-chip memory requirements are reduced, but device complexity increases
Solution Approach 1:
The patent merges multiple non-zero data elements into single packed words, combining several values into one storage unit. This merging operation reduces the total number of storage locations required in on-chip memory while the compression circuit handles the packing operation, balancing memory reduction against circuit complexity.
Solution Approach 2:
The patent changes the data organization parameter from one element per storage location to multiple elements per storage location through packing. This parameter change reduces the total memory capacity required by a factor equal to the number of elements packed per word, justifying the added compression circuit complexity.
Data Source
AI summary
Approaches for data compression involve a compression circuit packing non-zero data elements of a succession of words of a plurality of blocks into packed words by packing non-zero data elements of one or more words of the succession in each packed word, and restricting each packed word to data elements of one uncompressed block. The compression circuit writes each packed word in a RAM and within a compressed address range associated with the uncompressed block when the packed word is full of non-zero data elements, or before the packed word is full if the next input word is of another uncompressed block.