Tensor Compression Circuits for On-Chip Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale neural networks face computational inefficiencies due to limited on-chip memory, leading to memory spilling and reduced performance during convolution operations, as existing lossless compression methods do not effectively increase cache size or memory bandwidth and can result in less-than-ideal random access patterns.
Innovation Solution
The implementation of a lossy tensor compression and decompression circuit arrangement that determines the minimum and maximum values of tensor elements, allowing for efficient compression and decompression by representing each element as a function of these values and the number of bits available, thereby increasing on-chip memory capacity and bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If lossless compression is used to store tensor data in on-chip memory, then memory bandwidth is improved, but cache size does not effectively increase and random access patterns become less ideal
Solution Approach 1:
The patent changes the precision parameter of tensor data from full precision (e.g., 32-bit floating point) to lower precision (e.g., 8-bit integers) through quantization. This parameter change enables significant compression ratio improvement while maintaining acceptable accuracy for neural network inference, effectively increasing cache capacity without proportionally increasing cache structure complexity
Solution Approach 2:
The patent segments tensor data into multiple precision levels, storing frequently accessed data in high-speed on-chip memory at lower precision, while maintaining full precision copies in off-chip memory when needed. This segmentation allows the system to optimize for both speed and accuracy in different scenarios
2Productivity
If more on-chip memory is allocated to store large tensors and weights, then computational efficiency is improved, but the limited on-chip memory capacity causes spilling to off-chip memory
Solution Approach 1:
The patent applies lossy compression by changing the precision parameter of stored tensor data from high precision to low precision, achieving up to 4x compression ratios. This effectively multiplies the available on-chip memory capacity without physically expanding the memory hardware, allowing more data to reside in fast on-chip memory simultaneously
Solution Approach 2:
The compressed memory system serves multiple functions: it acts as both a compression engine and an extended cache memory, providing both space expansion and speed improvement simultaneously by keeping more data in on-chip memory
3Quantity of substance
If lossy compression is applied to tensor data, then on-chip memory capacity is effectively increased, but some precision information is lost
Solution Approach 1:
The patent systematically changes precision parameters from high to low through controlled quantization, mapping high-precision floating-point values to low-precision integer representations. This parameter transformation achieves compression while maintaining accuracy within acceptable bounds for neural network inference tasks
Solution Approach 2:
The patent applies partial precision reduction, keeping full precision for critical values (such as activation outputs and gradient computations) while applying lossy compression to less critical data (such as intermediate activations and weight gradients), optimizing the balance between compression ratio and accuracy
Data Source
AI summary
Lossy tensor compression and decompression circuits compress and decompress tensor elements based on the values of neighboring tensor elements. The lossy compression circuit scales each decompressed tensor element of a tile by a scaling factor that is based on the maximum value that can be represented by the number of bits used to represent a compressed tensor element, and the greatest value and least value of the tensor elements of the tile. The lossy decompression circuit performs the inverse of the lossy compression. The compression circuit and decompression circuit have parallel multiplier circuits and parallel adder circuits to perform the lossy compression and lossy decompression, respectively.