Hierarchical Sparse Tensor Compression for Lower DRAM Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI processing methods are inefficient due to the lack of effective compression techniques for sparse tensors, leading to high computation and power consumption, as well as increased DRAM bandwidth usage, as they primarily rely on single-layer compression schemes that fail to remove zero data effectively.
Innovation Solution
A layered sparse tensor compression method using a tile-based approach, where data is processed in a hierarchical cache structure, allowing for the identification and removal of zero values, reducing unnecessary data movements and storage, and only processing non-zero data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a single-layer compression scheme is used, then the device complexity is low, but the DRAM bandwidth usage is high and zero data cannot be effectively removed
Solution Approach 1:
The patent divides the compression scheme into multiple layers (first layer and second layer) with different compression ratios. The first layer uses a lower compression ratio for initial compression, while the second layer uses a higher compression ratio for further compression. This segmentation allows the system to achieve better overall compression and energy efficiency while maintaining manageable device complexity through modular design.
Solution Approach 2:
The patent introduces a hierarchical dimension to the compression scheme by organizing compression operations into multiple layers with different compression ratios. This dimensional approach allows the system to optimize for both energy efficiency and compression effectiveness, transforming a single-dimensional compression problem into a multi-dimensional solution space.
2Ease of operation
If a single-layer compression scheme is used, then the ease of operation is high, but the power consumption is high due to inability to skip zero neurons or weights
Solution Approach 1:
The compression operation is segmented into two distinct layers with different compression ratios. The first layer handles initial compression with a lower ratio, and the second layer performs further compression with a higher ratio. This segmentation enables the system to skip zero neurons or weights more effectively, reducing power consumption while maintaining operational simplicity through modular architecture.
Solution Approach 2:
The first layer compression is performed as a preliminary action before the second layer compression. By pre-compressing the data with a lower compression ratio first, the system prepares the data in a format that enables more effective second-layer compression, ultimately achieving better power efficiency while keeping the overall operation straightforward.
3Device complexity
If a single-layer compression scheme is used, then the device complexity is low, but the required buffer storage space is large
Solution Approach 1:
The buffer storage requirement is segmented across two compression layers. The first layer compresses data with a lower compression ratio, requiring certain buffer space, and the second layer further compresses the already-compressed data with a higher ratio, reducing the final buffer storage requirement. This segmented approach achieves better space efficiency while maintaining low device complexity.
Solution Approach 2:
The compression scheme implements a nested structure where the second layer compression operates on the output of the first layer compression. This nesting allows the system to achieve higher overall compression ratios, thereby reducing the required buffer storage space while keeping the device complexity manageable through hierarchical organization.
4Ease of operation
If a single-layer compression scheme is used, then the ease of operation is high, but the computation efficiency is low due to inability to save computation
Solution Approach 1:
The computation process is segmented into two layers of compression operations. The first layer performs initial compression with a lower ratio, and the second layer performs further compression with a higher ratio. This segmentation enables the system to skip zero neurons or weights more effectively, saving computation time and improving productivity while maintaining operational simplicity through modular design.
Solution Approach 2:
The first layer compression serves as a preliminary computation step that prepares data for more efficient second-layer processing. By performing initial compression first, the system reduces the data volume that needs to be processed in the second layer, thereby saving overall computation time and improving efficiency while keeping the method simple and modular.
Data Source
AI summary
A hierarchical sparse tensor compression method based on artificial intelligence devices, in DRAM, not only saves the storage space of the neuron surface, but also adds a meta-surface to the mask block. When reading data, the mask is first read, then the size of the non-zero data is calculated, and only these non-zero data are read to save DRAM bandwidth. In the cache, only non-zero data is stored, so the required storage space is reduced. When processing data, only non-zero data is used. The method uses a bit mask to determine if the data is zero. There are three levels in the hierarchical compression scheme: tiles, lines, and points, reading bitmasks and non-zero data from DRAM, and saving bandwidth by not reading zero data. When processing data, if their bit mask is zero, the tile data may be easily removed.