Sparse Tensor Storage Compression for Neural Network Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network accelerators inefficiently store and process sparse tensors, leading to increased memory usage, power consumption, and computational delays due to uncompressed dense formats and inefficient tensor rotations.
Innovation Solution
Implementing sparse tensor storage techniques that leverage sparsity by compressing tensor data before transfer and storage, rotating tensors within the accelerator, and replicating only required border regions to compute units, using local memory and efficient data handling mechanisms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If sparse tensor data is stored in uncompressed dense format, then storage and processing are simpler, but memory footprint increases and power consumption increases
Solution Approach 1:
The patent extracts and stores only the non-zero elements of sparse tensors, along with their position information, rather than storing the entire dense tensor structure. This selective extraction reduces memory footprint while maintaining the ability to reconstruct and process the original tensor data.
Solution Approach 2:
The patent applies different storage strategies to different parts of the tensor data based on sparsity characteristics. Dense regions are handled differently from sparse regions, with specialized data structures that adapt to the local density patterns, optimizing both memory usage and access efficiency.
2Speed
If sparse tensor data is stored in uncompressed dense format, then data access is faster, but power consumption increases
Solution Approach 1:
By extracting and storing only non-zero elements with their position metadata, the system reduces the total data volume that needs to be transferred and processed, thereby lowering power consumption while maintaining efficient access speeds through optimized data structures.
Solution Approach 2:
The patent performs preliminary compression and organization of sparse tensor data before transfer to compute units, pre-calculating and storing position information. This preliminary action reduces the computational burden and power consumption during actual processing operations.
3Productivity
If full tensor data is transferred to compute units, then computation can be performed, but data transfer volume increases
Solution Approach 1:
The system extracts and transfers only the essential non-zero elements and their position information to compute units, rather than transferring complete dense tensor data. This selective transfer maintains computation capability while dramatically reducing data transfer volume.
Solution Approach 2:
The patent segments tensor data into sparse and dense portions, transferring only the sparse non-zero elements to compute units. This segmentation allows compute units to process only the necessary data portions, reducing overall transfer volume while maintaining full computation capability.
4Productivity
If border regions are replicated to all compute units, then computation can proceed without waiting, but memory usage increases
Solution Approach 1:
The patent applies border region replication selectively based on local sparsity patterns and computational requirements of different compute units. Only the necessary border regions required by each compute unit are replicated, optimizing the balance between computation throughput and memory usage.
Solution Approach 2:
Instead of replicating complete border regions to all compute units, the system replicates only the partial border data that is actually needed by each compute unit for its specific computation tasks, reducing overall memory usage while maintaining computation continuity.
Data Source
AI summary
Methods, apparatus, systems and articles of manufacture are disclosed for sparse tensor storage for neural network accelerators. An example apparatus includes sparsity map generating circuitry to generate a sparsity map corresponding to a tensor, the sparsity map to indicate whether a data point of the tensor is zero, static storage controlling circuitry to divide the tensor into one or more storage elements, and a compressor to perform a first compression of the one or more storage elements to generate one or more compressed storage elements, the first compression to remove zero points of the one or more storage elements based on the sparsity map and perform a second compression of the one or more compressed storage elements, the second compression to store the one or more compressed storage elements contiguously in memory.