Sparse Tensor Storage Layout for Neural Network Accelerator Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network accelerators face inefficiencies due to the lack of consideration for sparsity in tensor storage, leading to increased memory usage, power consumption, and processing delays, especially when tensors need to be rotated and transferred between compute units.
Innovation Solution
Implementing sparse tensor storage by compressing tensor data based on sparsity, rotating tensors within the accelerator, and replicating only required border regions, allowing for efficient storage and transfer of data in a format optimized for neural network computations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If dense tensor storage is used without considering sparsity, then memory allocation is simplified, but memory usage and power consumption increase
Solution Approach 1:
The patent extracts and stores only the non-zero elements of sparse tensors along with their coordinate information, separating useful data from redundant zero elements. This extraction approach reduces memory usage while maintaining the ability to perform computations on sparse tensors efficiently.
Solution Approach 2:
The patent applies different storage strategies to different regions of tensor data based on sparsity patterns. By identifying and storing only non-zero elements in specific coordinate positions, the system optimizes memory usage for sparse regions while maintaining dense storage where appropriate.
2Ease of operation
If dense tensor storage is used without considering sparsity, then data transfer between compute units is simplified, but power consumption increases
Solution Approach 1:
The patent extracts and transfers only non-zero tensor elements between compute units, reducing the volume of data that needs to be transmitted. This extraction of essential information maintains computational accuracy while reducing power consumption during data transfer operations.
3Shape
If full tensor rotation is performed, then tensor orientation is correctly adjusted, but processing delays increase
Solution Approach 1:
The patent extracts and rotates only the non-zero elements of sparse tensors rather than performing full tensor rotation. By identifying non-zero elements and rotating only those with their coordinate transformations, the system achieves correct tensor orientation while significantly reducing processing time.
Solution Approach 2:
The patent segments the tensor rotation operation into identifying non-zero elements, extracting them, rotating their coordinates, and reassembling the rotated sparse tensor. This segmentation allows the system to avoid processing zero elements, reducing overall rotation time while maintaining correctness.
4Productivity
If border regions are replicated for all compute units, then parallel processing is facilitated, but memory usage increases
Solution Approach 1:
The patent applies border region replication selectively based on sparsity patterns and computational requirements of different compute units. By analyzing which compute units actually need border data from neighboring regions, the system replicates only necessary border elements, maintaining parallel processing capability while reducing redundant memory usage.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, apparatus, systems and articles of manufacture are disclosed for sparse tensor storage for neural network accelerators. An example apparatus includes sparsity map generating circuitry to generate a sparsity map corresponding to a tensor, the sparsity map to indicate whether a data point of the tensor is zero, static storage controlling circuitry to divide the tensor into one or more storage elements, and a compressor to perform a first compression of the one or more storage elements to generate one or more compressed storage elements, the first compression to remove zero points of the one or more storage elements based on the sparsity map and perform a second compression of the one or more compressed storage elements, the second compression to store the one or more compressed storage elements contiguously in memory.