Sparse Tensor Storage Layout for Faster Neural Network Rotation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network accelerators face inefficiencies due to the lack of consideration for sparsity in tensor data storage, leading to increased memory usage, power consumption, and processing delays, especially when tensors need to be rotated and transferred between compute units.
Innovation Solution
The implementation of sparse tensor storage methods that compress tensor data based on sparsity, allowing for efficient rotation and replication of only required data within the neural network accelerator, reducing memory usage and processing time by leveraging sparsity maps and dynamic storage elements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If dense tensor storage is used, then memory availability is maintained, but memory usage and power consumption increase
Solution Approach 1:
The patent extracts and stores only the non-zero elements of sparse tensors, removing redundant zero elements from storage. This is achieved through compressed sparse row (CSR) and compressed sparse column (CSC) formats that store only significant data elements, directly reducing memory usage while managing the complexity through structured storage organization.
Solution Approach 2:
The patent changes the storage parameter from dense format (storing all elements including zeros) to sparse format (storing only non-zero elements). This parameter change is implemented through sparsity maps that track the position and value of non-zero elements, transforming the storage approach to reduce memory consumption for sparse tensor data.
2Loss of time
If tensor rotation is performed using dense storage, then data accessibility is maintained, but processing time and power consumption increase
Solution Approach 1:
The patent performs preliminary rotation of sparse tensors before transfer between compute units, so that the tensor is already in the required orientation when needed. This preliminary action avoids the need for rotation during critical processing paths, reducing both processing time and power consumption by eliminating redundant rotation operations.
Solution Approach 2:
The patent extracts and transfers only the necessary non-zero elements during tensor rotation operations, rather than moving entire dense tensor blocks. This selective extraction of required data elements reduces the volume of data transferred and processed, directly decreasing processing time and energy consumption for rotation operations.
3Productivity
If sparsity is not considered in storage, then storage simplicity is maintained, but memory transactions and power consumption increase
Solution Approach 1:
The patent implements a universal storage system that can handle both dense and sparse tensors through a unified interface. The storage management infrastructure provides multi-functional capabilities including sparse storage (CSR/CSC formats), dense storage, rotation operations, and transfer operations, all accessible through consistent APIs that work across different tensor types and accelerator architectures.
Solution Approach 2:
The patent introduces sparsity maps as intermediary data structures that mediate between the sparse tensor data and the storage system. These maps track non-zero element positions and enable efficient access patterns, acting as an intermediary layer that simplifies the management of sparse data while improving processing efficiency through optimized memory access and reduced data movement.
Data Source
AI summary
Methods, apparatus, systems and articles of manufacture are disclosed for sparse tensor storage for neural network accelerators. An example apparatus includes sparsity map generating circuitry to generate a sparsity map corresponding to a tensor, the sparsity map to indicate whether a data point of the tensor is zero, static storage controlling circuitry to divide the tensor into one or more storage elements, and a compressor to perform a first compression of the one or more storage elements to generate one or more compressed storage elements, the first compression to remove zero points of the one or more storage elements based on the sparsity map and perform a second compression of the one or more compressed storage elements, the second compression to store the one or more compressed storage elements contiguously in memory.