Sparse Matrix Accelerator Tile Shuffling for Fanout Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network technologies face inefficiencies in storage and computation due to restricted sparsity patterns in weight tensors, leading to high computational costs and accuracy losses, as existing circuit designs struggle with irregular sparsity distributions and lack flexibility in pruning methods.
Innovation Solution
A hardware-friendly design for sparse matrix multiplication involves dividing weight tensors into tiles, shuffling, and generating bitmasks to remove zero elements, resulting in a compact tensor representation that reduces memory usage and fanout, allowing for more flexible sparsity patterns and improved computation efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If weight tensors are pruned to reduce non-zero elements, then computational cost is reduced, but accuracy loss occurs
Solution Approach 1:
The weight tensor is divided into multiple tiles, allowing selective pruning and processing of different segments. This segmentation enables the system to maintain important weights while removing less important ones, balancing accuracy and computational efficiency.
Solution Approach 2:
Different pruning strategies are applied to different tiles based on their local characteristics. The sparsity pattern is optimized locally for each tile rather than applying a uniform pruning approach, preserving local accuracy while achieving global efficiency gains.
2Productivity
If sparsity patterns are made more flexible to improve pruning efficiency, then circuit design complexity increases
Solution Approach 1:
The weight tensor is divided into multiple tiles, allowing selective pruning and processing of different segments. This segmentation enables the system to maintain important weights while removing less important ones, balancing accuracy and computational efficiency.
Solution Approach 2:
Different pruning strategies are applied to different tiles based on their local characteristics. The sparsity pattern is optimized locally for each tile rather than applying a uniform pruning approach, preserving local accuracy while achieving global efficiency gains.
3Quantity of substance
If zero elements are removed and non-zero elements are packed, then memory space is reduced, but computation complexity increases
Solution Approach 1:
A bitmask is introduced as an intermediary data structure to track the positions of non-zero elements. This bitmask acts as a mediator between the compact stored representation and the computation process, enabling efficient access to non-zero elements without requiring complex computation logic.
Solution Approach 2:
The non-zero elements are copied and packed into a compact representation, while the bitmask preserves the positional information. This copying approach allows the system to work with a smaller data set during computation while maintaining the ability to reconstruct the original tensor structure when needed.
Data Source
AI summary
This application describes an accelerator, a computer system, and a method for tensor product computation that facilitate circuit designs. The method may include: dividing a weight tensor into a matrix of tiles; shuffling the matrix of tiles in the weight tensor to obtain a shuffled weight tensor; computing a bitmask comprising a matrix of bits corresponding to elements in the shuffled weight tensor; removing the zero elements in the shuffled weight tensor and packing the non-zero elements in the shuffled weight tensor; generating a compact activation tensor based on the bitmask and an activation tensor; and performing tensor multiplication based on the compact weight tensor and the compact activation tensor. The shuffling step effectively reduces the fanout between the activations and the corresponding weights. A reduced fanout may reduce the wiring lengths and thus the energy consumption for signal transmission.