Dynamic Sparse Tensor Permutation for Dense Accelerator Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks with dynamic sparsity patterns, where sparsity is discovered at runtime, are inefficiently processed on commodity accelerators designed for dense computations, leading to poor performance in applications like natural language processing and image classification.
Innovation Solution
A compiling framework that transforms dynamic sparse tensors into dense tensors using data permutation and computation tiling, allowing for efficient execution on dense computation devices by decoupling sparse and dense computation stages, leveraging optimized dense computation kernels and hardware instructions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If dynamic sparse computation is performed on dense computation devices, then computation efficiency is improved, but device compatibility and performance are worsened
Solution Approach 1:
The patent introduces a sparse-to-dense transformation layer as an intermediary between sparse computation and dense hardware. This transformation converts sparse tensors into dense tensors with compacted non-zero elements, enabling dense computation devices to process sparse data efficiently without requiring specialized hardware modifications.
Solution Approach 2:
The patent changes the data representation parameter from sparse format to dense format through permutation operations. By transforming the sparsity pattern into a compacted dense representation, the system leverages optimized dense computation kernels while maintaining the benefits of sparse processing.
2Power
If sparse data is processed using traditional dense computation methods, then hardware utilization is improved, but computation performance is worsened
Solution Approach 1:
The patent performs preliminary compacting of sparse data into dense format before computation. This pre-processing step reorganizes non-zero elements to create compacted dense representations that can be efficiently processed by dense computation kernels, maximizing hardware utilization while maintaining performance.
Solution Approach 2:
The patent segments the computation process into distinct stages: sparse-to-dense transformation, compacted dense computation, and result reconstruction. This segmentation allows each stage to be optimized independently, with the transformation stage preparing data for efficient dense processing.
3Adaptability or versatility
If dynamic sparsity patterns are discovered at runtime, then adaptability is improved, but computation overhead is worsened
Solution Approach 1:
The patent implements dynamic sparsity handling by allowing the sparse-to-dense transformation to adapt to runtime sparsity patterns. The system dynamically discovers and compacts sparsity patterns during execution, maintaining adaptability while minimizing overhead through efficient permutation operations.
Data Source
AI summary
Embodiments of the present disclosure include techniques processing dynamically sparse neural networks as dense computations. A permutation is performed to translate an input tensor from a sparse format into a dense format. Once in a dense format, dense computation can be performed to generate output data that is also in the dense format. A reverse permutation may then be performed to translate the output data back into the sparse format. An analysis of the operator is performed prior to runtime to determine the one or more dimensions of the tensor expression associated with the operator that are permutation invariant. The permutation may permutate the input tensor across dimensions that are permutation invariant.