Neural Network Matrix Rearrangement for Sparse Tensor Core Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks are inefficient in computational performance due to high storage requirements and lack of optimization for sparse matrix operations, despite advancements in hardware for sparse neural networks.
Innovation Solution
The implementation of structured sparsity techniques, such as 2:4 structured sparsity, through permutations and pruning to rearrange data values in matrices representing neural network layers, optimizing them for faster processing by sparse tensor cores.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If structured sparsity techniques are applied to increase sparsity in neural network matrices, then computational performance is improved and storage requirements are reduced, but neural network accuracy may be lost
Solution Approach 1:
The patent applies parameter changes by systematically varying the sparsity pattern parameters (such as the 2:4 structured sparsity ratio) and pruning thresholds to optimize the balance between computational performance and accuracy retention. Different sparsity patterns are tested and selected based on their ability to maintain accuracy while improving performance.
Solution Approach 2:
The patent implements partial pruning where only specific portions of the matrix are pruned according to the structured sparsity pattern, rather than uniformly pruning all elements. This allows selective removal of less important connections while preserving critical pathways, thus maintaining accuracy while achieving sparsity benefits.
2Speed
If hardware is optimized for sparse tensor core operations, then processing speed increases, but device complexity increases
Solution Approach 1:
The patent segments the matrix operations into structured sparse patterns that can be efficiently processed by specialized hardware. By dividing the computation into regular patterns (such as 2:4 sparsity blocks), the hardware can use dedicated circuits for these specific patterns, reducing overall hardware complexity while maintaining high processing speed.
3Quantity of substance
If data values are rearranged through permutations to maximize sparsity, then storage requirements are reduced, but computational overhead increases
Solution Approach 1:
The patent performs preliminary permutation and rearrangement of data values during the training or preprocessing phase, organizing them into structured sparse patterns. This preliminary action ensures that subsequent computations can directly utilize the pre-organized sparse structure without incurring additional overhead during inference or processing phases.
Data Source
AI summary
Apparatuses, systems, and techniques to transform data sets, such as matrices representing layers of neural networks, to increase sparsity and/or other characteristics of said data sets to improve performance in computations, such as neural network computations. In at least one embodiment, one or more subsets of data in one or more sets of data are rearranged as part of a process to increase sparsity in said one or more sets of data to satisfy one or more one or more structural sparsity constraints.