Method and systems of dynamic sparse training of diagonally sparse networks with permutations

Abstract

An exemplary system and method for training neural networks under extreme or heavy sparsity constraints by (i) selecting and updating only the most critical diagonals during training via a diagonal-based sparse pattern, while additionally maintaining diagonal sparsity even during backpropagation, and / or (ii) using learned permutations to restore network expressivity while maintaining hardware efficiency, e.g., after a sparsity constraint having been applied, diagonal or otherwise. A structured sparse-to-sparse DST method and corresponding system are provided that enforce a diagonal sparsity pattern throughout the training process that preserves / maintains sparse computation in the forward and backward passes. Permutations can help bridge the gap between the performance of structured and unstructured sparsity in being implemented as co-learning for structure and permutations. Structure (e.g., diagonal or others) may be first applied in a training process, to which a second process based on permutations is applied.

Images