Weight Tensor Decomposition for Structurally Sparse Neural Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks (DNNs) have numerous weights that are not efficiently reduced, particularly in larger layers, leading to inefficiencies in computation and resource usage.
Innovation Solution
Decompose initial layers of a neural network into two successive layers, applying constraints to set scale values to zero, and insert activation functions, while using techniques like ADMM and probabilistic projection to enforce structural sparsity, allowing weight values to be restricted to specific sets, thereby reducing the number of weights required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks use many weights in large layers, then model accuracy is improved, but computational efficiency and resource usage deteriorate
Solution Approach 1:
The patent segments a single dense layer into multiple layers with structural sparsity, decomposing the weight tensor W into W = V × S where V contains structural sparsity patterns and S contains scale values. This segmentation allows the network to maintain accuracy while reducing computational complexity by organizing weights into a structured sparse format that enables efficient computation.
Solution Approach 2:
The patent applies local quality by introducing structural sparsity at specific locations within the network layers. By setting certain scale values to zero, entire groups of weights can be eliminated locally, creating sparse patterns that reduce computation in specific regions while preserving important weight connections elsewhere, thus improving efficiency without uniformly degrading accuracy.
2Quantity of substance
If the number of weights in neural network layers is reduced, then resource usage is improved, but model accuracy deteriorates
Solution Approach 1:
The patent changes parameters by transforming the weight representation from dense to structurally sparse format. By modifying the parameterization to W = V × S and applying constraints that drive scale values to zero, the network achieves parameter reduction while maintaining accuracy through the structured nature of the sparsity pattern and the remaining non-zero scale values that preserve important connections.
3Quantity of substance
If structural sparsity constraints are applied during training, then weight reduction is improved, but training complexity increases
Solution Approach 1:
The patent applies preliminary action by pre-defining the structural sparsity pattern through the decomposition W = V × S before training begins. The framework is prepared in advance with the factorized weight representation and constraints on scale values, so that during training the sparsity pattern emerges naturally through optimization rather than requiring complex post-processing or iterative pruning algorithms.
Data Source
AI summary
Some embodiments provide a method for improving structural sparsity of a machine-trained (MT) network. The method receives a network having multiple layers. Each layer of a set of the layers includes multiple filters of weight values. The method replaces the filters of a particular layer of the network with (i) a first set of filters of weight values, (ii) a set of scale values for the first set of filters, and (iii) a second set of filters of weight values. Each scale value corresponds to a different one of the filters of the first set of filters. The method trains the network by applying constraints to bias at least a subset of the scale values towards zero. When a particular scale value falls below a threshold value, the particular scale value is set to zero.