Neural Network Training With Compute-Time-Aware Weight Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep neural networks (DNNs) have a large number of weights that are computationally expensive and inefficient, with existing techniques for reducing weights being inadequate, especially in larger layers.
Innovation Solution
Decompose initial layers of the network into two successive layers, apply constraints to set scale values to zero, and insert activation functions, while using techniques like ADMM and probabilistic projection to enforce structural sparsity and weight quantization, reducing the number of weight values and improving computational efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks use many weights to maintain accuracy, then model performance is improved, but computational cost and memory requirements increase
Solution Approach 1:
The patent segments a single convolutional layer into two successive layers: a first convolutional layer followed by a second convolutional layer with 1x1 kernels. This segmentation allows the network to distribute computational tasks across multiple layers, reducing the number of weights required in each individual layer while maintaining overall model accuracy through the combined effect of both layers.
Solution Approach 2:
The patent changes the kernel size parameter of the second convolutional layer to 1x1, which dramatically reduces the number of weights per filter compared to traditional larger kernels. By adjusting this parameter and introducing scale values that can be quantized to discrete levels, the network achieves significant weight reduction while preserving representational capacity through the two-layer structure.
2Device complexity
If the network layer structure is simplified to reduce weights, then device complexity is reduced, but model accuracy may deteriorate
Solution Approach 1:
The patent introduces scale values as intermediary parameters between the two convolutional layers. These scale values act as mediators that control the contribution of each layer's output to the final result, allowing the simplified two-layer structure to maintain accuracy by dynamically adjusting the weight and importance of intermediate computations through the scale parameters.
3Quantity of substance
If weight values are quantized to reduce memory storage, then memory requirements are reduced, but training complexity increases
Solution Approach 1:
The patent applies different quantization strategies to different parameters within the network. Scale values are quantized to discrete levels (e.g., powers of 2), while weight values in the convolutional layers are treated differently. This localized differentiation of quality and precision requirements allows efficient memory usage without uniformly complicating the training process for all parameters.
Data Source
AI summary
Some embodiments provide a method for training a machine-trained (MT) network. The method receives a network having multiple layers. Each layer of a set of the layers includes multiple weight values. The method trains the network by alternately (1) propagating inputs through the network to generate outputs and adjusting the weight values based on differences between the generated outputs and expected outputs and (2) identifying sets of the weight values for removal according to a set of constraints that accounts for (i) a total number of weight values and (ii) an amount of time required to execute the network on a particular type of integrated circuit.