Neural Network Training With Activation Threshold Sparsification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high computational footprint of neural networks due to large numbers of input features, layers, and complexity makes them difficult to execute and train efficiently, particularly on hardware platforms like CPUs and GPUs.
Innovation Solution
A method and system for training neural networks by applying an activation threshold to select sparse activation matrices using a bimodal regularization function like the square Hoyer function, combined with Forced-Activation-Threshold Rectified Linear Unit (FATReLU) to increase sparsity, and performing iterative training to maintain accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of layers and complexity of each layer are increased to reach good accuracy, then the classification accuracy is improved, but the computational footprint increases making the neural network difficult to execute and train
Solution Approach 1:
The patent extracts and removes redundant weights from the neural network by applying sparsification techniques. During training, weights below a certain threshold are set to zero, effectively removing them from computation. This extraction of unnecessary elements reduces the computational footprint while preserving the network's accuracy on classification tasks.
Solution Approach 2:
The patent applies different sparsity levels to different parts of the neural network. Instead of uniformly sparsifying the entire network, it allows dense representations in critical layers and sparse representations in less critical layers. This local differentiation maintains accuracy in important regions while reducing computational burden in others.
2Productivity
If sparsification of weights is applied to reduce computational cost, then the execution efficiency is improved, but the number of non-zero elements decreases which may affect accuracy
Solution Approach 1:
The patent employs dynamic sparsification where the sparsity level is not fixed but adapts during training. The sparsity pattern evolves as the network learns, with weights being pruned iteratively based on their importance. This dynamic approach allows the network to maintain accuracy by preserving important weights while removing redundant ones, achieving a balance between execution efficiency and accuracy.
Solution Approach 2:
The patent changes the parameter of weight values by applying thresholding operations. Weights below a dynamically determined threshold are transformed to zero, while weights above the threshold are preserved. This parameter transformation reduces the number of non-zero elements (improving execution efficiency) while maintaining the essential weight structure needed for accurate classification.
3Adaptability or versatility
If the number of input features is increased to handle large data, then the model capacity is improved, but the computational cost increases making execution difficult
Solution Approach 1:
The patent extracts and removes redundant input features through sparsification. By setting many weight connections to zero, it effectively selects only the most important input features for each neuron. This extraction reduces the effective number of features processed during execution, lowering computational cost while maintaining model capacity to handle large datasets.
4Productivity
If iterative training with different activation threshold values is performed, then the sparsity of activation matrices is increased, but the training time increases
Solution Approach 1:
The patent performs preliminary sparsification by applying activation thresholds during the training process itself. Rather than training the network fully and then sparsifying it, the sparsification is integrated into the training iterations. This preliminary action allows the network to learn sparse representations from the beginning, reducing the total training time compared to post-training sparsification while achieving high sparsity in activation matrices.
Data Source
AI summary
A system and a method of training a Neural network (NN) model may include, receiving a pretrained NN model, that may include a plurality of layers, each associated with an activation matrix; selecting at least one, and performing an iterative training process on the layer. The iterative training process may include, applying an activation threshold to the activation matrix of the layer; measuring an accuracy value of the NN model; retraining the layer, while using a bimodal regularization function of one or more activation matrices of the NN model; and repeating the applying, measuring and retraining, while each repetition uses different activation threshold values. This repetition may be repeated until a maximal value of the activation threshold, where the NN model still converges, is found.