Unstructured Pruning for MLPs with Tanh Activation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks face challenges in scalability and practicality due to their high computational demands and memory footprints, particularly in resource-constrained environments like IoT and edge computing.
Innovation Solution
A method for optimizing trained neural network models by identifying and modifying neurons based on their activation values using a hyperbolic tangent (tanh) activation function, which helps in pruning unnecessary neurons and weights, thereby reducing model complexity without significantly affecting accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If deep neural networks are used to improve accuracy and performance, then predictive accuracy is improved, but computational demands and memory footprints increase
Solution Approach 1:
The patent extracts and removes unnecessary neurons from the neural network based on activation value analysis. By identifying neurons with consistently low or saturated activation values across training and validation datasets, the method prunes these neurons to reduce model complexity while preserving predictive accuracy on resource-constrained devices.
Solution Approach 2:
The patent changes the parameter of neuron activation by applying threshold-based pruning criteria. Neurons are evaluated based on their activation values, and those falling below certain thresholds or exhibiting saturation patterns are removed. This parameter-based selection enables systematic reduction of computational demands while maintaining model performance.
2Speed
If deep neural networks are deployed on edge devices, then real-time processing capability is improved, but energy consumption increases
Solution Approach 1:
The patent removes computationally expensive neurons that contribute minimally to the output, thereby reducing the overall computational load. This extraction of unnecessary processing elements directly decreases energy consumption while preserving the real-time processing capability needed for edge device deployment.
3Device complexity
If pruning is applied to reduce model size, then device complexity is reduced, but model performance may degrade
Solution Approach 1:
The patent employs feedback mechanisms by evaluating neuron activation patterns across both training and validation datasets. This feedback loop ensures that pruning decisions are based on actual performance impact, allowing the method to identify and remove only those neurons that do not significantly contribute to model performance, thus preventing degradation.
Solution Approach 2:
The patent performs preliminary analysis of neuron activation values before pruning. By pre-identifying neurons with consistently low or saturated activations across multiple datasets, the method prepares a targeted pruning strategy that minimizes the risk of removing performance-critical neurons, thereby preserving model performance while reducing size.
4Quantity of substance
If quantization is applied to reduce precision, then memory footprint is reduced, but numerical precision decreases
Solution Approach 1:
The patent extracts and removes neurons that require high precision but contribute minimally to the overall output. By eliminating these neurons, the method reduces the total memory footprint without requiring quantization of the remaining neurons, thereby preserving numerical precision while achieving memory efficiency.
Data Source
AI summary
Systems and methods for optimizing a trained neural network model are provided herein. A trained neural network model is obtained. The trained neural network model comprises a plurality of neurons. A plurality of output information of the hidden layers is received by performing a simulation of the trained neural network using a dataset. A plurality of mean activation values corresponding to neurons of the plurality of hidden layers is calculated, based on using a hyperbolic tangent (tanh) activation function. A subset of mean values corresponding to a plurality of ranges is calculated. A plurality of neurons and a plurality of associated layers corresponding to the subset of mean values is modified. An optimized neural network model that implements the modified plurality of neurons and the modified plurality of associated layers is generated.