Neural Network Activation Function Insertion for Pruning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Hidden layers in neural networks without activation functions cannot be efficiently pruned, limiting the effectiveness of model size reduction techniques like pruning.
Innovation Solution
A learning apparatus and method that perform adaptation processing by inserting, replacing, or relocating activation functions in neural networks to facilitate pruning, allowing for the reduction of parameters in hidden layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If pruning is applied to hidden layers without activation functions, then model size reduction is achieved, but inference accuracy deteriorates
Solution Approach 1:
The patent applies preliminary action by inserting activation functions into hidden layers before performing pruning. This preparation step enables the subsequent pruning operation to effectively reduce model size while maintaining accuracy, as the activation functions provide the necessary non-linearity for accurate parameter selection during pruning.
Solution Approach 2:
The patent changes the parameter state of hidden layers by introducing activation functions where they were previously absent. This parameter change transforms the layer from an inactive state to an active state, enabling effective pruning operations that balance model size reduction with accuracy preservation.
2Productivity
If activation functions are inserted into hidden layers, then pruning effectiveness is improved, but device complexity increases
Solution Approach 1:
The insertion of activation functions is performed as a preliminary action before pruning. This preparatory step, while adding temporary structural complexity, enables significantly more effective pruning that reduces the overall model size and improves productivity, making the added complexity worthwhile.
Solution Approach 2:
The patent applies segmentation by treating the insertion of activation functions as a separate, discrete operation from the pruning process. This allows the complexity of adding activation functions to be isolated and managed separately from the main pruning operation, enabling systematic control over the overall model modification process.
Data Source
AI summary
According to one embodiment, a learning apparatus includes a processor. The processor performs, on a neural network model, an adaptation processing that includes at least either insertion of an activation function, or correction of the activation function. The processor generates a trained model by training the neural network model on which the adaptation processing has been performed. The processor performs pruning on the trained model to generate a reconstructed model from which a parameter has been reduced.