Neural Network Activation Function Segmentation for Speed and Accuracy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network models face a trade-off between calculation speed and learning performance, as using complex activation functions improves learning performance but decreases calculation speed, and simple activation functions improve calculation speed but decrease learning performance.
Innovation Solution
An electronic apparatus generates a new activation function by combining two different activation functions, one with high calculation speed and one with high learning performance, using a gradient transformation function to reflect the features of the high-learning-performance activation function during backpropagation, thereby improving both calculation speed and learning performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a complex activation function is used to improve learning performance, then learning performance is improved, but calculation speed decreases
Solution Approach 1:
The patent segments the activation function into two distinct parts: a first activation function used during the forward propagation process that prioritizes calculation speed, and a second activation function used during the backward propagation process that prioritizes learning performance. This segmentation allows each part to be optimized independently for its specific function, resolving the contradiction between speed and performance.
Solution Approach 2:
The patent implements a dynamic selection mechanism where different activation functions are applied at different stages of the neural network training process. The system dynamically switches between the first activation function (for speed during forward pass) and the second activation function (for performance during backward pass), allowing the system to adapt its behavior based on the operational context.
2Speed
If a simple activation function is used to improve calculation speed, then calculation speed is improved, but learning performance decreases
Solution Approach 1:
The patent segments the activation function into two distinct parts: a first activation function used during the forward propagation process that prioritizes calculation speed, and a second activation function used during the backward propagation process that prioritizes learning performance. This segmentation allows each part to be optimized independently for its specific function, resolving the contradiction between speed and performance.
Solution Approach 2:
The patent implements a dynamic selection mechanism where different activation functions are applied at different stages of the neural network training process. The system dynamically switches between the first activation function (for speed during forward pass) and the second activation function (for performance during backward pass), allowing the system to adapt its behavior based on the operational context.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
An electronic apparatus is provided. The apparatus includes a communicator, a memory configured to store a neural network model, an activation function generating module that generates activation function information used in the neural network model, first activation function information, second activation function information, and third activation function information generated based on the first activation function information and the second activation function information, and a processor configured to receive fourth activation function information from an external device through the communicator, based on type information on the fourth activation function information corresponding to the second activation function information, input the first activation function information and the fourth activation function information to the activation function generating module to generate fifth activation function information, and control the neural network model to change the activation function information used in the neural network model from the third activation function information to the fifth activation function information.