Neural Network Training via Parametric Activation Function Transition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing embedded systems lack hardware support for state-of-the-art activation functions in neural networks, limiting the efficiency of neural network execution.
Innovation Solution
A training method that initializes neural networks with a second, parametrized activation function different from the hardware-supported first activation function, gradually adapting its parameters to match the hardware-supported function during a transition phase, followed by final training with the supported function.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If state-of-the-art activation functions (e.g., GELU, ELU, Swish) are used in neural networks, then network performance and accuracy are improved, but hardware support is lost on embedded systems
Solution Approach 1:
The patent applies parameter changes by transforming state-of-the-art activation functions into parametrized versions where parameters are optimized to match the characteristics of hardware-supported activation functions. This allows the network to achieve high accuracy with GELU, ELU, or Swish during training, then transition to ReLU variants for efficient embedded execution by adjusting the parametrized function's parameters to approximate the target hardware-supported function.
Solution Approach 2:
The patent introduces an intermediary approach by using a parametrized activation function as a bridge between state-of-the-art functions and hardware-supported functions. The parametrized function serves as a mediator that can be continuously adjusted during training to converge toward the hardware-supported function, enabling gradual adaptation without abrupt performance loss when transitioning to embedded deployment.
2Productivity
If ReLU variants are used for hardware support on embedded systems, then execution efficiency is improved, but network performance deteriorates compared to state-of-the-art activation functions
Solution Approach 1:
The patent applies preliminary action by performing the complex optimization of parametrized activation functions during the training phase on powerful hardware. The network is pre-trained with high-accuracy activation functions, and the parametrized versions are pre-optimized to approximate these functions. This preliminary work enables the network to achieve high accuracy during training, and the pre-optimized parameters enable efficient execution on embedded hardware without requiring complex runtime adjustments.
3Adaptability or versatility
If multiple activation functions are supported on embedded hardware, then versatility is improved, but hardware complexity increases
Solution Approach 1:
The patent applies universality by designing a single parametrized activation function implementation that can approximate multiple different activation functions through parameter adjustment. Instead of implementing separate hardware support for GELU, ELU, Swish, and other state-of-the-art functions, the system uses one universal parametrized function that can be configured to match the characteristics of any target activation function, thereby supporting multiple functions with a single hardware implementation.
Data Source
AI summary
The disclosure relates to training an artificial neural network, including providing a neural network for training, wherein after training the neural network is operated using a first activation function. An initial training of the neural network is executed based on a second activation function which is different from the first activation function and which is a parametrized activation function including at least one parameter to be changed which is chosen such that the second activation function is equal or essentially equal to the first activation function. The initial training is performed using a start parameter setup being chosen such that the second activation function differs from the first activation function. In a transition phase, further training steps are executed in which the at least one parameter is changed such that the second activation function approaches the first activation function. A final training step is executed using the first activation function.