Neural Network Training with Activation Transition for Embedded AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks using state-of-the-art activation functions cannot be efficiently executed on embedded systems due to limited hardware support, as most embedded chips only support ReLU variants, leading to inferior performance.
Innovation Solution
A training method that initially uses a second activation function unsupported by the target hardware, gradually transitioning to a first activation function supported by the hardware through a time-variant combination, ensuring efficient execution on embedded systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If state-of-the-art activation functions (e.g., GELU, ELU, Swish, Mish) are used for training neural networks, then training performance and model accuracy are improved, but the neural network cannot be efficiently executed on embedded systems due to lack of hardware support
Solution Approach 1:
The patent applies preliminary action by performing the activation function transformation during the training phase. The neural network is initially trained with state-of-the-art activation functions to achieve optimal performance, then the activation functions are transformed to hardware-supported variants before deployment. This preliminary transformation ensures both high training performance and efficient embedded execution without requiring hardware changes.
2Adaptability or versatility
If ReLU variants are used for neural network execution on embedded systems, then hardware compatibility is improved, but training performance and model accuracy deteriorate compared to state-of-the-art activation functions
Solution Approach 1:
The patent applies parameter changes by transforming the activation function parameters during the training process. Specifically, it transforms state-of-the-art activation functions (with superior performance characteristics) into equivalent ReLU-variant forms that maintain the essential functional behavior while becoming compatible with embedded hardware. This parameter transformation allows the network to achieve both high performance and hardware compatibility.
3Productivity
If multiple activation functions are supported by hardware, then neural network performance on embedded systems is improved, but device complexity increases
Solution Approach 1:
The patent applies the taking out principle by extracting the performance benefits of state-of-the-art activation functions during training, then separating these from the actual hardware execution requirements. The transformation process extracts the essential functional characteristics needed for high performance while removing the requirement for complex hardware support of multiple activation functions. This allows embedded systems to maintain simple hardware architectures while still achieving high execution efficiency through software-based activation function transformation.
Data Source
AI summary
The present disclosure relates to a method for training an artificial neural network, the method including providing a neural network to be trained, wherein after training the neural network is to be operated based on a first activation function. An initial training of the neural network is executed based on the at least a second activation function, the at least one second activation function being different to the first activation function. In a transition phase, further training steps are executed using a combination of the first activation function and the at least one second activation function. The combination of training functions is changed over time such that an overweighting of the second activation function at the beginning of the transition phase changes towards an overweighting of first activation function at the end of the transition phase. A final training step is based on the first activation function.