Ensemble Activation Function for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The selection of activation functions in neural networks for medical data inference is crucial but often arbitrary, affecting inference performance, and there is a need for a method that optimizes this selection to enhance performance.
Innovation Solution
The implementation of an ensemble activation function that combines multiple activation functions with mixing coefficients, applied across unit network structures to convert input vector elements to output vector elements, allowing for optimized inference data generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single activation function is selected by designer, then the neural network structure is simple, but the inference performance is limited
Solution Approach 1:
The patent combines multiple activation functions into a single ensemble activation function that processes inputs through parallel activation function paths. Each activation function (e.g., ReLU, sigmoid, tanh) operates on the same input, and their outputs are combined through mixing coefficients to produce the final output, thereby merging multiple functions' capabilities into one unified structure that improves inference performance while maintaining manageable complexity.
Solution Approach 2:
The ensemble activation function introduces dynamic adaptability by using learnable mixing coefficients that can be optimized during training. Instead of selecting a fixed activation function, the system dynamically adjusts the contribution of each activation function based on the input data characteristics, allowing the network to adaptively select the most suitable activation behavior for different scenarios.
2Reliability
If multiple activation functions are combined, then the inference performance improves, but the device complexity increases
Solution Approach 1:
The ensemble activation function serves multiple purposes simultaneously: it provides non-linear transformation, feature extraction, and adaptive weighting of different activation behaviors. By making the activation function structure universal and multi-functional, the patent achieves improved inference performance without proportionally increasing complexity, as the same structure handles multiple computational tasks.
Solution Approach 2:
The patent changes the parameters of the activation function system by introducing mixing coefficients as additional learnable parameters. Instead of changing the fundamental structure drastically, it modifies the parameter space to include weights that control the contribution of each activation function, allowing performance improvement through parameter optimization rather than structural complexity.
3Adaptability or versatility
If designer manually selects activation function, then the implementation is simple, but the adaptability to different purposes is limited
Solution Approach 1:
The ensemble activation function enables the neural network to self-select the optimal activation behavior through the learnable mixing coefficients. During training, the system automatically adjusts the coefficients to optimize performance for the specific task and data type, eliminating the need for manual designer selection while maintaining ease of implementation through automated parameter optimization.
Solution Approach 2:
The system incorporates feedback mechanisms through the training process, where the performance outcome feeds back into the optimization of mixing coefficients. This feedback loop allows the system to learn from its performance and automatically adapt the activation function composition to suit different purposes, achieving high adaptability without manual intervention.
Data Source
AI summary
According to one embodiment, an inference apparatus includes a processing circuit configured to: obtain processing target data; and calculate inference data by applying a trained neural network to the processing target data, wherein the trained neural network includes an ensemble activation function for each of a plurality of unit network structures configured to convert an input vector element to an output vector element, the ensemble activation function being configured to execute a calculation based on a plurality of activation functions and a plurality of mixing coefficients respectively corresponding to the activation functions.