Diverse Activation Neural Networks for Higher Model Expressiveness
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks lack diversity in activation functions within each layer, limiting their expressiveness and accuracy in classification and regression tasks.
Innovation Solution
Implementing learned diversity neural networks (LDNNs) with varying activation functions within each layer, trained using meta-learning to optimize inner and outer network parameters, allowing sub-networks to evolve diverse activation functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If identical neurons are used in each network layer, then the network structure is simple and easy to implement, but the network expressiveness and accuracy are limited
Solution Approach 1:
The patent applies local quality by introducing diversity in activation functions at different locations (layers) of the neural network. Each layer can have different activation functions selected from a set of candidate functions, allowing each layer to be optimized for its specific computational role rather than using a uniform activation function throughout the entire network.
Solution Approach 2:
The patent implements dynamics by making the activation functions learnable and adaptable during training. The network dynamically selects and adjusts activation functions from a set of candidates based on the training data and task requirements, allowing the network structure to evolve and optimize its expressiveness over time.
2Ease of manufacture
If identical neurons are used in each network layer, then the training process is simple, but the network accuracy in classification and regression tasks is limited
Solution Approach 1:
The patent applies parameter changes by varying the activation function parameters across different layers and positions in the network. Instead of using a fixed identical activation function everywhere, the system changes the activation function parameters to optimize performance for different computational tasks and data characteristics, thereby improving accuracy while maintaining manageable training complexity through systematic parameter variation.
3Adaptability or versatility
If diverse activation functions are introduced within each layer, then network expressiveness and accuracy are enhanced, but the device complexity increases
Solution Approach 1:
The patent applies segmentation by dividing the neural network into layers, where each layer can independently select from a set of candidate activation functions. This modular approach allows diversity to be introduced in a controlled manner at different segments of the network, managing complexity through systematic layer-by-layer configuration rather than attempting uniform diversity throughout the entire network.
Solution Approach 2:
The patent implements universality by providing a universal set of candidate activation functions that can be applied across different layers and positions in the network. This multi-functional approach allows the same set of candidate functions to serve multiple purposes across different computational tasks, reducing the need for completely different activation functions for each layer and thereby managing overall system complexity.
Data Source
AI summary
Various examples are provided related to diversity based deep learning. In one example, a learned diversity neural network (LDNN) system includes an input layer; an output layer; and at least one hidden layer including at least one activation function neuronal network. The at least one activation function neuronal network includes an input node, an output node, and a plurality of intermediate nodes coupled between the input and output nodes and isolated from other nodes or other activation function neuronal networks of the at least one hidden layer.