Differential Equations Network Neurons Learn Unique Activation Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face inefficiencies in size, storage, and processing speed due to the need for larger networks to solve complex problems, leading to increased costs and reduced performance as network size increases.
Innovation Solution
A differential equations network (DEN) where each neuron in a hidden layer learns a unique activation function, allowing for a more compact network structure with similar or greater problem-solving capabilities compared to DNNs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the network size is increased to solve more complex problems, then the problem-solving capability is improved, but the storage requirement and operating cost increase
Solution Approach 1:
Each neuron in the hidden layer is designed to learn multiple activation functions simultaneously rather than being assigned a single function. This multi-functionality allows the network to solve more complex problems with fewer neurons, reducing the overall network size and storage requirements while maintaining or improving problem-solving capability
Solution Approach 2:
The patent changes the parameter configuration by allowing neurons to learn multiple activation functions with different parameters (hyperparameters) simultaneously. This enables a compact network structure where each neuron adapts its behavior through learning multiple functions, achieving high versatility without increasing network size
2Adaptability or versatility
If the network size is increased to solve more complex problems, then the problem-solving capability is improved, but the processing speed decreases
Solution Approach 1:
By enabling each neuron to learn multiple activation functions simultaneously, the network achieves complex problem-solving capability without increasing the number of neurons or layers. This maintains a compact network structure that processes information faster compared to larger traditional networks
Solution Approach 2:
The patent merges the functionality of multiple activation functions into a single neuron's learning process. Instead of requiring separate neurons or layers for different activation functions, one neuron learns and utilizes multiple functions, reducing computational overhead and improving processing speed
3Device complexity
If each hidden layer is assigned a single activation function, then the network structure is simplified, but the number of layers and neurons must increase to learn multiple functions
Solution Approach 1:
Instead of assigning one activation function per neuron (traditional approach), the patent inverts the approach by enabling each neuron to learn and utilize multiple activation functions simultaneously. This inversion allows a single layer to achieve the functionality of multiple layers, reducing overall network complexity while maintaining versatility
Solution Approach 2:
Each neuron is designed as a universal unit capable of learning multiple activation functions rather than being specialized for a single function. This multi-functionality eliminates the need for multiple specialized layers, simplifying the network structure while enhancing the ability to learn diverse functions
Data Source
AI summary
Methods and systems are provided for a differential equations network. In one example, the differential equations network comprises one or more neuron within a single neural layer, where each of the neurons is configured to learn an activation function different or similar to an activation function learned by a different neuron within the same layer.