Neural Network Training With Dynamic Neuron Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional neural network training methods, such as backpropagation, face challenges in dynamically adjusting the network structure during training, leading to inefficiencies in learning data sets, particularly in determining the optimal number of neurons and layers, which can result in slow learning or failure to learn complex data sets.
Innovation Solution
The Dynamical Relational (DR) training method allows for dynamic allocation of neurons by adding new neurons to hidden layers or output layers based on the complexity of the data set, enabling learning in a single pass and providing a relational approach to rule extraction, where constraints define relationships between weights and thresholds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional backpropagation training methods are used, then the neural network can learn from data sets, but the training process is slow and may fail to learn complex data sets due to fixed network structure
Solution Approach 1:
The patent applies dynamics by enabling the neural network structure to change during training through dynamic neuron allocation. New neurons are added to hidden layers based on the complexity of the data set being learned, allowing the network to adapt its capacity in real-time. This resolves the contradiction by making the network structure dynamic rather than fixed, enabling both fast training and the ability to handle complex data sets.
Solution Approach 2:
The patent uses preliminary action by pre-allocating neurons to hidden layers before training begins. This preliminary structure provides a foundation for fast training while the dynamic addition of neurons during training ensures the network can handle complex data sets. The preliminary allocation avoids the need to start from scratch, improving training speed while maintaining adaptability.
2Device complexity
If the number of neurons and layers is fixed before training, then the network structure is simple and easy to implement, but it cannot dynamically adjust to the complexity of different data sets
Solution Approach 1:
The patent resolves this contradiction by making the network structure dynamic. While the initial structure is simple and fixed for ease of implementation, the network dynamically adds neurons to hidden layers during training based on data set complexity. This allows the structure to evolve from simple to complex as needed, maintaining ease of initial implementation while achieving adaptability to different data set complexities.
Solution Approach 2:
The patent applies preliminary action by establishing a basic fixed network structure before training begins. This preliminary structure provides simplicity and ease of implementation. During training, the system then dynamically adds neurons as needed, allowing the network to adapt to data set complexity without requiring complex pre-planning of the final structure.
3Adaptability or versatility
If neurons are dynamically added during training, then the network can adapt to data set complexity, but the training process becomes more complex and requires new allocation mechanisms
Solution Approach 1:
The patent applies dynamics by implementing dynamic neuron allocation mechanisms that automatically adjust the network structure during training. The system monitors training progress and data set complexity, then dynamically adds neurons to hidden layers as needed. This dynamic approach enables adaptability to data set complexity while managing training process complexity through automated decision-making rules for neuron allocation.
4Ease of manufacture
If traditional supervised training is used with fixed network structure, then the training algorithm is simple, but it may get stuck in local minima and fail to achieve optimal performance
Solution Approach 1:
The patent resolves this contradiction by making the training process dynamic through structural changes. While the basic training algorithm remains simple, the dynamic addition of neurons to hidden layers during training allows the network to escape local minima by changing its capacity and representation capabilities. This maintains algorithmic simplicity while improving convergence reliability through structural adaptability.
Data Source
AI summary
The present disclosure provides an artificial neural network communicatively-coupled to at least one computer having one or more processors, including a plurality of neurons arranged in layers. The artificial neural network is arranged to receive a new neuron into a layer of the artificial neural network during training; the new neuron being added to the neural network when no other neuron in that layer for a selected output can learn a relationship associated with an input vector of a data set being learnt. The new neuron is updated with both the relationship which could not be learnt by any other neuron in that layer and a modified data set from a last trained neuron in that layer that contributes to the selected output of the neural network, wherein the modified data set is formed by copying all learnt relationships from the last trained neuron into the new neuron and modifying the copied relationship based upon the relationship which could not be learnt by any other neuron in that layer; and, one or more output neurons are updated to accept input from the new neuron. Methods and computer-readable media are also disclosed.