Neural Network Ensembling via Genetic Algorithm Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for generating and training neural networks are slow and difficult, especially for large datasets or complex problems like regression, classification, and reinforcement learning, and often result in overfitting, requiring specialized expertise and resources.
Innovation Solution
An automated model building system using a genetic algorithm to generate and train neural networks, which aggregates multiple models through an ensembler to produce a more general output, mitigating overfitting and improving convergence speed by incorporating an ensembler as part of the genetic algorithm or using it as an input for backpropagation training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If genetic algorithms are used to generate neural networks, then model accuracy can be achieved, but the convergence process takes a large number of iterations
Solution Approach 1:
The patent applies preliminary action by using backpropagation to pre-train neural network models before they are evaluated by the genetic algorithm. This pre-training step provides a head start to the genetic algorithm, reducing the number of iterations needed to converge on an accurate solution while maintaining the ability to achieve high model accuracy through the evolutionary process.
2Quantity of substance
If backpropagation is used to train neural networks with large datasets or many nodes, then models can be generated, but the training process becomes slow
Solution Approach 1:
The patent merges two different training approaches: backpropagation and genetic algorithms. The system uses backpropagation to initially train models on large datasets, then employs genetic algorithms to evolve and optimize multiple models in parallel. This combination allows the system to handle large datasets effectively while improving training speed through the parallel evolutionary process and the ability to skip generations when convergence is achieved.
Solution Approach 2:
The patent applies dynamics by making the training process adaptive through the genetic algorithm framework. The system dynamically adjusts which models to train further based on their performance, allowing it to focus computational resources on promising models while discarding poor performers. This dynamic approach enables faster training on large datasets by avoiding the need to exhaustively train all possible model configurations.
3Measurement precision
If backpropagation is used to generate neural networks, then models can be trained, but overfitting occurs reducing generalization to other data sets
Solution Approach 1:
The patent applies feedback by using the genetic algorithm to evaluate model performance and provide feedback for further evolution. The system trains models using backpropagation, then uses the genetic algorithm to assess their generalization capability on validation data. Based on this feedback, the genetic algorithm selects, crosses over, and mutates models to improve generalization while preventing overfitting. This feedback loop ensures that models are not just memorizing training data but learning generalizable patterns.
4Measurement precision
If specialized data scientists are hired to create models, then accurate models can be developed, but the cost becomes prohibitive for many organizations
Solution Approach 1:
The patent applies self-service by creating an automated system that performs model generation, training, and optimization without requiring specialized human expertise. The genetic algorithm automatically evolves neural network architectures and parameters, and the system autonomously manages the training process including pre-training with backpropagation and final optimization. This self-service capability allows organizations to develop accurate models without hiring expensive data scientists, as the system performs all necessary tasks automatically.
Data Source
AI summary
A method includes determining, by a processor of a computing device, a subset of models included in a plurality of models generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The method includes aggregating the subset of models to generate an ensembler. The ensembler, when executed on an input, provides at least a portion of the input to each model of the subset of models to generate a plurality of intermediate outputs. An ensembler output of the ensembler is based on the plurality of intermediate outputs. The method further includes executing the ensembler on input data to determine the ensembler output.