Neural Network Training on Low-Power Controllers Using Genetic Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training neural networks on low-power controllers are inefficient due to high computational demands and power consumption, especially when dealing with large data sets and complex neural networks, and genetic algorithms and backpropagation techniques are slow and resource-intensive.
Innovation Solution
A combination of genetic algorithms and selective backpropagation techniques is used to generate and train neural networks on low-power hardware, optimizing memory usage and power consumption by filtering data, processing in batches, and selectively updating connection weights based on error contribution, while avoiding convergence issues.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If genetic algorithms or backpropagation are used to train neural networks on low-power controllers, then model accuracy can be achieved, but training time and power consumption increase significantly
Solution Approach 1:
The patent segments the training process into two distinct phases: (1) a genetic algorithm phase that performs coarse optimization of network architecture and weights, and (2) a backpropagation phase that performs fine-tuning. This segmentation allows each method to operate in its optimal regime, with the genetic algorithm handling large-scale search and backpropagation handling precise local optimization, thereby reducing total training time while maintaining accuracy.
Solution Approach 2:
The patent applies preliminary action by using the genetic algorithm to pre-optimize the neural network architecture and weights before applying backpropagation. This preliminary optimization creates a better starting point for backpropagation, reducing the number of iterations needed to achieve convergence and thereby reducing overall training time and power consumption.
2Measurement precision
If genetic algorithms or backpropagation are used to train neural networks on low-power controllers, then model accuracy can be achieved, but power consumption becomes prohibitive
Solution Approach 1:
The patent segments the training process into two distinct phases: (1) a genetic algorithm phase that performs coarse optimization of network architecture and weights, and (2) a backpropagation phase that performs fine-tuning. This segmentation allows each method to operate in its optimal regime, with the genetic algorithm handling large-scale search and backpropagation handling precise local optimization, thereby reducing total training time and power consumption.
Solution Approach 2:
The patent applies partial action by using backpropagation for only a limited number of iterations after the genetic algorithm phase, rather than performing exhaustive training. This partial application of backpropagation is sufficient to achieve acceptable accuracy while significantly reducing power consumption compared to using backpropagation alone.
3Measurement precision
If all input data is used for training, then model accuracy improves, but memory usage exceeds low-power controller capabilities
Solution Approach 1:
The patent extracts only the essential features and representative samples from the full input data set rather than processing all raw data. This extraction is achieved through the genetic algorithm's ability to identify important patterns and through selective sampling strategies, thereby reducing memory requirements while preserving the information necessary for accurate model training.
Solution Approach 2:
The patent processes only a subset of the available data during training, using the genetic algorithm to identify and focus on the most informative samples. This partial processing approach reduces memory usage while maintaining model accuracy by concentrating computational resources on the most critical data points.
4Measurement precision
If backpropagation is performed on all neural network connections, then model accuracy improves, but computational complexity and power consumption increase
Solution Approach 1:
The patent extracts and identifies the most critical connections and neurons that contribute most to the model's performance. During the backpropagation phase, computational resources are focused on optimizing these identified critical elements rather than uniformly processing all connections. This selective approach reduces computational complexity while maintaining model accuracy.
Solution Approach 2:
The patent applies local quality by differentiating the treatment of different neural network connections during backpropagation. Instead of applying uniform optimization to all connections, the method applies stronger optimization to connections that have greater impact on model performance and weaker or no optimization to less critical connections. This localized approach reduces overall computational complexity while preserving accuracy.
Data Source
AI summary
A method includes selecting a subset of models from a plurality of models. The plurality of models is generated based on a genetic algorithm and corresponds to a first epoch of the genetic algorithm. Each of the plurality of models includes data representative of a neural network. The method includes performing at least one genetic operation of the genetic algorithm with respect to at least one model of the subset to generate a trainable model. The method includes determining a rate of improvement associated with prior backpropagation iterations. The method includes selecting, based on the rate of improvement, one of the trainable model or a prior trainable model as a selected trainable model. The method includes generating the trained model including training the selected trainable model. The method includes adding the trained model as input to a second epoch of the genetic algorithm that is subsequent to the first epoch.