Scaled ANN Training Using Parameter Transfer Across Model Sizes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of implementing and training new artificial neural networks (ANNs) is burdensome, particularly for large ANNs with many neurons per layer, often requiring significant trial and error and extensive computing resources, and can take a long time to achieve desired accuracy.
Innovation Solution
A method and system for initializing and training ANNs by leveraging the training of an original ANN to initialize and scale a new ANN, using copied and scaled hyperparameters and learned parameters, reducing the need for extensive training and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a new ANN is created from scratch with more neurons per layer to achieve higher accuracy, then the model accuracy is improved, but the training time and computing resources required increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-training a source ANN model to convergence before using it to initialize the target model. This pre-training step performs the difficult work of finding good weight initializations and hyperparameters in advance, so that the target model can start from a proven good state rather than from random initialization, dramatically reducing the time needed to reach the same accuracy level.
Solution Approach 2:
The patent uses copying by transferring weights, biases, and hyperparameters from the source ANN model to the target ANN model. Instead of creating the target model from scratch, the system copies the learned parameters from the pre-trained source model and scales them appropriately, allowing the target model to inherit the source model's knowledge and performance characteristics while adapting to the new architecture.
2Measurement precision
If a new ANN is created from scratch with more neurons per layer to achieve higher accuracy, then the model accuracy is improved, but the computing resources required increase significantly
Solution Approach 1:
The patent uses copying by transferring weights, biases, and hyperparameters from the source ANN model to the target ANN model. Instead of creating the target model from scratch, the system copies the learned parameters from the pre-trained source model and scales them appropriately, allowing the target model to inherit the source model's knowledge and performance characteristics while adapting to the new architecture.
Solution Approach 2:
The patent applies parameter changes by systematically scaling hyperparameters (learning rate, batch size, number of epochs) and model parameters (weights, biases) when transitioning from the source model to the target model. This allows the model to be adapted to different architectures and computing environments while maintaining optimal training dynamics and reducing resource requirements.
3Measurement precision
If significant trial and error is performed to determine ANN initialization parameters, then the model performance is improved, but the time and complexity of the development process increase
Solution Approach 1:
The patent uses copying by transferring weights, biases, and hyperparameters from the source ANN model to the target ANN model. Instead of creating the target model from scratch, the system copies the learned parameters from the pre-trained source model and scales them appropriately, allowing the target model to inherit the source model's knowledge and performance characteristics while adapting to the new architecture.
Solution Approach 2:
The patent applies preliminary action by pre-training a source ANN model to convergence before using it to initialize the target model. This pre-training step performs the difficult work of finding good weight initializations and hyperparameters in advance, so that the target model can start from a proven good state rather than from random initialization, dramatically reducing the time needed to reach the same accuracy level.
4Measurement precision
If a powerful ANN is trained to achieve high accuracy, then the model accuracy is improved, but the difficulty of deploying it to environments with limited computing resources increases
Solution Approach 1:
The patent applies segmentation by separating the model development process into two distinct phases: training a source model to high accuracy, and then creating a scaled target model for deployment. This segmentation allows the high-accuracy model to be developed independently in the cloud, while the deployed model can be optimized for local resources, making deployment to resource-constrained environments much easier.
Solution Approach 2:
The patent applies parameter changes by systematically scaling hyperparameters (learning rate, batch size, number of epochs) and model parameters (weights, biases) when transitioning from the source model to the target model. This allows the model to be adapted to different architectures and computing environments while maintaining optimal training dynamics and reducing resource requirements.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer system is configured to scale an artificial neural network (ANN), by performing the steps of: initializing a first ANN based at least on first parameters and a first number of neurons per layer; training the first ANN using training inputs to adjust weights and biases of the first ANN; upon determining that an accuracy of the first ANN at generating outputs is greater than a threshold value, generating a second number of neurons per layer that is scaled from the first number of neurons per layer; initializing a second ANN based at least on the first parameters and on the second number of neurons per layer; training the second ANN using training inputs to adjust weights and biases of the second ANN; and executing the second ANN to generate inferences based on inference data input thereto.