Scaled Neural Network Training Using Transferred Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The process of implementing and training artificial neural networks (ANNs) at different scales is burdensome, requiring significant computational resources and time, and often involves extensive trial and error to achieve desired accuracy levels.
Innovation Solution
A method and system for initializing and training ANNs by leveraging the training of an original ANN to scale and initialize a new ANN, using copied and scaled hyperparameters and learned parameters, reducing the need for extensive training and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a new ANN is created entirely with more neurons per layer to achieve more accurate outputs, then the accuracy of outputs is improved, but the computational resources and training time increase significantly
Solution Approach 1:
The patent applies preliminary action by training a smaller initial ANN model first to establish a baseline architecture and parameters. This preliminary model's structure and learned parameters are then used to initialize the larger, more accurate ANN model, avoiding the need to train the larger model from scratch and significantly reducing computational resources required.
Solution Approach 2:
The patent uses copying by replicating the architecture structure and parameters from the smaller initial ANN model to create the larger ANN model. The weights, biases, and overall structure are copied and then scaled up, allowing the new model to start with proven configurations rather than requiring extensive trial and error.
2Measurement precision
If extensive trial and error is performed to determine structural parameters and activation functions, then the desired accuracy level is achieved, but the training time increases to several days
Solution Approach 1:
The patent performs preliminary action by establishing the ANN architecture and selecting activation functions in the initial smaller model. These decisions are then transferred to the larger model, eliminating the need to perform extensive trial and error again and dramatically reducing training time while maintaining the ability to achieve desired accuracy.
Solution Approach 2:
The patent applies parameter changes by systematically modifying the architecture parameters (number of neurons per layer, number of layers) from the initial model to create the scaled-up model. This structured approach to parameter changes avoids random trial and error and accelerates the optimization process.
3Measurement precision
If a powerful ANN is implemented in cloud computing environment with more computing resources, then more accurate outputs are generated, but the complexity of implementation increases
Solution Approach 1:
The patent applies segmentation by dividing the implementation process into two distinct phases: first training a smaller model on local resources, then using that model as a foundation for the larger cloud-based model. This segmentation simplifies the overall implementation complexity by breaking down the challenging task of creating a large ANN into manageable steps.
Solution Approach 2:
The smaller initial ANN model serves as an intermediary between the researcher and the final large-scale ANN implementation. It mediates the complexity by providing a simplified version that can be developed and tested locally, then used to bootstrap the more complex cloud-based implementation.
Data Source
AI summary
A computer system is configured to scale an artificial neural network (ANN), by performing the steps of: initializing a first ANN based at least on first parameters and a first number of neurons per layer; training the first ANN using training inputs to adjust weights and biases of the first ANN; upon determining that an accuracy of the first ANN at generating outputs is greater than a threshold value, generating a second number of neurons per layer that is scaled from the first number of neurons per layer; initializing a second ANN based at least on the first parameters and on the second number of neurons per layer; training the second ANN using training inputs to adjust weights and biases of the second ANN; and executing the second ANN to generate inferences based on inference data input thereto.