Neural Network Training With Estimation Networks to Reduce Time
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network training is resource-intensive and time-consuming, especially as applications and architectures become more complex, necessitating improved methods to accelerate the training process.
Innovation Solution
A training framework that utilizes an estimation network to generate estimated training values for neural networks based on previous model files, allowing for accelerated training without additional iterations, combined with traditional training techniques to refine the process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional neural network training methods are used, then training accuracy can be achieved, but training time and resource consumption increase significantly
Solution Approach 1:
The system performs preliminary actions by training an estimation network on a subset of training data before full training begins. This estimation network pre-calculates expected parameter updates, which are then used to accelerate the full training process, reducing overall training time while maintaining accuracy
Solution Approach 2:
The system creates a copy of the neural network architecture as an estimation network. This estimation network is trained on a subset of data and then used to generate estimated parameter updates for the main network, avoiding the need to perform all training iterations on the full network and thereby reducing training time
2Reliability
If traditional neural network training methods are used, then model convergence can be achieved, but computational resource consumption increases
Solution Approach 1:
The training process is segmented into distinct phases: (1) training the estimation network on a subset of data, (2) using the estimation network to generate parameter updates, and (3) applying these updates to the main network. This segmentation allows computational resources to be used more efficiently by distributing work across different network instances and data subsets
Solution Approach 2:
The estimation network serves as an intermediary between the training data and the main neural network. Instead of directly training the main network on all data, the estimation network processes the data first and generates estimated updates that are then applied to the main network, reducing its computational burden
Data Source
AI summary
Apparatuses, systems, and techniques to accelerate training of one or more neural networks. In at least one embodiment, training values for individual nodes in a first neural network are estimated using a second neural network or other estimation technique in order to decrease time required to train the first neural network.