Introspection Network Predicts Neural Network Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep neural networks is time-consuming due to the large number of parameters and the size of input datasets, often requiring days or weeks to reach convergence, even with optimization techniques like SGD.
Innovation Solution
An introspection network is trained to predict weight values at future training steps, allowing the target neural network to jump forward in accuracy by setting its weight values to the predicted values, thereby reducing the number of training steps needed to reach convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional iterative training with optimizers like SGD is used, then the neural network learns optimal weight values through thousands of training rounds, but the training period becomes extremely long (days to weeks)
Solution Approach 1:
The introspection network is trained in advance on weight histories from source neural networks to learn the evolution patterns of weight values. This preliminary training enables the introspection network to predict future weight values without requiring the target network to undergo the full iterative training process, thus performing the useful action of weight optimization in advance.
Solution Approach 2:
The invention creates training examples by copying weight histories from source neural networks. These copied weight sequences are transformed into training data (input sequences of weight values and corresponding future target values) that teaches the introspection network how weights evolve during training, enabling it to generalize to target networks.
2Productivity
If the number of training rounds is reduced using predicted weight values, then training time is significantly reduced, but the complexity of the training system increases due to the introspection network
Solution Approach 1:
The training system is segmented into two independent components: (1) the introspection network that predicts weight values, and (2) the target neural network that performs the actual task. This segmentation allows the prediction function to be separated from the training function, making the overall system more manageable despite the added complexity.
Solution Approach 2:
The introspection network serves as an intermediary between the training data and the target neural network. It translates historical weight patterns into predicted future weight values, which then guide the training process of the target network, reducing the number of iterations needed while maintaining training effectiveness.
Data Source
AI summary
An introspection network is a machine-learned neural network that accelerates training of other neural networks. The introspection network receives a weight history for each of a plurality of weights from a current training step for a target neural network. A weight history includes at least four values for the weight that are obtained during training of the target neural network up to the current step. The introspection network then provides, for each of the plurality of weights, a respective predicted value, based on the weight history. The predicted value for a weight represents a value for the weight in a future training step for the target neural network. Thus, the predicted value represents a jump in the training steps of the target neural network, which reduces the training time of the target neural network. The introspection network then sets each of the plurality of weights to its respective predicted value.