Generative Neural Network Fine-Tuning via Offline Reward Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for fine-tuning generative neural networks using reward functions are computationally demanding, especially when the network has a large number of parameters, as they often require frequent online sampling and scoring.
Innovation Solution
The approach involves fine-tuning a generative neural network in an offline manner using a reward function, dividing the training process into a 'grow' part for sampling and an 'improve' part for further training, allowing for parallelization across multiple hardware devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If online reinforcement learning is used to fine-tune the generative neural network with reward function, then the model can be trained with frequent sampling and scoring, but the computational cost and time consumption increase significantly
Solution Approach 1:
The patent pre-generates a batch of candidate outputs for each input example before training begins. This preliminary action stores the sampling results in advance, eliminating the need for frequent online sampling during training iterations and significantly reducing training time while maintaining training effectiveness.
Solution Approach 2:
The patent creates multiple copies of the generative neural network (first copy for generating candidate outputs, second copy for training). This allows the training process to use pre-generated candidates from the first copy while the second copy undergoes parameter updates, enabling parallel processing and reducing computational overhead during training.
2Reliability
If online reinforcement learning is used to fine-tune the generative neural network with reward function, then the model can adapt to reward-based optimization, but the computational resources required increase substantially
Solution Approach 1:
The patent pre-generates candidate outputs in batch mode before training, amortizing the computational cost of sampling across multiple training iterations. This preliminary batch generation reduces the per-iteration computational burden during training while maintaining the ability to optimize based on reward feedback.
Solution Approach 2:
The patent uses two separate network copies to distribute computational workload. The first network generates candidate outputs offline, while the second network performs training updates. This duplication allows computationally intensive sampling operations to be decoupled from the training loop, reducing overall computational cost.
3Adaptability or versatility
If the generative neural network has a large number of parameters (e.g., billions), then the model capacity and performance improve, but the sampling and training process becomes slower and more computationally demanding
Solution Approach 1:
The patent pre-generates candidate outputs for all training examples in batch mode before training begins. This preliminary action creates a complete set of training data that can be used throughout training without requiring additional forward passes through the large model, significantly improving training efficiency while maintaining model capacity.
Solution Approach 2:
The patent creates two copies of the large-parameter network to enable parallel processing. The first copy generates candidate outputs offline, and the second copy performs training updates. This copying strategy allows the system to leverage the model's full capacity while avoiding the sequential bottleneck of sampling during each training step.
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a generative neural network. One of the methods includes training a generative neural network by performing a sequence of a plurality training stages each generating an expanded training data set. The method also involves performing a sequence of improve steps, each comprising training the generative neural network on the training examples in a corresponding subset of the expanded training data set.