Generative Model Training With Iterative Preference Ranking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training quality of generative models is affected by the high cost and inefficiency of traditional preference optimization processes, which require significant manual annotation data.
Innovation Solution
A method involving the construction of a training prompt and performing multiple rounds of iterative training, where response contents are evaluated and parameters are adjusted to enhance the probability of preferred responses and reduce non-preferred ones, using synthetic instructions and a weighted loss function to improve data utilization and training stability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional preference optimization is used for training generative models, then model performance can be improved, but training cost increases and data efficiency decreases
Solution Approach 1:
The system uses the generative model itself to generate training data through self-distillation, where the model's own outputs are evaluated and used as training examples. This eliminates the need for extensive manual annotation data while maintaining training effectiveness, as the model learns from its own generated responses that are ranked and selected based on evaluation metrics.
Solution Approach 2:
An evaluation model acts as an intermediary between the generative model and the training process. The evaluation model ranks and selects responses without requiring manual annotations, serving as a bridge that enables automated preference optimization. This intermediary component allows the system to optimize model performance while reducing dependency on human-annotated data.
2Reliability
If traditional preference optimization is used for training generative models, then model performance can be improved, but training efficiency decreases
Solution Approach 1:
The training process operates continuously through multiple iterative rounds where the generative model generates responses, the evaluation model ranks them, and the parameters are updated. This continuous cyclic process maintains training efficiency by keeping all components actively engaged in each iteration, avoiding idle time and maximizing the utilization of computational resources throughout the training pipeline.
Solution Approach 2:
The system performs preliminary evaluation and ranking of generated responses before using them for parameter updates. By pre-ranking the responses based on evaluation metrics, the system prepares the training data in advance, which streamlines the subsequent parameter optimization process and improves overall training efficiency by avoiding redundant computations during the update phase.
3Manufacturing precision
If manual annotation data is used for preference optimization, then training quality can be maintained, but training cost increases
Solution Approach 1:
The system creates synthetic training examples by copying and reusing the generative model's own outputs as training data. Instead of relying on expensive manual annotations, the model generates multiple responses to the same prompt, ranks them, and uses these self-generated pairs as training examples. This copying approach maintains training quality while eliminating the need for human annotators, thereby reducing training costs.
Solution Approach 2:
The training process dynamically adjusts parameters including temperature, top-p sampling, and iteration counts to optimize the balance between training quality and cost. By changing these parameters across different training rounds, the system adapts the data generation and selection process to maintain high training quality while minimizing resource consumption and computational costs.
Data Source
AI summary
Embodiments of the disclosure relate to a method, an apparatus, a device, and a computer-readable storage medium for training a generative model. The method includes: constructing a training prompt; and performing a plurality of rounds of iterative training based on the training prompt, wherein each round of iterative training includes: obtaining a plurality of response contents generated by the generative model based on the training prompt; determining a first response content and a second response content from the plurality of response contents based on evaluation information of the plurality of response contents, wherein an evaluation of the first response content is superior to an evaluation of the second response content; and adjusting a parameter of the generative model to increase a first probability of outputting the first response content and reduce a second probability of outputting the second response content.