Synthetic Preference Data Generation for Reward Model Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current generative models require costly and time-consuming human feedback to align with human preferences, leading to inefficiencies in reward model training due to noisy and limited human labeled data.
Innovation Solution
The approach generates synthetic labeled/preference data by extracting preference pairs from generative outputs of a model, using a reward model to select positive and negative training examples, thereby creating a semi-supervised training framework that augments initial labeled datasets with high-quality on-policy data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human feedback data is collected to train the reward model, then the model can learn human preferences, but the process is costly and time-consuming
Solution Approach 1:
The patent creates synthetic copies of human preference data by generating multiple model outputs and using a reward model to simulate human preferences. Instead of collecting actual human feedback, the system copies the preference signaling process through automated model evaluations, thereby eliminating the time-consuming data collection step while maintaining preference signal quality.
Solution Approach 2:
The system performs self-training by generating its own training data. The generative model produces outputs, the reward model evaluates them, and the resulting preference pairs automatically become training examples. This self-service mechanism eliminates dependency on external human annotators, resolving the contradiction between data accuracy and collection time.
2Measurement precision
If human labeled data is used to train the reward model, then the model learns preferences, but the data exhibits a high level of noise
Solution Approach 1:
The patent replaces noisy human labels with synthetic preference signals generated by the reward model itself. By copying the evaluation process rather than relying on human annotations, the system produces cleaner, more consistent preference data without the noise inherent in human judgment variability.
Solution Approach 2:
The system uses the reward model to provide feedback on generated outputs, creating a closed-loop training process. This automated feedback mechanism ensures consistent and noise-free preference signals, as the same evaluation criteria are applied uniformly across all training examples, eliminating the inconsistency present in human-labeled data.
3Measurement precision
If more human feedback data is collected to improve reward model quality, then preference accuracy improves, but cost and time increase
Solution Approach 1:
The system achieves improved reward model quality through self-generated training data rather than external human feedback. The generative model and reward model work together to produce unlimited amounts of clean training examples automatically, eliminating the productivity bottleneck of manual data collection while maintaining or improving model quality.
Solution Approach 2:
The patent changes the fundamental parameter of data source from human annotators to automated model evaluation. This parameter change transforms the training process from a resource-intensive human-dependent operation to an efficient automated process, improving productivity while maintaining data quality through consistent reward-based evaluation.
Data Source
AI summary
Implementations are directed to generating synthetic labeled/preference data by extracting preference pairs from sets of N outputs to a given unlabeled input to a generative model. A plurality of generative outputs are generated by a generative model from a set of input data. A reward model is used to determine a plurality of reward values for the plurality of generative outputs. Based on the reward values, a pair of generative outputs from the plurality of generative outputs is selected for inclusion in a training example. The pair of outputs include a positive training example and a negative training example, where the reward values indicate that the positive training example is preferred over the negative training example. The process can be repeated for a plurality of sets of input data to generate a plurality of training examples for inclusion in a training dataset, which can be used to update reward model(s).