Generative Neural Network Training With Soft Preference Scores
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for fine-tuning generative neural networks using human feedback rely on 'hard' preferences, which are less efficient and less accurate than 'soft' preferences, leading to suboptimal model performance and increased computational costs.
Innovation Solution
The described techniques utilize 'soft' preferences, represented by non-binary preference scores, to train generative neural networks, allowing for more efficient and accurate alignment with user preferences, reducing training and inference costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods using hard preferences are used for fine-tuning, then the training process is simpler to implement, but the model performance is suboptimal and computational costs increase
Solution Approach 1:
The patent transforms binary preference labels into continuous preference scores, changing the parameter type from discrete to continuous. This allows the model to learn nuanced preferences rather than simple win/loss outcomes, improving performance while the efficient loss function design keeps computational costs manageable
Solution Approach 2:
The patent replaces the traditional reinforcement learning from human feedback (RLHF) mechanism with a direct preference optimization approach. This substitution eliminates the need for complex reward models and iterative reinforcement learning processes, reducing computational overhead while achieving better alignment
2Measurement precision
If soft preferences with non-binary scores are used, then alignment accuracy with user preferences improves, but training complexity increases
Solution Approach 1:
The patent changes the preference parameter from binary to continuous, enabling precise measurement of user preferences. The continuous preference scores capture subtle nuances in user feedback, significantly improving alignment accuracy
Solution Approach 2:
The patent incorporates continuous preference scores as feedback signals during training. This feedback mechanism provides gradient information that guides the model toward better alignment, and the efficient loss function design ensures this feedback can be processed without excessive complexity
3Ease of manufacture
If hard preferences are used in training examples, then data processing is more straightforward, but the resulting model generates outputs with lower quality
Solution Approach 1:
The patent transforms discrete preference labels into continuous preference scores, enriching the training data with nuanced information. This parameter change allows the model to learn from the degree of preference rather than just binary outcomes, significantly improving output quality
Solution Approach 2:
The patent replaces complex multi-stage fine-tuning processes with a unified direct preference optimization approach. This substitution simplifies the overall training pipeline while producing higher quality outputs through better preference alignment
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a generative machine learning machine learning models to perform a machine learning task. In one aspect, a method comprises receiving an input prompt; and processing the input prompt using a generative neural network to generate the data item, wherein the generative neural network is optimized to generate output data items in response to input prompts, the neural network being optimized such that a contribution of preference data to an objective function used for optimizing the generative neural network is determined based on a preference score associated with the preference data, the preference data comprising a first training data item, a second training data item, a training prompt, and the preference score representing a degree of preference for the first training data item over the second training data item as a response to the training prompt.