Sequence Model Preference Optimization With Tied Ranking Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle with overfitting to highly preferred sequences and misalignment between pairwise labels and modeling, particularly when human raters exhibit tied preferences, leading to a lack of diversity and reliability in sequence generation.
Innovation Solution
The proposed Tied Preference Optimization (TPO) techniques apply a pairwise ranking loss with tied labels, allowing training on all sequences, including those with tied preferences, and incorporate a general framework for generating preference optimization loss functions using reward functions, regularizers, and pairwise loss functions to align sequence generation with human preferences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing machine learning models are trained using conventional preference optimization, then the model learns to generate preferred sequences, but the model overfits to highly preferred sequences and loses diversity
Solution Approach 1:
The patent modifies the loss function parameters by introducing tied preference labels and adjusting the mathematical formulation to account for tie scenarios. This changes the optimization landscape to prevent overfitting while maintaining preference alignment, resolving the contradiction between reliability and diversity.
Solution Approach 2:
The patent applies partial action by using tied preference labels instead of forcing complete preferences in all cases. This allows the model to learn from scenarios where sequences are equally preferred, preventing overfitting to highly preferred sequences while maintaining adequate preference alignment.
2Ease of manufacture
If existing models ignore tied preferences in training, then the training process is simpler, but there is misalignment between pairwise labels and modeling
Solution Approach 1:
The patent performs preliminary action by establishing a framework that incorporates tied preference labels from the beginning of the training process. This preliminary setup ensures that the labeling system accurately reflects real-world preferences including ties, improving measurement precision without significantly complicating the training process.
3Productivity
If the model is trained to maximize reward function, then the model generates preferred outputs, but the model lacks reliability when human raters exhibit tied preferences
Solution Approach 1:
The patent changes the reward function parameters by incorporating tied preference labels into the loss function formulation. This modification allows the model to maintain generation efficiency while improving reliability in tied preference scenarios through adjusted optimization objectives.
Solution Approach 2:
The patent implements feedback by using tied preference labels that provide more accurate information about rater preferences. This feedback mechanism improves reliability by correctly signaling when sequences are equally preferred, preventing the model from making incorrect distinctions while maintaining productivity.
Data Source
AI summary
Provided are systems and methods for fine-tuning sequence processing models to human preferences. The approaches can account for tied preferences between pairs of sequences and, therefore, can be referred to as Tied Preference Optimization (TPO). Example sequence processing models include so-called large language models (LLMs), large multimodal models (LMMs), and other models that are configured to process inputs and/or generate outputs that are structured as a series of data elements such as tokens.