Language Model Alignment via Iterative Nash Policy Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing large language models (LLMs) face challenges such as the need for large amounts of human-labeled data, instability in training, and the inability of the Bradley-Terry model to accurately capture the complexity of human preferences, leading to inefficiencies and inaccuracies in aligning with human values.
Innovation Solution
The method employs iterative Nash policy optimization (INPO) to train language models by formulating the learning problem as a symmetric two-player game, using a no-regret learning algorithm like online mirror descent (OMD) to learn the Nash policy without requiring estimation of expected win rates, and incorporating a novel loss objective that minimizes over a preference dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If reinforcement learning with human feedback (RLHF) using PPO algorithm is used to align LLMs with human preferences, then the alignment capability is improved, but the training stability deteriorates and large amounts of human-labeled data are required
Solution Approach 1:
The patent uses a reference language model to generate responses that serve as training data for the target language model. Instead of requiring large amounts of human-labeled data, the system copies and adapts responses from the reference model, iteratively improving alignment while maintaining training stability through the reference model's guidance
Solution Approach 2:
The patent implements a feedback mechanism where the target language model generates responses, these responses are evaluated against human preferences, and the results are used to update the model via INPO optimization. This closed-loop feedback enables stable iterative improvement of alignment capability without requiring extensive human-labeled data
2Productivity
If direct preference optimization (DPO) algorithm is used to directly learn policy on human preference dataset, then computational efficiency is improved, but the ability to accurately capture complex human preferences deteriorates due to Bradley-Terry model limitations
Solution Approach 1:
The patent moves beyond the Bradley-Terry model's limited parameterization by using INPO optimization that can capture more complex preference structures. The method changes the optimization parameters and objective function to better represent human preferences while maintaining computational efficiency through direct policy optimization rather than two-stage training
Solution Approach 2:
The patent combines elements of direct preference optimization with iterative refinement and reference model guidance, creating a composite training approach that achieves both computational efficiency and accurate preference modeling. The hybrid method integrates multiple techniques to overcome the limitations of pure DPO
3Reliability
If iterative training with preference oracle is performed to improve alignment, then alignment accuracy is improved, but training time and computational resources increase
Solution Approach 1:
The patent performs preliminary action by using the reference language model to generate high-quality response pairs before training the target model. This pre-prepared training data from the reference model accelerates the iterative training process, reducing the time needed for each iteration while maintaining alignment accuracy
Solution Approach 2:
The patent implements periodic action through iterative training cycles where the target model is trained at regular intervals using preference data generated from previous iterations. This periodic training approach balances computational workload and enables progressive improvement of alignment accuracy without excessive time consumption in any single training phase
Data Source
AI summary
The present disclosure describes various methods, systems, and storage medium for training a language model to obtain an iterative Nash policy optimized (INPO) language model. The method includes initializing a first language model by a reference language model; for N-th iteration with N starting from 1 to M, generating a plurality of responses using the N-th language model for each prompt in a plurality of prompts, and constructing a preference dataset using a preference oracle based on the plurality of responses for each prompt, wherein the preference dataset comprises a winning response and a losing response; training the N-th language model to obtain a (N+1)-th language model by minimizing a value of an INPO function for all preference dataset for the plurality of prompts, wherein the INPO function comprises an expectation term, a regularization term, and a modification term; and outputting the (M+1)-th language model.