LLM Reward Function Reuse for Training Data Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for tuning large language models are inefficient and lack effective mechanisms for selecting training datasets and reward functions, leading to suboptimal performance in applications such as semantic matching, API matching, and entity matching.
Innovation Solution
A method that repurposes reward functions to compute similarity between textual prompts and ground truth labels, creating a data selection scoring function to select a training dataset for large language models using reinforcement learning, with options for generating reward functions when predefined ones are unavailable.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If existing methods are used for tuning large language models without effective data selection mechanisms, then the training process is simpler, but the model performance is suboptimal
Solution Approach 1:
The patent merges data selection and reward function into a unified reinforcement learning framework. The reward function is repurposed to serve dual purposes: evaluating model outputs during training and selecting training data through the data selection scoring function. This integration resolves the contradiction by combining previously separate operations into a single coherent process that improves model performance without proportionally increasing complexity.
Solution Approach 2:
The reward function is designed with multi-functionality, serving both as a evaluation metric for model outputs and as a scoring mechanism for data selection. By making the reward function universal, the patent eliminates the need for separate data selection criteria, thereby improving reliability while avoiding the complexity of maintaining multiple independent functions.
2Measurement precision
If manual annotation of training data is used, then data quality is high, but the process is time-consuming and costly
Solution Approach 1:
The system employs self-service mechanisms where the reinforcement learning agent automatically selects training data based on the repurposed reward function. Instead of requiring manual annotation for data selection, the model uses its own reward evaluation capabilities to autonomously identify high-quality training pairs, significantly reducing time consumption while maintaining data quality through consistent application of the reward criteria.
Solution Approach 2:
The patent changes the parameter usage of the reward function from solely output evaluation to also serving as a data selection criterion. By repurposing the reward function to compute similarity between textual prompts and ground truth labels, the system enables automatic data selection based on predicted performance metrics, eliminating manual annotation time while preserving quality through automated evaluation.
3Reliability
If multiple reward functions are used for training multiple large language models, then model alignment is improved, but computational resources are consumed
Solution Approach 1:
The patent merges the functionality of multiple reward functions into a single repurposed reward function that serves both data selection and model training evaluation. Instead of training multiple separate models with different reward functions, the system uses one model with a unified reward function that performs both data selection scoring and output evaluation, significantly reducing computational resource consumption while maintaining alignment quality.
Solution Approach 2:
The reward function is designed as a universal component that performs multiple functions: evaluating model outputs during training and selecting training data through the data selection scoring function. This multi-functionality eliminates the need for separate reward functions for each purpose, thereby reducing computational resource requirements while maintaining comprehensive model alignment capabilities.
Data Source
AI summary
A computer-implemented method, a computer program product, and a computer system for tuning large language models. A computer receives pairs of textual prompts and ground truth labels. A computer creates a data selection scoring function, by repurposing one or more reward functions to compute similarity between the textual prompts and the ground truth labels, where the one or more reward functions measure similarity between textual outputs produced by a large language model and the ground truth labels. A computer selects a training dataset from the pairs of the textual prompts and the ground truth labels, by using the data selection scoring function. A computer tunes the large language model using the training dataset and reinforcement learning with the one or more reward functions.