Judge Model Training for Bias-Resistant LLM Text Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for evaluating neural network-generated text are hindered by the need for expensive and difficult-to-scale human feedback, and biases in text length and position, which impede the training and fine-tuning of evaluation models.
Innovation Solution
A judge model training framework is developed, utilizing multiple datasets and engineered protocols to train a neural network-based judge model, reducing biases in text length and position, and enabling improved automatic evaluation of LLM-generated text.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human feedback is used to evaluate generated text, then evaluation quality is improved, but cost and scalability worsen
Solution Approach 1:
The patent creates synthetic training data by copying and adapting existing high-quality evaluation datasets. Multiple datasets are generated with different characteristics (length, position, content) to train the judge model to recognize quality patterns without requiring proportional human evaluation effort for each new evaluation task.
Solution Approach 2:
The judge model is trained to evaluate generated text autonomously without requiring human evaluators for each new text. The system uses synthetic training data to enable the model to self-assess quality, replacing the need for continuous human feedback while maintaining evaluation capability.
2Productivity
If traditional automatic evaluation methods are used, then scalability is improved, but bias in text length and position worsens
Solution Approach 1:
The patent applies local quality by creating synthetic training data with deliberately varied local characteristics - different text lengths, positions, and content distributions. This teaches the judge model to evaluate each local segment of text on its own merits rather than being biased by global properties like length or position.
Solution Approach 2:
The training datasets are intentionally designed with asymmetric properties - varying lengths, positions, and content types - to prevent the judge model from developing symmetric biases. By exposing the model to asymmetric training examples, it learns to evaluate text quality independently of these superficial characteristics.
3Reliability
If multiple datasets are used for training, then model performance is improved, but training complexity worsens
Solution Approach 1:
The judge model is designed with universal evaluation capabilities that work across multiple datasets and text types. The training framework uses multiple datasets with different characteristics to teach the model universal quality assessment skills that generalize across various text generation tasks and evaluation scenarios.
Data Source
AI summary
Embodiments described herein provide training a neural network based language model to generate content that aligns with user preference. The method may include: receiving a query and a corresponding response; generating a judgement indicating a preference level of the corresponding response and a critique indicating a reason of the judgement based on an input of the query, the corresponding response and an instruction indicating an evaluation protocol; constructing a preference judgment training sample comprising the query and the corresponding response; training a second neural network based language model using the preference training sample to judge whether a model-generated response to the query aligns with user preference; constructing a preference training dataset for a third neural network based language model based on judgment data generated from the trained second neural network based language model; training the third neural network based language model using the constructed preference training dataset.