Specific-Purpose LLM Training via Human Readability and Objective Compliance

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Large Language Models (LLMs) perform poorly when applied to specific purposes due to their training on general objectives, resulting in low-quality and irrelevant responses.

Innovation Solution

An online concierge system trains a specific-purpose LLM using a general-purpose LLM and an evaluation model, computing human readability and objective compliance scores to update the specific-purpose LLM's parameters, ensuring responses meet both general and specific objectives.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If a general-purpose LLM is trained on large datasets to achieve high performance on general objectives, then the model produces high-quality responses for general purposes, but the model performs poorly when applied to specific purposes with low-quality and irrelevant responses

Engineering Contradiction:

Improvegeneral-purpose performanceVSAvoidspecific-purpose response quality

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent segments the LLM training into two distinct phases: pre-training on general objectives (usefulness, truthfulness, harmlessness) and then fine-tuning on specific objectives. This segmentation allows the model to first acquire general capabilities and then specialize for specific purposes, resolving the contradiction between general versatility and specific-purpose reliability.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent changes the training parameters and objectives dynamically. The model is first trained with general objectives, then the training parameters are adjusted to include specific objectives with associated rewards. This parameter change enables the model to adapt from general-purpose performance to specific-purpose reliability by modifying the loss function and training data distribution.

Inventive Principle:

Principle #35Parameter changes

2Reliability

If the specific-purpose LLM is trained using reinforcement learning with human feedback to improve response quality, then the model meets general and specific objectives, but the training process becomes more complex

Engineering Contradiction:

Improveresponse qualityVSAvoidtraining process complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent implements reinforcement learning with human feedback (RLHF) where human annotators provide feedback on model responses. This feedback mechanism guides the model to improve response quality by maximizing rewards based on human preferences. The feedback loop creates a systematic approach to training that, while complex, provides clear guidance for model optimization.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent introduces an evaluation model as an intermediary component that automatically assesses response quality and provides scores for training. This intermediary simplifies the training process by providing automated feedback mechanisms, reducing the need for manual annotation while still maintaining high training standards. The evaluation model acts as a mediator between the model and the training process.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Reliability

If the system computes both human readability score and objective compliance score to evaluate responses, then the model aligns with both general and specific objectives, but the evaluation process increases computational cost

Engineering Contradiction:

Improvealignment with objectivesVSAvoidevaluation computational cost

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent computes both human readability score (comparing to general-purpose LLM response) and objective compliance score (using evaluation model), which is more than strictly necessary. This excessive evaluation ensures comprehensive alignment with both general and specific objectives, capturing nuanced performance aspects that single-score systems would miss, despite the increased computational cost.

Inventive Principle:

Principle #16Partial or excessive action

Solution Approach 2:

The patent replaces manual evaluation mechanisms with automated scoring systems. The human readability score uses automated comparison with general-purpose LLM responses, and the objective compliance score uses trained evaluation models. This substitution of mechanical/manual evaluation with automated systems reduces the computational burden compared to manual annotation while maintaining evaluation thoroughness.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20240289632A1Aligning large language models with specific objectives using reinforcement learning and human preference

Publication Date: 2024.08.29 MAPLEBEAR INC

AI summary

An online system trains a specific-purpose LLM. The online system obtains training examples and divides training examples across batches. The online system generates a specific response by applying parameters of the specific-purpose LLM to a batch of training examples. The online system generates a general response by applying parameters of a general-purpose LLM to the batch of training examples. The online system computes a human readability score representing the difference between the specific response and the general response. The online system computes an objective compliance score by applying an evaluation model to the specific response, the evaluation model trained to score the first response based on a specific objective. The online system updates the parameters of the specific-purpose LLM based on the human readability score and the objective compliance score.