Reward Model Trained Language Model for Transparent Reasoning

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current machine-learning models, particularly neural networks, face challenges in providing transparent and accurate reasoning processes for complex queries, often acting as 'black boxes' that lack interpretability and may generate inaccurate responses, which can have significant negative consequences in real-world applications.

Innovation Solution

A system that trains a language model using a reward model to generate and evaluate reasoning traces, providing both final answers and step-by-step explanations, employing reinforcement learning to improve the model's performance and transparency through outcome-supervised or process-supervised training.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Loss of information

If a language model generates reasoning traces for complex queries, then the transparency and interpretability of the model is improved, but the computational complexity and time required for processing increases

Engineering Contradiction:

Improvetransparency of reasoning processVSAvoidcomputational complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The reasoning process is segmented into discrete reasoning steps that are generated sequentially. Each step is evaluated independently by the reward model, allowing the complex task of generating transparent reasoning to be broken down into manageable units that can be processed efficiently

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

A reward model is introduced as an intermediary component that evaluates reasoning steps without requiring the main language model to perform the evaluation. This mediator provides feedback signals that guide the generation of transparent reasoning while keeping the computational burden distributed across specialized components

Inventive Principle:

Principle #24Intermediary (Mediator)

2Reliability

If reinforcement learning is used to train the language model with reward models, then the accuracy of reasoning tasks is improved, but the training time and computational resources required increase

Engineering Contradiction:

Improveaccuracy of reasoning tasksVSAvoidtraining time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The reward model provides immediate feedback signals during training based on the quality of generated reasoning steps. This feedback mechanism allows the language model to learn from errors and improve reasoning accuracy iteratively, with the reward signal guiding the optimization process more efficiently than trial-and-error approaches

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The reward model is trained separately on labeled reasoning data before being used to guide the language model training. This preliminary preparation of the evaluation component allows for more efficient subsequent training of the language model, as the feedback mechanism is already optimized and ready to provide accurate guidance

Inventive Principle:

Principle #10Preliminary action

3Reliability

If the model generates detailed reasoning traces with intermediate steps, then the reliability and safety of responses is improved, but the response length and processing time increase

Engineering Contradiction:

Improvesafety of responsesVSAvoidresponse generation speed

Core Design Contradiction:

ReliabilityVSProductivity

Solution Approach 1:

The generation of reasoning traces is made dynamic and adaptive. The model can adjust the level of detail in reasoning steps based on the complexity of the query and the confidence level of the predictions. For simpler queries, fewer reasoning steps are generated, maintaining speed while preserving safety for complex queries that require detailed reasoning

Inventive Principle:

Principle #15Dynamics

4Manufacturing precision

If outcome-supervised or process-supervised training is employed, then the quality of reasoning traces is improved, but the requirement for labeled training data and computational resources increases

Engineering Contradiction:

Improvequality of reasoning tracesVSAvoidamount of labeled training data

Core Design Contradiction:

Manufacturing precisionVSQuantity of substance

Solution Approach 1:

The reward model is trained to evaluate specific aspects of reasoning steps rather than requiring complete correctness of entire reasoning traces. This partial supervision approach allows the model to learn from incomplete or partially correct examples, reducing the need for extensively labeled training data while still improving reasoning quality

Inventive Principle:

Principle #16Partial or excessive action

Data Source

PatentUS20240104391A1Reward-model based reinforcement learning for performing reasoning tasks

Publication Date: 2024.03.28 GDM HOLDING LLC

AI summary

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for A training a language model for performing a reasoning task. The system obtains a plurality of training examples. Each training example includes a respective sample query text sequence characterizing a respective sample query and a respective reference response text sequence that includes a reference final answer to the respective sample query. The system trains a reward model on the plurality of training examples. The reward model is configured to receive an input including a query text sequence characterizing a query and one or more reasoning steps that have been generated in response to the query and process the input to compute a reward score indicating how successful the one or more reasoning steps are in yielding a correct final answer to the query. The system trains the language model using the trained reward model.