Reward modeling in machine learning model alignment

Abstract

A method is proposed for machine learning (ML) model alignment. In the method, a set of prompt- response pairs and a set of annotated rewards corresponding to the set of prompt-response pairs are obtained. A set of predicted rewards corresponding to the set of prompt-response pairs is determined by using a reward model, and the reward model is configured to receive a prompt-response pair and determine a reward for the prompt-response pair to guide alignment of the ML model. A training loss is determined based on an order consistency degree between the set of predicted rewards and the set of annotated rewards and the reward model is trained based on the training loss. The order consistency degree indicates a difference between an annotated preference order and a predicted preference order of two prompt-response pairs.

Description

REWARD MODELING IN MACHINE LEARNING MODEL ALIGNMENTFIELD

[0001] The present disclosure generally relates to the field of computer, and more specifically, to methods, devices, and computer program products for reward modeling in machine learning (ML) model alignment.BACKGROUND

[0002] The alignment of ML models, e.g., language models (LMs), is important for their safe and effective deployment across various applications. Current research on reinforcement learning from human feedback (RLHF) has largely focused on utilizing preference-based annotations provided by humans or general-purpose MLs. In general, there are two primary approaches to RLHF, namely the direct policy optimization that aligns MLs with supervised learning objectives, and the alternate method that constructs a reward model to guide the MLs optimization via either supervised learning or reinforcement learning.SUMMARY

[0003] In a first aspect of the present disclosure, there is provided a method of ML model alignment. The method includes: obtaining a set of prompt-response pairs and a set of annotated rewards corresponding to the set of prompt-response pairs, a response of the set of prompt-response pairs being generated by a ML model based on a prompt of the set of prompt-response pairs: determining, by using a reward model, a set of predicted rewards corresponding to the set of prompt-response pairs, wherein the reward model is configured to receive a prompt-response pair and determine a reward for the prompt-response pair to guide alignment of the ML model; determining a training loss based on an order consistency degree between the set of predicted rewards and the set of annotated rewards, the order consistency degree indicating a difference between an annotated preference order of two prompt-response pairs in the set of prompt-response pairs and a predicted preference order of the two prompt-response pairs; and training the reward model based on the training loss.

[0004] In a second aspect of the present disclosure, there is provided an electronic device. The electronic device comprises: a computer processor coupled to a computer -readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements a method according to the first aspect of the present disclosure.

[0005] In a third aspect of the present disclosure, there is provided a computer program product, the computer program product comprising a computer readable storage medium having instructions embodied therewith, the instructions executable by an electronic device to cause the electronic device to perform a method according to the first aspect of the present disclosure.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Through the more detailed description of some embodiments of the present disclosure in the accompanying drawings, the above and other objects, features, and advantages of the present disclosure will become more apparent, wherein the same reference generally refers to the same components in the embodiments of the present disclosure.

[0008] FIG. 1 illustrates a schematic diagram of an example process of LLM alignment according to some embodiments of the present disclosure;

[0009] FIG. 2 illustrates a schematic diagram of an example process of reward model training according to some embodiments of the present disclosure;

[0010] FIG. 3 illustrates an example flowchart of a method of ML alignment according to some embodiments of the present disclosure; and

[0011] FIG. 4 illustrates a block diagram of an electronic device in which various embodiments of the present disclosure can be implemented.DETAILED DESCRIPTION

[0012] Principle of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein can be implemented in various manners other than the ones described below.

[0013] In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0014] References in the present disclosure to “one embodiment,” “an embodiment,” “an example embodiment,” and the like indicate that the embodiment described may include a particular feature, structure, or characteristic, but it is not necessary that every embodiment includes the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an example embodiment, it is submitted that it is within the knowledge of one skilled in the art to affect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

[0015] It shall be understood that although the terms “first” and “second” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0016] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when usedherein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0017] Principle of the present disclosure will now be described with reference to some embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. The disclosure described herein can be implemented in various manners other than the ones described below. In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

[0018] It may be understood that data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with requirements of corresponding laws and regulations and relevant rules.

[0019] It may be understood that, before using the technical solutions disclosed in various embodiment of the present disclosure, the user should be informed of the type, scope of use, and use scenario of the information involved in the present disclosure in an appropriate manner in accordance with relevant laws and regulations, and the user’s authorization should be obtained.

[0020] For example, in response to receiving an active request from the user, prompt information is sent to the user to explicitly inform the user that the requested operation will need to acquire and use the user’s information. Therefore, the user may independently choose, according to the prompt information, whether to provide the information to software or hardware such as electronic devices, applications, servers, or storage media that perform operations of the technical solutions of the present disclosure.

[0021] As an optional but non-limiting implementation, in response to receiving an active request from the user, the way of sending prompt information to the user, for example, may include a pop-up window, and the prompt information may be presented in the form of text in the pop-up window. In addition, the pop-up window may also carry a selection control for the user to choose “agree” or “disagree” to provide the information to the electronic device.

[0022] It may be understood that the above process of notifying and obtaining the user authorization is only illustrative and does not limit the implementation of the present disclosure. Other methods that satisfy relevant laws and regulations are also applicable to the implementation of the present disclosure.

[0023] As used herein, the term “model” is referred to as an association between an input and an output learned from training data, and thus a corresponding output may be generated for a given input after the training. The generation of the model may be based on a machine learning technique. In general, a machine learning model may be built, which receives input information and makes predictions based on the input information. For example, a classification model may predict a class of the input information among a predetermined set of classes. As used herein, “model” may also be referred to as “machine learning model”, “learning model”, “machine learning network”, or “learning network,” which are used interchangeably herein.

[0024] As discussed above, currently preference-based annotations provided by humans or ML models have been used for ML alignment, e.g., for large language model (LLM) alignment. For example, promptresponse pairs annotated with human preferences / feedback may be used for training the reward model togenerate rewards for alignment of the MLs, such that the MLs can be aligned with specific requirements of various application scenarios, values and so on. The annotations reflecting human preferences may be obtained from humans and / or simulated by models. For example, a reward model may generate rewards (i.e., scores) for the prompt-response pairs to simulate or represent the human preferences for the prompt-response pairs.

[0025] However, reward modeling in the ML alignment may face various challenges. For example, during training of the reward model, pairwise comparisons (i.e., comparison between two prompt-response pairs) are used as training data but the reward model is configured to generate the rewards for the pairwise comparisons. The mechanism of converting the pairwise comparisons into rewards needs to be further investigated. Moreover, the training efficiency of the reward model needs to be further improved.

[0026] Embodiments of the present disclosure propose solutions for ML model alignment. A method for the ML model alignment is proposed according to embodiments of the present disclosure. In the method, a set of prompt-response pairs and a set of annotated rewards corresponding to the set of prompt-response pairs are obtained. A set of predicted rewards corresponding to the set of prompt-response pairs is determined by using a reward model, and the reward model is configured to receive a prompt-response pair and determine a reward for the prompt -response pair to guide alignment of the ML model. A training loss is determined based on an order consistency degree between the set of predicted rewards and the set of annotated rewards and the reward model is trained based on the training loss. The order consistency degree indicates a difference between an annotated preference order and a predicted preference order of two prompt-response pairs.

[0027] In embodiments of the present disclosure, the order consistency degree between the predicted rewards and the annotated rewards is used for determining the training loss of the reward model. Thus, the pairwise comparisons can be converted into the rewards in a simple and efficient way, such that the training efficiency of the reward model can be improved.

[0028] Example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0029] FIG. 1 illustrates a schematic diagram of an example process 100 of ML alignment according to some embodiments of the present disclosure. As illustrated in FIG. 1, an ML 110 and a reward model 120 are involved in the process 100.

[0030] The ML 110 may receive a prompt 131 and generate a response 133 corresponding to the prompt 131. The pair of the prompt 131 and the response 133 may be input to the reward model 120 and the reward model 120 may generate a reward 135 for the pair. Although one prompt and one response are illustrated in FIG.1, the reward 135 may be associated with two prompt-response pairs. As an example, the reward 135 may be a binary vector, e.g., (1, 0) or (0, 1), to represent the preference order of the two prompt-response pairs. As another example, the reward 135 may be a value in the range of [0, 1J to represent the preference order. The reward 135 then serves as feedback to the response 133 generated from the ML 110, such that the alignment of the ML 110 can be performed based on the reward 135.

[0031] In embodiments of the present disclosure, the reward model 120 is trained based on the order consistency degree between the predicted rewards and the annotated rewards, such that the training efficiency of the reward model 120 can be improved.

[0032] FIG. 2 illustrates a schematic diagram of an example process 200 of reward model training according to some embodiments of the present disclosure. For the purpose of illustration, the process 200 will be described with reference to the reward model 120 in FIG. 1. It is appreciated that the reward model 120 may be deployed in any suitable ML alignment and not limited to the ML alignment as illustrated in FIG. 1.

[0033] As illustrated in FIG. 2, a set of prompt-response pairs 210 and a set of annotated rewards 220 corresponding to the set of prompt-response pairs are obtained. Moreover, a set of predicted rewards 230 corresponding to the set of prompt-response pairs 210 is determined by using the reward model 120. The set of prompt-response pairs 210 may comprise a first prompt-response pair 211 and a second prompt-response pair 212. Accordingly, the set of predicted rewards 230 may comprise a first predicted reward 231 corresponding to the first prompt-response pair 211 and a second predicted reward 232 corresponding to the second promptresponse pair 212, and the set of annotated rewards 220 may comprise a first annotated reward 221 corresponding to the first prompt-response pair 211 and a second annotated reward 222 corresponding to the second promptresponse pair 212.

[0034] A training loss is determined based on an order consistency degree between the set of predicted rewards 230 and the set of annotated rewards 220 and the reward model 120 is trained based on the training loss. The order consistency degree indicates a difference between an annotated preference order and a predicted preference order of two prompt-response pairs of the set of prompt-response pairs 210.

[0035] For example, for the first prompt -response pair 211 (also referred to as first pair for short) and the second prompt -response pair 212 (also referred to as second pair for short), if the annotated preference order is that the first pair is superior to or better than the second pair, and the predicted preference order is also that the first pair is superior to or better than the second pair, the order consistency degree is high, e.g., represented by the value of “1”. In contrast, if the annotated preference order is that the first pair is superior to or better than the second pair, but the predicted preference order is that the first pair is inferior to or worse than the second pair, the order consistency degree is low, e.g., represented by the value of "O'. It will be appreciated that the order consistency degree may be represented in any suitable formula of the difference between an annotated preference order and a predicted preference order of two prompt-response pairs of the set of prompt -response pairs 210.

[0036] In some embodiments, the set of prompt-response pairs 210 may comprise cross-prompt pairs and / or same -prompt pairs. For example, a first prompt of the first prompt-response pair 211 and a second prompt of the second prompt-response pair 212 may be the same or different. Analysis of the cross-prompt pairs will be described in the following description.

[0037] In some embodiments, the set of annotated rewards 220 may be obtained by determining the set of annotated rewards 220 based on a threshold of variances among the set of annotated rewards 220. For example, annotators may be encouraged to increase the diversity of the annotated rewards so as to improve the annotation quality. Analysis of the annotation quality will be described in the following description.

[0038] Theoretical foundation of the present disclosure is provided below for better understanding of the proposed solutions. In the following description, the theoretical foundation may involve reward modeling objectives in ML model (e.g., LLM) alignment which inspire the proposed order consistency for the reward modeling; applicable / potential reward models based on the order consistency; and further analysis of the reward modeling based on the order consistency.Reward Modeling Objectives in ML Alignment

[0039] First, essential requirements of the reward model in ML alignment are analyzed theoretically. It is recognized in the present disclosure that, because the training data consists of binary' preferences, representing the relative order (ranking) between two prompt-response pairs, the reward modeling objectives can be viewed as a form of binary classification with the ultimate goal being to learn a continuous score for optimization.

[0040] In basic binary classification, accuracy is prioritized over modeling output probabilities precisely. For example, neural classifiers, despite being overconfident, are widely used for their accuracy. Similarly, the reward model is not required to predict comparison probabilities accurately', but rather to provide a reliable signal for ranking ML outputs at inference. Since the goal is response optimization using a reward proxy, it is sufficient to learn the reward model (also referred to as reward function) up to a monotonic transformation. While this might alter preference probabilities, it won’t affect optimization results. To this end, the learned reward function f only needs to satisfy the following condition: for any two distinct prompt -response pairs ( jq) and (x2, 2)’ isrequired that (f^x1, y1) — f(x2, 72))(r(xi> 71)— r(x2’ 72)) > 0, where r denotes a true reward function associated with the annotated rewards.

[0041] In other words, it is expected that the learned reward function preserves the ordering as the true reward function. Thus, the training loss for the reward model may be determined based on an optimization objective that the predicted preference order of two prompt-response pairs is consistent with the annotated preference order of the two prompt-response pairs.

[0042] This condition implies the existence of a strictly monotonic increasing function h such that (-) = Such an equivalence is sufficient for optimizing the reward in settings such as sampling-based optimization, and contextual bandits. Ideally, if the ground truth ordering is available, it may be defined that h = sign(r(x₁, y₁) − r(x₂, y₂)). If it is possible to ( I ) construct a model H̃: X × Y × X × Y ↦ {+1, −1} that predicts the correct ordering with high accuracy (i.e., H is order consistent) and (2) map this ordering into a continuous value, the requirements for downstream optimization (e.g., LLM alignment) can be satisfied.

[0043] The above analysis highlights the critical concept of the order consistency in the reward modeling, which inspires the solutions provided by the present disclosure. Detailed analysis of the reward modeling based on the order consistency is described in the following.Reward Models Based on Order Consistency

[0044] In some embodiments of the present disclosure, the reward model based on the order consistency may be a Bradley-Terry (BT) model. Although the BT model may have been used in the ML alignment, the theoretical justification for using the BT model in this context remains underexplored. In other words, currently it remains unclear why the BT model, which is originally developed for multi-player stochastic game matching, can be adopted to convert pairwise response comparisons to reward values and make predictions.

[0045] In contrast, in the present disclosure, the BT model is evaluated theoretically and it is determined that the BT model may serve as the reward model based on the order consistency. Details of the evaluation of the BT model will be described in the following. The mechanism of using the BT model to serve as the reward model based on the order consistency is described here.

[0046] In some embodiments, with reference to FIG. 2, a predicted order representation for the BT model may be determined based on a difference between the first predicted reward 231 and the second predicted reward 232, to represent the predicted preference order of the first prompt-response pair 211 and the second promptresponse pair 212. The predicted order representation may be, for example, o(rBT(x1,y1) — rBT(x2,y2)) where a is a sigmoid function. In some other examples, the predicated order representation may be any other suitable activation function of the difference between the first predicted reward 231 and the second predicted reward 232.

[0047] Based on the predicted order representation and the annotated preference order of the first promptresponse pair 211 and the second prompt-response pair 212, the training loss for the BT model may be determined. For example, the training loss may be determined based on a binary cross-entropy loss.

[0048] As a non-limiting example, tire BT model may be designed to enforce the order consistency by modeling the probability that h = 1 using c(rBT(x1,y1) — rBT(x2,y2)), where ff is a sigmoid function. This allows training the BT model with the following binary cross-entropy loss:

[0049] This structure guarantees that flipping the comparison order will also flip the prediction.

[0050] BT’s difference-in-reward structure inherently enforces anti-symmetry, i.e., if we exchange the roles of x1,y1with x2,y2the reward difference becomes negative. To better understand this, consider an order model H̃ that outputs a preference vector for both prompt-response pairs, i.e., H̃ := (H̃₁, H̃₂), where H̃₁, H̃₂: X × Y → {1, −1} and ideally align with ( / t, — / ?). The BT model imposes a hard constraint such that H̃₁ = −H̃₂.

[0051] Alternatively, in some embodiments of the present disclosure, the reward model based on the order consistency may be a classification-based model, i.e., a classifier. Compared to the BT model, the classifier may be simpler. In some embodiments, the classifier may be a neural network-based classification model or a treebased classification model.

[0052] It is recognized in the present disclosure that, with sufficient data, instead of explicitly enforcing the anti-symmetry constraint, the anti-symmetric structure could be learned implicitly by ensuring the order consistency, i.e., H̃₁ ≈ h and H̃₂ ≈ −h.

[0053] In some embodiments, with reference to FIG. 2, the classifier may determine a classification result for each of the first prompt-response pair 211 and the second prompt-response pair 212. The classification result may be a classification accuracy that the pair is classified as winning. For example, the classifier may determine a first classification accuracy of the first prompt-response pair 211 being classified as winning as the first predicted reward 231, and determine a second classification accuracy of the second prompt-response pair 212 being classified as winning as the second predicted reward 232.

[0054] The training loss for the classifier may be determined based on an expected probability that the first predicted reward 231 equals the first annotated reward 221 and the second predicted reward 232 equals the second annotated reward 222. The first annotated reward 221 is the negative of the second annotated reward 222. For example, if the first pair is annotated as being superior to the second pair, the first annotated reward221 may be of a value “1” and the second annotated reward 222 may be of a value “-1”. In this way, the antisymmetric structure can be learned implicitly by ensuring the order consistency.

[0055] As a non-limiting example, consider a classifier model H̃_clf, under this construction, the order consistency could be written as

[0056] Additionally, a union bound of the training loss may be determined based on a first expected probability that the first predicted reward 231 equals the first annotated reward 221 and a second expected probability that the second predicted reward 232 equals the second annotated reward 222. Then, the union bound may be considered as an updated training loss ℒ_clf andclassifier may be trained by minimizing the union bound of the training loss. The union bound of the training loss may be as follows:

[0057] In other words, the classifier may be trained as targeting on predicting these two parts separately, and if the model predicts well, it may satisfy the anti-symmetry constraint. Instead of directly enforcing the order consistency, the classification accuracy of each prompt-response pair may be used as a surrogate. In practice, this means training a classifier by treating the annotations and prompt-response pairs independently. Instead of having necessarily a pair, components of the pair may be split and the classification may be performed on each of the components of the pair. Then the logit can be used as a proxy for the reward model. For an alternative perspective: instead of learning the joint probability P(i j) that depends on both players i and j, this solution focuses on learning the marginal probability IP (i wins). These two are related via Jensen’s inequality, and the details is omitted in this specification.Further Analysis of Reward Models Based on Order ConsistencyRethinking the Usage of BT Models in ML Alignment

[0058] In the present disclosure, a comprehensive analysis of the application of the BT model in ML alignment is conducted, including contrasting its traditional use in multi-player arenas with the unique challenges posed in this context, analyzing the underlying rationale and offering a thorough justification for applying the BT model to ML reward modeling.

[0059] Two Different BT Models — Parameter Estimation and Prediction: The original BT model posits that in a simplified two-option scenario, the probability of selecting option t from a set l,j is proportional to the utility u(-) assigned to that option. Formally, this can be expressed as a softmax output of the log utilities r(-)P(i > j) = u(i) / (u(i)+u(j)) = exp(r(i)) / (exp(r(i))+exp(r(j))) = softmax(r(i), r(j)) (4)

[0060] LLM Arena with BT Model: One of the classical BT model applications is the LLM chatbot arenas, where multiple LLMs compete against one another based on human feedback through paired comparisons. Here, each LLM functions as a player, and the human -annotated preferences represent the outcome of these game matches. The goal is to assign a single performance score to each LLM player. In some literature, 130 models were compared across more than 1.7 million comparisons, with each model participating in over 26,000 matches on average.

[0061] In such a setting, estimating the values of r(-) is sufficient to achieve the primary goal of evaluating each chatbot's performance. This aligns closely with the original objective of BT model in ranking sports teams. Previous work has shown that, with enough pairwise competition, one can estimate these ability scores well up to a constant additive factor. It is shown that to estimate N scores via random pairwise comparisons, the theoretical lower bound on the number of comparisons is 풪(N log(N))¹, while the best-known methods require 풪(N log³(N)) comparisons.

[0062] Understanding Implicit Assumptions in Reward Modeling with BT Model: In contrast, the application of the BT model to reward modeling is not as straightforward. First, the implications of using the BT model in this context are not well-defined in the literature. For instance, if each prompt-response pair is treated as a player, how do we characterize the stochastic nature of human annotations as the game results? What assumptions need to be made? The answers to these questions were not elaborated in the literature. In the following, an analysis of the underlying assumptions of applying the BT model to preference -based annotation as a game is provided.

[0063] Besides the implicit assumptions, another challenge arises in the application of BT models in reward modeling lies in the comparison sparsity — any given prompt-response is compared to another one only once, resulting in far fewer comparisons than in the arena setting (i.e., much less than the 풪(N log(N)) theoretical lower-bound). To understand how a single model would handle these seemingly different tasks, an overview of the BT model and their variants is provided below.

[0064] Comprehending the BT Model Application in Preference Annotations: In the literature, several prior works have challenged the practical application of the BT model. While the previous analysis of whether the BT model is a good choice focused on the match-ness between the BT model and data. In this part, the basic assumptions of modeling human preference annotations with the BT model is revisited, and the following question is answered: What are the underlying assumptions when we assume the BT model can be used to model the preference annotations?

[0065] The canonical interpretation of how to apply equation (4) in preference-based learning is that: when randomly sampling a human annotator from the population, the human annotator’s choice of the preferred response is proportional to the response’s utility value. Formally, we use x,y±, y2to denote the prompt and responses, the above interpretation implies the following assumptions:

[0066] Assumption 1 (Existence of Deterministic Oracle Utility Values) The (log-)utility value r_{x,y} of any response y given x exists and is deterministic.

[0067] Definition 2 (Individual Annotations under Deterministic Biases) For annotator A the annotation result is deterministic and depends on the comparison of their biased evaluation of the utility values of both responses according to

[0068] Based on the above definitions, the following explicit assumption is provided to derive the BT model in this randomized annotation process:

[0069] Assumption 3 (Logistic Difference Assumption) The b(x, y₁, A) − b(x, y₂, A) is sampled i.i.d. from a standard logistic distribution for all x, y, A:

[0070] Remark 4 (Transitive property of difference) A reader might be (rightfully) worried if this assumption is consistent with transitive property of difference, e.g., when considering multiple comparisons we have to have b(x,y₄, A) − b(x,y₃, A) = b(x,y₄, A) − b(x,y₂, A) + b(x,y₂, A) − b(x,y₃, A) while knowing sum of (independent) logistic distributions is not logistic. One should be aware that the two terms being summed are not independent and the assumption can be achieved by assuming all annotator biases are independently Gumbel distributed with the same scale parameter.

[0071] With this assumption, the BT-type model is achieved.

[0072] Proposition 5 (Modeling Annotations under Logistic Difference Assumption)

[0073] While the above assumption on logistic bias differences lead to the BT-type models, it is also natural to check other assumptions such as the Gaussian difference assumption and its corresponding model:

[0074] Assumption 6 (Gaussian Difference Assumption) The b(x, y₁, A) − b(x, y₂, A) is sampled from a standard Gaussian distribution:b(x,y7, A) — b(x, y2, A) ~ W(0,l) (8)

[0075] Proposition 7 (Modeling Annotations under Gaussian Difference Assumption)where is the CDF of the Gaussian distribution.

[0076] To elaborate implications of those different assumptions and provide an alternative perspective to understand those assumptions, we have the following remark:

[0077] Remark 8 (Assumption on Performance in a Game) Assuming the performance of players A and B in a game is a Gaussian distribution centered at their ability scores uA,uBand the value of performance determines which player wins in the game, then:P(A wins) = P(xa> xb\xa~ N(uA,a2),xb~ JV'(wB, cr2)) = + j erf (^77^) (10)

[0078] Alternatively, when assuming the performance of players A and B in a game is a Gumbel distribution located at their ability scores uA, uB, and the value of performance determines which player wins in the game, then

[0079] While the latter corresponds to the BT model, the former is known to be the Thurstonian model. In the BT model, the tanh(·) rather than the error function erf(·) is used, for the sake of a better empirical fit and mathematical convenience. In both models, we use a game-dependent notation of the variance terms σ², b to characterize the player-agnostic performance variation. Those constants can be further absorbed in the utility values.

[0080] BT Regression: How BT Model Works with Sparse Comparisons: Additionally, estimating a separate r(-) for each prompt-response pair is impractical. In typical LLM alignment scenarios, we often have only / V / 2 comparisons for N pairs, far below the theoretical lower bound for consistent estimation. Furthermore, unlike the arena setting, there is no clear way to predict the score for a new, unseen pair. However,this challenge is not unique to LLM alignment; sports analysts also need to estimate a team’s ability before many competitions or predict the ability of a new team. A common approach in such cases is to use features or covariates, such as team composition or funding status, to predict scores. For LLM, one could have sentence embeddings as such covariates.

[0081] These extensions of the BT model on regression settings were explored shortly after its original introduction: some literature assumed that r(-) could be expressed as a linear combination of covariates. In this scenario, the problem reduces to a classic logistic regression on the difference between two sets of co variates. This allows predicting the score for a new team or prompt-response pair based on its covariates before any comparisons are made. More complex nonlinear models such as spline models have also been explored (De Soete and Winsberg, 1993).

[0082] In practice, reward modeling in ML alignment often employs neural networks, with multilayer perceptrons (MLPs) being a common choice to map embeddings to scores. However, there is currently no theoretical justification for why this particular choice of model and loss function is effective for learning reward functions. From a theoretical standpoint this model variant can be viewed as a nonparametric logistic regression problem with additional structural assumptions on the network and the analysis in the present disclosure builds on this framework. In the following, the asymptotic theory for learning reward models using MLPs and BT loss is established. Table 1 summarizes the key differences between the two usages.Table 1: Comparison of the BT model usage in LLM Arena and Reward Modeling

[0083] Asymptotic Theory on MLP-based BT Regression in Reward Modeling: In preference-based ML alignment, the dataset for use is under the form of ©pref =, where each tuple consists of the prompt,,the corresponding responses y1,iand y2,isampled from the LLM ℓ to be aligned y1,i,y2,i~, and the human-annotated preference h,, being 1 if yiis preferred and -1 otherwise.

[0084] Assume there is a known embedding function (-,-): X X 1 / H [0,1]^. such that there exists an unknown reward functioni-» R, and it can be expressed as t Wi' )) for all x,y. Without loss of generality, it is assumed that the embeddings are scaled within the range [0,1] — otherwise, the embeddings may be scaled into this range. Under this framework, reward modeling reduces to learning the function r. Note that under this formalism there is no need for a comparison to have the same prompt. The effects of using crossprompt and same -prompt will be explored in the following experiments.

[0085] The reward model is denoted as f, parameterized by 6. When there is no confusion, it will be abbreviated as r. The vector of two rewards is denoted as r and the class probability is then softmax (r).Thus, training this reward model reduces to training a classifier with a cross-entropy loss, where the predicted conditional class probabilities are computed as softmax(! / 7x2,y2))) - A special case of a model that preserves anti-symmetry is considered: if we exchange the roles of x1;with x2, y2the reward difference becomes negative.

[0086] It is demonstrated in the present disclosure that an MLP can approximate the probability that the first pair is preferred over the second, and subsequently show that this approach enables the model to learn the underlying reward function effectively. For notational simplicity, let Ψ1(i)and Ψ2(i)represent the embeddings of the i -th pair, where i = 1,Without loss of generality, we assume Ψ(i)∈ [0,1]d. Let the true preference probability for the i -th pair be p®, and the predicted probability be p̂(i)=(1,0) if the first response pair is preferred, and (0,1) otherwise. In this way, the BT model can be reduced to a pairwise classification problem, where the likelihood is given by:L̃CE(p) = -1 / n Σⁿᵢ₌₁(h⁽ⁱ⁾)ᵀ log(p⁽ⁱ⁾), p̂ = argmin L̃CE(p) (12)

[0087] It is unrealistic to assume we can find an NN that actually attends the global minimum, the difference between the fitted NN and the global minimum is denoted as

[0088] The truncated KL risk following Bos and Schmidt-Hieber is considered to overcome the divergence problem of KL risk.

[0089] Definition 9 (Truncated KL risk) The B — truncated KL risk for a probability estimator p Po, P) = E [pj min B, logy)] (14)

[0090] Theorem 10 is proposed in the present disclosure to establish that, with regularity assumptions on the true reward function, an MLP reward model can accurately predict preference probabilities, as measured by truncated KL risk.

[0091] Theorem 10 (Truncated KL risk bound, informal) Suppose the true utility function induced probability of preference satisfies smoothness and regularity assumptions detailed in Assumption 21 with smoothness characterized by constant β and regularity characterised by constant α with dimension of embedding being d. Let p be an estimator from the family of MLP satisfying regularity assumptions detailedin Assumption 18 with depth L. Define <pn' =sufficiently large n, there exists constants C, C" such that when Δn(p̂, p0) ≤ C″BφnLlog2(n) then RB(p0, p̂) ≤ C′BφnLlog2(n), where a ≲ b means there exists some constant C s.t. a ≤ Cb and a ≍ b means a ≲ b and b ≲ a.

[0092] A detailed formal statement and proof is omitted in this specification.

[0093] Corollary 11 (Connecting probability to reward) Given that (√a − √b)2= (a − b)2 / (√a + √b)2, it can be demonstrated that in a large subset of the embedding space,where p is a probability between p0and p, the second line is due to mean value theorem. This indicates that the comparison should be between those pairs that are relatively close in reward to avoid diverging behavior of the logit function. Formal proofs and detailed theoretical analyses are omitted in this specification.Rethinking Reward Modeling Objectives in ML Alignment

[0094] In the present disclosure, a first asymptotic theory for neural network-based BT regression in preference -based reward modeling is introduced, which includes establishing the first risk bound for BT model reward estimation in the context of ML alignment.

[0095] Practical implementation of the BT model poses several requirements including the paired data, the specially designed anti-symmetric model structure, and the inherent assumptions of the BT model itself. This leads a question that whether it is possible to have alternative approaches to reward modeling. Inspired by the above evaluation of the BT model, it is recognized in the present disclosure that although the BT model can serve as the reward model through its log-likelihood loss and anti-symmetric structure, some formal elements for the BT model are indispensable and can be simplified. For example, the BT model may not need to predict comparison probabilities accurately, but rather to provide a reliable signal for ranking ML outputs at inference.

[0096] In classical classification problems, accuracy is often optimized indirectly using losses like crossentropy. This hints applying techniques for improving prediction accuracy to improve reward modeling. While the BT model uses cross-entropy loss and an antisymmetric structure, it is recognized in the present disclosure that a simple classification-based algorithm using a classification accuracy as a surrogate can also serve as the reward model in ML alignment. Details have been discussed above and will be omitted in this part.

[0097] Moreover, noise from annotators is analyzed and discussed. Generally, observed ordering (i.e., annotated ordering) is often subject to noise from human annotators. Drawing on insights from the psychological bottleneck literature, it is reasonable to assume that when the true underlying scores of two responses are similar, it becomes more difficult for annotators to distinguish between them. Formally, an assumption of imperfect preference annotation in approximating true scores is provided below.

[0098] Assumption 12 (Imperfect Preference Annotation in Approximating True Scores) The true utility difference is denoted as Δr := |r(x1, y1) − r(x2,y2)|, and the annotator function h(x1, x2, y1, y2) provides feedback that probabilistically aligns with the oracle utility r(x, y). It is assumed that it is harder for the annotators to assign correctly when the reward difference between two pairs is Ar according to:P( / i(x1,x2,y1,y2)(r(x1,y1) - r(x2,y2)) > 0|dr) = f(dr) (17) where r is the unknown oracle utility function, and ( (•) is any monotonic increasing function to [0.5,1].

[0099] With those noisy annotations, an order consistent with the noisy ordering is considered.

[0100] Definition 13 (Order Consistency) The training loss over an ordering model H may be defined as:Loc(f̂) = Ex₁,x₂,y₁,y₂,hퟙ[h = Ĥ] (18)

[0101] That is, the probability that a reward model ordering agrees with annotation.

[0102] Moreover, this definition is shown with the following proposition that minimizing this loss would help achieve order consistency with the true reward function:

[0103] Proposition 14 (Lower bound on population level order consistency) Suppose a learned model H achieves objective equation (1) up to 1 — 6e error for some small 0 < 6 < 1 and e < 3 / 20, i.e.,

[0104] Then, with probability at least 1 − δ over Δr := |r(x1,y1) − r(x2,y2)| for any given Δr the order consistency of f̂ with respect to the oracle utility r(x, y) is bounded below by:

[0105] Further if we assume that with probability at least 1 − κ, that ξ(Δr) ≥ √(ε² + 1) − 3ε + ε, we have Ex₁,x₂,y₁,y₂~ℓ(x)[ퟙ[Ĥ·[r(x1,y1) − r(x2,y2)] > 0)] ≥ 1 − 4ε − κ − δ (21)Rethinking the Preference Annotation Process for Reward Modeling

[0106] In the present disclosure, an analysis of preference annotation process for reward modeling is conducted, which includes an analysis of cross-prompt pairs and effects of diverse annotations on the annotation quality.

[0107] In both BT and classification-based reward modeling, there is no theoretical requirement to limit comparisons to the same prompts. For classification models, this is straightforward, as they do not rely on paired data at all. Similarly, in traditional BT applications, random pairwise comparisons among players are common. This further motivates investigation into how randomized comparisons across different prompts (i.e., crossprompt pairs) affect reward modeling performance. To further motivate the usage of cross-prompt comparison, the following notation is introduced on annotation quality and analysis as a case study under a Gaussian assumption on score distributions. In equation (17), a special case of / (■) = <?( / ?(•)) is considered where the annotators’ ability is characterized by p. When? = 0, random annotations are obtained:0.5 (22)

[0108] and when? -» oo, perfect annotations are obtained:

[0109] In a nutshell, the annotators abilities and the differences among prompt-response pairs together determine how much preference are correctly labeled in annotation. In the following, a special case when two responses of a prompt x are randomly sampled from a single LLM f is shown.

[0110] Example 1 (Annotation Quality under Gaussian Score) When data for pairwise annotation is generated through random sampling of two responses yx,y2~ (x), it is further assumed that the utility of those two responses are sampled from a Gaussian distribution with variance ffx, i.e., y ~ t?(x), r(x,y) ~ N fix, a2). Then the annotation quality Qpaircan be defined as the averaged annotation order consistency

[0111] where TX= ff(P\r x,yP) — r(x, y2)|) is a random variable (over y1,y2) and the probability density function of TXis

[0112] The derivation is omitted in this specification. In this special case, it is easy to get some numerical results: when β2σx2= 1, we have Qpair≈ 0.6749 e.g., roughly 67.5% of data are correctly labeled by annotators. Similarly, when β2σx2= 2 Qpair≈ 0.7251; when β2σx2= 4, Qpair≈ 0.7781; when β2σx2= 10, Qpair≈ 0.8428. This suggests that the effect of better annotators and the effect of response utility diversity are always coupled: in order to improve data quality, we may either improve the ability of annotators or further diversify the generation utilities — as both of those parameters control the annotation quality. Thus, it is demonstrated in the present disclosure that, determining the set of annotated rewards based on a threshold of variances (i.e., diversity) among the set of annotated rewards is beneficial for annotation quality.

[0113] Next, it is demonstrated in the present disclosure that cross-prompt comparison in this example can be an effective practice to increase response utility diversity, and show that cross-prompt annotation can improve annotation quality.

[0114] It is demonstrated in the present disclosure that cross-prompt annotation improves quality under Gaussian score. Specifically, when considering multiple prompts xt, i = 1,2,..., N, the corresponding responses are denoted asand scores are denoted as r(xi, yi) ~In the following, it is shown that cross-prompt annotation can improve annotation quality.

[0115] Proposition 15 (Cross-Prompt Comparisons Increase Utility Diversity) When data for pairwise annotation is generated through random sampling of two responses y1,y2~ ℓ(x), and the utility of those two responses are sampled from a Gaussian distribution with variance a2, i.e., y ~ ℓ(x), rx,y~ 풩(μx, σx2), when there are multiple prompts x, we haveExEy₁,y₂|x[|rx,y₁− rx,y₂|] ≤ Ex₁,x₂Ey₁|x₁,y₂|x₂[|rx₁,y₁− rx₂,y₂|] (26)

[0116] The proof is omitted in this specification. This gives an intuitive result that, in expectation, the reward differences between cross-prompt comparisons are larger than the reward differences between promptresponse pairs sharing the same prompt. More generally, cross-prompt comparisons improve data quality when the utility distribution of different randomly sampled responses given a single prompt is unimodal and symmetric (e.g., Gaussian).

[0117] Theorem 16 (Cross-Prompt Annotation Improves Annotation Quality) When data for pairwise annotation is generated through random sampling of two responses y1,y2~ ℓ(x), and the utility of those two responses are sampled from a location-scale family with probability density function gx(x) = f((x − μx) / σx) for f being unimodal and symmetric to 0. For any ξ: ℝ+→ [1 / 2,1], first order differentiable, monotone increasing and concave we have

[0118] Theorem 16 highlights that cross-prompt comparisons improve annotation quality for a broad class of utility distributions and (. The proof is omitted in this specification.Related Experiments

[0119] In the present disclosure, various experiments are conducted to validate the proposed solutions according to the present disclosure. In some experiments, results indicate that classification-based reward models not only perform better than the BT reward models but also offer greater flexibility by allowing the use of diverseoff-the-shelf machine learning algorithms. This makes them a competent alternative to BT models in reward modeling.

[0120] In some experiments, results show that as annotation error rates increase, the classification models exhibit greater robustness compared to the BT models, experiencing smaller performance drops. On the other hand, the BT models outperform classification models when annotation quality is high (i.e., less than 10% wrong labels). The results indicate that the classification models consistently outperform BT models, not only delivering superior performance with the same number of annotations but also demonstrating more consistent improvements as the number of annotations increases.

[0121] In some experiments, results show that cross-prompt comparisons outperform same -prompt comparisons, offering substantial improvements in reward model performance. The results indicate that crossprompt comparisons are essential when responses for a single prompt lack diversity, and the superiority of crossprompt comparison is reflected in its general capability regardless of response diversities — the performance achieved by cross-prompt annotations is more stable and has lower dependence of the response diversities.

[0122] Additionally, results show that in practice, when randomly selecting two responses for a single prompt to be annotated, those pairs are likely to face the challenge of high response similarity. And this is the case when cross-prompt annotation can be applied to improve performance.

[0123] In a nutshell, the experimental results highlight the advantages of classification-based reward models over traditional BT models, particularly in terms of flexibility and robustness to annotation quality and quantity. While BT models perform better under high-quality annotations, classification models demonstrate superior overall performance and resilience to increasing annotation error rates.

[0124] Additionally, empirical studies on cross-prompt comparisons show it significantly improves reward modeling, especially when responses to the same prompt lack diversity. Through synthetic experiments, it is further demonstrated that the challenge of limited diversity is likely to occur in practice, providing additional justification for exploring this annotation method in future research.Example Process and Device

[0125] FIG. 3 illustrates a flowchart of a method 300 for multi-agent debate in accordance with some example implementations of the present disclosure. The method 300 may be implemented by any suitable device with computing capability, for example, a processor.

[0126] In the method 300, at block 310, a set of prompt-response pairs and a set of annotated rewards corresponding to the set of prompt-response pairs are obtained. A response of the set of prompt-response pairs is generated by a ML model based on a prompt of the set of prompt-response pairs. At block 320, a set of predicted rewards corresponding to the set of prompt-response pairs is determined by using a reward model. The reward model is configured to receive a prompt-response pair and determine a reward for the prompt -response pair to guide alignment of the ML model. At block 330, a training loss is determined based on an order consistency degree between the set of predicted rewards and the set of annotated rewards. The order consistency degree indicates a difference between an annotated preference order of two prompt-response pairs in the set of prompt -response pairs and a predicted preference order of the two prompt-response pairs. At block 340, the reward model is trained based on the training loss.

[0127] In some embodiments, the two prompt-response pairs may comprise a first prompt-response pair and a second prompt-response pair, and a first prompt of the first prompt-response pair and a second prompt of the second prompt-response pair are the same or different.

[0128] In some embodiments, the set of predicted rewards may comprise a first predicted reward corresponding to the first prompt-response pair and a second predicted reward corresponding to the second prompt-response pair, the set of annotated rewards comprise a first annotated reward corresponding to the first prompt-response pair and a second annotated reward corresponding to the second prompt-response pair, and determining the training loss based on the order consistency degree comprises: determining the predicted preference order of the first and second prompt-response pairs based on the first and second predicted rewards; determining the annotated preference order of the first and second prompt-response pairs based on the first and second annotated rewards; and determining the training loss based on an optimization objective that the predicted preference order is consistent with the annotated preference order.

[0129] In some embodiments, the reward model may be a first reward model based on classification, and determining, by using the reward model, the set of predicted rewards corresponding to the set of prompt -response pairs comprises: determining, by using the first reward model, a first classification accuracy of the first promptresponse pair being classified as winning as the first predicted reward; and determining, by using the first reward model, a second classification accuracy of the second prompt-response pair being classified as winning as the second predicted reward; and wherein determining the training loss may comprises: determining the training loss based on an expected probability that the first predicted reward equals the first annotated reward and the second predicted reward equals the second annotated reward, and wherein the first annotated reward is the negative of the second annotated reward.

[0130] In some embodiments, the method 300 may further comprise: determining a union bound of the training loss based on a first expected probability that the first predicted reward equals the first annotated reward and a second expected probability that the second predicted reward equals the second annotated reward, and wherein training the reward model based on the training loss comprises training the reward model by minimizing the union bound of the training loss.

[0131] In some embodiments, the first reward model may be a neural network-based classification model or a tree -based classification model.

[0132] In some embodiments, the reward model may be a second reward model based on a Bradley-Terry model, and determining the training loss may comprises: determining a predicted order representation based on a difference between the first predicted reward and the second predicted reward, to represent the predicted preference order of the first and second prompt-response pairs; and determining the training loss based on the predicted order representation and the annotated preference order of the first and second prompt -response pairs.

[0133] In some embodiments, obtaining the set of annotated rewards may comprise: determining the set of annotated rewards based on a threshold of variances among the set of annotated rewards.

[0134] In some embodiments of the present disclosure, there is provided a non-transitory computer program product, the non-transitory computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by an electronic device to cause the electronic device to perform a method of multi-agent debate. The method comprises: obtaining a setof prompt-response pairs and a set of annotated rewards corresponding to the set of prompt-response pairs, a response of the set of prompt-response pairs being generated by a ML model based on a prompt of the set of prompt -response pairs; determining, by using a reward model, a set of predicted rewards corresponding to the set of prompt-response pairs, wherein the reward model is configured to receive a prompt-response pair and determine a reward for the prompt-response pair to guide alignment of the ML model; determining a training loss based on an order consistency degree between the set of predicted rewards and the set of annotated rewards, the order consistency degree indicating a difference between an annotated preference order of two promptresponse pairs in the set of prompt-response pairs and a predicted preference order of the two prompt-response pairs; and training the reward model based on the training loss. In some implementations of the present disclosure, the method further comprises other steps as described in the present disclosure.

[0135] FIG. 4 illustrates a block diagram of an electronic device 400 in which various embodiments of the present disclosure can be implemented. It would be appreciated that the electronic device 400 shown in FIG.4 is merely for purpose of illustration, without suggesting any limitation to the functions and scopes of the present disclosure in any manner. The electronic device 400 may be used to implement the above method 400. As shown in FIG. 4, the electronic device 400 may be a general-purpose electronic device. The electronic device 400 may at least comprise one or more processors or processing units 410, a memory 420, a storage unit 430, one or more communication units 440, one or more input devices 450, and one or more output devices 460.

[0136] The processing unit 410 may be a physical or virtual processor and can implement various processes based on programs 425 stored in the memory 420. In a multi-processor system, multiple processing units execute computer executable instructions in parallel so as to improve the parallel processing capability of the electronic device 400. The processing unit 410 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0137] The electronic device 400 typically includes various computer storage medium. Such medium can be any medium accessible by the electronic device 400, including, but not limited to, volatile and non-volatile medium, or detachable and non-detachable medium. The memory 420 can be a volatile memory (for example, a register, cache, Random Access Memory (RAM)), a non-volatile memory (such as a Read-Only Memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), or a flash memory), or any combination thereof. The storage unit 430 may be any detachable or non-detachable medium and may include a machine-readable medium such as a memory, flash memory drive, magnetic disk, or another other media, which can be used for storing information and / or data and can be accessed in the electronic device 400.

[0138] The electronic device 400 may further include additional detachable / non-detachable, volatile / non-volatile memory medium. Although not shown in FIG. 4, it is possible to provide a magnetic disk drive for reading from and / or writing into a detachable and non-volatile magnetic disk and an optical disk drive for reading from and / or writing into a detachable non-volatile optical disk. In such cases, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0139] The communication unit 440 communicates with a further electronic device via the communication medium, In addition, the functions of the components in the electronic device 400 can be implemented by a single computing cluster or multiple computing machines that can communicate via communication connections. Therefore, the electronic device 400 can operate in a networked environment using a logicalconnection with one or more other servers, networked personal computers (PCs) or further general network nodes.

[0140] The input device 450 may be one or more of a variety of input devices, such as a mouse, keyboard, tracking ball, voice-input device, and the like. The output device 460 may be one or more of a variety of output devices, such as a display, loudspeaker, printer, and the like. By means of the communication unit 440, the electronic device 400 can further communicate with one or more external devices (not shown) such as the storage devices and display device, with one or more devices enabling the user to interact with the electronic device 400, or any devices (such as a network card, a modem, and the like) enabling the electronic device 400 to communicate with one or more other electronic devices, if required. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0141] In some embodiments, instead of being integrated in a single device, some, or all components of the electronic device 400 may also be arranged in cloud computing architecture. In the cloud computing architecture, the components may be provided remotely and work together to implement the functionalities described in the present disclosure. In some embodiments, cloud computing provides computing, software, data access and storage service, which will not require end users to be aware of the physical locations or configurations of the systems or hardware providing these services. In various embodiments, the cloud computing provides the services via a wide area network (such as Internet) using suitable protocols. For example, a cloud computing provider provides applications over the wide area network, which can be accessed through a web browser or any other computing components. The software or components of the cloud computing architecture and corresponding data may be stored on a server at a remote position. The computing resources in the cloud computing environment may be merged or distributed at locations in a remote data center. Cloud computing infrastructures may provide the services through a shared data center, though they behave as a single access point for the users. Therefore, the cloud computing architectures may be used to provide the components and functionalities described herein from a service provider at a remote location. Alternatively, they may be provided from a conventional server or installed directly or otherwise on a client device.

[0142] The functionalities described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

[0143] Program code for carrying out the methods of the subject matter described herein may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general-purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely or partly on a machine, executed as a stand-alone software package partly on the machine, partly on a remote machine, or entirely on the remote machine or server.

[0144] In the context of this disclosure, a machine-readable medium maybe any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.The machine-readable medium may be a machine-readable signal medium or a machine -readable storage medium. A machine -readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine -readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] Further, while operations are illustrated in a particular order, this should not be understood as requiring that such operations are performed in the particular order shown or in sequential order, or that all illustrated operations are performed to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the subject matter described herein, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single implementation. Rather, various features described in a single implementation may also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0146] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter specified in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

[0147] From the foregoing, it will be appreciated that specific embodiments of the presently disclosed technology have been described herein for purposes of illustration, but that various modifications may be made without deviating from the scope of the disclosure. Accordingly, the presently disclosed technology is not limited except as by the appended claims.

[0148] Embodiments of the subject matter and the functional operations described in the present disclosure can be implemented in various systems, digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible and non-transitory computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine -readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing unit” or “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0149] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0150] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0151] It is intended that the specification, together with the drawings, be considered exemplary only, where exemplary means an example. As used herein, the use of “or” is intended to include “and / or”, unless the context clearly indicates otherwise.

[0152] While the present disclosure contains many specifics, these should not be construed as limitations on the scope of any disclosure or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular disclosures. Certain features that are described in the present disclosure in the context of separate embodiments can also be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0153] Similarly, while operations are illustrated in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. Moreover, the separation of various system components in the embodiments described in the present disclosure should not be understood as requiring such separation in all embodiments. Only a few embodiments and examples are described and other embodiments, enhancements and variations can be made based on what is described and illustrated in the present disclosure.

Claims

CLAIMSWhat is claimed is:

1. A method of machine learning (ML) model alignment, comprising:obtaining a set of prompt-response pairs and a set of annotated rewards corresponding to the set of prompt -response pairs, a response of the set of prompt -response pairs being generated by a ML model based on a prompt of the set of prompt-response pairs;determining, by using a reward model, a set of predicted rewards corresponding to the set of promptresponse pairs, wherein the reward model is configured to receive a prompt-response pair and determine a reward for the prompt-response pair to guide alignment of the ML model;determining a training loss based on an order consistency degree between the set of predicted rewards and the set of annotated rewards, the order consistency degree indicating a difference between an annotated preference order of two prompt-response pairs in the set of prompt-response pairs and a predicted preference order of the two prompt-response pairs; andtraining the reward model based on the training loss.

2. The method of claim 1, wherein the two prompt-response pairs comprise a first prompt-response pair and a second prompt-response pair, and a first prompt of the first prompt-response pair and a second prompt of the second prompt-response pair are the same or different.

3. The method of claim 2, wherein the set of predicted rewards comprise a first predicted reward corresponding to the first prompt-response pair and a second predicted reward corresponding to the second prompt-response pair, the set of annotated rewards comprise a first annotated reward corresponding to the first prompt-response pair and a second annotated reward corresponding to the second prompt-response pair, and determining the training loss based on the order consistency degree comprises:determining the predicted preference order of the first and second prompt-response pairs based on the first and second predicted rewards;determining the annotated preference order of the first and second prompt-response pairs based on the first and second annotated rewards; anddetermining the training loss based on an optimization objective that the predicted preference order is consistent with the annotated preference order.

4. The method of claim 3, wherein the reward model is a first reward model based on classification, and determining, by using the reward model, the set of predicted rewards corresponding to the set of prompt -response pairs comprises:determining, by using the first reward model, a first classification accuracy of the first prompt-response pair being classified as winning as the first predicted reward; anddetermining, by using the first reward model, a second classification accuracy of the second promptresponse pair being classified as winning as the second predicted reward; andwherein determining the training loss comprises:determining the training loss based on an expected probability that the first predicted reward equals the first annotated reward and the second predicted reward equals the second annotated reward, and wherein the first annotated reward is the negative of the second annotated reward.

5. The method of claim 4, wherein the method further comprises:determining a union bound of the training loss based on a first expected probability that the first predicted reward equals the first annotated reward and a second expected probability that the second predicted reward equals the second annotated reward, andwherein training the reward model based on the training loss comprises training the reward model by minimizing the union bound of the training loss.

6. The method of any of claims 3 to 5, wherein the first reward model is a neural network -based classification model or a tree -based classification model.

7. The method of claim 3, wherein the reward model is a second reward model based on a Bradley -Terry model, and determining the training loss comprises:determining a predicted order representation based on a difference between the first predicted reward and the second predicted reward, to represent the predicted preference order of the first and second prompt -response pairs; anddetermining the training loss based on the predicted order representation and the annotated preference order of the first and second prompt-response pairs.

8. The method of any of claims 1 to 7, wherein obtaining the set of annotated rewards comprises: determining the set of annotated rewards based on a threshold of variances among the set of annotated rewards.

9. An electronic device, comprising a computer processor coupled to a computer-readable memory unit, the memory unit comprising instructions that when executed by the computer processor implements a method of machine learning (ML) model alignment, the method comprising:obtaining a set of prompt-response pairs and a set of annotated rewards corresponding to the set of prompt-response pairs, a response of the set of prompt-response pairs being generated by a ML model based on a prompt of the set of prompt-response pairs;determining, by using a reward model, a set of predicted rewards corresponding to the set of promptresponse pairs, wherein the reward model is configured to receive a prompt-response pair and determine a reward for the prompt-response pair to guide alignment of the ML model;determining a training loss based on an order consistency degree between the set of predicted rewards and the set of annotated rewards, the order consistency degree indicating a difference between an annotatedpreference order of two prompt-response pairs in the set of prompt-response pairs and a predicted preference order of the two prompt-response pairs; andtraining the reward model based on the training loss.

10. The device of claim 9, wherein the two prompt-response pairs comprise a first prompt-response pair and a second prompt-response pair, and a first prompt of the first prompt -response pair and a second prompt of the second prompt-response pair are the same or different.

11. The device of claim 10, wherein the set of predicted rewards comprise a first predicted reward corresponding to the first prompt-response pair and a second predicted reward corresponding to the second prompt -response pair, the set of annotated rewards comprise a first annotated reward corresponding to the first prompt-response pair and a second annotated reward corresponding to the second prompt-response pair, and determining the training loss based on the order consistency degree comprises:determining the predicted preference order of the first and second prompt-response pairs based on the first and second predicted rewards;determining the annotated preference order of the first and second prompt-response pairs based on the first and second annotated rewards; anddetermining the training loss based on an optimization objective that the predicted preference order is consistent with the annotated preference order.

12. The device of claim 11, wherein the reward model is a first reward model based on classification, and determining, by using the reward model, the set of predicted rewards corresponding to the set of prompt -response pairs comprises:determining, by using the first reward model, a first classification accuracy of the first prompt-response pair being classified as winning as the first predicted reward; anddetermining, by using the first reward model, a second classification accuracy of the second promptresponse pair being classified as winning as the second predicted reward; andwherein determining the training loss comprises:determining the training loss based on an expected probability that the first predicted reward equals the first annotated reward and the second predicted reward equals the second annotated reward, and wherein the first annotated reward is the negative of the second annotated reward.

13. The device of claim 12, wherein the method further comprises:determining a union bound of the training loss based on a first expected probability that the first predicted reward equals the first annotated reward and a second expected probability that the second predicted reward equals the second annotated reward, andwherein training the reward model based on the training loss comprises training the reward model by minimizing the union bound of the training loss.

14. The device of any of claims 11 to 13, wherein the first reward model is a neural network-based classification model or a tree -based classification model.

15. The device of claim 11, wherein the reward model is a second reward model based on a Bradley-Terry model, and determining the training loss comprises:determining a predicted order representation based on a difference between the first predicted reward and the second predicted reward, to represent the predicted preference order of the first and second prompt -response pairs; anddetermining the training loss based on the predicted order representation and the annotated preference order of the first and second prompt-response pairs.

16. The device of any of claims 9 to 15, wherein obtaining the set of annotated rewards comprises: determining the set of annotated rewards based on a threshold of variances among the set of annotated rewards.

17. A computer program product, the computer program product comprising a non-transitory computer readable storage medium having instructions embodied therewith, the instructions executable by an electronic device to cause the electronic device to perform a method of machine learning (ML) model alignment, the method comprising:obtaining a set of prompt-response pairs and a set of annotated rewards corresponding to the set of prompt -response pairs, a response of the set of prompt-response pairs being generated by a ML model based on a prompt of the set of prompt-response pairs;determining, by using a reward model, a set of predicted rewards corresponding to the set of promptresponse pairs, wherein the reward model is configured to receive a prompt -response pair and determine a reward for the prompt-response pair to guide alignment of the ML model;determining a training loss based on an order consistency degree between the set of predicted rewards and the set of annotated rewards, the order consistency degree indicating a difference between an annotated preference order of two prompt-response pairs in the set of prompt-response pairs and a predicted preference order of the two prompt-response pairs; andtraining the reward model based on the training loss.

18. The computer program product of claim 17, wherein the two prompt -response pairs comprise a first prompt-response pair and a second prompt-response pair, and a first prompt of the first prompt-response pair and a second prompt of the second prompt-response pair are the same or different.

19. The computer program product of claim 18, wherein the set of predicted rewards comprise a first predicted reward corresponding to the first prompt -response pair and a second predicted reward corresponding to the second prompt-response pair, the set of annotated rewards comprise a first annotated reward correspondingto the first prompt-response pair and a second annotated reward corresponding to the second prompt -response pair, and determining the training loss based on the order consistency degree comprises:determining the predicted preference order of the first and second prompt-response pairs based on the first and second predicted rewards;determining the annotated preference order of the first and second prompt-response pairs based on the first and second annotated rewards; anddetermining the training loss based on an optimization objective that the predicted preference order is consistent with the annotated preference order.

20. The computer program product of claim 19, wherein the reward model is a first reward model based on classification, and determining, by using the reward model, the set of predicted rewards corresponding to the set of prompt -response pairs comprises:determining, by using the first reward model, a first classification accuracy of the first prompt-response pair being classified as winning as the first predicted reward; anddetermining, by using the first reward model, a second classification accuracy of the second promptresponse pair being classified as winning as the second predicted reward; andwherein determining the training loss comprises:determining the training loss based on an expected probability that the first predicted reward equals the first annotated reward and the second predicted reward equals the second annotated reward, and wherein the first annotated reward is the negative of the second annotated reward.

Images