Reliability assessment of machine learning model outputs

Abstract

The present disclosure relates to a computer-implemented method for assessing reliability of outputs from machine learning models, comprising: - generating, by a first machine learning model, a sequence of items, wherein each item is selected from a corresponding set of candidate items based on at least one selection criterion, wherein each candidate item of a set of candidate items is associated with probability information; and - performing, by an assessment model, a factuality check on the generated sequence of items based on the probability information of the sets of candidate items associated with the items of the sequence of items.

Description

RELIABILITY ASSESSMENT OF MACHINE LEARNING MODEL OUTPUTSFIELD

[0001] The present disclosure relates to a computer-implemented method for assessing reliability of outputs from machine learning models, a computer program comprising instructions for carrying out the computer-implemented method for assessing reliability of outputs from machine learning models, a computer-readable storage medium comprising instructions for carrying out the computer-implemented method for assessing reliability of outputs from machine learning models, and a data processing system for assessing reliability of outputs from machine learning models. More particularly, the present disclosure relates to LLM hallucination detection using probability distribution arithmetic.BACKGROUND

[0002] Large Language Models (LLMs) are a category of machine learning models capable of generating text in response to a given input, commonly referred to as a prompt. These models are fine-tuned to produce outputs that are both high in probability and have good human readability. For example, in response to the prompt “What is the first month of the year?”, an LLM is likely to generate outputs such as “January” or “The first month of the year is January.” This capability allows LLMs to provide answers, explanations, or other contextually relevant responses to a wide variety of queries.

[0003] LLMs generate text outputs by processing and concatenating smaller units known as tokens. Tokens can represent single characters, multi -character strings, or segments of words. A model predicts the next token in a sequence based on the context provided by preceding tokens. Rather than selecting a single token outright, LLMs calculate a probability distribution over its entire vocabulary, assigning a probability to each possible token. This distribution reflects the model’s confidence in itspredictions. For instance, if one token has a significantly higher probability than the rest, the model can be considered confident in that choice. Conversely, if probabilities are distributed more evenly, the model exhibits uncertainty. These probability distributions play a crucial role in determining the sequence of generated tokens.

[0004] Despite their capabilities, LLMs are prone to errors, including generating factually incorrect or fabricated information, often referred to as “hallucinations.” Recent studies have highlighted the frequency of such errors, with estimates indicating that the LLM underpinning ChatGPT fabricates facts approximately 3% of the time, while another LLM developed by Google is estimated to fabricate facts up to 27% of the time. These inaccuracies can pose significant risks to individuals and businesses relying on LLM-generated information. Consequently, there is a need for technologies capable of detecting and mitigating LLM hallucinations to enhance their reliability and utility in practical applications.SUMMARY

[0005] In light of the above, a computer-implemented method for assessing reliability of outputs from machine learning models, a computer program comprising instructions for carrying out the computer-implemented method for assessing reliability of outputs from machine learning models, a computer-readable storage medium comprising instructions for carrying out the computer-implemented method for assessing reliability of outputs from machine learning models, and a data processing system for assessing reliability of outputs from machine learning models are provided.

[0006] It is an object of the present disclosure to assess the reliability of outputs generated by machine learning models. Another object of the present disclosure is to enhance the efficiency and reliability of machine learning models by providing a dedicated processor architecture or extending the capabilities of commonly used processor architectures through the use of dedicated software that is able to operate and perform arithmetic directly on distribution representations.

[0007] The objects are solved by the features of the independent claims. Preferred embodiments are defined in the dependent claims.

[0008] According to an independent aspect of the present disclosure, a computer- implemented method for assessing reliability of outputs from machine learning models is provided. The method includes generating, by a first machine learning model, a sequence of items, wherein each item is selected from a corresponding set of candidate items based on at least one selection criterion, wherein each candidate item of a set of candidate items is associated with probability information; and performing, by an assessment model, a factuality check on the generated sequence of items based on the probability information of the sets of candidate items associated with the items of the sequence of items.

[0009] According to some embodiments, which can be combined with other embodiments described herein, the first machine learning model is a Large Language Model (LLM).

[0010] According to some embodiments, which can be combined with other embodiments described herein, the sequence of items generated by the first machine learning model, such as the Large Language Model, is a sequence of tokens, e.g., text tokens.

[0011] According to some embodiments, which can be combined with other embodiments described herein, the factuality check includes determining at least one quality parameter for the generated sequence of items. For example, the at least one quality parameter could be a sequence coherence as investigated by research teams like Google DeepMind. This parameter assesses the factuality of items within the sequence, ensuring that the generated items align with the intended narrative or task. Other quality parameters include, but are not limited to, fluency, factual accuracy, and diversity.

[0012] According to some embodiments, which can be combined with other embodiments described herein, the factuality check includes determining a factuality score for the generated sequence of items. For example, the factuality score could be defined as a metric used to evaluate the accuracy and factual integrity of the generated sequence of items. A factuality score of 1 could indicate that the entire generated sequence is factually accurate and aligns perfectly with the reference truth. A factuality score of 0 could indicate that the sequence contains significant factual inaccuracies or is completely misaligned with the reference truth.

[0013] According to some embodiments, which can be combined with other embodiments described herein, the assessment model includes, or is, a second machine learning model.

[0014] According to some embodiments, which can be combined with other embodiments described herein, the second machine learning model is configured to quantify correctness of the generated sequence of items.

[0015] According to some embodiments, which can be combined with other embodiments described herein, the second machine learning model uses information, such as the probability information, about the set of candidate items as input and returns a factuality score as output.

[0016] According to some embodiments, which can be combined with other embodiments described herein, the factuality score is between 0 and 1. A factuality score of 1 could indicate that the entire generated sequence is factually accurate and aligns perfectly with the reference truth. A factuality score of 0 could indicate that the sequence contains significant factual inaccuracies or is completely misaligned with the reference truth.

[0017] According to some embodiments, which can be combined with other embodiments described herein, the second machine learning model is trained based on pairs of inputs and outputs to the first machine learning model that have been given afactuality score. In some embodiments, the input to the second machine learning model is sequences of sets of candidate items and associated probability values, where each set of candidate items and associated probability values is the output of the first machine learning model corresponding to the output the first machine learning model generates when generating a single token. For example, the second machine learning model can be trained with input / output pairs that look like (243: probability 0.1) or (input={(l) ["house" : probability 0.1, "dog" : probability 0.6, ...], (2) ["ran" : probability 0.5, "exploded": probability 0.01], (3) [], ...}, output = {FACTUAL}).

[0018] According to some embodiments, which can be combined with other embodiments described herein, the pairs of inputs and outputs are prompt / response pairs that have been given a factuality score. For example, the pairs of inputs and outputs to the second machine learning model can be pairs of inputs to, and outputs from, the first machine learning model, where the outputs from the first machine learning model are sets comprising one or more tokens each paired with a probability, and where the output of the first machine learning model, over multiple output tokens, have been given a factuality score.

[0019] According to some embodiments, which can be combined with other embodiments described herein, each set of candidate items output by the first machine learning model represents a probability distribution of candidate items, and the method further includes generating a distribution representation from the probability distribution of each set of candidate items. The distribution representation can be a representation of probability mass / density functions / distributions on which a processor unit can operate. In one embodiment, the distribution representations enable commonly utilized processor architectures to operate on the probability distributions of candidate items. Non-limiting exemplary methods for generating such representations from a data set are explained in WO 2022 / 248719 Al and WO 2022 / 248714 Al.

[0020] According to some embodiments, which can be combined with other embodiments described herein, the distribution representation includes candidate item position parameters and associated candidate item probability parameters. In particular, a representation generation algorithm can be used to generate the distribution representation of the probability distributions of candidate items. In preferred embodiments, the representation generation algorithm can be configured to generate the distribution representation according to the position parameter and the probability parameter, wherein each position of the distribution representation is represented by a Dirac delta associated with a corresponding probability mass. In other words, each real number in the domain of a distribution may be described by a Dirac delta with some probability mass located at the value of the real number. As a result, the distribution representation is in the form of a sum of Dirac deltas.

[0021] According to some embodiments, which can be combined with other embodiments described herein, the distribution representation is a Telescoping Torques Representation (TTR). The Telescoping Torques Representation (TTR) builds on Dirac mixture representation and is based on the intuition that the mean value of a set of samples splits the samples into two sets with equal products of the distance of the sample from the mean and the sample probability. This allows the representation to capture the effects of unlikely, yet high-value samples even at small representation sizes.

[0022] According to some embodiments, which can be combined with other embodiments described herein, the distribution representation is a Joint Telescoping Torques Representation (jTTR). The Joint Telescoping Torques Representation (jTTR) builds on TTR and further tracks correlations between distributional variables. Briefly summarized, jTTR effectively tracks correlations between distributional variables by tracking the origins of each distributional variable from other distributional variables. This capability allows jTTR to accurately propagate uncertainty through many arithmetic operations where TTR falls short. For instance, a conversion algorithm can be used to take in a sequence of multidimensional samples(each of the same dimensionality) and output a jTTR for each variable corresponding to a dimension.

[0023] According to some embodiments, which can be combined with other embodiments described herein, the method further includes arranging or re-arranging the candidate items of the set of candidate items based on at least one ordering criterion before performing the factuality check.

[0024] According to some embodiments, which can be combined with other embodiments described herein, the at least one ordering criterion relates to the probabilities associated with the candidate items.

[0025] According to some embodiments, which can be combined with other embodiments described herein, arranging or re-arranging the candidate items of the set of candidate items based on the at least one ordering criterion includes arranging or rearranging the candidate items in descending order of probability, starting with the candidate item having the highest probability and proceeding to the candidate item with the lowest probability.

[0026] Alternatively, arranging or re-arranging the candidate items of the set of candidate items based on the at least one ordering criterion may include arranging or re-arranging the candidate items in ascending order of probability, starting with the candidate item having the lowest probability and proceeding to the candidate item with the highest probability.

[0027] According to some embodiments, which can be combined with other embodiments described herein, arranging or re-arranging of the candidate items of the set of candidate items includes mapping a default order of the candidate items of the set of candidate items to the distribution representation, such a Dirac delta mixture.

[0028] According to some embodiments, which can be combined with other embodiments described herein, the at least one ordering criterion relates to a position of the candidate items on a number line. In particular, the criterion for ordering maybe based on how the tokens (items) are mapped to real-valued numbers (positions on the number line). The ordering of items / tokens can be designed to ensure that the resulting distribution is as smooth as possible. One possible way to achieve this quantitatively is to determine the ordering that maximizes the Shannon entropy of the rearranged distribution of positions (items / tokens) and their associated probabilities.

[0029] According to another independent aspect of the present disclosure, a computer program is provided. The computer program has instructions which, when the program is executed by a computer, cause the computer to carry out the method for assessing reliability of outputs from machine learning models according to the embodiments described in this document.

[0030] According to another independent aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has instructions which, when the program is executed by a computer, cause the computer to carry out the method for assessing reliability of outputs from machine learning models according to the embodiments described in this document.

[0031] Preferably, the computer-readable storage medium is a non-transitory computer-readable storage (or memory) medium. The term “non-transitory computer- readable storage (or memory) medium” includes, but is not limited to, magnetic or optical media, e.g., disks, hard drives, or DVDs / CD-ROMs, volatile or non-volatile media such as RAM (e g., SDRAM, DDR, RDRAM, SRAM, etc ), ROM, etc. The non-transitory computer-readable storage (or memory) medium may include other types of non-transitory memory as well or combinations thereof.

[0032] According to another independent aspect of the present disclosure, a data processing system for assessing reliability of outputs from machine learning models is provided. The data processing system includes one or more processors configured to generate, using a machine learning model, a sequence of items, wherein each item is selected from a corresponding set of candidate items based on at least one selection criterion, wherein each candidate item of a set of candidate items is associated withprobability information; and perform, using an assessment model, a factuality check on the generated sequence of items based on the probability information of the sets of candidate items associated with the items of the sequence of items.

[0033] According to some embodiments, which can be combined with other embodiments described herein, the one or more processors of the data processing system are configured to perform the computer-implemented method for assessing reliability of outputs from machine learning models according to the embodiments described in this document.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] So that the manner in which the above recited features of the present disclosure can be understood in detail, a more particular description of the disclosure, briefly summarized above, may be had by reference to embodiments. The accompanying drawings relate to embodiments of the disclosure and are described in the following:FIG. 1 illustrates a schematic view of a conventional processor unit;FIG. 2 illustrates a schematic view of a processor unit according to embodiments of the present disclosure;FIG. 3 shows a per-token distribution of a Large Language Model;FIG. 4 shows a flowchart of a method for assessing reliability of outputs from machine learning models according to embodiments of the present disclosure;FIG. 5 illustrates the construction of an uncertainty representation using Telescoping Torques Representation (TTR) according to embodiments of the present disclosure;FIG. 6 illustrates a bit-level layout of an uncertainty representation according to embodiments of the present disclosure; andFIG. 7 shows a re-arranged per-token distribution of a Large Language Model according to embodiments of the present disclosure.DETAILED DESCRIPTION OF EMBODIMENTS

[0035] Reference will now be made in detail to the various embodiments of the disclosure, one or more examples of which are illustrated in the figures. Within the following description of the drawings, the same reference numbers refer to same components. Generally, only the differences with respect to individual embodiments are described. Each example is provided by way of explanation of the disclosure and is not meant as a limitation of the disclosure. Further, features illustrated or described as part of one embodiment can be used on or in conjunction with other embodiments to yield yet a further embodiment. It is intended that the description includes such modifications and variations.

[0036] This document includes references to “one embodiment” or “an embodiment”. The appearances of the phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0037] The term “comprising” is open-ended. As used in the appended claims, this term does not foreclose additional structure or steps.

[0038] Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that the units / circuits / components include structure (e.g., circuitry) that performs those task or tasks during operation. As such, the unit / circuit / component can be said to be configured to perform the task even when the specified unit / circuit / component is not currently operational (e.g., is not on). Theunits / circuits / components used with the “configured to” language include hardware for example, circuits, memory storing program instructions executable to implement the operation, etc. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software and / or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue.

[0039] As used herein, the terms “first”, “second”, etc. are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.).

[0040] As used herein, the term “based on” is used to describe one or more factors that affect a determination. This term does not foreclose additional factors that may affect a determination. That is, a determination may be solely based on those factors or based, at least in part, on those factors. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A may be determined based solely on B.

[0041] As used herein, the term “uncertainty tracking” (or uncertainty propagation) is the process of monitoring how uncertainties in data affect the outputs of sequences of arithmetic operations that act on the data. For a given data item, such as a measurement, the term uncertainty quantification (UQ) is usually used to refer to the process of quantifying the uncertainty of that single data item.

[0042] FIG. 1 illustrates a schematic view of a conventional processor unit 100.

[0043] A conventional processor unit 100 takes as input integer and floating-point values and performs operations (e g., ADD, SUBSTRACT, MULTIPLY, XOR, etc.) on these data. The results of these operations are other integers or floating-point values.

[0044] Uncertainties in data, such as those used in machine learning applications, are not uncommon, but the conventional processor unit 100 can only deal with integerand floating-point values which are certain and unambiguous. Therefore, the software running over the conventional processor unit 100 eliminates the uncertainties by averaging the variations across readings into a single number, to get the data into a form that the conventional processor unit 100 can handle, before feeding the averaged data into subsequent steps.

[0045] In one exemplary embodiment of the present disclosure, the conventional processor unit 100 according to FIG. 1 can be extended without any hardware reconfiguration using the concept of representation described later, particularly the so- called Telescoping Torques Representation (TTR). TTR enables the embodiments of the present disclosure on the conventional processor unit 100 without any hardware re-configuration.

[0046] FIG. 2 illustrates a schematic view of a processor unit 200 according to embodiments of the present disclosure, which is capable of processing and operating on probability distributions. The processor unit 200 is a dedicated architecture, such as the architecture disclosed in WO 2022 / 248719 Al operating on a representation according to WO 2022 / 248719 Al and WO 2022 / 248714 Al.

[0047] The processor unit 200 of the present disclosure can take in as input the same kinds of integer and floating-point values and perform the same kinds of operations (e g., ADD, SUBSTRACT, MULTIPLY, XOR, etc.) as the conventional processor unit. In addition to that, the values taken as input include additional information about distributions associated with each value, and the processor unit 200 may also, in some embodiments, enable a number of additional operations that act on the distribution information (e g., BAYESLAPLACE, DISTFROMSAMPLES, etc ).

[0048] In some embodiments, the processor unit 200 includes additional and / or specifically configured hardware to enable the uncertainty tracking of the present disclosure. For example, the processor unit 200 may include a floating-point register file. The floating-point register file may include a first register file for storing particle data items and a second register file for storing distribution data. The distribution datarepresents distributions (e.g., TTR) that encode the uncertainty associated with the corresponding particle data items in the first register file.

[0049] A first arithmetic logic unit (ALU) can be configured to perform arithmetic operations on particle data items selected from the first register file, while a second arithmetic logic unit (ALU) can be configured to perform arithmetic operations on distribution data selected from the second register file. For example, the first arithmetic logic unit may execute an arithmetic operation (e.g., addition, subtraction, multiplication, or division) on two floating-point particle data items selected from the first register file and output the result. Simultaneously, the second arithmetic logic unit may execute the same arithmetic operation on two corresponding distribution data items selected from the second register file, which are associated with the particle data items used in the first arithmetic operation. The second arithmetic logic unit then outputs the result.

[0050] Notably, the arithmetic operation applied to the distribution data in the second register file is identical to the arithmetic operation applied to the particle data items in the first register file. Consequently, the output of the second arithmetic logic unit represents updated distributional data that quantifies the uncertainty associated with the result of the arithmetic operation performed on the particle data items.

[0051] A detailed description of an exemplary embodiment of the processor unit 200 is provided in WO 2022 / 248719 Al.

[0052] In some embodiments, the processor unit 200 of the present disclosure can be implemented as a virtualization layer. By moving the uncertainty tracking into a virtualization layer below the user’s software, applications can be significantly simplified. Furthermore, the implementation of the processor unit 200 as a virtualization layer means users can benefit from advances in the technology as it evolves, without needing to rewrite the user’s software.

[0053] The embodiments of the present disclosure utilize the processor unit 200 to process distributions, such as per-token distributions, to assess the accuracy of the output generated by a machine learning model, as will be further explained below. While the description is provided with reference to a Large Language Model (LLM), it should be understood that the present disclosure is not limited to LLMs and can be applied to other machine learning models capable of generating and outputting a sequence of items associated to probability information, such as text tokens.

[0054] Large Language Models (LLMs) are a type of machine learning model designed to process and generate human-like text. These models are trained on datasets containing text from a variety of sources, allowing them to learn patterns, relationships, and contextual nuances in language. When presented with an input, often called a prompt, an LLM generates a sequence of text that is both highly likely based on patterns it has learned during training and has good human readability. For example, if given the prompt “What is the capital of France?” the model is likely to generate a response such as “Paris” or “The capital of France is Paris.”

[0055] The process of generating text involves several stages, with one component being the computation of what is known as a per-token distribution. In particular, LLMs break down text into smaller units called tokens. A token can represent a single character, a sequence of characters, or even parts of a word, depending on how the model’s tokenizer processes the input text.

[0056] When an LLM generates a response, it does not immediately select a single token to append to the sequence. Instead, it calculates a set of scores for every token in its vocabulary, producing what is referred to as logits. These logits are raw, unnormalized values that represent the model’s assessment of how likely each token is to follow the sequence so far. To convert these logits into probabilities, the model applies a mathematical function called softmax. This operation normalizes the logits, producing a probability distribution over all possible tokens. This probabilitydistribution is the per-token distribution, which represents the likelihood of each token being the next in the sequence.

[0057] FIG. 3 shows an exemplary per-token distribution. The vocabulary index of the per-token distribution refers to the specific position or numerical identifier assigned to each token in the model’s vocabulary. The vocabulary is a predefined set of all possible tokens that the model can generate or interpret, and each token is associated with a unique index for referencing. The per-token distribution assigns a probability value to each index in the vocabulary. The distribution is essentially a vector of size N, where each element corresponds to the probability of the token associated with that vocabulary index being the next token in the sequence.

[0058] The per-token distribution is central to the generation process, as it determines the model’s level of certainty about its predictions. If the distribution is sharply peaked, with one token having a high probability and the rest having near-zero probabilities, it indicates that the model is confident in its choice. Conversely, if the distribution is flatter, with several tokens having comparable probabilities, it reflects uncertainty in the model’s next token prediction. This variability in the distribution can influence the diversity and consistency of the text the model generates.

[0059] Once the per-token distribution is computed, the model uses it to select the next token based on at least one selection criterion. The specific token selection process can vary depending on the application. The model may pick the token with the highest probability for deterministic outputs, or it might sample from the distribution for more varied and creative responses. Adjustments to the distribution, such as applying temperature scaling to make it sharper or flatter, can further refine the nature of the generated text.

[0060] In essence, the per-token distribution is a probabilistic representation of all possible continuations at each step of the generation process. It reflects the model’s understanding of context and provides the flexibility to produce both predictable and diverse outputs. By continually computing and updating this distribution for eachtoken in the sequence, LLMs are able to construct coherent and contextually relevant text, demonstrating their capacity for complex language understanding and generation.

[0061] The present disclosure introduces an assessment model, such as a machine learning model, configured to evaluate the factual accuracy of outputs generated by a Large Language Model (LLM) in response to a given prompt. The assessment model achieves this by analyzing the probability distributions associated with the individual tokens generated by the LLM, particularly the per-token distributions. By utilizing the sequence of these distributions, the present disclosure determines whether the LLM is more likely (or less likely) to produce an output that is factually incorrect.

[0062] To provide this functionality, the present disclosure could take advantage of the computational capabilities of the processor unit illustrated in FIG. 2, which can perform operations directly on probability distributions. This approach addresses a significant inefficiency in current methods, such as Monte Carlo-based beam search, which are computationally expensive and suboptimal for real-time or large-scale applications. The method disclosed herein offers a practical and scalable solution for evaluating the veracity of LLM-generated outputs, overcoming the limitations and inefficiencies of existing approaches, such as Monte Carlo-based beam search.

[0063] FIG. 4 shows a flowchart of a method 400 for assessing reliability of outputs from machine learning models according to embodiments of the present disclosure.

[0064] The method 400 includes generating, by a first machine learning model, a sequence of items, wherein each item is selected from a corresponding set of candidate items based on at least one selection criterion, wherein each candidate item of a set of candidate items is associated with probability information (Block 410); and performing, by an assessment model, a factuality check on the generated sequence of items based on the probability information of the sets of candidate items associated with the items of the sequence of items (Block 420).

[0065] The first machine learning model can be a Large Language Model (LLM) as described above, with the generated sequence of items corresponding to a sequence of text tokens. However, it should be understood that the present disclosure is not limited to LLMs and can be applied to other machine learning models capable of generating and outputting a sequence of items, such as text tokens.

[0066] In some embodiments, the assessment model includes, or is, a second machine learning model, particularly a trained machine learning model, configured to quantify correctness of the generated sequence of items. For example, the second machine learning model could use information about the set of candidate items as input and return a factuality score as output, wherein the factuality score is between 0 and 1. A factuality score of 1 could indicate that the entire generated sequence is factually accurate and aligns perfectly with the reference factuality or reference truth. A factuality score of 0 could indicate that the sequence contains significant factual inaccuracies or is completely misaligned with the reference factuality or reference truth.

[0067] In some embodiments, the second machine learning model is trained based on input / output pairs that have been given a factuality score, wherein the input corresponds to candidate items and associated probability values, and the output corresponds to generated sequences of items. For example, the input / output pairs could be prompt / response pairs that have been given a factuality score.

[0068] Each candidate item within a given set of candidate items is associated with corresponding probability information, such that the set collectively represents the likelihood of each token being the next in the sequence. In other words, each set of candidate items represents a probability distribution over potential candidates for the next item in the sequence (per-token distribution, see FIG. 3). The embodiments of the present disclosure utilize the processor unit of FIG. 2 to process probability distributions and assess the accuracy of generated sequences of items. To facilitate this, the probability distribution of each set of candidate items may be transformed intoa respective distribution representation, enabling the processor unit to perform calculations, such as arithmetic operations (e.g., ADD, SUBSTRACT, MULTIPLY, XOR, etc ).

[0069] A representation generation algorithm can be used to generate the distribution representation from the per-token distribution. In preferred embodiments, the representation generation algorithm can be configured to generate the distribution representation according to a position parameter and a probability parameter, wherein each position of the distribution representation is represented by a Dirac delta associated with a corresponding probability mass. In other words, each real number in the domain of the per-token distribution may be described by a Dirac delta with some probability mass located at the value of the real number.

[0070] It should be noted that this representation of a distribution in terms of Dirac deltas within its domain is different from that of a probability mass function (PMF), where the domain of the distribution is by definition discrete-valued and integrals can be replaced with sums. Because the Dirac delta distribution is not a function but rather is a distribution, the representation of distributions as a collection of Dirac deltas is also different from the concept of a probability density function (PDF). In general, one can call a representation made up of Dirac deltas that integrate to 1.0 (i.e., their probability masses sum up to 1.0) as probability density distribution (PDD) which is a generalization of a probability density function (PDF) (i.e., a PDF is a PDD but not vice versa).

[0071] The so-called Telescoping Torques Representation (TTR) of preferred embodiments of the present disclosure is described below with reference to FIGs. 5 and 6 as an example of a distribution representation of a per-token distribution in terms of Dirac deltas. It is to be understood that the present disclosure is not limited thereto and that other methods may also be used to construct representations, such as Moment- Quantized Histogram Representation (MQHR) described in WO 2022 / 248714 Al.

[0072] The Telescoping Torques Representation (TTR) builds on Dirac mixture representation of a set of samples of a data set, and is based on the intuition that the mean value of a set of samples splits the samples into two sets with equal products of the distance of the sample from the mean and the sample probability. This allows the representation to capture the effects of unlikely, yet high-value samples even at small representation sizes.

[0073] Let srcDM be a Dirac delta mixture representing a set of samples from a random variable X. Let TV be a positive integer power of two. srcDM is converted to a distribution representation of size N using a recursive conversion algorithm of log2N steps. The number of steps is what is referred to as the “order” of the distribution representation. At each step, the algorithm splits a source Dirac mixture on its mean value m, creating two Dirac mixtures, which are the inputs of two new calls to the algorithm.

[0074] FIG. 5 illustrates an example for N = 4, starting from a SFCDM of 10 data samples (e.g., particles). The top part of FIG. 5 illustrates the source Dirac mixture. The mixture consists of 10 Dirac deltas, each one positioned at the value of each unique source sample. The mass of each Dirac delta is 0.1, equal to the relative frequency of each data sample (e.g., particle).

[0075] In step 0 of FIG. 5(a), the conversion algorithm creates the zeroth-order distribution representation of SI-CDM, which is a single Dirac delta located at the mean (m = 14.85) of srcDM (i.e., t / po^O] = 14.85) with probability mass equal to the sum of the probability masses of all Dirac deltas in srcDM (t / ma^[0] = 1.0).

[0076] In step 1 of FIG. 5(b), the conversion algorithm splits SFCDM into two mixtures (each with five Dirac deltas in this example) using m as a boundary, and repeats the process for each sub-mixtures. FIG. 5(b) illustrates the first-order distribution representation of SI-CDM, which has two Dirac deltas positioned at the mean value of the Dirac mixture splits of step 0, each one with probability mass equal to 0.5.

[0077] In step 2 of FIG. 5(c), the conversion process repeats, further splitting the sub-mixtures of step 1, into two parts. This results in four sub-mixtures and four Dirac deltas that represent them and form the second-order distribution representation of srcDM. FIG. 5(c) illustrates that the split in step 1 results in sub-mixtures that each contain a different number of source particle samples. Step 2 represents these submixtures with Dirac deltas with unequal probability masses. FIG. 5(c) illustrates that the outcome Dirac deltas retain the property of equal probability torque on the two sides of the mean value of the source Dirac mixture (m = 14.85).

[0078] A Dirac mixture requires only a series of positions x, and probability masses pt for its description. It should be noted that the terms “position” and “support position” can be used interchangeably in this context. For a random variable d, its Dirac mixture of size N is an array of IN floating-point values. For the zth Dirac delta of the mixture, the position dpos\i\ and probability mass dmass[i\ are stored.

[0079] FIG. 6 illustrates a one bit-level layout for such a Dirac mixture. The set of bits in the layout allows a concise in-processor representation for the per-token distribution.

[0080] The above sum-of-Dirac-deltas representation may be described in the form of a set of tuples (i.e., a set of n ^-tuples). A reference herein to a “tuple” may be considered to include a reference to a data structure consisting of multiple parts defining an ordered set of data constituting a record, as is commonly understood in the art. In mathematics, a tuple is a finite sequence of elements such as a sequence of k elements, where & is a non-negative integer (e.g., k=2).

[0081] In a preferred embodiment of the present disclosure, each tuple of the set of tuples is a real or integer value (i.e., the position parameter) paired with a probability (i.e., the probability mass, which is a real number between 0 and 1), and the probabilities in the set of tuples add up to 1. Accordingly, each tuple provides a data structure within which distributional information is encoded. These tuples are distinct from the probability distribution itself.

[0082] Tuples other than those described above may be used in further embodiments. For example, (m+ l)-tuples could be defined where the first m numbers tell the m- dimensional position of a Dirac delta, and the last number is the probability mass of the Dirac delta.

[0083] In further embodiments, the TTR is a Joint Telescoping Torques Representation (jTTR). The Joint Telescoping Torques Representation (jTTR) builds on TTR and further tracks correlations between distributional variables. Briefly summarized, jTTR effectively tracks correlations between distributional variables by tracking the origins of each distributional variable from other distributional variables. This capability allows jTTR to accurately propagate uncertainty through many arithmetic operations where TTR falls short. For instance, a conversion algorithm can be used to take in a sequence of multidimensional samples (each of the same dimensionality) and output a jTTR for each variable corresponding to a dimension. The Joint Telescoping Torques Representation is described in detail in PCT / EP2025 / 067326.

[0084] The memory layout of the Dirac deltas of a jTTR follows a tZ-dimensional discrete hypercube where d is the number of ancestors the jTTR has (each of the distributional variables that a given distributional variable originates from can be referred to as its ancestor, and the totality of such distributional variables as its ancestors'). The d ancestors are thought to make up the dimensions of the discrete hypercube. Each dimension of the discrete hypercube has N entries where N is the representation size for a TTR (which is equivalent to a single-dimensional jTTR). One can refer to N as the base representation size for the j TTR. In general, a j TTR will have NdDirac deltas, coming from the N possible distinct entries per dimension of the d- dimensional discrete hypercube. Based on this model, each Dirac delta of a jTTR will have a tZ-dimensional index that uniquely identifies it. The jTTR algorithm makes use of this indexing framework to track the origins of Dirac deltas in order to compute with them in a way that respects correlations.

[0085] The challenge with jTTR is the exponential computer memory and compute requirements as the number of ancestors d and representation size N increases. Some embodiments of the present disclosure represent the output tensor (hypercube) in computer memory as a tensor train, such as a low-rank tensor train. In particular, the hypercube can be decomposed according to a tensor train. This reduces the computer memory requirements from exponential to linear complexity. Another feature of the tensor train representation is that a large subset of statistics of the output distribution can be computed directly in tensor train format, avoiding the need to either form the full tensor (losing the memory gained) or access each element of the output tensor (exponential in compute). As a result, a large number of user applications can be addressed with linear complexity in memory and compute, without losing any correlation information.

[0086] The tensor can be represented by a collection of small three-dimensional cores that can be strung together to individually access elements of the final tensor. That means, a tZ-dimensional tensor in tensor train form will have d cores (one for each dimension) and that the index T{n, t2, , id}, can be computed by multiplying together slice number ix from the x-th core with the other slices. For this reason, the left dimension of the first core must be one and so also the right dimension of the last core, so that the output is a scalar.

[0087] The efficiency of the tensor train representation depends on the matrix rank of each respective dimension. Since jTTR multiplication is an outer product of two vectors, which is a rank 1 matrix operation, the tensor train representation can be kept in that exact rank 1 form. Addition, on the other hand, is a rank 2 operation, and hence the maximum dimension of the tensor train cores that involve an addition operation will be two. As a result, for latent variables formed by only multiplication or division, all d tensor train cores will have dimension 1 x Nx 1. For latent variables that have undergone an addition or subtraction operation as well, the extreme cores will have dimensions 1 x N x 2 and 2 x N x 1 respectively, whereas the central cores will havedimension 2 xNx 2. As a result, one ends up storing a linear number of A-vectors with an increase in the number of dimensions.

[0088] The tensor train concept is described in detail in EP25189608.

[0089] The present disclosure is not limited to the above sum-of-Dirac-deltas representation, and other representations can be constructed from a data set, such as a per-token distribution. Another example of such an uncertainty representation is an interval-based representation (i.e., a piecewise constant approximation of the probability density function, which is equivalent to the linear approximation of the CDF) having n intervals and k = 3 (start of interval, end of interval, density value).

[0090] Yet another example is a mixture of these, e.g., consider a distribution density with half of its mass in a uniform [0, 1] (thus, exactly represented by the 3 -tuple (0, 1, 0.5)) and the other half is degenerate at the value 2 (thus, exactly represented by the 2- tuple (2, 0.5)). A degenerate distribution can be understood as a probability distribution that has a non-trivial probability mass (i.e., greater than 0) concentrated in a lower dimensional manifold than the dimension of the range of the corresponding random variable (a.k.a., the domain of the distribution). In particular, any 1-dimensional PDD that contains a Dirac delta with probability mass greater than 0 is a degenerate distribution (which has nontrivial mass concentrated at a point, a 0-dimensional manifold).

[0091] The representations described above make it unnecessary to generate large numbers of samples in order to sample the tails of a distribution (i.e., to converge) in the terminology of Monte Carlo methods. In particular, the representation (and subsequent computation using it) makes irrelevant the whole idea of having to keep running Monte Carlo iterations until one has sampled far enough out into the tails (of the inputs as well as of the results).

[0092] In some embodiments, the arrangement of values within the support of the per-token distributions may be modified from the default ordering of tokens in theirsupport space to an ordering that facilitates more efficient representation using a distribution representation, such as Dirac mixtures. By default, per-token distributions are defined based on a pre-determined ordering of tokens in the vocabulary. Under this ordering, the distributions are typically “spiky”, as illustrated in FIG. 3. However, by reordering the vocabulary indices, for instance, according to their probability, the resulting distribution transitions into a smooth decay pattern, as shown in FIG. 7. For example, the candidate items can be reordered in descending order of probability, starting with the candidate item having the highest probability and proceeding to the candidate item with the lowest probability. Alternatively, the candidate items could be reordered in ascending order of probability, starting with the candidate item having the lowest probability and proceeding to the candidate item with the highest probability. This transformation allows for more efficient processing and representation of the distributions.

[0093] The methods described herein may be implemented in software, hardware, or a combination thereof, in different embodiments. In addition, the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and changes may be made as would be obvious to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are meant to be illustrative and not limiting. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances may be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations are illustrated in the context of specific illustrative configurations. Other allocations of functionality are envisioned and may fall within the scope of claims that follow. Finally, structures and functionality presented as discrete components in the example configurations may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of embodiments as defined in the claims that follow.

Claims

CLAIMS1. A computer-implemented method for assessing reliability of outputs from machine learning models, comprising: generating, by a first machine learning model, a sequence of items, wherein each item is selected from a corresponding set of candidate items based on at least one selection criterion, wherein each candidate item of a set of candidate items is associated with probability information; and performing, by an assessment model, a factuality check on the generated sequence of items based on the probability information of the sets of candidate items associated with the items of the sequence of items.

2. A computer-implemented method of claim 1, wherein the first machine learning model is a Large Language Model, and wherein the generated sequence of items is a sequence of text tokens.

3. A computer-implemented method of claim 1 or 2, wherein the factuality check includes determining at least one quality parameter for the generated sequence of items.

4. A computer-implemented method of any one of claims 1 to 3, wherein the factuality check includes determining a factuality score for the generated sequence of items.

5. A computer-implemented method of any one of claims 1 to 4, wherein the assessment model includes, or is, a second machine learning model.

6. A computer-implemented method of claim 5, wherein the second machine learning model is configured to quantify correctness of the generated sequence of items.

7. A computer-implemented method of claim 5 or 6, wherein the second machine learning model uses information about the set of candidate items as input and returns a factuality score as output, in particular wherein the factuality score is between 0 and 1.

258. A computer-implemented method of claim 7, wherein the second machine learning model is trained based on input / output pairs to the first machine learning model that have been given a factuality score, in particular wherein the input / output pairs are prompt / response pairs that have been given a factuality score.

9. A computer-implemented method of any one of claims 1 to 8, wherein each set of candidate items represents a probability distribution of candidate items, and wherein the method further includes: generating a distribution representation from the probability distribution of each set of candidate items based on the probability information of each candidate item, wherein the distribution representation is in the form of a sum of Dirac deltas, in particular wherein the distribution representation is a Telescoping Torques Representation.

10. A computer-implemented method of any one of claims 1 to 9, further including arranging or re-arranging the candidate items of the set of candidate items based on at least one ordering criterion before performing the factuality check.

11. A computer-implemented method of claim 10, wherein the at least one ordering criterion relates to the probability information associated with the candidate items or a position of the candidate items on a number line.

12. A computer-implemented method of claim 10 or 11, wherein the arranging or rearranging of the candidate items of the set of candidate items includes mapping a default order of the candidate items to a distribution representation, in particular Dirac delta mixtures.

13. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 12.

14. A computer-readable storage medium comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any one of claims 1 to 12.

15. A data processing system for assessing reliability of outputs from machine learning models, comprising one or more processors configured to: generate, using a machine learning model, a sequence of items, wherein each item is selected from a corresponding set of candidate items based on at least one selection criterion, wherein each candidate item of a set of candidate items is associated with probability information; and perform, using an assessment model, a factuality check on the generated sequence of items based on the probability information of the sets of candidate items associated with the items of the sequence of items.

Images