Parameter-free method for efficient and accurate LLM inference acceleration via speculative decoding

Abstract

In some examples, apparatus and methods are provided for selecting a draft token sequence for verification by using a large language model, LLM. Different sources of statistics on text data (prompt, generated output, large dataset of text data) can be utilized in order to choose candidates to use for speculative decoding via look-ups.

Description

[0001] PARAMETER-FREE METHOD FOR EFFICIENT AND ACCURATE LLM INFERENCE ACCELERATION VIA SPECULATIVE DECODING

[0002] TECHNICAL FIELD

[0003] The present disclosure relates, in general, to inference in large language models (LLM). More specifically, although not exclusively, the present disclosure relates to speculative decoding in natural language processing.

[0004] BACKGROUND

[0005] As the demand for LLMs increases, they become ever more complex, with (currently) hundreds of billions of parameters. During the inference phase of an LLM, output text is generated by the model in response to a text input query. In order to reduce the latency between an input query and generation of the corresponding output, which increases as the complexity of the underlying models increases, various techniques can be implemented. One such technique is speculative decoding, in which there is an assumption that, when generating the next token in response to an input query, the choice of the token to pick from a vocabulary (the set of all possible tokens) is simple, and should therefore not require the full complexity of the LLM to be determine. Speculative Decoding consists in trying to predict more than one token at a time, by proposing multiple tokens and using any free computing power to verify, in parallel, the proposed tokens.

[0006] SUMMARY

[0007] An objective of the present disclosure is to provide a parameter-free method and apparatus for efficient and accurate LLM inference acceleration via speculative decoding.

[0008] The foregoing and other objectives are achieved by the features of the independent claims.

[0009] Further implementation forms are apparent from the dependent claims, the description and the Figures.

[0010] A first aspect of the present disclosure provides an apparatus comprising at least one processor, and at least one memory including computer program code (902) for selecting a draft token sequence for verification by using a large language model, LLM, wherein a token comprises a text portion, the at least one memory and computer program code being configured to, with the at least one processor, cause the apparatus to select an input token sequence from an input for the LLM, generate a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data, generate a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM, select, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value, generate a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure, and select the draft token sequence for input to the LLM by using the tree data structure.

[0011] Using different sources of statistics on text data (such as an input prompt (or part thereof), generated output, large dataset of text data etc.) candidates to use for SD via look-ups can be chosen. Different sources are good at predicting tokens in different settings, and using the input and a large dataset presents almost complementary prediction capabilities, boosting the acceptance rate of candidate tokens.

[0012] In an example, a calibration method to combine the data sources to achieve the best acceptance rate is provided. The calibration method is easy to run, and gives better acceptance rates than other approaches. As candidates coming from the large dataset can be stored on disk for fast retrieval (or subsampling if using a suffix array implementation), any disadvantages usually presented by SD in terms of the cost of retrieving candidates are reduced. That is, according to an example, the cost becomes negligible.

[0013] In an implementation of the first aspect, the at least one memory and computer program code can be configured to, with the at least one processor, further cause the apparatus to assign a first weight to at least a proportion of the corpus of text data of the first datastore, assign a second weight to at least a proportion of the data representing the input for the LLM and the set of verified token sequences of the second datastore, and modify the probability associated with the first and second subsets of candidate tokens on the basis of the first and second weights. The at least one memory and computer program code can be configured to, with the at least one processor, further cause the apparatus to provide, to the LLM, the generated tree structure and a corresponding mask, wherein the corresponding mask comprises an indication of valid token selections from the tree data structure for use by the LLM. The at least one memory and computer program code can be configured to, with the at least one processor, further cause the apparatus to generate, using the corpus of text data of the first datastore, a suffix array comprising a sorted array of all suffixes, wherein a continuation of a selected prefix is constructed by sampling over continuations of the prefix found in the suffix array. The at least one memory and computer program code can be configured to, with the at least one processor, further cause the apparatus to organize the data representing the input for the LLM and the set of verified token sequences of the second datastore into a second tree data structure comprising nodes connected by edges, wherein each node comprises a prefix corresponding to a portion of text of the input for the LLM and the set of verified token sequences and wherein each edge represents a number of occurrences of the portion of text of the input for the LLM and the set of verified token sequences of the second datastore for a given path of the second tree data structure.

[0014] In an example, the at least one memory and computer program code can be configured to, with the at least one processor, further cause the apparatus to match the input token sequence by searching, in at least one of the first and second datastores, for a set of prefix continuations, wherein a probability of a prefix continuation comprises a measure for the number of occurrences of a prefix and a continuation for the prefix, divided by the number of occurrences of the prefix.

[0015] A second aspect of the present disclosure provides a method for selecting a draft token sequence for verification using a large language model, LLM, wherein a token comprises a text portion, the method comprising generating an input token sequence from an input for the LLM, generating a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data, generating a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM, selecting, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value, generating a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure, and selecting the draft token sequence for input to the LLM by using the tree data structure.

[0016] In an implementation of the second aspect, the method can further comprise assigning a first weight to at least a proportion of the corpus of text data of the first datastore, assigning a second weight to at least a proportion of the data representing the input for the LLM and the set of verified token sequences of the second datastore, and on the basis of the first and second weights, modifying the probability associated with the first and second subsets of candidate tokens. The method can further comprise providing, to the LLM, the generated tree structure and a corresponding mask, wherein the corresponding mask comprises an indication of valid token selections from the tree data structure for use by the LLM. The method can further comprise generating, using the corpus of text data of the first datastore, a suffix array comprising a sorted array of all suffixes from which the tree of a selected prefix is constructed by sampling over continuations of the prefix found in the suffix array. The method can further comprise organizing the data representing the input for the LLM and the set of verified token sequences of the second datastore into a second tree data structure comprising nodes connected by edges, wherein each node comprises a prefix corresponding to a portion of text of the input for the LLM and the set of verified token sequences and wherein each edge represents a number of occurrences of the portion of text of the input for the LLM and the set of verified token sequences of the second datastore for a given path of the second tree data structure.

[0017] In an example, the method can further comprise matching the input token sequence by searching, in at least one of the first and second datastores, for a set of prefix continuations, wherein a probability of a prefix continuation comprises a measure for the number of occurrences of a prefix and a continuation for the prefix, divided by the number of occurrences of the prefix.

[0018] A third aspect of the present disclosure provides a computer program stored on a non-transitory medium and including code instructions, which, when executed on one or more processors, cause the one or more processors to execute a method for selecting a draft token sequence for verification using a large language model, LLM, wherein a token comprises a text portion, the method comprising selecting an input token sequence from an input for the LLM, generating a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data, generating a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM, selecting, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value, generating a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure, and selecting the draft token sequence for input to the LLM by using the tree data structure.

[0019] In an implementation of the third aspect, the code instructions, which, when executed on one or more processors, can cause the one or more processors to execute a method, further comprising assigning a first weight to at least a proportion of the corpus of text data of the first datastore, assigning a second weight to at least a proportion of the data representing the input for the LLM and the set of verified token sequences of the second datastore, and on the basis of the first and second weights, modifying the probability associated with the first and second subsets of candidate tokens. The code instructions, which, when executed on one or more processors, can cause the one or more processors to execute a method, further comprising providing, to the LLM, the generated tree structure and a corresponding mask, wherein the corresponding mask comprises an indication of valid token selections from the tree data structure for use by the LLM.

[0020] These and other aspects of the invention will be apparent from the embodiment(s) described below.

[0021] BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order that the present disclosure may be more readily understood, embodiments will now be described, by way of example, with reference to the accompanying drawings, in which:

[0023] Figure 1 is a schematic representation of a system according to an example;

[0024] Figure 2 is a schematic representation of a (pruned) prefix tree of characters (representing tokens) according to an example;

[0025] Figure 3 is a flowchart of a method according to an example; and

[0026] Figure 4 is a schematic representation of a machine according to an example.

[0027] DETAILED DESCRIPTION

[0028] Example embodiments are described below in sufficient detail to enable those of ordinary skill in the art to embody and implement the systems and processes herein described. It is important to understand that embodiments can be provided in many alternate forms and should not be construed as limited to the examples set forth herein.

[0029] Accordingly, while embodiments can be modified in various ways and take on various alternative forms, specific embodiments thereof are shown in the drawings and described in detail below as examples. There is no intent to limit to the particular forms disclosed. On the contrary, all modifications, equivalents, and alternatives falling within the scope of the appended claims should be included. Elements of the example embodiments are consistently denoted by the same reference numerals throughout the drawings and detailed description where appropriate.

[0030] The terminology used herein to describe embodiments is not intended to limit the scope. The articles “a,” “an,” and ‘The” are singular in that they have a single referent, however the use of the singular form in the present document should not preclude the presence of more than one referent. In other words, elements referred to in the singular can number one or more, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and / or “including,” when used herein, specify the presence of stated features, items, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, items, steps, operations, elements, components, and / or groups thereof. The term “and / or” is only an association relationship for describing associated objects and represents that three relationships may exist such that A and / or B may indicate that A exists alone, A and B exist at the same time, or B exists alone. The character “ / ” generally represents that the associated objects are in an “or” relationship.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used herein are to be interpreted as is customary in the art. It will be further understood that terms in common usage should also be interpreted as is customary in the relevant art and not in an idealized or overly formal sense unless expressly so defined herein.

[0032] The following contains specific information related to implementations of the present disclosure. The drawings and their accompanying detailed disclosure are merely directed to implementations. However, the present disclosure is not limited to these implementations. Other variations and implementations of the present disclosure will be obvious to those skilled in the art.

[0033] The phrases “in one implementation,” or “in some implementations,” may each refer to one or more of the same or different implementations. The term “coupled” is defined as connected whether directly or indirectly through intervening components and is not necessarily limited to physical connections. The expression “at least one of A, B and C” or “at least one of the following: A, B and C” means “only A, or only B, or only C, or any combination of A, B and C.”

[0034] The terms “system” and “network” may be used interchangeably.

[0035] For the purposes of explanation and non-limitation, specific details such as functional entities, techniques, protocols, and standards are set forth for providing an understanding of the present disclosure. In other examples, detailed disclosure of well- known methods, technologies, systems, and architectures are omitted so as not to obscure the present disclosure with unnecessary details.

[0036] Persons skilled in the art will immediately recognize that any network functions) or algorithm(s) disclosed may be implemented by hardware, software or a combination of software and hardware. Disclosed functions may correspond to modules which may be software, hardware, firmware, or any combination thereof.

[0037] A software implementation may include machine- and / or computer- readable and / or executable instructions stored on a machine- and / or computer-readable medium such as memory or other types of storage devices. One or more microprocessors or general-purpose computers with communication processing capability may be programmed with corresponding executable instructions and perform the disclosed network functions) or algorithm(s).

[0038] The microprocessors or general-purpose computers may include Applications Specific Integrated Circuitry (ASIC), programmable logic arrays, and / or using one or more Digital Signal Processor (DSPs). Although some of the disclosed implementations are oriented to software installed and executing on computer hardware, alternative implementations implemented as firmware or as hardware or as a combination of hardware and software are well within the scope of the present disclosure. The computer readable medium includes but is not limited to Random Access Memory (RAM), Read Only Memory (ROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory, Compact Disc Read-Only Memory (CD-ROM), magnetic cassettes, magnetic tape, magnetic disk storage, or any other equivalent medium capable of storing computer-readable instructions.

[0039] In the following description, the following terms may be used:

[0040] • FLOPs: (Floating Point Operations per Second), is a measure of a computer system's performance, particularly in tasks involving numerical computations. It indicates how many floating-point calculations (like addition, subtraction, multiplication, division of decimal numbers) a system can perform in one second.

[0041] • Batch size: The number of input samples (e.g. user requests) processed simultaneously during training or inference.

[0042] • Throughput: In this context (LLM inference), the number of tokens which are generated by the LLM serving system per unit of time. To a certain degree of approximation, this can be seen as the inverse of cost.

[0043] • Latency: In the present disclosure this is defined as the time it takes for each sequence (user) to see the next token. Can also be called Time Between Tokens (TBT).

[0044] • Device: A specialized hardware component for LLM inference, usually called Al accelerator. Typically, this is a General-Purpose GPU, an NPU (Neural Processing Unit) or a TPU (Tensor Processing Unit).

[0045] • Host: Hardware device (mainly composed of CPUs and memory) that manages and coordinates the execution of the inference process.

[0046] • Tokenizer: a function or a computational tool that processes a sequence of text, to divide it into distinct units or tokens, based on predefined rules.

[0047] • Suffix array: a data structure that provides a sorted list of all suffixes of a given string, facilitating efficient search and pattern matching. Natural language models (NLPs), including LLMs, process natural language in discrete units, called tokens. Natural language sentences are broken into a word-sized tokens (i.e. byte pair tokenization) and then each token is embedded into a d-dimensional vector of real numbers. Language models always expect sequences of embedded tokens as their input (also known as prompt or context) and may output a sequence of embedded tokens, such as generating a new token. This is called text generation, a fundamental capability of perhaps the most popular NLP tasks: chatbots, summarization, question answering and more.

[0048] Once an LLM has been pretrained and fine-tuned for a generative task it can be deployed for inference service. The LLM inference procedure undergoes two main phases: (i) a prefill phase, and (ii) a generative decoding phase. In the prefill phase, the whole prompt is processed in one shot and the intermediate results of this phase are cached in a key value- (KV) cache to speed-up the subsequent generative decoding phase. From a performance perspective, the prefill phase is known to be computationally bound (i.e., memory subsystem is idling). Once the prefill is done, the generative decoding phase starts, where the LLM processes as an input the prefill phase’s output and produces a new output token. To speed up the generation, the previously computed intermediate results are streamed from the KV-cache. From a performance perspective, the generative decoding phase is characterized by being memory-bandwidth-bound (i.e., the computational resources are waiting for the memory to provide the data) because each generated token performs computation on the entire KV-context. The decoding phase continues for several rounds in an auto-regressive fashion: the output of the previous round is fed into the LLM in the current round. The decoding phase terminates when a special "end of sequence" (EOS) token is generated, which signals the system to return the full output (the initial prompt + all the generated tokens) back to the user.

[0049] Each step of the generative decoding phase, which gives as output one single token, requires going through multiple transformer layers, each of which contains an attention block and a feedforward block. These blocks require loading the KV Cache of each layer, the model parameters, and computing some matrix-vector multiplications. During each of these steps, the available FLOPs are highly underutilized, while loading the KV Cache and model weights from memory to the compute units takes most of the time, making each step last several milliseconds. The hardware features of Al accelerators are particularly suited for training (and prefill), where matrix-matrix multiplications are performed and become the bottleneck (compute-bound), but not for decoding.

[0050] In this context, speculative decoding is used to decrease the latency (intended as Time Between Tokens) of autoregressive generation. In autoregressive decoding the cost of predicting the next token is roughly constant and independent of its complexity. Speculative Decoding consists in trying to predict more than one token at a time, by proposing multiple tokens and using the “free FLOPs” to verify, in parallel, the proposed tokens. Note that this increases the computation performed, but not the memory utilization, since the KV Cache is (roughly) the same for all tokens to verify, and the model weights are the same. This makes also the autoregressive step consist of matrix-matrix multiplications, making it closer to the prefill step in terms of hardware utilization.

[0051] If N tokens of the proposed ones are accepted in sequence by the LLM (because they are the same tokens that it would have predicted in standard autoregressive decoding), the process can proceed to N+l tokens. The model used to propose the tokens (the draft model) is usually a smaller LLM that runs sequentially multiple times to propose multiple tokens. Over time, different methods have been introduced to generate the candidate sequence of tokens to verify (the draft), for example training additional attention heads for this task. Other research has proposed to verify multiple drafts in parallel rather than a single draft of subsequent tokens, taking advantage of the free FLOPs to increase the probability of accepting a token.

[0052] Most SD methods are focused on increasing the acceptance rate of tokens to improve the latency. These methods usually do not try to keep the complexity simple enough for easy adoption and deployment in practical cases. Moreover, in typical scenarios, LLM inference is run at large scales, and a key factor to optimize is cost (often under some fixed latency constraints, for example for good user experience). Scaling SD for real- world inference systems has encountered significant challenges. For example, the complexity of deploying the draft model, the cost of speculating with the draft model, and the limited free FLOPs available when doing inference on large batches, compared to the single sequence case, implying lower acceptance rates (need to divide the free FLOPs by the batch size to find how much speculation budget there is for each sequence).

[0053] Note that the throughput gains that can be achieved by speculating are typically lower than those achievable by batching. Still, batching is limited by a device memory, as there is a need to store a new KV Cache for each sequence, while speculation does not increase memory utilization. Also, while batching only increases throughput, speculative decoding improves both throughput and latency. Due to these factors, SD is an invaluable technique for LLM serving systems. However, none of the current solutions have fully addressed its inherent complexities.

[0054] According to an example, a fast, easy to build and easy to deploy method for SD is provided that provides high-quality candidates for verification with a high acceptance rate. In an example, a combination of different sources of statistics on text continuations can be used in order to pick candidates to provide to an LLM for verification during SD. The different sources of statistics can be combined in order to increase the acceptance rate. For example, fast calibration of the different sources relative to some ground truth text can be performed.

[0055] Figure 1 is a schematic representation of a system according to an example. With reference to the example of figure 1, a speculative decoding process for an LLM 101 is described. In the example of figure 1, an apparatus 100 is provided for selecting a draft token sequence for verification using a large language model, LLM, 101. The LLM 101 can be implemented in a device 102 for example.

[0056] An input 103 for LLM comprises a set of tokens 105. Each token of the set of tokens 105 comprises a portion of text, which may be a letter, a word, part of a word, or part of sentence.

[0057] In an example, an input token sequence 107 is generated or otherwise derived from the input 103 for the LLM 101. The input token sequence 107 can comprise, e.g., the last few (e.g., three) tokens 106 of the input 103, such as the (or part of the) input prompt plus the output generated up until the point that the input token sequence 107 is provided. For example, in response to an input from a user for example, the LLM 101 can infer an output comprising a set of output tokens. The, or some, of the output tokens can be provided for a subsequent inference iteration, such that the next input for the LLM comprises a combination of the original input and an output based on the original input. In an example, the input token sequence 107 can comprise a predefined number of tokens from the input 103, ranging from, e.g., one to ten.

[0058] According to an example, the input token sequence 107 can be used to find a set of possible continuations using a data from a first datastore 109 and from the continuations that already occurred in the prompt and output (and possibly from other sources too) in a second datastore 111, or some weighted or otherwise predefined combination of these sources.

[0059] A first set of candidate tokens are selected on the basis of the input token sequence 107, from the first datastore 109, In an example, the first datastore 109 comprises a corpus of text data. According to an example, the first datastore 109 is created by providing a set of textual sentences, represented as sequences of characters, to a model tokenizer that is configured to transform the textual data into sequences of numbers, exactly in the same way as it is done for the input that is fed to the LLM. The result of the tokenization is a set of sequences of numbers, which are appended to form a single long sequence. Each sentence normally (but not necessarily) starts with a special BOS (Beginning Of Sentence) token. The single long sequence is used to construct a suffix array to efficiently find all the occurrences of a given (short) sequence of tokens in the long sequence, with the continuations sorted numerically by increasing token ID. The suffix array consists of a list of integers representing positions in the long sequence.

[0060] A second set of candidate tokens are selected on the basis of the input token sequence 107, from the second datastore 111. As noted above, the second datastore 111 comprises data representing the input 103 for the LLM 101 and a set of verified token sequences comprising a set of tokens previously output by the LLM in response to the input for the LLM.

[0061] From the first and second sets of candidate tokens, first 113 and second 115 subsets of candidate tokens are selected, in which the first 113 and second 115 subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input 107 for the LLM 101 above a predefined threshold value.

[0062] According to an example, a tree data structure 117 is generated using a combination of the first 113 and second 115 subsets of candidate tokens. The tree data structure comprises a set of nodes, each node comprising a token of the first 113 and second 115 subsets of candidate tokens. At least a subset of the nodes are logically linked to one another with edges, with each edge representing a probability value associated with a likelihood of continuation from one node to the next as the tree data structure is traversed from a root node 119 of the tree data structure.

[0063] In an example, the tree data structure 117 comprises a trie with the most likely continuations coming from the different sources (109, 111), each edge owning a probability given by a combination of the data source probability estimation with some scaling weight. The scaling weight is based on the kind of source (datastore / input), on the size of the prefix, on the depth in the tree and so on. Weight values can be calculated by calibrating the probabilities given by the sources to match the corresponding LLM acceptance rate. The best continuations selected to put in the trie are the ones having the highest probability in the incoming edge.

[0064] Using the trie 117, a draft token sequence 119 for input to the LLM 101 can be determined and passed to the LLM 101 along with a decoding mask 121. The draft token sequence 119 and decoding mask 121 can be used by the LLM 101 to verify the candidates 119.

[0065] In an example, in greedy decoding (the next token accepted by the LLM 101 is always the one with the highest probability), it is possible to obtain the highest probability token from the LLM distribution and check if it occurs in the drafts, then reiterate if found. When not doing greedy decoding, instead of picking the argmax token, it is possible to sample 123 from the distribution given by the LLM, then use the same approach.

[0066] The input 103 can then be updated using the output 125 of the LLM 101, the output comprising accepted tokens, which are also provided to the second datastore 111.

[0067] In a scale-out case (Tensor Parallel) instead of replicating the speculation process, a single process can be used on a host for the speculation. This makes it particularly easy to scale to any number of devices.

[0068] The term 'probability' as used herein refers to the likelihood of certain token sequences occurring in a dataset, primarily conditioned on being the continuation of another sequence. This is to be consistent with the terminology used for LLM predictions (before sampling), which are usually considered probabilities over the vocabulary.

[0069] According to an example, it is therefore possible to use a combination of sources of text statistics to provide speculation candidates for the LLM 101. That is, given multiple text sources (e.g., the prompt 107, the self-generated output 125, a large dataset of text data in the first datastore 109), the information contained in each of the sources can be aggregated in such a way that it is possible to query the information as if it was in a prefix-tree, with the edges containing the number of occurrences of that path in the data source. Note that the data structure 117 used does not need to be implemented in software as a prefix tree. It is enough to be able to extract the same information in a fast way.

[0070] As noted above, some tokens (e.g., the last few tokens) 106 of the latest input 103 to the LLM 101 (comprising a prompt and some already generated tokens) can be matched in each of the data sources 109, 111. If the prefix (of the search, corresponding to the input suffix) is of size N then, in an example, a search for prefixes of size [N-l ... 1], or a subset of them is performed. Then, the best continuations of the prefixes, based on their probabilities (e.g., a higher probability is better) are selected. The probability of a continuation given a prefix, is, in an example, given by the number of occurrences of the full sequence [prefix + continuation] divided by the number of occurrences of the prefix.

[0071] Figure 2 is a schematic representation of a (pruned) prefix tree of characters (representing tokens). In the example of figure 2, the prefix tree 201 is created from some characters 203 , stemming from a root node (such as 119 in figure 1 ). Note that it is not a complete prefix trie, since the depth is limited, and the sequence [f| is not inserted. However, this is typically how data sources of the present disclosure can be logically represented.

[0072] For example, the probability of [c] being the continuation of prefix [c, d] is:

[0073] P([c] | [c, d]) = occ([c, d, c]) / occ([c, d]) =1 / 2 where “occ” is occurrences. Note that P([c, d] | [c, d]) is also Yi.

[0074] The possible continuations from the sources 109, 111 can therefore be combined to form a unique draft 119. Logically, the process can be described, in an example, as: a. Initialize an empty vector of candidates. b. After matching the prefix in each data source 109, 111, insert all the continuations of the prefix of depth 1 into a single priority queue, prioritized by their probability. c. Pick the most likely continuation among all sources from the priority queue and insert it in the vector of candidates. Then insert all the continuations of the prefix + the newly added token into the priority queue. Note that the probability can be given by: occ([prefix + already _added_token(s) + candidate_token]) / occ[prefix], d. Reiterate the process until the candidates’ vector contains the desired number of elements.

[0075] In the process, the parent indices in candidates’ vector is kept track of in order to construct the mask for the LLM for verification.

[0076] According to an example, for better results, the sources 109, 111 can be weighted differently, when inserted in the same priority queue, since the probability estimate of each source might be of different quality (e.g., something giving probability of 1 all the time may be weighted lower). When inserting the weights in the priority queue, the probability obtained as described above is multiplied by the weight assigned to that source. To find good weights, a grid-search process can be implemented on the weights, choosing the weights that give the best acceptance rate. This grid search can be done without using an accelerator, if there is some data generated by the LLM, simulating the LLM speculative decoding progression. The LLM can then be fed with the candidates’ vector 119 and the corresponding mask 121. The same approach can be used both for greedy decoding and for sampling methods: in case of sampling, sample from the output distribution of the LLM instead of doing the argmax, and then, as in greedy decoding, check if the sampled token occurs in the drafts.

[0077] Note that to maximize the effect of SD, the desired candidates’ vector size should be given by hardware constraints on FLOPs and memory bandwidth utilization and expected acceptance rate. More advanced techniques can involve using the expected acceptance rate of each candidate, combined with the hardware features, to decide whether to stop earlier or later when picking the candidates.

[0078] According to an example, the prompt and output generated up to the current point can be inserted in the same data source, and a concrete prefix tree can be used as a data structure to query and update it. A large dataset of text (e.g., in the first datastore 109) can be used, indexed by a suffix array, to efficiently find all the possible continuations of a certain prefix in the whole dataset. The prefix tree can then be built with the prefix as root, using these continuations. To reduce retrieval time, instead of using all continuations to build the trie, at most K continuations can be used. Given the sorted structure of the suffix array, the sampling can be done by picking candidates at regular intervals (if there are M > K continuations, intervals will on average of size M / K).

[0079] In the above, the sources are combined with a priority-queue fashion. According to another example, a fixed maximum budget can be provided over all the chosen candidates to each of the sources. For example, if the candidates’ budget is N, give a fixed budget to the first data source, for example N / 2. Start with the sources that are more likely not to use all the budget, so that if they do not fill the budget this can be used by the following sources. A similar approach can be used as described above (i.e., it can be performed with a priority queue) to fill the candidates’ vector up to the budget allocated to the sources used until then. With this approach, the priority queue contains candidates from one source only at a time.

[0080] According to an example, an advanced combination of different text sources with calibration can be used. This can, for example, improve grid- search-based weighting of different sources to better choose from which source to pick the next candidate in the shared priority-queue based approach as described above. This can be applied both by first generating some data given some prompts using the LLM, and then simulating the LLM inference process, or by actually running the LLM each time (which is costlier).

[0081] Figure 3 is a flowchart of a method according to an example. In block 301 a dictionary D is initialised that maps each depth in the drafts to a list of pairs, representing the probability estimated from the data source, and which tokens were accepted or not (1 or O).

[0082] In block 303, iteration is performed over the prompts, and for each prompt iteration over the tokens of the answer is performed. The ‘cursor’ can then be moved to the start of the first answer.

[0083] In block 305, it is determined whether the last sequence is finished. If yes, in block 307, the regression line for each of the depths in D is computed using as data points the points in the lists. The x axis is the probability, and the y axis is 0 or 1. In block 309, the coefficients of the regression line are stored.

[0084] In the other case from block 305 (no), in block 309, the data source is queried, and the aggregated candidates are obtained with their estimated probabilities. The counter A of accepted tokens is initialised to 0. In block 311 it is determined whether any candidates are left. If not, then in block 313 the process moves forward in the LLM output of A + 1 tokens, and the process returns to block 305. Else, in block 315, the next candidate is obtained, along with its probability p, and depth d. In block 317, it is determined whether the candidate is acceptable (corresponds to the LLM ground truth). If not, then in block 319 the list corresponding to the entry d in D is obtained, and append the pair (p, 0). The process then returns to block 311. Else, in block 321 the list corresponding to the entry d in D is obtained, and append the pair (p, 1 ). In block 323 the counter A is increased to A+l and the process returns to block 311.

[0085] The process described with reference to figure 3 is performed for each of the data sources (prompt and self-output 111, large dataset 109), and for each prefix size in the range of interest (typically not longer than 6). Once the files with the calibration coefficients are stored, the information can be loaded by the datastore. After computing the probability of a continuation, as described above, this probability is, in an example, scaled to match the corresponding regression line, before inserting it into the priority queue. As such, the balancing between different data sources is done automatically by calibrating the data source probability estimates to match the expected acceptance rate.

[0086] As described above, prefixes of size P can be searched in order to obtain all of the continuations of length C, from which a prefix tree of continuations (of depth C) can be generated. In an alternative example, the continuations of length 1 of the prefix can be determined, with their estimated conditional probabilities. It is then possible to iterate over these continuations, add the continuation to the prefix, and remove the first token from the prefix. Now the prefix is again of size P, but “shifted to the right”. Then, search again for the continuations with the new prefix (iterate C times for similar results to the other method). This alternative does not give the same results for the probability estimates (it would give the same result if the first element of the old prefix were not removed each time). To speed up the process, a trivial solution is to avoid iterating over all continuations and rather go in depth only with the ones with higher probability. Calibration can be used also for this. In this case, though, it is not necessary to calibrate for different continuation depths, as this is done inherently by the method.

[0087] In another example, in order to obtain faster implementation of the data structure 109 to retrieve candidates, all of the continuations of the prefixes up to a certain size (e.g. 4) can be retrieved and stored with their probabilities with the information to reconstruct the mask (e.g., on disk). At runtime, this dramatically reduces the retrieval time with no acceptance rate drop or additional host memory usage, at the cost of disk space usage and having to run in advance (only once) the process to precompute all candidates (the suffix-array data structure can be used to build the precomputed one).

[0088] In another example, which can improve the datastore-based part of the speculation, the overall process can be integrated in the following way:

[0089] 1. Divide the set of documents to be used in RAG by topic.

[0090] 2. For each topic, create a datastore to retrieve candidates from (as explained above, it can be suffix-array based or precomputed). This is done exactly as in the large general dataset case.

[0091] 3. When a document is retrieved (given a prompt), get the topic-based datastore corresponding to the topic of the document (typically the one containing also the document that was retrieved), and use it as additional source of speculation.

[0092] This provides high quality data sources for speculation in specific settings. These single-topic datastores could be auto-tuned (each one separately, or all together) with the calibration process.

[0093] In another example, dynamic speculation parameters based on the prompt can be provided in which the weights are adjusted based on the type of prompt. The process is the following:

[0094] 1. Use a simple classifier to assign the prompt to a category (the classifier should be trained beforehand). 2. The classifier will have specific weights to be given to the prompt / datastore / other sources. Use the corresponding weights when selecting the candidates.

[0095] 3. Alternatively, or in conjunction with point (2), each category can have a specific datastore to use to retrieve candidates.

[0096] For example, one can have a specific datastore containing, e.g., Python code: when the prompt is detected to be about providing some Python code, the Python datastore can be added to the sources, and the weights of the sources can be updated, to adapt the speculation to the context.

[0097] In another example, using small ML model speculation, in addition to the data sources already discussed (prompt, selfgenerated output, large datastore, RAG-retrieved document, RAG document group), it is also possible to use a small ML model (it can even be a neural network) that gives probability distributions over the vocabulary, that can be weighted and merged with (some of, or all) the other sources of text statistics. In principle, this could even run on device.

[0098] In another example, live creation of the large datastore can be implemented. That is, given that finding high-quality data for the LLM to speculate (in a specific context) is often hard, this makes it hard to build the large datastore (109) in many cases. In these cases, in particular (but this also holds in general), the best data source to use is data generated by the LLM itself through real interactions with users.

[0099] This kind of data is generally available to the provider of the LLM serving system. It can be collected and used to create the datastore. This consists in the following main steps:

[0100] 1. Every time the LLM finishes generating a sequence, this sequence is added to a collection of generated sequences.

[0101] 2. At regular time intervals (e.g., once a day), the sequences collected are used to construct new data chunks to be added to the datastore (or to update the available chunks with new data). Once the data is updated (new suffix array or new precomputed data on disk), the set of sequences to be added is cleared, and can start to be filled again. In a real inference system, this should run asynchronously.

[0102] 3. When the data collected is too much (retrieval time increases, but the speculation quality saturates), one can get rid of the least recent data (or use another method to reduce the data size), in particular for the suffix array-based solution.

[0103] Examples in the present disclosure can be provided as methods, systems or machine-readable instructions, such as any combination of software, hardware, firmware or the like. Such machine-readable instructions may be included on a computer readable storage medium (including but not limited to disc storage, CD-ROM, optical storage, etc.) having computer readable program codes therein or thereon.

[0104] The present disclosure is described with reference to flow charts and / or block diagrams of the method, devices and systems according to examples of the present disclosure. Although the flow diagrams described above show a specific order of execution, the order of execution may differ from that which is depicted. Blocks described in relation to one flow chart may be combined with those of another flow chart. In some examples, some blocks of the flow diagrams may not be necessary and / or additional blocks may be added. It shall be understood that each flow and / or block in the flow charts and / or block diagrams, as well as combinations of the flows and / or diagrams in the flow charts and / or block diagrams can be realized by machine readable instructions.

[0105] The machine-readable instructions may, for example, be executed by a machine such as a general-purpose computer, a platform comprising user equipment such as a smart device, e.g., a smart phone, a special purpose computer, an embedded processor or processors of other programmable data processing devices to realize the functions described in the description and diagrams. In particular, a processor or processing apparatus may execute the machine-readable instructions. Thus, modules of apparatus may be implemented by a processor executing machine readable instructions stored in a memory, or a processor operating in accordance with instructions embedded in logic circuitry. The term 'processor' is to be interpreted broadly to include a CPU, processing unit, ASIC, logic unit, or programmable gate set etc. The methods and modules may all be performed by a single processor or divided amongst several processors.

[0106] Such machine-readable instructions may also be stored in a computer readable storage that can guide the computer or other programmable data processing devices to operate in a specific mode. For example, the instructions may be provided on a non- transitory computer readable storage medium encoded with instructions, executable by a processor.

[0107] Figure 4 is a schematic representation of a machine according to an example. The machine 400 can be, e.g., a system or apparatus, user equipment, or part thereof. The machine 400 comprises a processor 403, and a memory 405 to store instructions 402, executable by the processor 403. The machine comprises a storage 409 that can be used to store data 401 for datastores 109 and / or 111 as described above with reference to figure 1 for example.

[0108] The instructions 407, executable by the processor 403, can cause the machine 400 to select an input token sequence from an input for the LLM, generate a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data, generate a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM, select, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value, generate a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure, and select the draft token sequence for input to the LLM by using the tree data structure.

[0109] Accordingly, the machine 400 can implement a method for selecting a draft token sequence for verification using a large language model, LLM.

[0110] Such machine-readable instructions may also be loaded onto a computer or other programmable data processing devices, so that the computer or other programmable data processing devices perform a series of operations to produce computer- implemented processing, thus the instructions executed on the computer or other programmable devices provide an operation for realizing functions specified by flow(s) in the flow charts and / or block(s) in the block diagrams.

[0111] Further, the teachings herein may be implemented in the form of a computer or software product, such as a non-transitory machine-readable storage medium, the computer software or product being stored in a storage medium and comprising a plurality of instructions, e.g., machine readable instructions, for making a computer device implement the methods recited in the examples of the present disclosure. In some examples, some methods can be performed in a cloud-computing or network-based environment. Cloud-computing environments may provide various services and applications via the Internet. These cloud-based services (e.g., software as a service, platform as a service, infrastructure as a service, etc.) may be accessible through a web browser or other remote interface of the user equipment for example. Various functions described herein may be provided through a remote desktop environment or any other cloud-based computing environment.

[0112] The preceding description has been provided to enable others skilled in the art to best utilize various aspects of the exemplary embodiments disclosed herein. This exemplary description is not intended to be exhaustive or to be limited to any precise form disclosed. Many modifications and variations are possible without departing from the spirit and scope of the instant disclosure. The embodiments disclosed herein should be considered in all respects illustrative and not restrictive. Reference should be made to the appended claims and their equivalents in determining the scope of the instant disclosure.

Claims

CLAIMS1. Apparatus comprising: at least one processor; and at least one memory including computer program code (902) for selecting a draft token sequence for verification by using a large language model, LLM, wherein a token comprises a text portion, the at least one memory and computer program code being configured to, with the at least one processor, cause the apparatus to: select an input token sequence from an input for the LLM; generate a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data; generate a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM; select, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value; generate a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure; and select the draft token sequence for input to the LLM by using the tree data structure.

2. The apparatus as claimed in claim 1 , wherein the at least one memory and computer program code are configured to, with the at least one processor, further cause the apparatus to: assign a first weight to at least a proportion of the corpus of text data of the first datastore; assign a second weight to at least a proportion of the data representing the input for the LLM and the set of verified token sequences of the second datastore; and modify the probability associated with the first and second subsets of candidate tokens on the basis of the first and second weights.

3. The apparatus as claimed in claim 1 or 2, wherein the at least one memory and computer program code are configured to, with the at least one processor, further cause the apparatus to: provide, to the LLM, the generated tree structure and a corresponding mask, wherein the corresponding mask comprises an indication of valid token selections from the tree data structure for use by the LLM.

4. The apparatus as claimed in any preceding claim, wherein the at least one memory and computer program code are configured to, with the at least one processor, further cause the apparatus to: generate, using the corpus of text data of the first datastore, a suffix array comprising a sorted array of all suffixes, wherein a continuation of a selected prefix is constructed by sampling over continuations of the prefix found in the suffix array.

5. The apparatus as claimed in any preceding claim, wherein the at least one memory and computer program code are configured to, with the at least one processor, further cause the apparatus to: organize the data representing the input for the LLM and the set of verified token sequences of the second datastore into a second tree data structure comprising nodes connected by edges, wherein each node comprises a prefix corresponding toa portion of text of the input for the LLM and the set of verified token sequences and wherein each edge represents a number of occurrences of the portion of text of the input for the LLM and the set of verified token sequences of the second datastore for a given path of the second tree data structure.

6. The apparatus as claimed in claim 4 or 5, wherein the at least one memory and computer program code are configured to, with the at least one processor, further cause the apparatus to: match the input token sequence by searching, in at least one of the first and second datastores, for a set of prefix continuations, wherein a probability of a prefix continuation comprises a measure for the number of occurrences of a prefix and a continuation for the prefix, divided by the number of occurrences of the prefix.

7. A method for selecting a draft token sequence for verification using a large language model, LLM, wherein a token comprises a text portion, the method comprising: generating an input token sequence from an input for the LLM; generating a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data; generating a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM; selecting, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value; generating a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure; and selecting the draft token sequence for input to the LLM by using the tree data structure.

8. The method as claimed in claim 7, further comprising: assigning a first weight to at least a proportion of the corpus of text data of the first datastore; assigning a second weight to at least a proportion of the data representing the input for the LLM and the set of verified token sequences of the second datastore; and on the basis of the first and second weights, modifying the probability associated with the first and second subsets of candidate tokens.

9. The method as claimed in claim 7 or 8, further comprising: providing, to the LLM, the generated tree structure and a corresponding mask, wherein the corresponding mask comprises an indication of valid token selections from the tree data structure for use by the LLM.

10. The method as claimed in any of claims 7 to 9, further comprising: generating, using the corpus of text data of the first datastore, a suffix array comprising a sorted array of all suffixes from which the tree of a selected prefix is constructed by sampling over continuations of the prefix found in the suffix array.

11. The method as claimed in any of claims 7 to 10, further comprising: organizing the data representing the input for the LLM and the set of verified token sequences of the second datastore into a second tree data structure comprising nodes connected by edges, wherein each node comprises a prefix corresponding to a portion of text of the input for the LLM and the set of verified token sequences and wherein each edge represents a number of occurrences of the portion of text of the input for the LLM and the set of verified token sequences of the second datastore for a given path of the second tree data structure.

12. The method as claimed in claim 10 or 11, further comprising: matching the input token sequence by searching, in at least one of the first and second datastores, for a set of prefix continuations, wherein a probability of a prefix continuation comprises a measure for the number of occurrences of a prefix and a continuation for the prefix, divided by the number of occurrences of the prefix.

13. A computer program stored on a non-transitory medium and including code instructions, which, when executed on one or more processors, cause the one or more processors to execute a method for selecting a draft token sequence for verification using a large language model, LLM, wherein a token comprises a text portion, the method comprising: selecting an input token sequence from an input for the LLM; generating a first set of candidate tokens, wherein the first set of candidate tokens are selected from a first datastore on the basis of the input token sequence, wherein the first datastore comprises a corpus of text data; generating a second set of candidate tokens, wherein the second set of candidate tokens are selected from a second datastore on the basis of the input token sequence, wherein the second datastore comprises data representing the input for the LLM and a set of verified token sequences, wherein the set of verified token sequences comprises a set of tokens previously output by the LLM in response to the input for the LLM; selecting, from the first and second sets of candidate tokens, first and second subsets of candidate tokens, wherein the first and second subsets of candidate tokens comprise tokens having a probability of representing a continuation to the input for the LLM above a predefined threshold value; generating a tree data structure using a combination of the first and second subsets of candidate tokens, wherein the tree data structure comprises a set of nodes, each node comprising a token of the first and second subsets of candidate tokens, wherein at least a subset of the nodes are logically linked to one another with edges, wherein each edge represents a probability value associated with a continuation from one node to the next as the tree data structure is traversed from a root node of the tree data structure; and selecting the draft token sequence for input to the LLM by using the tree data structure.

14. The computer program as claimed in claim 13, wherein the code instructions, which, when executed on one or more processors, cause the one or more processors to execute a method, further comprising: assigning a first weight to at least a proportion of the corpus of text data of the first datastore; assigning a second weight to at least a proportion of the data representing the input for the LLM and the set of verified token sequences of the second datastore; and on the basis of the first and second weights, modifying the probability associated with the first and second subsets of candidate tokens.

15. The computer program as claimed in claim 13 or 14, wherein the code instructions, which, when executed on one or more processors, cause the one or more processors to execute a method, further comprising: providing, to the LLM, the generated tree structure and a corresponding mask, wherein the corresponding mask comprises an indication of valid token selections from the tree data structure for use by the LLM.17

Images