Method and system of predicting a token over a vocabulary of tokens of a language model

By clustering token embeddings and dynamically selecting relevant clusters, the method addresses computational bottlenecks in large language models, enhancing efficiency and scalability while preserving accuracy and flexibility in token prediction.

WO2026235581A1PCT designated stage Publication Date: 2026-11-12EMBEDL AB
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/SE2026/010179
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-09
Filing Date
2026-04-27
Publication Date
2026-11-12

AI Technical Summary

Technical Problem

Large language models face computational bottlenecks during inference due to the need to evaluate similarity between the model's internal state and each token in a large vocabulary, which is inefficient and resource-intensive, especially in high-performance and resource-constrained environments.

Method used

A method involving one-time offline clustering of token embeddings into clusters using centroid-based clustering, followed by dynamic selection of a subset of clusters based on similarity scores, reducing the need for full-vocabulary token scoring during inference, and utilizing inner product similarity to maintain compatibility with existing models.

Benefits of technology

This approach significantly reduces computational burden and memory usage while maintaining prediction accuracy and scalability, allowing for flexible trade-offs between latency and accuracy, and supports both deterministic and probabilistic token selection strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure SE2026010179_12112026_PF_FP_ABST
    Figure SE2026010179_12112026_PF_FP_ABST
Patent Text Reader

Abstract

A computer-implemented method (100) of predicting a token (102) over a vocabulary of tokens of a language model in response to a provided embedding vector (104) without modifying or retraining underlying language model parameters, the method comprising, by processing circuitry of a computer system: performing (200) a one-time offline clustering of token embeddings of the language model into a plurality of clusters (105) using centroid-based clustering, each cluster being associated with a respective cluster centroid and comprising a subset of the vocabulary; receiving (202), at inference time, an input embedding vector (104); computing (204) similarity scores (108) between the input embedding vector and the cluster centroids; selecting (206) a subset of clusters based on the computed similarity scores; identifying (208) a reduced candidate token set (112) comprising tokens from the selected clusters; computing (210) similarity scores (114) between the input embedding vector and the embeddings of the tokens in the reduced candidate token set; and selecting (212) a token (102) in response to the computed similarity scores.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] METHOD AND SYSTEM OF PREDICTING A TOKEN OVER A VOCABULARY OF TOKENS OF A LANGUAGE MODEL

[0002] Field of the Invention

[0003] The present invention relates to a method of predicting a next token in a language model. In particular, the invention relates to a method of predicting a token over a vocabulary of tokens of a language model in response to a provided embedding vector without requiring any data nor modifying or retraining underlying language model parameters.

[0004] Background of the Invention

[0005] Large language models (LLMs) have become foundational tools, enabling significant advancements across a variety of tasks such as text generation, machine translation, question answering, and as components in larger systems such as visual-language models. These models typically operate by processing input sequences through multiple layers of neural network components, resulting in either the generation of a probability distribution over a large vocabulary of tokens or directly producing a token of maximum probability without computing the final probabilities (greedy selection). In both cases, the next token is derived from the model’s internal representation of context.

[0006] The processing of the model’s internal state is commonly performed by a dense classification layer, and typically requires evaluating the similarity between the model's internal state and the embedding of each token in the vocabulary. As the size of language models and their vocabularies increase, this final stage may become a significant computational bottleneck, particularly during inference. The need to efficiently process large vocabularies while maintaining prediction accuracy poses ongoing challenges for the deployment of such models in both high-performance and resource-constrained environments.

[0007]

[0008] In view of above-mentioned and other drawbacks of the prior art, it is an object of the present disclosure to provide an improved method of predicting a token of a language model.

[0009] According to a first aspect, there is provided a computer implemented method of predicting a token over a vocabulary of tokens of a language model in response to a provided embedding vector without modifying or retraining underlying language model parameters, the method comprising, by processing circuitry of a computer system: performing a one-time offline clustering of token embeddings of the language model into a plurality of clusters using centroid-based clustering, each cluster being associated with a respective cluster centroid and comprising a subset of the vocabulary; receiving, at inference time, an input embedding vector; computing similarity scores between the input embedding vector and the cluster centroids; selecting a subset of clusters based on the computed similarity scores; identifying a reduced candidate token set comprising tokens from the selected clusters; computing similarity scores between the input embedding vector and the embeddings of the tokens in the reduced candidate token set; and selecting a token in response to the computed similarity scores.

[0010] The present invention is based on the realization that significant improvements in inference efficiency and scalability can be achieved by avoiding full-vocabulary token scoring during inference in large language models, and instead utilizing a two-stage retrieval process based on a onetime offline clustering of token embeddings. By partitioning the vocabulary into a large number of semantically meaningful clusters and dynamically selecting a subset of these clusters in response to an input embedding vector, the computational burden of evaluating all possible tokens is substantially reduced. This approach allows the model to focus only on a reduced candidate token set derived from the selected clusters, thereby minimizing the number of similarity computations required at each inference step.

[0011] Furthermore, by employing similarity measures that are consistent with theoriginal model architecture, such as inner product similarity, the method ensures compatibility with existing language models while eliminating the need for retraining. Furthermore, clustering can be based purely on already learned embeddings, eliminating any need to make adjustments based on new data. The dynamic adjustability of the number of clusters selected at inference time further enhances scalability, enabling a configurable trade-off between latency and prediction accuracy.

[0012] In particular, the method can utilize inner product similarity, commonly used in language models, to produce logits with magnitudes well-suited for direct application of a softmax function. By retaining this similarity metric, the method allows the reduced candidate token set to be evaluated in a manner consistent with the original model design, thereby maintaining output fidelity without requiring architectural modification. The one-time clustering step produces a fixed cluster centroid matrix, enabling efficient and repeated reuse during inference without incurring additional computational overhead.

[0013] When using a much larger number of clusters compared to other approaches, it is observed that a more semantically meaningful partitioning of the vocabulary is achieved, allowing for finer distinctions between token groups. In such a scenario, a naive inverted file index (IVF) approximate nearest neighbor (ANN) approach, which typically selects only a single cluster for retrieval, would result in suboptimal performance due to insufficient coverage of relevant tokens. To effectively leverage the partitioning enabled by a large number of clusters while maintaining a meaningful representation of next possible tokens, the present method selects multiple clusters, referred to as probes, during each inference step. The number of probes corresponds to the number of clusters selected, and all tokens belonging to these selected clusters are gathered in a single operation for further evaluation.

[0014] This multi-probe strategy is particularly advantageous for inference on hardware since it only increases the computational load of a single matrix multiplication involving the cluster centroids, which is not a performance bottleneck because the total number of clusters remains significantly smaller than the vocabulary size. Moreover, the subsequent operations, i.e. theretrieval of token embeddings from the selected clusters (index selection) and the computation of similarity scores for the reduced candidate token set (second matrix operation), are unaffected by the number of clusters or probes. Both of these operations are inherently parallelizable and can be executed efficiently on hardware accelerators such as GPUs. These steps occur as single operations rather than requiring iterative processing for each selected cluster, which further enhances inference efficiency.

[0015] The ability to select multiple clusters, while maintaining low overhead, allows for both detailed semantic retrieval and high prediction accuracy, without compromising the speed or scalability of the method. Moreover, the method supports both deterministic and probabilistic token selection strategies, thereby providing flexibility for various application scenarios.

[0016] Overall, the invention delivers a scalable, efficient, and flexible mechanism for accelerating token prediction in large language models. It offers full vocabulary coverage, preserves compatibility with existing architectures, and achieves significant reductions in computation and memory usage during inference.

[0017] According to an example embodiment, one-time offline clustering is performed such that each cluster of the plurality of clusters comprises the same number of tokens. Enforcing equal-sized clusters enables representation of clusters as contiguous tensors with fixed dimensions, thereby facilitating regular memory layouts and eliminating the need for irregular memory access patterns and dynamic indexing. Such structuring improves compatibility with vectorized processing and parallel execution units. As a result, cache utilization and memory bandwidth efficiency are enhanced, leading to improved processing efficiency and reduced latency during inference.

[0018] According to an example embodiment, selecting a subset of clusters comprises applying a softmax function to the similarity scores to provide a probability vector and applying multinomial sampling to the probability vector. This allows for a probabilistic selection of clusters based on their relevance to the input embedding vector. The number of clusters selected, also referred toas the number of probes, is a configurable hyperparameter that may be adjusted dynamically at inference time. This dynamic control enables a tradeoff between latency and accuracy, where a higher number of probes generally improves prediction quality but increases computational cost. The method thereby provides flexibility, allowing users to optimize the system based on specific operational requirements or hardware constraints.

[0019] According to an example embodiment, selecting a token comprises applying a softmax function to the similarity scores to provide a probability vector and sampling a single token from the probability vector. This approach supports stochastic token prediction, which is essential for tasks requiring diverse outputs, such as natural language generation or conversational Al. By sampling according to the probability vector, the method allows different tokens to be selected across inference runs, even given the same input context, thereby enhancing the variability and naturalness of generated text.

[0020] According to an example embodiment, sampling the single token from the probability vector comprises performing probabilistic sampling, whereby the probability values directly influence the likelihood of each token being selected. This method aligns with standard practices in language modeling where softmax outputs are interpreted as probabilities, and tokens are drawn in a manner that reflects their contextual likelihood.

[0021] According to an example embodiment, computing similarity scores between the input embedding vector and the cluster centroids comprises using inner product similarity. Inner product similarity refers to the dot product between vectors and is a common method for computing logits in language models. Using inner product similarity ensures that the resulting scores are of a magnitude appropriate for direct application of a softmax function. This maintains consistency with the underlying language model architecture and avoids the need for additional scaling or normalization, thereby preserving both accuracy and computational efficiency.

[0022] According to an example, similarity scores between the input embedding vector and the cluster centroids are computed using a first numerical precision, and similarity scores between the input embeddingvector and embeddings of tokens in a reduced candidate token set are computed using a second numerical precision, the first numerical precision being lower than the second numerical precision. Performing first-stage similarity computations at reduced precision decreases computational complexity, memory usage, and energy consumption, since the first-stage computation is used only for cluster selection. Performing second-stage similarity computations at higher precision preserves numerical fidelity for final token selection, thereby maintaining prediction accuracy despite sensitivity of language model output computations to numerical precision. In one example, the first numerical precision is 8-bit integer precision and the second numerical precision is 16-bit floating-point precision, which may correspond to a default precision of the model. Such asymmetric allocation of numerical precision improves efficiency of the computer system while preserving model output quality.

[0023] According to an example embodiment, selecting a subset of clusters comprises selecting a predetermined number of clusters having the highest similarity scores. This represents a greedy selection strategy, where the clusters most similar to the input embedding vector are deterministically chosen. As with probabilistic selection, the number of clusters selected can be configured dynamically, providing the same latency-accuracy trade-off benefits. However, the deterministic nature of greedy selection ensures predictable performance and is particularly suited for real-time or resource-constrained applications since it does not require transforming the similarity scores to a probability distribution.

[0024] According to an example embodiment, selecting a token comprises selecting the token having the highest similarity score. This is another form of greedy decoding, which is computationally efficient and deterministic, resulting in consistent outputs for a given input context. Such an approach is beneficial when low-latency inference is critical, as it avoids the overhead associated with probabilistic sampling.

[0025] According to an example embodiment, computing similarity scores between the input embedding vector and the cluster centroids comprisesusing cosine similarity. Cosine similarity measures the angular distance between vectors and is independent of their magnitudes. This method is especially effective when spherical k-means clustering has been used in the offline clustering step, as it maintains consistency between the clustering and retrieval phases. Using cosine similarity in this context helps ensure that cluster selection accurately reflects the semantic structure imposed during clustering.

[0026] According to an example embodiment, each cluster centroid is computed as the mean of the token embeddings belonging to the respective cluster, according to a selected distance metric. The distance metric may be selected from Euclidean distance, cosine similarity, or inner product similarity, depending on the clustering algorithm used. The resulting centroids are stored in a fixed matrix for reuse during inference, ensuring that the clustering step introduces no additional computational load during token prediction.

[0027] According to an example embodiment, computing similarity scores between the input embedding vector and the cluster centroids comprises performing a dot product between the input embedding vector and the matrix of cluster centroids, resulting in a centroid logit vector. This vector contains the similarity scores for all clusters and is used to determine which clusters are most relevant for the current context. Dot product operations are highly parallelizable and can be executed efficiently on modern hardware accelerators.

[0028] According to an example embodiment, identifying a reduced candidate token set comprises selecting all token embeddings that belong to the identified clusters, and wherein the token embeddings are retrieved by a single slicing of the weight matrix to extract cluster tokens. This batch retrieval minimizes memory access and allows for efficient processing of the reduced token set, which is critical for achieving high throughput during inference.

[0029] According to an example embodiment, computing the similarity scores between the input embedding vector and the embeddings of the tokens in the reduced candidate token set comprises calculating an inner product between the input embedding vector and each token embedding in the reducedcandidate token set. The resulting similarity scores are directly usable as logits for application of a softmax function or for selecting the token having the highest similarity score. This ensures compatibility with existing model architectures and avoids unnecessary transformations, preserving both the accuracy and speed of the system.

[0030] There is also provided a computer program product comprising program code for performing, when executed by the processing circuitry, the method, and a non-transitory computer-readable storage medium comprising instructions, which when executed by the processing circuitry, cause the processing circuitry to perform the method of any of the aforementioned embodiments.

[0031] According to a second aspect of the invention, there is provided a computer system comprising processing circuitry for predicting a token over a vocabulary of tokens of a language model in response to a provided embedding vector without modifying or retraining underlying language model parameters, wherein the processing circuitry is configured to: perform a onetime offline clustering of token embeddings of the language model into a plurality of clusters using centroid-based clustering, each cluster being associated with a respective cluster centroid and comprising a subset of the vocabulary; receive, at inference time, an input embedding vector; compute similarity scores between the input embedding vector and the cluster centroids; selecting a subset of clusters based on the computed similarity scores; identifying a reduced candidate token set comprising tokens from the selected clusters; computing similarity scores between the input embedding vector and the embeddings of the tokens in the reduced candidate token set; and selecting a token in response to the computed similarity scores.

[0032] Effects and features of the second aspect of the invention are largely analogous to those described above in connection with the first aspect of the invention.

[0033] Further features of, and advantages with, the present invention will become apparent when studying the appended claims and the following description. The skilled person realizes that different features of the presentinvention may be combined to create embodiments other than those described in the following, without departing from the scope of the present invention.

[0034] Brief Description of the Drawings

[0035] These and other aspects of the present invention will now be described in more detail, with reference to the appended drawings showing an example embodiment of the invention, wherein:

[0036] Fig. 1 schematically illustrates features of the method according to embodiments of the invention;

[0037] Fig. 2 is a flow chart outlining steps of a method according to example embodiments;

[0038] Figs. 3A-B schematically illustrate features of methods according to example embodiments; and

[0039] Fig. 4 schematically illustrates a computer system comprising processing circuitry configured to perform the method.

[0040] Detailed Description of Example Embodiments

[0041] In the present detailed description, various embodiments of the method and system according to the present invention will be described.

[0042] Fig. 1 schematically illustrates a method 100 of predicting a token 102 according to example embodiments and the method will be described with further reference to Fig. 2 which is a flow chart outlining steps of the method.

[0043] In the present context, token embeddings refer to numerical vectors representing tokens (words, subwords, punctuation marks, etc.) in the embedding space of the language model. Each token has a unique embedding capturing semantic properties. The cluster centroids are central points representing each cluster, computed typically as the mean of the embeddings belonging to that cluster.

[0044] In an example implementation, the method begins with performing 200 a one-time offline clustering of token embeddings of the language model into a plurality of clusters using centroid-based clustering, each cluster beingassociated with a respective cluster centroid and comprising a subset of the vocabulary. Clustering 200 results in a cluster centroid matrix 106 representing a set of centroids obtained by clustering the vocabulary token embeddings offline. Each centroid corresponds to a semantic grouping of tokens.

[0045] Next, the method comprises receiving 202 an embedding vector e e Rd104, where d represents the embedding size of the language model. The embedding vector 104 represents the current contextual state produced by the language model, typically after processing an input sequence. The vocabulary V of tokens is pre-clustered offline into k clusters, and each cluster is associated with a respective centroid. These centroids are stored in the cluster centroid matrix C e Rk*d106, where each row corresponds to the centroid of a cluster.

[0046] At inference time, an input embedding vector 104 is received 202 and similarity scores 108 are computed 204, also referred to as centroid logits, by performing a matrix multiplication between the centroid matrix Cand the embedding vector e. This results in a score vector ceRk, where each value represents the similarity between the input embedding and a corresponding cluster centroid.

[0047] Based on these centroid logits, a subset of clusters is selected 206. Specifically, a number p of centroids is sampled from the score vector c, where p represents the number of probes and is a configurable hyperparameter. The selected centroids cpe 7?P 110 determine which clusters of tokens will be considered for further evaluation.

[0048] Once the relevant clusters are identified, the method retrieves all token embeddings that belong to the selected clusters, forming a reduced candidate token set V e Rx*d112, where x is the number of tokens in the selected clusters. This determination of a reduced vocabulary matrix 112 is performed efficiently through a single slicing operation 111 on the vocabulary decoding matrix 113.

[0049] Next, similarity scores (logits) 114 are computed 210 between the input embedding vector e and the embeddings in the reduced token set V. Thisresults in a vector v’e Rx, where each value corresponds to the similarity between the input context and a candidate token.

[0050] Finally, the method selects 212 the next token t based on the computed logits v'. This selection may be performed by choosing the token with the highest similarity score (argmax) in a greedy approach 124, or by applying a softmax function and sampling 126 from the resulting probability distribution, depending on the specific decoding strategy employed.

[0051] The input embedding vector is the context-specific hidden state produced by the language model, used to compute the similarity with token embeddings or centroids. Similarity scores are numerical values representing the degree of similarity between vectors, typically computed using inner product or cosine similarity. Probes refers to the number of clusters selected for detailed evaluation during inference. Higher probe counts improve accuracy but increase latency, and the reduced candidate token set is the subset of tokens whose embeddings are evaluated in detail after the initial cluster selection, significantly smaller than the full vocabulary.

[0052] Referring to Fig. 3A in combination with Fig. 1, an example of a greedy token prediction approach is illustrated. Once the score vector 108 containing similarity scores for each cluster centroid is computed, a subset of centroids 110 is selected 206. In the greedy approach, this selection 206 involves choosing the top p centroids 120 with the highest similarity scores, where p is a predefined number of centroids, referred to as probes. These centroids correspond to clusters of tokens that are most relevant based on the context represented by the embedding vector 104.

[0053] Following the selection of centroids 110, a reduced vocabulary matrix 112 is calculated and a second matrix multiplication is performed between the reduced vocabulary matrix 112 and the embedding vector 104.

[0054] To determine the next token, the method applies a greedy selection 124 by identifying the token with the highest score in the candidate tokens score vector 114. This token is then selected 212 and output as the next token 102. The greedy approach offers a deterministic and fast mechanismfor token selection, making it particularly suitable for applications that require low latency and predictable outputs.

[0055] Referring now to Fig. 3B in combination with Fig. 1 , an example of a sampling approach is described introducing probabilistic elements into both cluster selection and token prediction. As in the greedy approach, selection 206 is done based on the calculated score vector 108.

[0056] In the sampling approach, a softmax function is applied 122 to the score vector 108 to transform the similarity scores into a probability distribution over all centroids. Based on this distribution, a subset of centroids 110 is sampled, with the number of centroids sampled defined by the configurable parameter p (the number of probes). This introduces diversity into the selection process, allowing for dynamic exploration of different clusters based on their relevance.

[0057] Following the selection of centroids 110, a reduced vocabulary matrix 112 is calculated and a second matrix multiplication is performed between the reduced vocabulary matrix 112 and the embedding vector 104.

[0058] Another softmax function is applied 126 to this candidate tokens score vector 114 to generate a probability distribution over the tokens in the reduced set. A token is then sampled 126 from this distribution and selected 212 as the next token 102. The sampling approach allows for non-deterministic token selection, leading to varied outputs across different inference runs, even for the same input context. This is particularly valuable in natural language generation tasks where diverse and creative responses are desirable.

[0059] In an example embodiment, each cluster centroid in the cluster centroid matrix 106 is computed as the mean of the token embeddings belonging to the respective cluster. The clustering process utilizes a selected distance metric to assign tokens to clusters, ensuring semantically meaningful groupings. The distance metric used for clustering may include, for example, Euclidean distance, cosine similarity, or inner product similarity, depending on the nature of the embedding space and clustering algorithm applied. Once the centroids are computed, they are stored as part of a fixed matrix 106, referredto as the cluster centroid matrix, which is used during inference for centroidlevel scoring 204. This matrix remains unchanged during inference, allowing for efficient reuse and rapid similarity computation

[0060] According to an example embodiment, computing the similarity scores 108 between the input embedding vector 104 and the cluster centroids stored in the cluster centroid matrix 106 comprises performing a dot product operation. This dot product is a standard matrix multiplication, resulting in a score vector 108, also known as a centroid logit vector, which contains a similarity score for each cluster centroid. The dot product efficiently captures the degree of alignment between the input context and each cluster centroid, and the resulting scores are subsequently used for cluster selection 206.

[0061] In an example embodiment, identifying a reduced candidate token set 112 comprises selecting all token embeddings that belong to the identified clusters 110. Once the relevant centroids have been selected, the method retrieves the corresponding tokens by performing a single slicing operation 111 on the vocabulary decoding matrix 113. This slicing operation 111 extracts all token embeddings from the matrix 113 that are associated with the selected clusters 110, consolidating them into a reduced vocabulary matrix 112. This approach ensures that the retrieval is efficient and hardwarefriendly, enabling batch processing of token embeddings in preparation for subsequent similarity computations 210.

[0062] In an example embodiment, computing the similarity scores 114 between the input embedding vector 104 and the embeddings in the reduced candidate token set 112 comprises calculating an inner product between the input embedding vector 104 and each token embedding in the reduced set 112. The resulting similarity scores 114 are directly usable as logits, which can be either passed through a softmax function to obtain a probability distribution for probabilistic sampling 126, or used in an argmax operation for deterministic token selection 124. This approach ensures compatibility with standard language model architectures, as the use of inner product similarity produces logits that maintain the expected numerical properties for immediate use in downstream token selection processes.Fig. 4 illustrates a computer system 400 according to an embodiment. The steps of a method performed by the computer system 400 are in practice performed by processing circuitry 401 embodied in the form of one or more microprocessors arranged to execute a computer program 402 downloaded to a non-transitory computer readable storage medium 403 associated with the microprocessor, such as a Random Access Memory (RAM), a Flash memory or a hard disk drive. The processing circuitry 401 is arranged to cause the computer system 400 to carry out the method according to embodiments when the appropriate computer program 402 comprising computer-executable instructions is downloaded to the storage medium 403 and executed by the processing circuitry 401. The storage medium 403 may also be a computer program product comprising the computer program 402. Alternatively, the computer program 402 may be transferred to the storage medium 403 by means of a suitable computer program product, such as a Digital Versatile Disc (DVD) or a memory stick. As a further alternative, the computer program 402 may be downloaded to the storage medium 403 over a network. The processing circuitry 401 may alternatively be embodied in the form of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), etc. The computer system 400 further comprises a communication interface 404 (wired or wireless) over which it is configured to transmit and receive data.

[0063] Even though the invention has been described with reference to specific exemplifying embodiments thereof, many different alterations, modifications and the like will become apparent for those skilled in the art.

[0064] Additionally, variations to the disclosed embodiments can be understood and effected by the skilled person in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependentclaims does not indicate that a combination of these measured cannot be used to advantage.

Claims

CLAIMS1. A computer-implemented method (100) of predicting a token (102) over a vocabulary of tokens of a language model in response to a provided embedding vector (104) without modifying or retraining underlying language model parameters, the method comprising, by processing circuitry of a computer system:performing (200) a one-time offline clustering of token embeddings of the language model into a plurality of clusters using centroid-based clustering, each cluster being associated with a respective cluster centroid and comprising a subset of the vocabulary;receiving (202), at inference time, an input embedding vector (104); computing (204) similarity scores (108) between the input embedding vector and the cluster centroids;selecting (206) a subset of clusters based on the computed similarity scores;identifying (208) a reduced candidate token set (112) comprising tokens from the selected clusters;computing (210) similarity scores (114) between the input embedding vector and the embeddings of the tokens in the reduced candidate token set; andselecting (212) a token (102) in response to the computed similarity scores.

2. The method according to claim 1 , one-time offline clustering is performed such that each cluster of the plurality of clusters comprises the same number of tokens.

3. The method according to claim 1 or 2, wherein selecting a subset of clusters comprises applying a softmax function to the similarity scores to provide a probability vector and applying multinomial sampling to the probability vector.

4. The method according to any one of the preceding claims, wherein selecting a token comprises applying a softmax function to the similarity scores to provide a probability vector and sampling a single token from the probability vector.

5. The method according to claim 4, wherein sampling the single token from the probability vector comprises performing probabilistic sampling.

6. The method according to any one of claims 3 to 5, wherein computing similarity scores between the input embedding vector and the cluster centroids comprises using inner product similarity.

7. The method according to any of the preceding claims, wherein: the similarity scores between the input embedding vector and the cluster centroids are computed using a first numerical precision; andthe similarity scores between the input embedding vector and the embeddings of the tokens in the reduced candidate token set are computed using a second numerical precision, the first numerical precision being lower than the second numerical precision.

8. The method according to claim 1 or 2, wherein selecting a subset of clusters comprises selecting a predetermined number of clusters having the highest similarity scores.

9. The method according to claim 8, wherein selecting a token comprises selecting the token having the highest similarity score.

10. The method according to claim 8 or 9, wherein computing similarity scores between the input embedding vector and the cluster centroids comprises using cosine similarity.

11. The method according to any one of the preceding claims, wherein each cluster centroid is computed as the mean of the token embeddings belonging to the respective cluster according to a selected distance metric, and storing a cluster centroid matrix comprising centroid embeddings as a fixed matrix for centroid-level scoring.

12. The method according to any one of the preceding claims, wherein computing similarity scores between the input embedding vector and the cluster centroids comprises performing a dot product between the input embedding vector and the matrix of cluster centroids, resulting in a centroid logit vector.

13. The method according to any one of the preceding claims, wherein identifying a reduced candidate token set comprises selecting all token embeddings that belong to the identified clusters, and wherein the token embeddings are retrieved by a single slicing of the weight matrix to extract cluster tokens.

14. The method according to claim 1 or 2, wherein computing the similarity scores between the input embedding vector and the embeddings of the tokens in the reduced candidate token set comprises calculating an inner product between the input embedding vector and each token embedding in the reduced candidate token set, wherein the resulting similarity scores are directly usable as logits for application of a softmax function or for selecting the token having the highest similarity score.

15. A computer program product comprising program code for performing, when executed by the processing circuitry, the method of any of claims 1 to 14.

16. A non-transitory computer-readable storage medium comprising instructions, which when executed by the processing circuitry, cause the processing circuitry to perform the method of any of claims 1 to 14.

17. A computer system comprising processing circuitry for predicting a token over a vocabulary of tokens of a language model in response to a provided embedding vector without modifying or retraining underlying language model parameters, wherein the processing circuitry is configured to:perform a one-time offline clustering of token embeddings of the language model into a plurality of clusters using centroid-based clustering, each cluster being associated with a respective cluster centroid and comprising a subset of the vocabulary;receive, at inference time, an input embedding vector;compute similarity scores between the input embedding vector and the cluster centroids;selecting a subset of clusters based on the computed similarity scores; identifying a reduced candidate token set comprising tokens from the selected clusters;computing similarity scores between the input embedding vector and the embeddings of the tokens in the reduced candidate token set; and selecting a token in response to the computed similarity scores.