Attention head functionalities in machine learning models

By identifying and managing parametric and contextual attention heads in transformer models through relevance scores, the reliability and efficiency of LLM predictions are improved, addressing the unclear mechanism of in-context learning in LLMs.

WO2026238942A1PCT designated stage Publication Date: 2026-11-19FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/EP2026/064174
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-15
Filing Date
2026-05-14
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

The mechanism behind in-context learning (ICL) for knowledge retrieval in Large Language Models (LLMs) is not fully understood, affecting the reliability and efficiency of predictions.

Method used

Identify and distinguish between parametric and contextual attention heads in transformer models by analyzing predictions under closed-book and open-book settings, using relevance scores to monitor and control their contributions to prediction results.

Benefits of technology

Enhances the reliability and efficiency of LLM predictions by accurately identifying and managing the functionalities of different attention heads, allowing for better prediction control and understanding of knowledge sources.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure EP2026064174_19112026_PF_FP_ABST
    Figure EP2026064174_19112026_PF_FP_ABST
Patent Text Reader

Abstract

An apparatus is configured for identifying, out of a plurality of attention heads of a machine learning model, one or more of the attention heads of a first type and / or one or more of the attention heads of a second type using a first set of predictions and a second set of predictions. The first set of predictions are performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model. The second set of predictions are performed by providing the machine learning model with input data to perform the respective prediction, and, for each of the predictions of the second set, a prediction result of the respective prediction includes information that originates from the input data provided to perform of the respective prediction.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Attention head functionalities in machine learning models

[0002] Description

[0003] Embodiments of the present invention relate to apparatuses and methods for identifying different types of attention heads of a machine learning (ML) model. In other words, embodiments may relate to analyzing a machine learning model to identify portions of the model, which relate to specific functionalities. Further embodiments relate to apparatuses and methods for performing a prediction using a machine learning model. Further embodiments relate to an apparatus and a method for inferring a source of an output token of a prediction performed by an ML model. Further embodiments relate to an apparatus and a method for analyzing a prediction performed by using an ML model.

[0004] Many, if not most language tasks can be framed as a sequence to sequence problem [52, 60], This view is integral to how modern Large Language Models (LLMs) operate, as they are able to approximate relations between an input and and output sequence not only as a continuation of text, but also as a response to a stimulus

[0055] , In a sense, input prompts serve as a query to search and induce function(s) in a vast, high-dimensional latent space, where the corresponding process can be cast as question answering

[0043] or instruction following [49, 68],

[0005] This capability is brought forth with the introduction of in-context learning (ICL)

[0012] that enables LLMs to adapt to new tasks with few demonstrations at inference time, without additional fine-tuning. Previous work has investigated ICL from various perspectives, including its relation to induction heads that can replicate token patterns during prediction

[0047] , the ability to compress attention heads to function vectors (FVs) representing a specific task [26, 62], and how it can emerge when transformers

[0063] implicitly learn to perform gradient-based optimization [4, 65], Besides meta-learning, ICL can be used for retrievalaugmentation

[0053] , where external knowledge from web retrieval corpora [11, 24, 27] or dialogue information [57, 75, 78] is given instead of input-output pairs to ground LLMs during generation. However, the mechanism behind ICL for knowledge retrieval is not yet fully understood.

[0006] It is an objective of embodiments of the present invention to provide apparatuses and methods for analyzing internal processes of a machine learning model and / or using knowledge about internal processes or properties of an ML model for controlling and / or

[0007] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0008] analyzing a prediction performed by an ML model, e.g., to improve reliability and / or efficiency of predictions performed by ML models.

[0009] This objective is achieved by the subject-matter of the independent claims.

[0010] Embodiments of a first aspect of the invention rely on the idea to identify attention heads of a machine learning model, such as a transformer model, which attention heads perform specific functions. In particular, attention heads of a first type, e.g., referred to as parametric head type, are identified by analyzing predictions performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model, in other words, performed in a closed-book setting. Attention heads of a second type, e.g., referred to as contextual head type, or in-context head type, are identified by analyzing predictions performed by providing the machine learning model with input data to perform the respective prediction, the prediction result of the predictions used for analyzing the ML model for identifying the heads of the second type including, or resulting in, an information, which originates form the input data. In particular, the ML model is analyzed by determining relevance scores for the attention heads of the ML model, the relevance scores measuring a contribution of the respective attention head to a prediction result of the respective prediction, the identification of the attention heads of the first and second types being performed by comparing the predictions performed for identifying the first type of attention heads with the predictions performed by identifying the second type of attention heads.

[0011] In other words, embodiments of the first aspect of the invention may rely on the inventors’ finding that relevance scores of attention heads, which relevance scores are determined by back-propagating prediction results obtained by performing predictions using the ML model, can reveal different functionalities of different attention heads, when the predictions used for the analysis are performed under predefined conditions. In particular, by comparing relevance scores obtained in a closed-book setting, in which the model needs to rely on embedded knowledge, to relevance scores obtained in open-book settings, in which additional input data is provided to the model, attention heads, which contribute embedded knowledge of the ML model to a prediction result can be distinguished from attention heads, which extract knowledge from the additional input data. Identifying the attention heads of the first type and of the second type of an ML model opens the possibility to analyze any prediction performed by the model, by monitoring the activation of the identified attention heads of the first and / or second type during the prediction and / or by analyzing the

[0012] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0013] contributions, the identified attention heads and / or second type contributing to the result of a prediction. Even further, identifying the attention heads of the first and / or the second type opens the possibility of actively controlling the behavior of an ML model during a prediction.

[0014] Embodiments of the first aspect of the invention provide an apparatus (e.g., apparatus for classifying one or more attention heads of a machine learning model (e.g., a LLM or a transformer model), configured for identifying, out of a plurality of attention heads of a machine learning model, one or more of the attention heads of a first type (e.g., a parametric head type) and / or one or more of the attention heads of a second type (e.g., a contextual head type) by, for each of a first set of predictions and a second set of predictions performed by the machine learning model, determining, for each of the plurality of attention heads, for each of the predictions of the first set and the second set, a relevance score, which measures a contribution of the respective attention head to a prediction result of the respective prediction. The first set of predictions are performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model (e.g., the first set of predictions are performed in a closed-book setting). The second set of predictions are performed by providing the machine learning model with input data (e.g., the input data including an instruction information, also referred to as task information, and contextual information; e.g., the instruction information comprising a question to be answered or an instruction for a task to be performed; for example, the task is to be performed on the contextual information) to perform the respective prediction (e.g., the first set of predictions are performed in an open-book setting); and wherein, for each of the predictions of the second set, a prediction result (e.g., a predicted answer to a question, which is input to the machine learning model for the respective prediction) of the respective prediction includes information that originates (e.g., exclusively originates, e.g., the information is not included in the embedded knowledge of the machine learning model) from the input data provided to perform of the respective prediction. The apparatus is further configured for identifying the one or more of the attention heads of the first type (e.g., a contextual head type) and / or the one or more of the attention heads of the second type by assigning one or more of the attention heads to the first type of attention heads and / or assigning one or more of the attention heads to the second type of attention heads by comparing, for each of the attention heads, an average relevance of the respective attention head in the first set of predictions and an average relevance of the respective attention head in the second set of predictions.

[0015] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0016] According to an embodiment, the apparatus is configured for obtaining the first set and the second set by performing a plurality of predictions, including the first set and the second set, using the machine learning model to obtain respective prediction results by: for each of the first set of predictions, requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model; and for each of a subset out of the plurality of predictions, providing the machine learning model with input data to perform the respective prediction. The apparatus is further configured for selecting the second set of predictions out of the subset of the plurality of predictions by selecting predictions, the prediction results of which include information that originates (e.g., exclusively originates, e.g., the information is not included in the embedded knowledge of the machine learning model) from the input data provided to perform of the respective prediction.

[0017] According to an embodiment, the apparatus is configured for obtaining the first set of predictions by: performing a first plurality of predictions using the machine learning model to obtain respective prediction results by, for each of the first plurality of predictions, requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model; and for each of the first plurality of predictions, assessing the prediction result of the respective prediction with respect to a ground truth criterion (e.g., rating an extent, to which the prediction result aligns with a ground truth) to obtain an assessment result, and selecting the first set of predictions out of the first plurality of predictions based on the assessment results (e.g., including prediction, (e.g., a predetermined count of predictions), the prediction results of which align best to the ground truth, in the first set).

[0018] In other words, the first set of predictions may be selected to include predictions, the prediction result aligning, e.g., in terms of a criterion based on score rating the alignment, with the ground truth for a question or task, on which the respective prediction is performed, i.e., which is input to the prediction. Thus, the predictions, on which the identification of the attention heads of the first and / or second type is based, are filtered with respect to the degree of correctness of the predictions, thereby increasing the quality of the identification in the sense that the identified attention heads are attention heads, which contribute to providing reliable prediction results.

[0019] According to an embodiment, the apparatus is configured for obtaining the second set of predictions by: performing a second plurality of predictions using the machine learning

[0020] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0021] model to obtain respective prediction results by, for each of the second plurality of predictions, providing the machine learning model with input data to perform the respective prediction; and for each of the second plurality of predictions, assessing the prediction result of the respective prediction with respect to an extent, to which the prediction result aligns with an information included in the input data, to determine an assessment result; and selecting the second set of predictions out of the second plurality of predictions based on the assessment results determined for the second plurality of predictions (e.g., including prediction results (e.g., a predetermined count of prediction results), which align best to the ground truth, in the second set).

[0022] By selecting, out of the predictions performed by providing the additional input information, e.g., performed in an open-book setting, predictions, the result of which includes the information included in the input data, attention heads, which extract information from the input data, can be identified. For example, the input data includes information not included in the embedded knowledge of the model, so that the origin of the prediction result can be attributed to the input data. To this end, for example, the input data may include counterfactual information, so that, assuming that the ML model is trained using correct information leading to the embedded knowledge of the model being correct, a prediction result indicating the counterfactual information can be assumed to be extracted from the input data. In other words, for example, the contextual information is an information, which is not part of the embedded knowledge of the ML model.

[0023] According to an embodiment, the apparatus is configured for, in comparing the average relevance of the respective attention head in the first set of predictions and the average relevance of the respective attention head in the second set of predictions, determining, for each of the attention heads, a first average relevance score by forming an average over the relevance scores determined for the respective attention head for the first set of predictions, and a second average relevance score by forming an average over the relevance scores determined for the respective attention head for the second set of predictions.

[0024] According to an embodiment, the apparatus is configured for, in comparing the average relevance of the respective attention head in the first set of predictions and the average relevance of the respective attention head in the second set of predictions, determining a difference score representing a difference between the first and the second average relevance scores determined for the respective attention head.

[0025] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0026] By forming the difference score between the relevance of an attention head in the first set of predictions, performed by enforcing the usage of embedded knowledge, and the relevance of the attention head in the second set of predictions, selected in a manner that the prediction requires usage of contextual information, the difference score may represent a score, which contrasts the provision of embedded knowledge against the extraction of contextual information. The inventors found that such a difference score, which contrasts the two scenarios represented by the first set of predictions and the second set of predictions, is a reliable indicator for assigning an attention head to the first type or the second type. In particular, the usage of the relevance score as such might lead to less reliable classification, as a high relevance score in a prediction performed under certain conditions does not necessarily imply that the high relevance is specific to this condition. This uncertainty is overcome by comparing the relevance scores achieved under the conditions of the first set of predictions and the relevance scores achieved under the conditions of the second set of predictions.

[0027] According to an embodiment, the apparatus is configured for determining the relevance scores for the attention heads by backpropagating an output of the machine learning model through the machine learning model to the attention heads.

[0028] According to an embodiment, the apparatus is configured for, in determining the relevance score for the respective attention head: determining, for each coefficient of an attention head output of the respective attention head, a relevance share, which measures a contribution of the respective coefficient to the prediction result of the respective prediction; and in summing the relevance shares determined for the respective attention head for the respective prediction, ignoring (or set to zero) negative relevance shares.

[0029] For example, the output of the machine learning model, which is back-propagated, may be a score, which is assigned by the respective prediction to the prediction result, or to a portion thereof, which represents the answer to a question or to a task of the input to the prediction, in particular, the answer, representing the ground truth, or the input information, respectively, in the cases of the first set of predictions and the second set of predictions.

[0030] Embodiments of a second aspect of the invention rely on the idea to identify attention heads of a predetermined type of an ML model by analyzing the attention head’s contribution to a prediction result, which contribution originates from a set of tokens out of a plurality of input tokens provided to the ML model for the prediction. For example, the input tokens provided

[0031] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0032] to the ML model for the prediction may include different types of information, such as a task information, also referred to as instruction information, which comprises a task or instruction, e.g., a question to be answered, for the prediction, and further comprises contextual information, e.g., information which may be used by the ML model for performing the instruction, or information, on which the instructions are to be performed, such as, for example, a text to be translated. According to embodiments of the second aspect of the invention, a relevance score is determined, which represents a contribution of the set of tokens to the result of the prediction, which contribution is provided by the respective attention head. That is, for example, the relevance score is determined in a manner, which measures the contribution of the set of tokens to the prediction result while only considering contributions which pass through the attention head to be analyzed. The inventors found that such relevance scores can be used for identifying attention heads, which can be assigned to a function in dependence on what type of information the set of tokens, on which the analysis is performed, represents. For example, if the set of tokens represents a task information, attention heads with a high relevance score determined for the set of tokens may be assigned the function of interpreting the task information. As another example, by defining the set of tokens in a manner that the set of tokens represents contextual information, an attention head with a high relevance score may be identified as performing the function of retrieving contextual information from the input. Thus, for example, depending on the choice of the set of tokens within the input information, different functionalities of attention heads can be checked and attention heads of the ML model may be identified to perform one of these functions. For example, the plurality of attention heads, out of which the attention heads of the predetermined type are identified, are attention heads of the second type, e.g., the in-context type, e.g., as identified using one of the embodiments of the first aspect. In other words, attention heads of the second type as described with respect to the first aspect may be further sub-classified using embodiments according to the second aspect.

[0033] Embodiments according to the second aspect provide an apparatus configured for identifying, out of a plurality of attention heads of a machine learning model (e.g., a LLM or transformer model), one or more attention heads of a predetermined type (e.g., retrieval head type or a task head type) by: determining, for a set of tokens out of a plurality of input tokens provided to the machine learning model as input information for a prediction, for each of the plurality of attention heads, a relevance score (e.g., \rho^h_task / ret, e.g., according to equation (8), e.g., in combination with (7)), which represents a contribution of the set of tokens to a prediction result (e.g., y_t) of the prediction, which contribution is

[0034] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0035] provided by the respective attention head; and assigning one or more of the attention heads to the predetermined type based on the relevance scores assigned to the attention heads.

[0036] According to an embodiment, the apparatus is configured for, in determining the relevance score for the respective attention head: determining, for each of the set of tokens, a tokenindividual relevance score (e.g., \rho^h_j, e.g., equation (7)), which represents a tokenindividual contribution of the respective token to the prediction result (e.g., y_t), which tokenindividual contribution is provided by the respective attention head; and aggregating the token-individual relevance scores determined for the set of tokens for the respective attention head.

[0037] According to an embodiment, the set of tokens are part of a task information (or a question) included in the input information. Selecting the set of tokens being part of the task information may identify attention heads, which interpret the task information. According to this embodiment, the predetermined type may be a task type, and the attention heads identified to be of the task type may be referred to as task heads.

[0038] According to an embodiment, the set of tokens represent a part of the input information, which is provided in the prediction result (or which is retrieved from the input information to provide the prediction result). In other words, for example, the set of tokens may be part of contextual information of the input information. For example, the predictions used for determining the relevance scores of the attention heads may be selected such that a task of the prediction requires the retrieval of contextual information included in the input information, and in a manner that the prediction result aligns with the ground truth, that is, the prediction results include a piece of information from the contextual information of the input information. Defining the set of tokens to present a part of the input information, which is provided in the prediction result, allows to identify attention heads, which perform the function of retrieving information from the input information, which information is to be used for generating the prediction result.

[0039] According to an embodiment, the apparatus is configured for assigning one or more of the attention heads to the predetermined type by: selecting a predetermined count of attention heads having assigned the highest relevance scores among the plurality of attention heads (e.g., according to equation (9)); or using a threshold value for the relevance scores assigned to the plurality of attention heads.

[0040] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0041] According to an embodiment, the apparatus is configured for determining the relevance score by backpropagating an output of the machine learning model through the machine learning model to set of tokens, wherein the apparatus is configured for excluding propagation paths, which circumvent the respective attention head.

[0042] Embodiments according to a third aspect of the invention rely on the idea to use, for selecting an output token of a prediction performed by a machine learning model, a classifier to classify candidate tokens out of which the output token is to be selected. In particular, the selecting of the output token out of the candidate tokens is performed by weighting probability scores of the candidate tokens, which probability scores are determined by the ML model to predict the output token, in dependence on the class, to which the respective candidate token is assigned by the classifier. In other words, the decision, which of the candidate tokens to select as the output token, depends, in addition to the probability scores determined for the output tokens, on the classification of the output tokens. In even other words, for example, the classifier introduces an additional instance, which influences the selection of the output token. For example, the output token is the next token to be selected for generating a prediction result of the prediction. Using the classifier and weighting the probability score in dependence on the classification of the candidate tokens allows to influence the prediction result, e.g., in a manner, which is defined by the construction of the classifier. For example, the classifier may evaluate the prediction process, which yields the probability scores for the candidate tokens, thereby yielding an information about functional parts of the ML model, which contribute to the probability score for the respective candidate token. Such information may provide an indication on the nature of the candidate token, e.g., what type of functions of the ML model contributed to the probability score for the respective output token. In examples, the first class is a class of output tokens, which represent information retrieved from contextual information, which is input to the prediction. Additionally, or alternatively, the second class may be a class of tokens, which represent information retrieved from embedded knowledge of the ML model.

[0043] Embodiments according to the third aspect of the invention provide an apparatus for performing a prediction using a machine learning model (e.g., a LLM or transformer model), wherein the apparatus is configured for predicting an output token (e.g., a result of the prediction; e.g., a token of the result of the prediction, e.g., the next token to predict) by: deriving, for each of a set of candidate tokens (e.g., all tokens of a vocabulary of the machine learning model), a respective probability score (e.g., representing a probability for the respective candidate token being the ground truth for the predicted output token, or

[0044] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0045] representing a probability for the respective candidate token being selected as the output token); classifying candidate tokens of at least a subset of the set of candidate tokens using a classifier, which distinguishes between a set of classes, the set of classes comprising (or consisting of) a first class and a second class (wherein, e.g., in classifying the candidate tokens, for each of the candidate tokens, distinguishing between the probability score being attributable to (or caused by, e.g., predominantly contributed by) a first type of attention heads (e.g., retrieval heads) or a second type of attention heads (e.g., parametric heads) of the machine learning model); selecting one of the candidate tokens of the subset as the output token based on the probability scores; and in selecting one of the candidate tokens of the subset as the output token, weighting the probability scores of candidate tokens assigned to the first class differently than the probability scores of candidate tokens assigned to the second class.

[0046] According to an embodiment, the apparatus is configured for setting weights for weighting the probability scores of the candidate tokens assigned to the first class relative to the probability scores of the candidate tokens assigned to the second class.

[0047] According to an embodiment, the apparatus is configured for setting the weights in dependence on a user input.

[0048] For example, setting the weights in dependence on a user input may allow a user to configure the behavior of the ML model in influencing, by the user input used for setting the weights, the decision between the candidate tokens.

[0049] According to an embodiment, the user input is indicative of a preference, which distinguishes between, for generating a prediction result of the prediction: preferring information included in embedded knowledge of the machine learning model; and preferring information included in input data provided to the machine learning model for the prediction. For example, by configuring the ML model to prefer information included in the input data, hallucination of the model may be suppressed by pushing the model towards using information provided as input for generating the prediction result.

[0050] According to an embodiment, the weights are global within each of the classes (e.g., one (e.g., exactly one) weight is used for weighting the probability scores for the tokens of assigned to one class). In other words, for example, the classifier differentiates between

[0051] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0052] exactly two classes, each of which uses exactly one weight for weighting the probability scores assigned to the respective class.

[0053] According to an embodiment, the weights consist of a first weight for weighting the probability scores of the candidate tokens assigned to the first class and a second weight for weighting the probability scores of the candidate tokens assigned to the second class.

[0054] According to an embodiment, a sum of the weights has a predetermined value.

[0055] According to an embodiment, the apparatus is configured for, in setting the weights, setting a ratio between a total of weights for weighting the probability scores of the candidate tokens assigned to the first class, and a total of weights for weighting the probability scores of the candidate tokens assigned to the second class. For example, in setting this ratio, in cases, in which the number of candidate tokens assigned to the two classes deviate from each other, an overweighting of one of the classes may be avoided, such that the decision between choosing one of the candidate tokens of the first class and choosing one of the candidate tokens of the second class is set by the ratio between the total of weights for weighting the first class and the total of weights for weighting the second class.

[0056] According to an embodiment, the apparatus is configured for selecting the subset of candidate tokens out of the set of candidate tokens by selecting a number (e.g., predefined count) of candidate tokens having the highest probability scores associated therewith. In other words, out of the set of candidate tokens, a subset of candidate tokens may be preselected based on the probability scores, and the classification may be performed on the subset of candidate tokens. This preselection may avoid the computational effort, and additionally, this preselection may avoid promoting candidate tokens with low probability scores.

[0057] According to an embodiment, a probability score assigned to the first class is contributed by a first set of attention heads (e.g., of a retrieval head type) (e.g., a function of the attention heads of the first set is retrieving, or copying, information from input data of the prediction) and / or wherein a probability score assigned to the second class is contributed by a second set of attention heads (e.g., of a parametric type) (e.g., a function of the attention heads of the second set is providing embedded knowledge of the machine learning model).

[0058] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0059] According to an embodiment, the apparatus is configured for classifying each of the candidate tokens of the subset by: determining, for each of a set of attention heads of the machine learning model (e.g., the attention heads of the first type, e.g., retrieval heads), an activation score (e.g., L, equation (10)), which measures a component of an output of the respective attention head during the prediction, which component is associated to the respective candidate token (e.g., associated in terms of an unembedding matrix of the machine learning model) (e.g. which component promotes a logit of the token); determining an aggregated activation score by aggregating the activation scores determined for the set of attention heads wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to (e.g., the probe comprises, for each of the set of attention heads, a probe weight; e.g., the probe weight represents a score, which indicates a balance between the attention head being of a first type (e.g. performing the function of retrieving information from the input data) and the attention head being of a second type (e.g., performing the function of providing embedded knowledge of the model)); classifying the respective candidate token (or the probability score thereof) based on the aggregated activation score (e.g., the classification distinguishes between a first class and a second class; E.g., the token is assigned to the first class, if the aggregated activation score exceeds a predetermined threshold, and to the second class, if the aggregated activation score does not exceed the predetermined threshold).

[0060] In other words, for example, the classification is performed by analyzing, for each of the candidate tokens of the subset, deviation of the attention heads of the set of attention heads during the prediction. For example, the probe weight for an attention head may indicate to which extent a functionality of the respective attention head is attributable to a first type or a second type, e.g., the retrieval type or the parametric type. For example, the probe weight may differentiate between the first type and the second type, wherein the probe weight is not necessarily binary, but may represent a score indicating a balance between the first type and the second type. In weighting the activation scores determined for attention heads by the probe weights, the aggregated activation score forms an indicator, which represents a probability for the candidate token to be the result of the functionality of attention heads of the first type or the result of the functionality of attention heads of the second type. Accordingly, the classification according to this embodiment may provide a classification, which differentiates between candidate tokens being the result of a functionality of a first type of attention heads, such as retrieval heads, and the candidate token being the result of the functionality of a second type of attention heads, e.g., the parametric type.

[0061] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0062] According to an embodiment, the apparatus is configured for setting the weights to be zero for the candidate tokens of the first class, or to be zero for the candidate tokens of the second class. In other words, one or the other class of candidate tokens may be disregarded in selecting the output token.

[0063] Embodiments according to a fourth aspect of the invention rely on the idea to evaluate a prediction performed using an ML model with respect to the alignment or compliance with a task or instruction for the prediction, which is part of input data for the prediction by measuring, whether an activation of a predetermined attention head, or one or more predetermined attention heads, fit to a function vector, which is associated with the task information, the function vector representing expected activation of the predetermined attention head for the task information. Embodiments according to the fourth aspect may rely on the inventors’ finding that for a specific task, a function vector may be determined, which is characteristic for the output of the predetermined attention head when performing predictions on the specific task. For example, the function vector may be determined by evaluating the output of the predetermined attention head for a plurality of predictions on the task.

[0064] An embodiment according to the fourth aspect of the invention provides an apparatus configured for performing a prediction using a machine learning model (E.g., a LLM or a transformer model), the machine learning model comprising a plurality of attention heads, wherein the apparatus is configured for, in performing the prediction: inputting input data comprising task information (e.g., a question) to the machine learning model, measuring an activation of a predetermined attention head (e.g., an attention head of a task head type, e.g., as determined among the attention heads according to embodiments according to the second aspect) of the plurality of attention heads the attention head; and checking, whether the activation of the predetermined attention head fits to a function vector 448, which is associated with the task information, and which represents an expected activation of the predetermined attention head for the task information.

[0065] According to an embodiment, the apparatus is configured for checking, whether the activation of the predetermined attention head fits to a function vector (448), by: calculating a metric, which measures a distance or a deviation between the activation of the predetermined attention head and the function vector.

[0066] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0067] Embodiments according to a fifth aspect of the invention rely on the idea to exploit the assignment of attention heads to specific functionalities for determining a source of an output token of a prediction, e.g., for determining if the source of the output token is contextual information provided as input for the prediction or if the source of the output token is embedded knowledge of the ML model. In particular, embodiments according to the fifth aspect of the invention rely on the idea to determine activation scores of a set of attention heads, which activation scores measure a component of an output of the respective attention head for the prediction of the output token, which component is associated to the output token, e.g., in terms of an activation, which promotes the probability of the token to be selected as output token.

[0068] For example, the set of attention heads, for which the activation scores are determined, and aggregated, may be a set of attention heads of a predetermined type, e.g., being assigned to specific functionalities, such as extracting information from the context or from the knowledge of the model. For example, the set of attention heads may be retrieval heads and parametric heads, e.g., identified as described with respect to embodiments of the first and second aspect of the invention. According to embodiments of the fifth aspect, the activation scores for this set of attention heads are aggregated, and the aggregated activation score is used for classifying in the output token, with respect to which the activation scores are determined. The aggregated activation score may then be used for classifying the output token, or the probability score thereof. For example, this classification may be performed as described with respect to the fourth aspect. In other words, according to the fifth aspect, a classification of an output token may be performed, which classification uses, as classification criterion, the activation of a predetermined set of attention heads during the prediction of the output token. In other words, according to the fifth aspect, a classification, e.g., as described with respect to the fourth aspect, may be used to classify the association of output tokens with specific functionalities, such as context retrieval or embedded knowledge retrieval.

[0069] Embodiments according to the fifth aspect provide an apparatus configured for inferring (or predicting or estimating or determining) a source of an output token resulting from a prediction performed by a machine learning model (or a source of a probability score (or a prediction result) for the prediction) (e.g., a LLM or transformer model), wherein the apparatus is configured for: determining, for each of a set of attention heads of the machine learning model, an activation score (e.g., L, equation (10)), which measures a component of an output (326) of the respective attention head for the prediction of the output token,

[0070] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0071] which component is associated to the output token (e.g., associated in terms of an unembedding matrix of the machine learning model) (e.g. which component promotes a logit of the token); determining an aggregated activation score by aggregating the activation scores determined for the set of attention heads, wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to; and classifying the output token (or the probability score thereof) based on the aggregated activation score (e.g., the classification distinguishes between a first class and a second class; E.g., the output token is assigned to the first class, if the aggregated activation score exceeds a predetermined threshold, and to the second class, if the aggregated activation score does not exceed the predetermined threshold).

[0072] According to an embodiment, the apparatus is configured for deriving the activation score for the respective attention head from an output of the respective attention head (e.g., the output during the prediction of the output token) using an unembedding matrix of the machine learning model (e.g., by multiplying the output with a portion of the unembedding matrix, which projects on the output token).

[0073] According to an embodiment, the apparatus is configured for, in classifying the output token, using a classifier, which distinguishes between a set of classes, the set of classes comprising (or consisting of) a first class and a second class.

[0074] According to an embodiment, the apparatus is configured for, if the output token is assigned to the first class, identifying one of input tokens of the prediction as a source of the output token by: for each of a plurality of token positions (e.g., all token positions of an input sequence of the prediction, or a portion thereof) (e.g., an input of the prediction is mapped or distributed onto a sequence of tokens having respective token positions in the sequence of tokens; E.g., the sequence of tokens is propagated through layers of the machine learning model by sequentially updating, e.g., layer by layer, embedding vectors for each of the token positions of the sequence), determining an attention score by aggregating attention weights of respective attention maps of the set of attention heads, which attention weights define, for the respective attention heads, a contribution of the respective token position (j) to a token position (S) of the output token (e.g., the attention score for one of the token positions is formed by aggregating, for each of the token positions, one attention weight of each of the attention heads), wherein, in aggregating the attention weights, each of the attention weights is weighted using the probe weight of the probe, which probe weight

[0075] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0076] is associated with the attention head the respective attention weight belongs to, and weighted using the activation score determined for the attention head the respective attention weight belongs to (e.g., as defined by equation (15)); and identifying the source of the output token based on the attention scores determined for the token positions (e.g., by selecting the input token for which the aggregation of weights is maximum).

[0077] For example, the attention weights of the attention map of an attention head may indicate the extent to which the attention head maps a token of the input of the attention head at a first token position of the sequence to a second token position of the sequence in the output of the attention head. In other words, the attention maps may represent a transfer function, which indicates the transfer or coupling between input and output of the attention head for all combinations of token positions of the sequence. Thus, the attention scores represent a coupling between the currently evaluated token position to the token position of the output token.

[0078] By weighting the attention weights using the probe weight of the probe, different attention heads are weighted differently, so that a functionality or type of the attention heads may be accounted for in the determining of the attention score. For example, the weighting using the probe weights may allow to only consider attention heads of a predetermined type, such as the retrieval type, or to account for an extent to which the attention head is assignable to a certain functionality, such as contextual input retrieval. Thus, when aiming for localizing the source of the output token in the input sequence, contributions from attention heads, which provide embedded knowledge, may be suppressed, while promoting the contribution of attention heads to the attention score, which are of the retrieval type.

[0079] Furthermore, by weighting the attention weights using the activation score determined for the attention head, the attention heads contribution to the logit or probability, which promoted or caused the selection of the output token, may be selectively considered while excluding or underweighting activations, which map to other tokens.

[0080] For example, the plurality of token positions, for which the attention scores are determined, may be the input tokens, in other words, the attention score may represent, for each of the plurality of token positions, a score, which rates the probability, to which the respective token position is the source for the output token.

[0081] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0082] According to an embodiment, the apparatus is configured for identifying the source of the output token by identifying the token position, for which maximum the attention score is maximum (among the token positions).

[0083] According to an embodiment, the apparatus is configured for, in classifying the output token, comparing the aggregated activation score to a threshold.

[0084] According to an embodiment, the apparatus is configured for the classification distinguishing between the source of the output tokens being one of input tokens of the prediction or the source of the output token being an embedded knowledge of the machine learning model.

[0085] According to an embodiment, the probe is linear, e.g., a linear classifier, or the probe is nonlinear, e.g., a non-linear classifier (for example, the probe may be implemented by means of a non-linear network (e.g., as a probe for the task of determining context-groundedness vs parametric groundedness of a token). For example, a weight for each input attention head may be approximated via gradient computations. Alternatively, relevance computations may be used for determining the weights, e.g., as described in appendix A.2 of the description).

[0086] According to an embodiment, the probe is configured for distinguishing between a first type of attention heads (e.g., retrieval heads) and a second type of attention heads (e.g., parametric heads).

[0087] According to an embodiment, the probe is configured to comprise a probe weight for each of the attention heads of the set of attention heads.

[0088] According to an embodiment, the set of attention heads is a subset of a plurality of attention heads of the machine learning model (e.g., the attention heads of the set of attention heads have a predetermined type, e.g., in terms of their functionality; E.g., the attention heads of the set of attention heads are retrieval heads, e.g., the set of attention heads is defined as the set of attention heads assigned to the second type according to the means defined in any of the embodiments of the first aspect, or as the set of attention heads assigned to the predetermined type according to the means defined in any of the embodiments of the second aspect.).

[0089] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0090] Embodiments according to a sixth aspect of the invention rely on the idea to compare an output of an attention head during a prediction performed by an ML model to each of a plurality of function vectors. For example, a function vector may be a vector, which is characteristic for the output of the attention head when performing a task, for which the function vector is determined or, in other words, to which the function vector is associated. For example, each of the plurality of function vectors may be a function vector as described with respect to the fourth aspect, wherein each of the plurality of function vectors may be associated with a different task. Accordingly, by comparing the output of the attention head to the plurality of function vectors, an indication for the task, which is actually performed by the ML model during the prediction may be obtained. For example, such analysis of the prediction of the ML model may be used to evaluate if the ML model, or to which extent the ML model performs the task included in the input information.

[0091] Embodiments according to the sixth aspect provide an apparatus for analyzing a prediction performed by using a machine learning model (e.g., a LLM or transformer model), configured for: comparing an output (e.g., an output for, or referring to, a currently predicted token, e.g., an output for a predetermined token position, e.g., z^h_S) of a predetermined attention head of the machine learning model during the prediction (or caused by the prediction) to each of a plurality of function vectors (e.g., a function vector refers to a vector of predetermined values, the vector having the same dimensionality as the output).

[0092] According to an embodiment, the apparatus is configured for, in comparing the output of the predetermined attention head (h) of the machine learning model to each of the plurality of function vectors, determining respective distance measures indicating a distance between the output and the respective function vector.

[0093] For example, the distance measure may be a measure as described with respect to the fourth aspect.

[0094] According to an embodiment, the predetermined attention head is of a predetermined attention head type (e.g., an attention head of a task head type, e.g., as determined among the attention heads according to embodiments of the second aspect).

[0095] According to an embodiment, the predetermined attention head is part of a set of attention heads (e.g., of a predetermined type, e.g., a task head type, e.g., as determined among the attention heads according to embodiments according to the second aspect), and wherein

[0096] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0097] the apparatus is configured for comparing, for each of the attention heads of the set of attention heads, an output of the respective attention head during the prediction (or caused by the prediction) to each of the a plurality of function vectors (e.g., a respective plurality of function vectors, which is specific to the respective attention head; e.g., each of the respective pluralities of function vectors for the attention heads may comprise a function vector for each of a plurality of task).

[0098] By checking a plurality of attention heads with respect to respective function vectors, a precise indication for the functionality performed by the ML model during the prediction can be obtained. For example, for each of a plurality of tasks, a respective function vector for each of the set of attention heads may be available, which is characteristic for the output of the respective attention head when performing the task, for which the function vectors for the attention heads are determined.

[0099] According to an embodiment, each of the function vectors has a prediction task associated therewith.

[0100] According to an embodiment, the apparatus is configured for using a result of the comparison (e.g., using the distance measures) for selecting one of the function vectors out of the plurality of function vectors, and providing (or outputting, e.g., on a user interface) an indication of the function vector, or an indication of a prediction task associated with the function vector.

[0101] In other words, for example, the selected function vector may be indicative of the function or task performed by the ML model during the prediction.

[0102] Embodiments according to a seventh aspect of the invention rely on the idea to use a function vector, which, e.g., is associated with a specific function or task, and, e.g., represents a characteristic output of a predetermined attention head in a prediction performed by an ML model on this function or task, for controlling a prediction performed using the ML model. For example, the function vector may be used to set an output of the predetermined attention head in performing the prediction to be controlled. The inventors found that controlling the output of predetermined attention heads, e.g., task heads as identified using embodiments of the second aspect, during a prediction may increase the adherence to the task or function, e.g., even without explicitly including the task in the prediction input. Even further, independent of whether or not the task is included in the

[0103] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0104] prediction input, the usage of function vectors may improve the adherence to the task, as this manner of imposing an instruction to the prediction is independent of a correct interpretation of a task information included in the prediction input.

[0105] Embodiments according to the seventh aspect provide an apparatus for performing a prediction using a machine learning model, the machine learning model comprising a plurality of attention heads. The apparatus is configured for setting a plurality of output activations (or output values) of an output of a predetermined attention head (e.g., a task head determined according to embodiments of the second aspect) using predetermined values (e.g. defined by a function vector), which are independent of an input of the predetermined attention head (e.g., setting the plurality of output activations to the predetermined values).

[0106] According to an embodiment, the apparatus is configured for selecting the predetermined values based on an input information (e.g., representing a setting for the prediction), wherein the input information differs from (e.g., is additional to) a prediction input provided as an input to the machine learning model (e.g., to an input projection of the ML model) for performing the prediction (e.g., in other words, the input information is not part of a sequence of input tokens of the prediction, which is propagated through the model).

[0107] According to an embodiment, the output of all attention head (e.g., all layers of the ML model), which precede the predetermined attention head in a propagation stream of the machine learning model, is independent of the input information.

[0108] According to an embodiment, the predetermined values represent a function vector, which is associated with a function or a task.

[0109] According to an embodiment, the predetermined values comprise, for each of the plurality of output activations, a respective predetermined value.

[0110] According to an embodiment, the plurality of output activations comprises, or consists of, output activations for the ultimate token position of a sequence of token positions, the output of the predetermined attention head comprising, for each of the sequence of token positions, a respective output activation vector.

[0111] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0112] According to an embodiment, the apparatus comprises an input interface, which is configured for receiving a prompt information and a setting information, wherein the apparatus is configured for: determining a sequence of input tokens from the prompt information, and providing the sequence of input tokens as an input of the machine learning model, the input being independent of the setting information; and selecting the predetermined values based on the setting information.

[0113] According to an embodiment, the predetermined attention head is configured for determining, for each token position of a sequence of tokens (e.g., of a current recursion of a prediction), an output vector comprising a plurality of output activations (e.g., output activation values, e.g., the vector has the length of a dimension of the attention head). According to this embodiment, the apparatus is configured for setting each of the output activations of the output vector for the ultimate token position of the sequence of tokens to a respective predetermined value (e.g., defined by a function vector having the length of the dimension of the attention head).

[0114] According to an embodiment of any of the aspects of the invention, an input for the prediction (e.g., the input data) is provided in form of one or more of an audio signal, a video signal, an image signal, or a communication signal (e.g., a signal of a communication channel, e.g., aG, broadband communication, LTE, e.g., as measured by an antenna, e.g., an antenna of the apparatus).

[0115] According to an embodiment of any of the aspects of the invention, an input for the prediction is a multimodal signal (e.g., questions and system instructions could be text based („you are a helpful assistant with the purpose to retrieve data”), yet the given context is image, video, and / or audio.

[0116] A further embodiment according to the first aspect provides a method for comprising identifying, out of a plurality of attention heads of a machine learning model, one or more of the attention heads of a first type and / or one or more of the attention heads of a second type by: for each of a first set of predictions and a second set of predictions performed by the machine learning model, determining, for each of the plurality of attention heads, for each of the predictions of the first set and the second set, a relevance score, which measures a contribution of the respective attention head to a prediction result of the respective prediction. The first set of predictions are performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine

[0117] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0118] learning model. The second set of predictions are performed by providing the machine learning model with input data to perform the respective prediction; and wherein, for each of the predictions of the second set, a prediction result of the respective prediction includes information that originates from the input data provided to perform of the respective prediction. The method further comprises assigning one or more of the attention heads to the first type of attention heads and / or assigning one or more of the attention heads to the second type of attention heads by comparing, for each of the attention heads, an average relevance of the respective attention head in the first set of predictions and an average relevance of the respective attention head in the second set of predictions.

[0119] A further embodiment according to the second aspect provides a method for comprising: identifying, out of a plurality of attention heads of a machine learning model, one or more attention heads of a predetermined type by: determining, for a set of tokens out of a plurality of input tokens provided to the machine learning model as input information for a prediction, for each of the plurality of attention heads, a relevance score, which represents a contribution of the set of tokens to a prediction result of the prediction, which contribution is provided by the respective attention head; and assigning one or more of the attention heads to the predetermined type based on the relevance scores assigned to the attention heads.

[0120] A further embodiment according to the third aspect provides a method for comprising: performing a prediction using a machine learning model, wherein the method comprises predicting an output token by: deriving, for each of a set of candidate tokens, a respective probability score; classifying candidate tokens of at least a subset of the set of candidate tokens using a classifier, which distinguishes between a set of classes, the set of classes comprising a first class and a second class; selecting one of the candidate tokens of the subset as the output token based on the probability scores; and, in selecting one of the candidate tokens of the subset as the output token, weighting the probability scores of candidate tokens assigned to the first class differently than the probability scores of candidate tokens assigned to the second class.

[0121] A further embodiment according to the fourth aspect provides a method for comprising performing a prediction using a machine learning model, the machine learning model comprising a plurality of attention heads, wherein the method comprises, in performing the prediction: inputting input data comprising task information to the machine learning model; measuring an activation of a predetermined attention head of the plurality of attention heads the attention head; and checking, whether the activation of the predetermined attention

[0122] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0123] head fits to a function vector, which is associated with the task information, and which represents an expected activation of the predetermined attention head for the task information.

[0124] A further embodiment according to the fifth aspect provides a method for inferring a source of an output token resulting from a prediction performed by a machine learning model (or a source of a probability score, wherein the method comprises: determining, for each of a set of attention heads of the machine learning model, an activation score, which measures a component of an output of the respective attention head for the prediction of the output token, which component is associated to the output token; determining an aggregated activation score by aggregating the activation scores determined for the set of attention heads wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to; classifying the output token based on the aggregated activation score.

[0125] A further embodiment according to the sixth aspect provides a method for analyzing a prediction performed by using a machine learning model, wherein the method comprises: comparing an output of a predetermined attention head (h) of the machine learning model during the prediction to each of a plurality of function vectors.

[0126] Brief description of the figures

[0127] Embodiments of the present disclosure are described in more detail below with respect to the figures, among which:

[0128] Fig. 1 illustrates an example of an ML model,

[0129] Fig. 2 illustrates an example of a multi-head attention,

[0130] Fig. 3 illustrates an apparatus for identifying an attention head of a first type and / or an attention head of a second type according to an embodiment of the first aspect,

[0131] Fig. 4A illustrates a determination of the first set of predictions according to an embodiment,

[0132] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0133] Fig. 4B illustrates a determination of the second set of predictions according to an embodiment,

[0134] Fig. 5 illustrates an apparatus for identifying an attention head of a predetermined type according to an embodiment of the second aspect,

[0135] Fig. 6 illustrates an apparatus for performing a prediction according to an embodiment of the third aspect,

[0136] Fig. 7 illustrates an apparatus for performing a prediction according to another embodiment of the third aspect,

[0137] Fig. 8 illustrates an apparatus for performing a prediction according to an embodiment of the fourth aspect,

[0138] Fig. 9A illustrates an apparatus for inferring a source of an output token according to an embodiment of the fifth aspect,

[0139] Fig. 9B illustrates an apparatus for inferring a source of an output token according to another embodiment of the fifth aspect,

[0140] Fig. 10 illustrates an apparatus for analyzing a prediction according to an embodiment of the sixth aspect,

[0141] Fig. 11 illustrates an apparatus for performing a prediction according to an embodiment of the seventh aspect,

[0142] Fig. 12 illustrates a functional map of in-context and parametric heads according to an embodiment,

[0143] Fig. 13 illustrates a recoil analysis according to an embodiment,

[0144] Fig. 14 illustrates sorted in-context scores according to an embodiment,

[0145] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0146] Fig. 15A illustrates an extraction and insertion of task function vectors according to an embodiment,

[0147] Fig. 15B illustrates an extraction and insertion of parametric function vectors according to an embodiment,

[0148] Fig. 16A illustrates a fine grained detection of knowledge source according to an embodiment,

[0149] Fig. 16B illustrates causal source tracking of retrieved documents according to an embodiment,

[0150] Fig. 17A illustrates a functional map of in-context and parametric heads in Mistral-7B- lnstruct-v0.3 according to an embodiment,

[0151] Fig. 17B illustrates a functional map of in-context and parametric heads in Gemma-2- 9B-it according to an embodiment,

[0152] Fig. 18A illustrates sorted in-context scores for Mistral-7B-Instruct-v0.3 according to an embodiment,

[0153] Fig. 18B illustrates sorted in-context scores for all heads of Gemma-2-9B-it according to an embodiment,

[0154] Fig. 19 illustrates a distribution of logit lens scores according to an embodiment,

[0155] Fig. 20 illustrates heat maps of the weighted aggregation of retrieval head attention maps according to an embodiment,

[0156] Fig. 21 illustrates an example of a counterfactual prediction input according to an embodiment,

[0157] Fig. 22 shows an example for a biography entry,

[0158] Fig. 23A shows Table 1: Zero-shot recall scores for task, parametric, and retrieval heads,

[0159] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0160] Fig. 23B shows Table 2: Performance of the retrieval head probe across models,

[0161] Fig. 24 shows Table 3: An example of our data set with the original (orig) and paraphrased (para) biography entry,

[0162] Fig. 25 shows Table 4: Overview of models’ performance,

[0163] Fig. 26 shows Table 5: Task Heads’ FVs evaluation,

[0164] Fig. 27A shows Table 6: Recall scores.

[0165] Fig. 27B shows Table 7: Cloze-style statements and question prompts used to extract and patch parametric function vectors,

[0166] Fig. 28 shows Table 8: Qualitative examples of parametric function vectors patching for all models.

[0167] Detailed Description

[0168] Embodiments of the present invention are now described in more detail with reference to the accompanying drawings, in which the same or similar elements or elements that have the same or similar functionality have the same reference signs assigned or are identified with the same name. In the following description, a plurality of details is set forth to provide a thorough explanation of embodiments of the disclosure. However, it will be apparent to one skilled in the art that other embodiments may be implemented without these specific details. In addition, features of the different embodiments described herein may be combined with each other, unless specifically noted otherwise.

[0169] Fig. 1 illustrates an example of an M L model 110. Embodiments described below may, when referring to an ML model, optionally operate on ML models according to the ML model 110 described with respect to Fig. 1. However, details described with respect to Fig. 1 are not to be understood limiting. Further embodiments may operate on ML models different from the one described with respect to Fig. 1. ML model 110 comprises a transformer block 120, which comprises a multi-head attention 124. For example, transformer 120 comprises a plurality of layers, each of which comprises a transformer block, each transformer block

[0170] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0171] comprising a multi-head attention. ML model 110 may also be referred to as transformer model. For example, ML model 110 may be a large language model LLM. Alternatively, ML model may be another type of generative model, e.g., a vision transformer. ML model 110 receives a model input 108, which may be tokenized, that is, model input 108 may comprise a sequence of tokens. ML model 110 determines a prediction result 112 based on the model input 108. In other words, ML model performs a prediction on model input 108 to obtain the prediction result 112. For example, ML model 110 performs the prediction autoregressively. To this end, ML model 110 may input the sequence of tokens of the model input 108 to transformer 120 to obtain a transformer output 122. The transformer output 102 is subjected to an output projection 130, for example, a linear protection to obtain logits 132. For example, the logits 132 may comprise one column or vector of scores for each token of the sequence of tokens, the size of the vector per token corresponding to the size of a vocabulary of the ML model 110. In other words, the logits may be unnormalized probability scores. Logits 132 are subjected to a normalization, e.g., by a softmax function, see block 140 of Fig. 1, to obtain probability scores 142. The probability scores 142 may comprise, for each token of the sequence of tokens, a probability vector, which assigns a probability to each entry of the vocabulary of the ML model 110. For example, the probability scores for a token of the sequence may indicate the probabilities for the subsequent token of the respective position in the sequence of tokens. The probability scores for the last position of the current sequence of tokens may be used in a token selection 150 to select a new token to be appended to the sequence of tokens. The such extended sequence of tokens is then again input to transformer 120. This loop may be continued until a stop criterion is met, the thus obtained sequence of tokens representing the output of the prediction. For example, the prediction result 112 may be the part of the sequence of tokens appended to the sequence of tokens during the prediction period.

[0172] Fig. 2 illustrates an example of a multi-head attention, e.g., as it may optionally be implemented in multi-head attention 124 of Fig. 1. The multi-head attention of Fig. 2 comprises a plurality of attention heads. Note that while in Fig. 2, h indicates the number of heads, in the following description, index h may be used to index the attention heads. Each of the attention heads receives the input of the multi-head attention and provides an intermediate attention output, which comprises, for each token of the sequence, a vector of dimension Dh. The intermediate attention outputs of the attention heads are subjected to an output transformation Wo. The output transformation may comprise a concatenation and transformation, or, alternatively, a transformation of the individual intermediate attention outputs followed by a summation, e.g., according to equation (3) below. The output X of

[0173] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0174] each head of the multi-head attention comprises for each position of the sequence of tokens a vector, which has the length corresponding to the model dimension.

[0175] In the following, embodiments of the invention will be described. The embodiments described in the following may optionally be implemented using one or more ML models as described with respect to Fig. 1 and / or using multi-head attention as described with respect to Fig. 2. For example, attention heads 24 described below may optionally be implemented as described in Fig. 2. However, different implementations of ML models and multi-head attentions may be used alternatively.

[0176] Fig. 3 illustrates an apparatus 10 according to an embodiment of the first aspect. Apparatus 10 is configured for identifying, out of a plurality of attention heads 24 of a machine learning model 14, one or more attention heads of a first type, e.g., a parametric type, and / or one or more attention heads of a second type, e.g., a contextual or in-context type. In other words, apparatus 10 may classify one or more attention heads of the ML model 14, e.g., a transformer model, e.g., a LLM or a vision transformer. ML model 14 comprises a plurality of attention heads, which may be part of a plurality of multi-head attention blocks 26, each comprising a plurality of attention heads 24. For example, ML model 14 may be implemented as described with ML model 110 of Fig. 1 and / or attention heads 24 may be implemented like the attention heads shown in Fig. 2. Apparatus 10 comprises a relevance determination block 20, which determines, for each of a first set of predictions and a second set of predictions performed by the ML model 14, for each of the plurality of attention heads of the ML model 14 a relevance score 22, which measures a contribution of the respective attention head to a prediction result 16 of the respective prediction. In other words, for example, relevance determination block 20 may analyze the first set of predictions and the second set of predictions by determining, based on the prediction results for the respective predictions of the first set of predictions and the second set of predictions, for each of the attention heads, a relevance score 22, which measures a contribution of the respective attention head to the respective prediction result 16.

[0177] For each of the predictions, a prediction input may be input to ML model 14 to obtain the prediction result 16 for the respective prediction, for example, as described with respect to the model input 108, from which prediction result 112 is predicted as described with respect to Fig. 1. For illustrative purposes, in Fig. 3, the predictions are identified using the respective input signal, i.e., in the prediction input. In other words, for each of the first set of

[0178] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0179] prediction, a respective prediction input 28 is input to the ML model, and for each of the second set of prediction, a respective prediction input 38 is input to the ML model.

[0180] For example, the prediction input 28 for the first set of predictions may comprise an instruction information, also referred to as task information, generally speaking, an information on which the respective prediction is performed, which may be regarded as an instruction to ML model to generate the prediction result 16. Throughout all embodiment, an instruction information or task information may refer to any information, which represents a task to be performed for the respective prediction, e.g., a question to be answered, an instruction, which describes content to be generated, or a request for a translation of an input text.

[0181] In case of the prediction input 28 for the first set of predictions of embodiment of the first aspect, the task information may be limited to tasks, which can be performed in a closed-book setting, that is, without contextual information being provided in the prediction input. In other words, for example, prediction input 28 may consist of the task information.

[0182] The prediction input for each of the second set of predictions 38 includes input data 36. For example, input data 36 for a prediction 38 of the second set includes an instruction information, and additionally includes contextual information, e.g., information which is additional to the instruction information, e.g., which does not represent an instruction, e.g., a task or a question, but information, which may be used by the model to obtain the prediction result 16.

[0183] The first set of predictions 28 are performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model. For example, the prediction input for the first set of predictions does not include contextual information, e.g., the prediction input for the first set of predictions may consist of an instruction information, which does not include the information, which, according to the instruction represented by the instruction information is to be used or retrieved to generate the prediction result for the respective prediction.

[0184] Each of the second set of predictions 38 is performed by providing the machine learning model 14 with input data 36 to perform the respective prediction. Furthermore, for each of the predictions of the second set, the respective prediction result includes information that originates from the input data 36 provided to perform the respective prediction. For example,

[0185] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0186] the latter criterion may be achieved by selecting the predictions of the second set in a manner, which ensures that the prediction result is not part of the embedded knowledge of the ML model 14. In this case, as the information is part of the input data 36, a prediction result including this information, must originate from the input data 36.

[0187] The relevance scores 22 determined for each of the attention heads 24 for each of the predictions of the first set and the second set of predictions are provided to attention head identifier 30, which assigns one or more of the attention heads of ML model 14 to the first type of attention heads and / or signs one or more attention heads of the ML model 14 to the second type of attention heads by comparing, for each of the attention heads an average relevance of the respective attention head in the first set of predictions and an average relevance of the respective attention head in the second set of predictions. For example, the average relevance for an attention head may be determined by an averaging metric.

[0188] In the following, further optional details on how to derive the first set of predictions and the second set of predictions will be described.

[0189] Fig. 4A illustrates a determination of the first set of predictions 28 according to an embodiment. According to this embodiment, apparatus 10 obtains the first set of predictions 28 by performing a first plurality of predictions 28’ using the machine learning model 14 to obtain respective prediction results 18. The predictions of the first plurality of predictions are performed by, for each of the first plurality of predictions, requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model. In other words, the first plurality of predictions may be performed in a closed-book setting. For example, prediction input of the predictions 28’ does not include information, from which a prediction result, which aligns with a ground truth for the instruction information included in the prediction input, is derivable. For example, the prediction input for each of the predictions 28’ consists of instruction information only.

[0190] According to the embodiment of Fig. 4A, apparatus 10 further comprises a prediction selector 40, which selects the first set of predictions 28 out of the first plurality of predictions 28’. To this end, prediction selector 40 assesses each of the first plurality of predictions 28’ by assessing the prediction result of the respective prediction 28’ with respect to a ground truth information 19 to obtain an assessment result for the respective prediction. For example, the prediction result 18 is compared to the ground truth information for the respective prediction, i.e., a ground truth for the task (e.g., question) of the prediction input

[0191] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0192] for the respective prediction. E.g., the ground truth may be provided as part of a the data set, which comprises the prediction input for the predictions, and on which apparatus 10 performs the identification of the attention heads of the first and second types. For example, the assessment of the prediction results may be performed by checking, if and / or to which extent the prediction result matches the ground truth information 19. For example, the assessment may be performed by checking to which extent the prediction result includes the ground truth information. E.g., the assessment may be performed by a recall method, the assessment result being a recall score. Prediction selector 40 then selects the first set of predictions 28 out of the first plurality of predictions 28’ based on the assessment results determined for the predictions 28’. For example, prediction selector 40 may include predictions, the prediction results of which align best to the ground truth information, in the first set of predictions, for example, by including predictions, the assessment result of which exceeds a predetermined threshold, or by including a predefined count of predictions into the first set of predictions.

[0193] Fig. 4B illustrates a determination of the second set of predictions 38 according to an embodiment. According to this embodiment, apparatus 10 obtains the second set of predictions 38 by performing a second plurality of predictions 38’ using the machine learning model 14 to obtain respective prediction results. For each of the second plurality of predictions 38’, apparatus 10 provides the machine learning model 14 with input data 36’ to perform the respective prediction 38’. For example, the input data includes instruction information and contextual information, e.g., as described before. For example, the contextual information may be information, which is not part of the embedded knowledge of model 14. For example, the instruction information, e.g., the task or question, may be of a kind that carrying out the instruction carried in the instruction information requires information of the contextual information included in the input data 36’.

[0194] According to the embodiment of Fig. 4B, apparatus 10 comprises a prediction selector 40, e.g., as described with respect to Fig. 4A, which, in the case of selecting the predictions of the second set, assesses the prediction results 18 by, determining, for each of the second plurality of predictions 38’, an assessment result, by assessing the prediction result of the respective prediction 38’ with respect to an extent, to which the prediction result aligns with an information included in the input data 36’. In other words, prediction selector 40 may assess whether the prediction result of the respective prediction 38’ aligns with an information, which according to the contextual information provided as input, is a ground truth information for the prediction, e.g., the extent to which a question is answered in

[0195] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0196] accordance with the information included in the contextual information. For example, assessing the alignment of the prediction result with the information, is performed as described with respect to Fig. 4A for the first plurality of predictions. For example, as described with respect to Fig. 4A, a recall method may be used to determine a recall score, which rates the alignment of the prediction results 18 of the predictions 38’ with the information included in the input data 36’.

[0197] In the following, continuing with the description of Fig. 3, further details of the identification module 30, which identifies the attention heads of the first type and / or the second type based on the relevance scores 22 determined for the attention head are described.

[0198] According to an embodiment, identification module 30 determines, for each of the attention heads 24, a first average relevance score by forming an average over the relevance scores determined for the respective attention head 24 for the first set of predictions, and a second average relevance score by forming an average over the relevance scores determined for the respective attention head for the second set of predictions.

[0199] For example, the relevance score for an attention head for one of the predictions may be determined as described with respect to equation 4 below. In other words, the relevance score assigned to an attention head may be determined by summing over relevance scores assigned to each of the output coefficients of the respective attention head, which is, in the example described below, a vector of Dhcoefficients for each of the number of S token positions.

[0200] According to an embodiment, identification module 30 determines a difference score representing a difference between the first and the second average relevance scores determined for the respective attention head.

[0201] For example, the difference score may be calculated as defined in equation (5) below. For example, in this equation, Exmay represent an averaging metric, such as, for example, the mean.

[0202] In the following, further optional details for determining the relevance scores for the attention heads are described.

[0203] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0204] According to an embodiment, relevance determinator 20 determines the relevance scores for the attention heads 24 by backpropagating an output of the machine learning model through the machine learning model to the attention heads 24.

[0205] For example, the output, which is backpropagated, may be an activation or a score of a token of the prediction result 16, which token represents, or aligns with, the ground truth information for the respective prediction, e.g., the ground truth information, based on which the respective prediction was selected to be part of the first or second set of predictions. For example, the score, which is backpropagated may be a score or activation assigned to the vocabulary entry selected for this token. In other words, the output, which is backpropagated, may be the score, e.g., the logit, assigned to the token at the token position of the output sequence, which token matches or represents the ground truth information.

[0206] For example, the score, which is backpropagated may be the logit, e.g., as described in Fig.

[0207] 1.

[0208] According to an embodiment, in determining the relevance score for the respective attention head, apparatus 10 determines, for each coefficient of an attention head output of the respective attention head, a relevance share, which measures a contribution of the respective coefficient to the prediction result 16 of the respective prediction. The relevance shares for the coefficients of the attention head may be summed, wherein, in forming the sum of the relevance shares, negative relevance shares may be ignored, or set to zero. In other words, negative relevance shares may be excluded from determining the relevance score of the attention head, e.g., as described with respect to equation 1 below.

[0209] For example, the backpropagation of the output, e.g., referred to as initial relevance score for the backpropagation, may be backpropagated by distributing a relevance score at a node of the ML predictor 14 onto predecessor nodes of the node according to, for each of the predecessor nodes, a fraction associated with the respective predecessor node. The fraction associated with the respective predecessor node is determined depending on a further fraction at which the activation of the respective predecessor node contributes to the activation of the node in the prediction. For example, the relevance score for the node is determined by summing up, according to a summation function, the relevance scores distributed from successor nodes of a further node so as to obtain the relevance score for the further node.

[0210] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0211] For example, the back-propagation may be performed by a layer-wise relevance propagation, e.g., as described in WO 2025 / 163210 A1, or in reference [2],

[0212] In the following, embodiments of further aspects of the invention will be described. In any of these embodiments, when referring to attention heads of the first type and / or of the second type, e.g., when referring to parametric heads or in-context heads, these heads may optionally be determined as described with respect to Fig. 3, Fig. 4A, and Fig. 4B.

[0213] Fig. 5 illustrates an apparatus 210 according to an embodiment of the second aspect. Apparatus 210 is configured for identifying, out of a plurality of attention heads 24 of a machine learning model 40, one or more attention heads of a predetermined type. For example, the predetermined type is a retrieval head type or a task head type. The apparatus 210 comprises a relevance score determinator 220, which determines, for a set of tokens 229 out a plurality of input tokens 228 provided to the machine learning model 40 an input information for a prediction, for each of the plurality of attention heads 24, a relevance score 222. Relevance score 222 represents a contribution of the set of tokens to a prediction result 16 of the prediction, which contribution is provided by the respective attention head. Apparatus 210 further comprises an attention head identification 230 which assigns one or more of the attention heads 24 to the predetermined type based on the relevance scores 222 assigned to the attention heads 24.

[0214] For example, with respect to ML model 14 and attention heads 24, the description of Fig. 3 may optionally apply equivalently.

[0215] For example, the plurality of attention heads, for the relevance score 222 is determined, and out of which the attention heads of the predetermined type are determined, may be a subset of all attention heads of an ML model 14, e.g., attention heads identified as attention heads of the second type or an in-context type as described above with respect to apparatus 10. In other words, apparatus 210 may perform a sub-classification of attention heads classified as in context heads, e.g., as described with respect to apparatus 10.

[0216] For example, the input information for the prediction, also referred to as prediction input, comprises, e.g., represented by the input tokens 228, instruction or task information and, in addition to the instruction information, contextual information, wherein the latter is optional.

[0217] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0218] For example, the relevance score 222 may be determined as described with respect to relevance score 22 of Fig. 3, however, for determining relevance score 222, the back propagation may be performed back to the set of tokens. In back propagating the output of the machine learning model for the prediction through the machine learning model to the set of tokens, only propagation paths, which pass through the respective attention head, for which relevance score 222 is determined, may be considered. In other words, propagation paths, which circumvent the respective attention head, may be excluded from the determination of the relevance score 222. In yet other words, the relevance score 222 may be a relevance score which represents a contribution of the set of tokens to the output of the prediction, wherein, in determining the relevance score, the propagation paths are filtered with respect to the attention head, for which the relevance score is to be determined.

[0219] According to an embodiment, apparatus 10 determines the relevance score 222 for the respective attention head by determining, for each of the set of tokens, a token individual relevance score, which represents a token individual contribution of the respective token to the prediction result, which token individual contribution is provided by the respective attention head. Apparatus 10 may aggregate the token individual relevance scores determined for the set of tokens for the respective attention head to obtain the relevance score 222 for the respective attention head.

[0220] According to an embodiment, the set of tokens are part of a task information included in the input information.

[0221] According to another embodiment, the set of tokens 229 represent a part of the input information, which is provided at the prediction result. In other words, the set of tokens represents a part of the input information, which represents contextual information, based on which the prediction result is determined, i.e., which is retrieved from the input information to provide the prediction result.

[0222] For example, the prediction, based on which the relevance score 222 is determined, may be a prediction of the type as described for predictions 38 of the second set of predictions of Fig. 3. In other words, the prediction may be a prediction, for which the prediction input includes instruction information and contextual information, and for which the prediction result 16 comprises, or consists of, an information included in the contextual information of the input information, the prediction result aligning to the instruction information, e.g., as described with respect to Fig. 3. In particular, the combination of instruction information and

[0223] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0224] contextual information may be of a manner that excludes the prediction result to be retrieved from the embedded knowledge of the model 14, e.g., by using counterfactual information as contextual information. This type of prediction may be used in particular in cases in which the predetermined type is a retrieval pad type, referring to attention heads, which retrieve contextual information from the input data. However, same predictions may also be used for identifying attention heads of a task type. However, in case of the task type, the input information does not necessarily include contextual information.

[0225] According to an embodiment, attention head identification 230 assigns one or more of the attention heads 24 to the predetermined type by selecting the predetermined count of attention heads having assigned the highest relevance scores among the plurality of attention heads. Alternatively, attention head identification 230 may use a threshold value for the relevance scores assigned to the plurality of attention heads to assign attention heads to the predetermined type, e.g., assigning attention heads having a relevance score 222 exceeding the threshold to the predetermined type.

[0226] For example, the identification of attention heads of the predetermined type may be performed as described in section 6, in particular section 6.1 below. For example, the token individual relevance score may be calculated as described with respect to equation 7 below. The relevance score 222 may be determined as described with respect to equation 8 below. The assignment of attention heads to the predetermined type performed by attention head identification 230 may be performed in accordance with equation 9 below.

[0227] Fig. 6 illustrates an apparatus 310 for performing a prediction using a machine learning model 14 according to an embodiment of the third aspect. Apparatus 310 subjects input information 328 of the prediction to ML model 14 to generate a prediction result 16. Apparatus 310 predicts an output token 352, e.g., one of one or more output tokens of the prediction result 316, by deriving, for each of a set of candidate tokens, a respective probability score 337, see block 335 of Fig. 6. Block 335 providing the probability scores 337 may be an output projection of the ML model, optionally including a normalization. In other words, the probability scores 337 according to Fig. 6 may, for example, be logits, e.g., logits 132 described with respect to Fig. 1, or, alternatively, normalized probability scores, e.g., probability scores 142 as described with respect to Fig. 1. Optionally, apparatus 310 may select a subset 342 out of the set of candidate tokens, indicated as optional block 340 in Fig. 6. Alternatively, the subset 342, on which the classification is performed, corresponds to the entire set of candidate tokens. Apparatus 310 comprises a classifier, which classifies

[0228] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0229] candidate tokens of at least a subset 342 of the set of candidate tokens, the classifier 370 distinguishing between a set of classes, which comprises a first class and a second class. In examples, the set of classes consists of the first class and the second class. The classification may distinguish between the probability score being attributable to a first type of attention heads, e.g., retrieval heads, or a second type of attention heads, e.g., parametric heads. The apparatus 310 further comprises a token selector 350, which selects one of the candidate tokens of the subset 342 based on the probability scores 337. In selecting the output token 352 out of the candidate tokens of the subset 342, token selector weights the probability scores of the candidate tokens in dependence on the class, to which the respective candidate token is assigned, namely by weighting the probability scores of candidate tokens assigned to the first class differently than the probability scores of candidate tokens assigned to the second class.

[0230] For example, token selector 350 may select, after weighting the probability scores, the candidate token having the highest probability score, or may randomly select one out of a subset of the subset 342 of candidate tokens, the subset of the subset 342 comprising one or more most probable candidate tokens according to the weighted probability scores.

[0231] According to an embodiment the token selector 350 is configured for setting weights for weighting the probability scores of the candidate tokens assigned to the first class relative to the probability scores of the candidate tokens assigned to the second class.

[0232] According to an embodiment the token selector 350 is configured for setting the weights in dependence on a user input. According

[0233] According to an embodiment the weights are global within each of the classes (e.g., one (e.g., exactly one) weight is used for weighting the probability scores for the tokens of assigned to one class).

[0234] According to an embodiment the weights consist of a first weight for weighting the probability scores of the candidate tokens assigned to the first class and a second weight for weighting the probability scores of the candidate tokens assigned to the second class.

[0235] According to an embodiment a sum of the weights has a predetermined value.

[0236] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0237] According to an embodiment, in setting the weights, setting a ratio between a total of weights for weighting the probability scores of the candidate tokens assigned to the first class, and a total of weights for weighting the probability scores of the candidate tokens assigned to the second class.

[0238] According to an embodiment, apparatus 310 comprises a user input interface to receive a user input. According to this embodiment, token selector 350 selects the weight in dependence on the user input. For example, the user input may be an indication, which distinguishes between preferring information included in embedded knowledge of the ML model and preferring information included in the input data 328 provided to the machine learning model for the prediction.

[0239] As already mentioned above, apparatus 310 may optionally include a subset selector 340. According to an embodiment, subset selector 340 selects the subset of candidate tokens out of the set of candidate tokens by selecting a number of candidate tokens having the highest probability scores associated therewith. For example, the number of candidate tokens may be a predefined count. Alternatively, the number of candidate tokens included in the subset 340 may include all candidate tokens, the probability scores of which exceed a predefined threshold.

[0240] According to an embodiment, the classifier 370 distinguishes between the probability score being attributable to, or caused by, a first type of attention heads, e.g., retrieval heads, or a second type of attention heads, e.g., parametric heads. For example, according to this embodiment, the classification may be performed on a score, determined for each of the candidate tokens of the subset 342, which is determined in dependence on activations of attention heads of a set of attention heads, the set of attention heads comprising, or consisting of, attention heads of the first type and the second type.

[0241] For example, according to an embodiment, a probability score assigned to the first class is contributed by a first set of attention heads (e.g., of a retrieval head type) (e.g., a function of the attention heads of the first set is retrieving, or copying, information from input data of the prediction) and / or wherein a probability score assigned to the second class is contributed by a second set of attention heads (e.g., of a parametric type) (e.g., a function of the attention heads of the second set is providing embedded knowledge of the machine learning model).

[0242] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0243] Fig. 7 illustrates another embodiment of apparatus 310 of Fig. 6, according to which the classification of the candidate tokens of the subset 342 is performed on the basis of activation of attention heads of a set of attention heads of the ML model 14. According to the embodiment of Fig. 7, apparatus 10 comprises an activation score determination 350 which determines, for each of a set of attention heads of the ML model 14, an activation score 352. The activation score 352 for an attention head of the set 342 measures a component of an output 326 of the attention head during the prediction, which component is associated to the respective candidate token. For example, the association is manifested in an embedding matrix of the machine learning model. For example, the component of the output measured by the activation score is a component which promotes the logit associated with the token. For example, the output 326 of the attention head during the prediction associated to the respective candidate token refers to the output for the last position of sequence S, z, and the component of the output 326 of the attention head during the prediction, which component is associated to the respective candidate token being the t-the entry of W^, when t is the position of the candidate token in the vocabulary of the model.

[0244] For example, the set of tokens may represent a vocabulary of the ML model, the probability score representing a score which indicates the probability of the respective token as being the next output token to be selected. The activation score may measure the component of the output of the respective attention head, which component maps, according to an embedding matrix, e.g., an output projection, e.g., as used in block 335 to generate the probability scores 337 based on an output of a transformer of the ML model 14, which maps the output of the transformer from a dimension of the transformer to the dimension of the vocabulary of the ML model (or the dimension of the set of candidate tokens), e.g., output projection 130 described with respect to Fig. 1. For example, the activation score may be determined as described with respect to equation 10 below.

[0245] According to the embodiment of Fig. 7, apparatus 310 further comprises a weighted aggregation 360, which receives the activation scores 352 determined for the attention heads of the set of attention heads. Weighted aggregation 360 may determine an aggregated activation score 362 by aggregating the activation scores 352 determined for the set of attention heads. In aggregating the activation scores 352, the weighted aggregation 360 weights each of the activation scores 352 using a probe weight of a probe which probe weight is associated with the attention head, for which the respective activation score 352 is determined.

[0246] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0247] For example, the process of determining the activation scores 352 and the aggregated activation score 362 is performed for each candidate token of the subset 342 of candidate tokens. In other words, for each candidate token of the subset 342, activation scores 352 are determined for the respective attention heads of the set of attention heads, and based on the activation scores 352, an aggregated activation score is determined for the candidate token. The aggregated activation score 362 is used by classifier 372 providing the classification 372 for the candidate token.

[0248] For example, the set of attention heads may comprise, or consist of, attention heads of the first type, e.g., as determined using any of the embodiments of the first aspect of the invention, e.g., parametric heads, and attention heads of a second type, e.g., a retrieval type, e.g., as determined as described with respect to embodiments of the second aspect, optionally combined with embodiments of the first aspect described above.

[0249] For example, the probe may comprise, for each of the attention heads of the set of attention heads, a weight which differentiates between the attention head being of the first type or of the second type, e.g., by indicating a balance, which indicates to which extent the respective attention head fulfils one or the other functionality. For example, such a definition of the probe weights results in the aggregated activation score 362 representing a probability, or an extent, to which the respective candidate token is the result of a functionality represented by the first type of attention heads or the result of the functionality represented by the second type of attention heads. For example, in assigning a probe weight equal to one to attention weights which fully represent the first type, and assigning the probe weight equal to zero to attention head entirely assigned to the second type, for a candidate token being the result of the functionality of the first type, e.g., retrieval type, the weighted aggregation score 362 will be large, while for a candidate token, which is the result of the functionality of the second type, e.g., parametric, low activations of attention heads of the first type will lead to a low contribution to the weighted aggregation score 362, while high activations of attention heads of the second type, due to the weighting with weight zero, or low weights, will lead to low contributions to the weighted aggregation score 362, resulting in an overall low aggregated activation score 362. In a similar manner, a candidate token being the result of the functionality of the first type of attention heads, will result in a high aggregated activation score 362. Accordingly, the aggregated activation score 362 provides for a differentiation between contributions from the first and the second types of attention heads and

[0250] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0251] accordingly allows for a classification according to the functionalities of these attention heads.

[0252] For example, the probe including the probe weights for the set of attention heads is obtained as described in section 7 below, or as described in section D.1 below, in particular, equation 14.

[0253] Fig. 8 illustrates an apparatus 410 according to an embodiment of the fourth aspect of the invention. Apparatus 410 performs a prediction using an ML model 14, e.g., the ML model 14 described with respect to any of the previous embodiments. The ML model 14 comprises a plurality of attention heads 24. In performing the prediction, apparatus 410 inputs input data 428 to the ML model 14. Input data 428 comprises task information 432, e.g., the previously described instruction information. Optionally, input data 422 further comprises contextual information. Apparatus 410 comprises a prediction analyser 540. Prediction analyser 450 measures an activation of a predetermined attention head of the attention heads 24 during the prediction. For example, the activation of the predetermined attention head is the output of the respective attention head during the prediction. Prediction analyser 450 checks whether the activation 436 determined for the predetermined attention head during the prediction fits to a function vector 448. Function vector 448 is associated with the task information 432 included in the input data 428 and represents an expected activation of the predetermined attention head for the task information.

[0254] For example, the function vector 448 is specific to the task information.

[0255] By checking whether the activation of the attention head fits to the function vector 448, it can be checked, for example, if the ML model 14 performs the task given to the ML model by the task information 432.

[0256] For example, the predetermined attention head is a task head, e.g., identified as described with respect to embodiments of the second aspect.

[0257] For example, the function vector may be a vector obtained by averaging the outputs of the predetermined attention head for a plurality of predictions using the task information as input.

[0258] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0259] For example, the function vector may have the dimensionality of an output of the predetermined attention head.

[0260] For example, checking whether the activation 436 fits to the function vector 448 may be formed by calculating a metric, which measures a distance or a deviation between the activation 436 and the function vector 448.

[0261] Fig. 9A illustrates an apparatus 510 according to an embodiment of the fifth aspect of the invention. Apparatus 510 infers a source of an output token 582 resulting from a prediction performed by an ML model 14, e.g., the ML model 14 as described before. To this end, apparatus 510 comprises an activation score determination, which determines, for each of a set of attention heads 24 of the machine learning model, an activation score 352, which measures a component of an output 326 of the respective attention head for the prediction of the output token 582, which component is associated to the output token 582. For example, the determination 350 of the activation scores 326 determined for the attention heads of the set of attention heads with respect to the output token 582 is performed as described with respect to Figs. 6 and 7. Thus, in other words, the activation score may indicate the strength of an activation of the attention head for the token position, e.g., the token position in the vocabulary, of the output token 582.

[0262] Apparatus 510 further comprises a weighted aggregation 360, which determines an aggregated activation score 362 by aggregating the activation scores 352 determined for the set of attention heads, wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to. Apparatus 510 further comprises a classifier 370, which classifies the output token based on the aggregated activation score 362. For example, the weighted aggregation 360 and the classifier 370 may operate as described with respect to Fig. 6 and Fig. 7.

[0263] As described with respect to Fig. 6 and Fig. 7, the set of attention heads, for which the aggregated activation scores are determined, may be retrieval heads and parametric heads, such that the classification 370 may distinguish between the output token being attributable to context information included in the input information 328 and the output token 582 being information extracted from the embedded knowledge of an ML model 14. Accordingly, according to the fifth aspect of the invention, a source of output token 582 may be determined or classified.

[0264] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0265] Fig. 9B illustrates another embodiment of apparatus 510. According to the embodiment of Fig. 9B, apparatus 510 identifies one out of input tokens 522 of the prediction as a source of the output token, if the output token 582 is assigned to the first class, e.g., if the output token 582 is assigned to a class, according to which the output token 582 is retrieved from the input information 328. In other words, according to this embodiment, the input information 328 comprises a plurality of input tokens 522, each having a token position in the sequence of input tokens. According to this embodiment, apparatus 510 further comprises an attention score determination 590. Attention head determination 590 determines, for each of a plurality of token positions, e.g., all token positions of an input sequence of tokens of the prediction included in the input information 328, or a portion thereof, such as a portion, which represents contextual information, an attention score 592. For determining the attention score 592 for a token position, attention score determination 590 aggregates attention weights of respective attention maps 532 of the set of attention heads, which attention weights define, for the respective attention head, a contribution of the respective token position to a token position of the output token 582. For example, the attention maps are referred to as Ah, and the attention weight, which defines the coupling between input token positionj and output token position I, i.e., the contribution of the input at positionj to the output at position I, as A^, e.g., as described in Section 3.2, in particular, Equation (2), below.

[0266] In aggregating the attention weights, attention score determination 590 weights each of the attention weights using the probe weight of the probe, which probe weight is associated with the attention head the respective attention weight belongs to, and further weighted using the activation score 352 determined for the attention head, the respective attention weight belongs to. For example, the attention score 592 is determined according to Equation 15, see Section D.1 below.

[0267] According to the embodiment of Fig. 9B, apparatus 510 further comprises a source token determination 596, which identifies the source of the output token based on the attention scores 592 determined for the token positions of the plurality of token positions.

[0268] According to an embodiment, source token determination 596 identifies the token position, for which the attention score is maximum among the attention scores determined for the plurality of token positions as the source of the output token 582.

[0269] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0270] As already described with respect to Fig. 7, the set of attention heads, for which the activation scores are determined, and which are considered for determining the attention score, may be a subset of a plurality of attention heads of the ML model 14, the subset comprising attention heads of one or more predetermined times, such as retrieval heads and parametric heads.

[0271] It is noted that the aspect described with respect to Fig. 9B, i.e., the localization of the source of the output token 582 in the sequence of input tokens may be implemented independently from the classification described with respect to Fig. 9A. In this case, for example, the set of attention heads may exclusively include attention heads, which are classified as retrieval heads. In this case, for example, the weighting of the attention weights in the determination of the attention score 592 by the probe weights for the attention heads may be omitted, e.g., the attention weights of different attention heads may be weighted equally, that is, it is not discriminated between the attention heads.

[0272] In the following, further optional details about the probe and the probe rates are described, which details may optionally applied to any of the embodiments according to any of the aspects described herein, in particular, aspects 3, 4 and 5.

[0273] According to an embodiment, the probe is linear, i.e., a linear classifier. Alternatively, the probe may be non-linear, e.g., a non-linear classifier. For example, the probe may distinguish between a first type of attention heads and a second type of attention heads. For example, the probe weights may be determined as described in Section D.1.

[0274] Fig. 10 illustrates an apparatus 610 according to an embodiment of the sixth aspect. Apparatus 610 analyzes a prediction performed using an ML model 14, e.g., the ML model described with respect to any of the previous aspects. Apparatus 10 according to the embodiment of Fig. 10 comprises a function analyzer 650, which compares an output 636 of a predetermined attention head 24 of the machine learning model 14 during the prediction to each of a plurality of function vectors 648.

[0275] For example, the output 636 of the predetermined attention head refers to the activation, or the attention head output of attention head 24, e.g., the entire output, or the output for a predetermined token position, e.g., the last token position of the currently processed sequence, based on which the subsequent token is predicted, e.g., as described with

[0276] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0277] respect to Fig. 1. For example, the output 636 is the attention head output for the last token position S, e.g., referred to as z.

[0278] For example, each of the function vectors 648 may be a function vector for the predetermined attention head 24, which represents or is associated with a different function or task. For example, each of the function vectors may have been determined using the output of the predetermined attention head during predictions, for which the ML model was given the task as input information. For example, the function vector may have been obtained by averaging the outputs observed during these predictions. In other words, each of the function vectors 648 may represent a characteristic output of the predetermined attention head during predictions performing the task associated with the respective function vector.

[0279] According to an embodiment, when comparing the output of the predetermined attention head 24 to each of the plurality of function vectors, function analyzer 650 determines respective distance measures indicating a distance between the output and the respective function vector, e.g., as already described with respect to the fourth aspect. For example, the predetermined attention head 24 is of a predetermined attention head type, e.g., of the task head type. For example, the predetermined attention head is determined as described with respect to embodiments of the second aspect.

[0280] For example, the predetermined attention head may be part of a set of attention heads, and the function analyzer 650 may compare, for each of the attention heads of the set of attention heads, an output of the respective attention head during the prediction to each of the plurality of function vectors, or to each of a respective plurality of function vectors for each of the attention heads.

[0281] In other words, for example, the function vectors may be attention-head specific, i.e., determined individually for each of the attention heads. Alternatively, the function vectors may be common to all attention heads.

[0282] Apparatus 610 may use a result of the comparison for selecting one of the function vectors out of the plurality of function vectors, e.g., the one, which matches best to the output 636 of the attention head. Apparatus 610 may provide an indication of the function vector or an indication of a prediction task associated with the function vector, e.g., as an output, e.g., on a user interface.

[0283] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0284] Fig. 11 illustrates an apparatus 710 for performing a prediction using a machine learning model 14 according to an embodiment of the seventh aspect. For example, machine learning model 14 may be implemented as described with respect to any of the previous embodiments. Machine learning model 14 comprises a plurality of attention heads 24. In performing the prediction, apparatus 710 according to Fig. 11 is configured for setting a plurality of output activations of an output 226 of a predetermine attention head 24* of the plurality of attention heads 24 using predetermined values 748. The predetermined values 748 are independent of an input 722 of the predetermine attention head 24*.

[0285] For example, for performing the prediction, apparatus 710 provides prediction input 708 to the ML model 14. For example, prediction input 708 may correspond to prediction input 108 described with respect to Fig. 1, or may correspond to any of the prediction inputs described with respect to any of the previous embodiments. For example, prediction input 708 may include a task information.

[0286] For example, ML model 14 may use the output 726 of the predetermined attention head 24* for determining an input of a plurality of the attention heads 24*, which are located downstream of the predetermined attention head 24*, e.g., downstream in a sequence of attention blocks of the ML model 14, which are sequentially processed. In other words, the output 726 may be used for determining the input of a plurality of attention heads, which follow the predetermined attention head in a sequential order defined among blocks, in which the plurality of attention heads are arranged.

[0287] For example, apparatus 710 sets the plurality of output activations of the output 726 of the predetermined attention head to the predetermined values.

[0288] For example, the plurality of output activations may be a subset of all output activations of the predetermined attention head. For example, the predetermined attention head 24* determines, for each token position of a sequence of tokens of a current recursion of the prediction, an output vector comprising a plurality of output activations, e.g., as described with respect to Fig. 1 and Fig. 2. For example, the output vector has a length of a dimension of the attention head, e.g., denoted as dhabove. For example, the plurality of output activations set using the predetermined values is the vector denoted as zhsabove.

[0289] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0290] According to an embodiment, apparatus 710 sets each of the output activations of the output vector for the ultimate token position of the sequence of tokens to a respective predetermined value. For example, the respective predetermined values are defined by a function vector having the length of the dimension of the attention head.

[0291] In other words, the predetermined values may represent a function vector, which is associated with a function or a task. For example, the function vector may be determined as described above, e.g., based on the output of the predetermined attention head during a plurality of predictions, in which the ML model performs a specific task, e.g., as described above.

[0292] In other words, for example, apparatus 710 may replace the plurality of output activations of the predetermined attention head 24* by predetermined values 748, e.g., of a function vector.

[0293] Accordingly, in performing the prediction, the plurality of output activations may be set to be independent of the prediction input 708.

[0294] According to an embodiment, apparatus 710 selects the predetermined values, e.g., the function vector, for setting the plurality of output activations based on an input information, which differs from the prediction input 708 provided as input to the ML model 14 for performing the prediction. In other words, the input information, based on which the predetermined values are selected, is additional to the prediction input 708. In other words, the input information is not part of a sequence of input tokens of the prediction, e.g., the input sequence as described with respect to Fig. 1. Accordingly, the output of all attention heads which precede the predetermined attention head 24* in a propagation stream of the ML model 14 may be independent of the input information.

[0295] For example, the plurality of output activations comprises, or consists of, output activations for the ultimate token position S of a sequence of token positions. According to this embodiment, the output of the predetermined attention head 24* comprises, for each of the sequence of token positions, a respective output activation vector. For example, the plurality of output activations set to the predetermined values may consist of the output activations of the output activation vector of the ultimate token position.

[0296] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0297] According to an embodiment, apparatus 710 comprises an input interface, which is configured for receiving a prompt information and setting information (e.g., the above-mentioned input information, which is independent of the prediction input 708). According to this embodiment, apparatus 710 is configured for determining a sequence of input tokens from the prompt information, and providing the sequence of input tokens as an input of the ML model 14, the input being independent of the setting information. According to this embodiment, apparatus 710 selects the predetermined values for setting the plurality of output activations based on the setting information.

[0298] By setting the output activations of the predetermined attention head to the predetermined values, the ML model 14 may be forced to perform a certain function or task, which is associated with a function vector comprising the predetermined values.

[0299] For example, the predetermined head 24* is a task head, e.g., as identified according to any of the embodiments of the second aspect.

[0300] In the following, further optional details are described, which may be combined, or which may apply to any of the embodiments of any of the aspects described above.

[0301] For example, in determining the activation score, e.g., activation scores 352, as an alternative to considering activations of the respective attention head during prediction, a relevance score may be determined, which measures a contribution, e.g., an actual contribution, of the attention head to the currently evaluated prediction output or output token, e.g., to the logit or probability score for the output token. For example, the relevance score may be determined by backpropagating the logit or probability score through the ML model to the attention head.

[0302] According to an embodiment, an input for the prediction is provided in form of one or more of an audio signal, a video signal, an image signal, or a communication signal. For example, a signal of a communication signal may be a signal of a communication channel, e.g., a 4G, a 5G, a broadband communication channel, an LTE channel, e.g., is measured by an antenna, such as an antenna, which may be part of any of the apparatuses described above.

[0303] According to an embodiment, the input for the prediction is a multi-model signal, e.g., the signal may comprise a mixture of one or more of text, image, video, audio. For example,

[0304] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0305] questions and system instructions could be text based, and the context information, e.g., the given context, is image, video, and / or audio.

[0306] In the following, further embodiments of the invention will be described. The embodiments described in the following may be implemented independently from the embodiments described above. Alternatively, any of the embodiments described above may be supplemented by any of the details and features described below, or any of the embodiments described below may be supplemented or combined with any of the features described above.

[0307] Embodiments described herein may related to the linking of instructions, contextual and parametric retrieval, what may be considered as the anatomy of in-context learning.

[0308] Large language models are able to exploit in-context learning to access external knowledge beyond their training data through retrieval-augmentation. While promising, its inner workings remain unclear. In this work, we shed light on the mechanism of in-context retrieval augmentation for question answering by viewing a prompt as a composition of informational components. We propose an attribution-based method to identify specialized attention heads, revealing in-context heads that comprehend instructions and retrieve relevant contextual information, and parametric heads that store entities’ relational knowledge. To better understand their roles, we extract function vectors and modify their attention weights, showing how they influence the answer generation process. Finally, we leverage the gained insights to trace the sources of knowledge used during inference, paving the way towards more safe and transparent language models.

[0309] 1. Introduction

[0310] In this work, we aim to shed a light on the mechanism behind ICL for knowledge retrieval.

[0311] Viewing a prompt as a composition of informational components, we show that in opendomain question answering, certain attention heads perform various operations on the prompt at different stages of inference and layers (see Figurel). Our method identifies two groups of heads based ont heir functions: parametric heads that encoe relationship knowledge [22, 50] and in-context heads responsible for processings inforamtion in the

[0312] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0313] prompt. Furhter, as in-context heads need to understand which prompt components to process and how, we hypthesize that they specialize to fill their respective roles.

[0314] Fig. 12 illustrates a functional map of in-context and parametric heads in Llama-3.1-8B-Instruct. They are surprisingly well-structured and operate on the input prompt at various levels, with in-context heads processing information in the prompt, including instruction comprehension and retrieval operations - and parametric heads that encode relational knowledge. In-context heads can specialize to task heads to parse instructions (blue) or retrieval heads for verbatim copying (green). Together with parametric heads, they affect the answer generation process through function vectors that they transport (a, c) or their attention scores (b). Our relevance analysis (bar plot) shows that instruction-following capabilities emerge in middle layers, while answer retrieval occurs in later layers. See Section C.1.

[0315] Our analysis shows that in-context heads can indeed execute specialized functions such as instruction comprehension and retrieval of relevant contextual information. To investigate this further, we curate a controlled biography dataset with entities extracted from Wikidata

[0066] , Remarkably, we find that both in-context and parametric heads can be compacted to form FVs, which can be applied in zero-shot settings to induce targeted functions.

[0316] Building on these insights, we probe for sources of knowledge used during retrieval-augmented generation (RAG) and show where it is localized within the input context. Our attempt shows promising results, serving as an initial step towards more transprent RAG systems.

[0317] Overall, our contributes can be summarized as follows:

[0318] • We describe an attribution-based method to determine attention heads that play a key role during in-context retrieval augmentation, revealing that they operate on the prompt in a distinct manner. Our method can thus be used to reliably categorize attention heads into in-context and parametric heads.

[0319] • We analyze how in-context heads specialize in reading instructions and retrieving information, mapping their localization across model layers. Additionally, we demonstrate the influence of in-context and parametric heads on the answer generation process, by compating them into function vectors or modifying their attention weights.

[0320] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0321] We present preliminary results on enabling casual tracing of source information for retrieval-augmented LMs, suggesting fruitful directions for intrepretability of RAG systems.

[0322] 2. Related Work

[0323] Retrieval Augmentation: Retrieval-augmented language models (RALMs) [34, 38] address inherent limitations of LMs by providing external knowledge sources. Despite this advantage, issues such as discrepancies between contextual and parametric knowledge may occur [41, 74], Some works have studied mechanisms behind knowledge preference in RALMs [31, 44, 48, 77], but they mostly focus on simple domains where the answer is limited to a single word. Besides, RALMs cannot guarantee faithful answer attribution to contextual passages

[0020] , which necessitates a shift to interpretability. For example, the term answer attribution means the use of external documents to support the generated response, which is different from feature attribution used throughout this word to describe interpretability techniques.

[0324] The Role of Attention: Attention mechanisms have been previously observed to encode many kinds of information. Clark et al.

[0014] showed that they correspond well to linguistic properties such as syntax and coreference. Similarly, Voita et al

[0064] found that syntactic heads play an important role in machine translation models. In relation to world knowledge, Geva et al.

[0022] proposed that certain heads store factual associations and demonstrated how they extract an attribute of a given subject-relation pair. Interestingly, attention also appears to encode a sense of “truthful” directions

[0039] , With the exception of Voita et al

[0064] , the above works make use of attention weights, which might not fully capture the model’s prediction [10, 29, 69], Our work can be seen as an attempt to reconcile both perspectives: analyses based on attention weights and feature attribution methods [9],

[0325] In-Context Learning: Numerous works have studied ICL since its introduction. Liu et al.

[0326]

[0040] studied what constitutes a good example for demonstrations. Dai et al.

[0016] suggested that ICL can be understood as an implicit fine-tuning. ICL is a general phenomenon, although it is commonly assumed to be unique to autoregressive models

[0054] , At the component level, ICL is primarily associated with induction heads [17, 47], However, recent findings showed that certain heads can also be compressed to form FVs that represent a demonstrated task [26, 62], Yin and Steinhardt

[0076] investigated the connection between FV heads and induction heads, showing that they are distinct from each other and how

[0327] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0328] models’ ICL performance is mainly driven by FV heads. ICL can also be viewed as a combination of meta-learning and retrieval

[0045] , In that regard, we study the latter perspective to understand its mechanism, with a focus on the retrieval augmentation paradigm.

[0329] 3. Background and Preliminaries

[0330] The self-attention mechanism in transformers poses a challenge in understanding which heads actually contribute during in-context retrieval augmentation, and how they process various components in the prompt. This is mainly due to the fact that information from different tokens gets increasingly mixed as layers go deeper and how several attention heads may implement redundant functions

[0067] , A natural option is to analyze attention weights, as they are an inherent part of a model’s computation. However, attention can be unfaithful

[0028] , which questions its validity as an explanation [10, 56], This problem is further exacerbated by “attention sinks” [36, 72] - a phenomenon where heads heavily attend to the first token and obscure the weights of other tokens in the sequence.

[0331] An alternative would be to use feature attribution methods [9], as they trace the contribution of each input element to the model’s output prediction. Propagation-based feature attribution [8, 58, 59] especially takes the entire computation path into account, which can be used to characterize attention heads

[0064] or identify latent concepts [1], Furthermore, feature attribution is able to estimate causal relations

[0021] and thus enables to observe how a specific attention head affects a model’s prediction.

[0332] In this section, we provide a description of AttnLRP [2], on which our method is based, due to its superior performance and efficiency in transformer architectures compared to other attribution methods. We also provide an overview of the multi-head attention mechanism in transformers, which we leverage through AttnLRP to identify both in-context and parametric heads (§5). Additionally, we analyze the specialization of in-context heads, extract function vectors from the identified heads (§6), and use this information for reliable and efficient source tracking of facts in RAG systems (§7).

[0333] 3.1 Layer-wise relevance propagation

[0334] Feature attribution methods aim to quantify the contribution of input features x to the overall activation of an output y in linear but also highly non-linear systems. We define a function

[0335] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0336] 5? that maps the input activations x to relevance scores indicating their causual effect on a model’s output logit y.

[0337]

[0338] In principle, any feature attribution method can be employed for 5?, though trade-offs between faithfulness and computational efficiency must be carefully considered. Perturbation-based approaches

[0042] typically offer high faithfulness but incur exponential computational costs, as each ablated latent component requires at least one dedicated forward pass

[0025] , In contrast, gradient-based methods

[0058] are computationally more efficient, requiring only a single backward pass, which makes them well suited for large-scale interpretability studies. However, they are susceptible to noisy gradients, which are distorted by non-linear components such as layer normalization [5, 73], To address these limitations, we adopt AttnLRP [2], an extension of Layer-wise Relevance Propagation (LRP) [8] designed for transformer architectures. As a backpropagation-based technique, AttnLRP propagates relevance scores from a model output to its inputs in a layer-wise manner and can be implemented efficiently via modified gradient computation in a single backward pass. Importantly, it incorporates stabilization procedures for non-linear operations, thereby improving the faithfulness of relevance distributions compared to standard gradient- or other decomposition-based methods [7],

[0339] Relevance scores produced by (Attn) LRP can be either positive or negative. Positive relevance indicates an amplifying effect on the final model logit y, whereas negative relevance reflects an inhibitory influence. Without loss of generality, we focus our analysis on signal-amplifying components by considering only positive relevance scores. Formally, we define:

[0340] 7?.+(x|y) = max (' (xly), 0) (D

[0341] This yields a clearer separation between in-context and parametric heads in the subsequent analysis.

[0342] 3.2 Attention Mechansim

[0343] While the original formulation of the multi-head attention mechanism

[0063] concisely summarizes the parallel computation of attention heads, our goal is to isolate their individual

[0344] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0345] contributions. To this end, we reformulate the equations to make the influence of each head more explicit [17, 18], Let X = (x^..., xs) e IRf / x' the matrix of hidden token representations for a sequence S with dimension d, and suppose our model employs H parallel heads, each of dimension dh= d / H. Then, the computation of the Multi-Head Attention layer can be reformulated into H complementary operations, where each head h produces an intermediate attention output z-1e ]Rd / l:

[0346]

[0347] where is the attention weight from token j to token i, and

[0348]

[0349] is the per-head value projection. The final attention output is obtained by multiplying the pre-output of each head with their corresponding output projection matrix WQ e Rdxdfollowed by summing over all heads:

[0350] H

[0351] Xi -j2Wozi (3) h=l

[0352] We leverage the multi-head attention mechanism in transformers through the lens of AttnLRP to identify both in-context and parametric heads in §5 and how in-context heads specialize in §6.

[0353] 4. Experimental Setup

[0354] Models: We use instruction-tuned LLMs due to their increased capability on question answering (QA) tasks in our preliminary experiments: Llama-3.1-8B-lnstruct

[0023] , Mistral-7B-lnstruct-v0.3

[0030] , and Gemma-2-9B-lnstruct

[0061] , We apply AttnLRP based on their huggingf ace implementations

[0070] , For the rest of this work, we refer to each model by their family prefix.

[0355] Datasets: To perform our analyses, we use two popular open-domain QA datasets: NQ-Swap

[0041] and TriviaQA (TQA)

[0032] , NQ-Swap is derived from Natural Questions

[0037] , a QA dataset with questions collected from real Google search queries and answer spans annotated in Wikipedia articles. TQA contains trivia questions with answers sourced from the Web. Both datasets are based on the MRQA 2019 shared task version

[0019] ,

[0356] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0357] Similar to Petroni et al.

[0051] , we use different types of contextual information to see how they affect in-context and parametric heads. We use oracle contexts as they are always relevant to the question and contains the true answer. In addition, we use counterfactual contexts as they contain information that is usually not seen during pre-training and fine-tuning stages, thus forcing models to rely on it to answer the question correctly. Oracle context is often not available; therefore we also use Dense Passage Retriever (DPR)

[0033] with a Wikipedia dump from December 2018 as our retrieval corpus. For simplicity, we only select the top one retrieved document. We show results for oracle and counterfactual contexts and retrieved context.

[0358] Inspired by Allen-Zhu and Li [6], we build a human biography datasets to allow us to better understand the characteristic of in-context and parametric heads and conduct controlled experiments. Using Wikidata

[0066] , we collect profiles for random 4,255 notable individuals containing their date of birth, birth place, educational institute attended, and occupation. We concatenate the attributes of each individual in a random order to form a biographical entry and ask Llama-3.1-8B-lnstruct to paraphrase it. See below for more details.

[0359] 5. Localization of In-Context Parametric Heads

[0360] In retrieval-augmented generation, LLMs are faced with the option to generate responses by using a wealth of knowledge they learn during pre-training or by relying on contextual information provided in the prompt through ICL. Here, we categorize attention heads that are responsible for both capabilities.

[0361] Method: We aim to identify the sets of in-context headsjfctxand parametric headsjfparamas depicted in Fig. 1. We define in-context heads are defined as those that contribute to the model’s prediction during RAG by using contextual information, whereas parametric heads primarily contribute upon reliance on internal knowledge. We hypothesize that each head type contributes maximally under a specific condition while having minimal influence in others, i.e., in-context heads are expected to contribute the most in open-book settings and the least in closed-book settings, and vice versa. We analyze questions with counterfactual contexts forcing retrievel to produce a counterfactual prediction ycfthat disagrees with the parametric answer. Conversely, we also focus on closed-book settings, where contextual information is minimized, to identify parametric heads and reduce the chance that in-context heads contribute. We restrict our analysis to instances where a gold

[0362] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0363] reference answer ygoid is predicted, to ensure that relevance attribution reflects genuine parametric behavior.

[0364] We use AttnLRP to quantify the contribution of each attention head h to the prediction by summing the positive relevance scores assigned to its latent output zhacross its dimension dhand over all i token positions, when explaining the targeted prediction yt, which can be either the gold reference answer ygoid or a counterfactual output ycf, depending on the setting:

[0365]

[0366] To contrast heads across settings, we compute a difference score D representing their average contribution in open-book versus closed-book conditions for all Nhheads in the model:

[0367] ■D = fe ^Cytf)] - ExJ^Cygoid)]: h = 1,2. ivj (5)

[0368] We then identify the most distinctive heads for each behavior by selecting the top 100 heads (around 10%= 15% of total heads) with the highest positive and lowest negative difference scores:

[0369] ^ctx = {argsortdesc(2))}i° ijTparam = {a^sort^CD)}^ (6)

[0370] Experiments: To ensure that the identified in-context and parametric heads play a role in QA tasks, we ablate both sets of heads and measure performance drops in settings where they are expected to be mostly active (open- and closed-book, respectively). We also measure if the removal of in-context heads affects the models’ capabilities to answer in closed-book setting and vice versa, since this informs to what extent both sets of heads are dependent on each other. Furthermore, we want to know if the identified in-context and parametric heads can generalize to other datasets. To test this, we compute the score of both heads only over NQ-Swap and reuse the same sets of heads on TQA. To evaluate the aforementioned criteria, we report recall [3] as instruction-tuned models tend to produce verbose responses. As baselines, we select random heads, and also adopt the Attention Weight Recall (AWR) method based on attention maps’ activations, as described in

[0071] ,

[0371] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0372] Results: Fig. 13 illustrates results for Llama. Fig. 13 shows how the recall score evolves when either heads identified as in-context or parametric heads are ablated. We observe that the removal of 20 heads (100 heads) reduces the performance by 13.86%-63.84% (44.26%-68.66%) for open- and closed-book settings across different configurations, indicating the causal influence these heads have on the answers’ correctness. Moreover, the performance drops on TQA hold even though the heads are computed on NQ-Swap, showing that the identified in-context and parametric heads are transferrable to other datasets.

[0373] We compare in-context heads as identified with our method against AWR heads, and find that the removal of 20 in-context heads results in a roughly similar reduction of recall as removing 100 AWR heads.

[0374] Fig. 13 illustrates a recall analysis for Llama when either in-context or parametric heads are ablated. Removing identified in-context heads noticeably affects the model’s performance in open-book QA across various configurations. Conversely, removal of identified parametric heads most strongly affects the model’s closed-book QA capabilities. Compared to Wu et al.

[0071] that only yield AWR (retrieval) heads, our method allows to obtain both incontext and parametric heads.

[0375] Furthermore, ablating in-context heads yield a more drastic performance decrease compared to the removal of AWR heads, suggesting that our method is more suitable than those based on attention scores alone to study heads that contribute to response generation. Ablating randomly chosen heads barely affects the model’s ability to answer correctly.

[0376] We examine whether in-context and parametric heads are independent of each other. As expected, ablating parametric heads has little influence to the model’s performance in our open-book setting. Interestingly, this leads to a slight performance increase on NQ-Swap with counterfactual contexts, which suggest that the ablation forces the model to rely more on the given context instead of its own parametric knowledge. Surprisingly, ablating incontext heads in the closed-book setting incurs a non-negligible performance reduction. This is likely due to the influence in-context heads have when processing the input prompt. We explore this in the next section.

[0377] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0378] 6. Functional Roles of In-Context and Parametric Heads

[0379] Given that ablating in-context heads yields a non-negligible drop in closed-book QA performance, where no external documents are available, we posit that in-context heads not only process the context but also interpret the intensional frame - the semantic structure imposed by the instruction itself

[0055] , In the counterfactual shown in Fig. 21, the intensional frame (the question prompt) is shown in italics, the object instance in bold, and two equally plausible answers.

[0380] To answer correctly, the model must map the intensional frame onto the knowledge triple (s,r,o*), where s is the subject (“Mike Tyson), r is the semantic relation (the predicate specified by the question, here “has worked as”), and o* is the object (the unknown answer, “boxer” or firefighter”). Depending on where the answer resides, o* may be retrieved from the model’s parametric memory (op) or from the context (oc). By treating (s,r,o*j, as the complete task specification, we analyze how in-context and parametric heads specialize both to comprehend the intensional frame and to retrieve the object o* needed to generate the correct answer.

[0381] 6.1 Disentangling the Functionality of In-Context Heads

[0382] Our goal is to identify heads specialized in processing the intensional frame and those specialized in retrieving the answer object from the context. Inspired by the work [1], which demonstrates that relevance is effective for separating functional components in latent space, we measure how much relevance of an attention head is assigned to the question and retrieved answer tokens.

[0383] Method: Specifically, for each head h, we compute the total relevance attributed to the attention weight A’jj when explaining the logit output yt. Since relevance flows backwards from the output to the input, our goal is to obtain relevance at the input level of each layer. Given that each head transfers information from the key at position j to the query at position i, we aggregate this backward relevance over all possible query positions i to obtain a single relevance score for the source token at key position / :

[0384]

[0385] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0386] Here, p-1represents the total relevance that head h assigns to token j when contributing to logit yt. Next, we aggregate the relevance scores separately for two sets of token positions within the context: the intensional-frame tokens, denoted as j ejtask, which comprise the question token positions, and the answer object tokens, denoted as j ejret, which represent positions of the retrieved object.

[0387] Ptask

[0388]

[0389] Finally, to obtain the sets of specialized heads, we rank heads by their aggregated relevance and select the top K

[0390]

[0391] Results: We compute the task relevance score p^askand the retrieval relevance score p^eiover the NQ-Swap with counterfactual contexts to minimize influences of parametric heads, and aggregate their distributions across the model layers. In Fig. 12, we observe that p^askinitially increases in the early layers where few parametric heads are located, suggesting that early parametric heads enrich the question with relational knowledge. The relevance peaks in the middle layers, where in-context heads dominate, aligning with the transition to more context-dependent reasoning. In contrast, the retrieval relevance score p^eipeaks in deeper layers, reflecting the point where the model extracts the final answer object oc. Fig.

[0392] 14 further illustrates the sorted average difference D between open-book and closed-book settings for all heads, alongside the top 40 task headsjftaskand retrieval headsjfret. We observe that retrieval heads and task heads consistently rank among the highest-scoring in-context heads, emphasizing the critical role of in-context heads for retrieval augmented generation.

[0393] Fig. 14 illustrates sorted in-context scores for 1024 heads of Llama, comparing open-book and closed-book settings via score D. Positive scores indicate in-context behavior, while negative scores reflect parametric behavior. Retrieval heads and task heads are predominantely high-scoring in-context heads. See Fig. 18 for other models.

[0394] 6.2 Causal Effects of In-Context and Parametric Heads

[0395] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0396] An important question is whether the heads we identify truly reflect their assigned functionalities. We examine this by investigating their causal effects on the answer generation process.

[0397] Experiments: We conjecture that task headsjftaskencode the intensional frame (s,r,o*) and that parametric headsparamcontain information of the subject s, which depending on the training data may or may not include op. On the other hand, retrieval headsjfretsearch for oc, allowing them to copy any tokens from the context verbatim, without being restricted to only plausible answers. For task and parametric heads, we compute FVs (A function vector can be defined as a sum of averaged heads outputs over a task

[0062] or computed individually

[0026] , Here, we consider a function vector to be an individual output of a task or parametric head.) for each head and insert them into various settings to trigger the execution of their functions. Following Wu et al.

[0071] , we opt for a needle-in-a-haystack (NIAH) setting for retrieval heads and determine their ability to retrieve relevant information from the context by modifying the attention weights. We conduct our analysis on the biography dataset described in §4 and measure recall score [3], For comparison, we also consider random heads for FV extraction and attention modification. See the below for more details.

[0398] Fig. 15 illustrates extraction and insertion of task and parametric FVs. The induced generation is highlighted in italic.

[0399] Table 1 shown in Fig. 23 shows zero-shot recall scores for task, parametric, and retrieval heads.

[0400] Task Heads: We demonstrate that task heads encode intensional frames. In a zero-shot manner, we extract task FVs from each head injftaskfor four questions relating to all recorded attributes from the biography dataset. Then, we insert them to another biographical entry without a question at the final token position, and also for all subsequent token generations (Fig. 15, top). We examine whether they reliably induce responses aligned with the original question. In Table 1 (left), we show that applying FVs in a zeroshot manner allows all models to response accordingly wrt. the intensional frame, yielding an average improvement of 78.75 points over random heads.

[0401] Parametric Heads: Parametric heads contain factual, relational knowledge. To show this, we first select a random attribute of an individual and convert it to a cloze-style statement.

[0402] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0403] Then, we extract FVs fromjfparam, which are inserted to a question prompt of another unrelated individual (Fig. 15, bottom). We observe if the generated response contains information of the original entity conditioned on the intensional frame. For simplicity, we restrict extraction to cases where the closed-book answer is correct wrt. gold reference. We see that in Table 1 (middle), adding parametric FVs allows all models to recover the original attributes significantly, with an increase of 30.41 points compared to random.

[0404] Retrieval Heads: We assess retrieval heads’ ability to copy verbatim by using famous poem titles as needles, inserted at a random position in the biographical entries. At the last token of the entry and for the following generations, we increase the attention weights of all heads injfreton every token of the needle to force the model to copy. Our results (Table 1, right) show a drastic increase of 83.15 points over the random baseline, indicating that retrieval heads are indeed able to perform copy operations independent of the token position.

[0405] 7. Source Tracking

[0406] Our experiment in §6.2 demonstrates that retrieval heads reliably perform verbatim copying of text spans when their corresponding attention maps focus on the retrieved tokens. We now aim to investigate if we can (i) detect when retrieval heads initiate the copying process for the first answer toke ( / .e., whether a token is derived from external contexts rather than from model parameters), and (ii) accurately localize its position within that context using the attention maps. To this end, we train a linear probe using the NQ-Swap with counterfactual contexts. Each retrieval head’s output at last token position

[0407]

[0408] is decoded via logit lens

[0046] , converting each head’s activation into a score for token t e N using the model’s unembedding matrix Wye ]Rl lxdand layer normalization LN( • ):

[0409]

[0410] where Wye ]Rdxdis the head’s output projection and Wy[t] the row of the unembedding matrix corresponding to token t. As such, (z |t) computes how stronly head h writes token t into the residual stream. In Fig. 19, we illustrate histograms of the logit lens scores. Next, we train the probe by learning weights {wh}hE^ret. For the localization task, we aggregate each head’s attention map A j weighted with the head’s activity, as their attention maps pinpoint to the retrieved token, and predict the source token position as the index with the

[0411] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0412] highest aggregated attention weight. For details, please refer to Section D. In Table 2, the retrieval head probe achieves an Area Under the Receiver Operating Characteristic Curve (ROC AUC) of at least 94%, reliably distinguishing contextual from parametric predictions.

[0413] We can use the probe to determine if the next predicted token should be parametric or from context, i.e., we can force the model to read from context and not hallucinate.

[0414] Fig. 16: (a) When asked “Where does llama originate from?”, the retrieval head probe classifies “South America” and “Africa” as parametric, while “Meta” as contextual. The LIMAP projection of retrieval head activations displays the linear probe’s decision boundary (dashed line) separating parametric from contextual clusters, (b) The weighted aggregation of retrieval head attention maps at the final query position is superimposed on the RAG document to pinpoint the retrieved source span.

[0415] Simultaneously, each model attains a top-1 localization accuracy of at least 84%, confirming a linearly separable representation of the retrieval task. In Fig. 20, we illustrate heatmaps of the aggregated attention maps superimposed on the input, which highlights the position of the predicted token. LRP localization can be used as baseline.

[0416] 8. Conclusions

[0417] We propose a method to explore the inner workings of ICL for retrieval-augmentation, revealing in-context and parametric heads that operate on the input prompt in a distinct manner and find that in-context heads can specialize to read instructions or retrieve relevant information. We study the roles of the identified heads by converting them into FVs or modifying their weights, showing how they can affect the generation process. Finally, we present a probe to track for knowledge provenance, opening up a path towards more interpretable retrieval-augmented LMs.

[0418] Broader Impacts While our work is intended to illuminate the mechanism behind ICL in retrieval augmented LMs, we caution that it may also enable potential for abuse, such as using the identified heads for inducing malicious behavior in the model.

[0419] 8.1 Findings, which may be exploited by embodiments

[0420] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0421] While some embodiments are described with respect to QA tasks, e.g., because they induce a wide variety of function vectors in challenging real-world applications to robustly extract retrieval heads, embodiments are not limited to QA task, but also work on other tasks. While using QA, the copy behavior is more pronounced in the histograms, we also show it still transfers to different tasks like translation.

[0422] Relevance attention maps can have negative relevance, while activation not.

[0423] Many heads not visible in logit lens, because they write function vectors. But LRP shows them well. We achieve this kind of analysis, because we take advantage of classical XAI tools, which is often overlooked by Mechlnterp community.

[0424] Attribution methods enable the disambiguation of a single, task-relevant information pathway amidst numerous parallel - and potentially conflicting - computational processes. This level of clarity is made possible by XAI, which would be much more difficult to achieve with raw activation patterns alone.

[0425] Because of the compositional nature of text, we assume that head functions are also compositional. Thererfore, w can patch arbitrary QA question of more complexity

[0426] The efficiency of LRP allows us to do more large scale experiments.

[0427] We manually inspected instances where Layer-wise Relevance Propagation (LRP) failed to accurately track the top-1 token. Predominantly, these cases involved contexts with multiple similar tokens, leading to ambiguity in token relevance.

[0428] The better trained the model, the more pronounced the effect of FVs

[0429] Embodiments rely on the idea categorize different heads types in the model. Each head type should be specific to a behavior. The most prominent behavior in LLMs is in-context learning, ICL. Focusing on ICL, we find 2 broad categories: in-context heads & parametric heads.

[0430] A Further details

[0431] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0432] Licenses Llama-3.1-8B-lnstruct is released under the Llama 3.1 Community License. Gemma-2 9B-it is released under the Gemma license agreement. Mistral-7B-lnstruct-v0.3 is released under Apache 2.0. NQ-Swap and TriviaQA that we use are derived from MRQA

[0019] , which is released under the MIT license. We construct the biography dataset using Wikidata Query Service (https: / / query.wikidata.org / ), which is available under CCO.

[0433] Datasets For NQ-Swap, we use the preprocessed data and split available on HuggingFace (https: / / huggingface.co / datasets / pminervini / NQ-Swap)(4,746 examples). As for TriviaQA, we use the dev split from the MRQA repository (https: / / github.com / mrqa / MRQA-Shared-Task-2019)(7,785 examples).

[0434] To build our biography dataset, we start by selecting 100,000 random individuals that have the following attributes in Wikidata: date of birth (P569), place of birth (P19), education (P69), and occupation (P106). Furthermore we filter for individuals that have notable contributions (P800), in an effort to maximize the chance that all LLMs we employ can answer questions regarding them. An entity may have multiple occupations and educations. For simplicity, we only select one of them in a random manner. We also only choose entities that have complete labels for all attributes and ensure that they are distinct individuals, resulting in the final 4,255 examples.

[0435] In Table 3, we show an example of our dataset. The paraphrased biography entry is obtained with Llama-3.1-8B-lnstruct through greedy decoding. We set the maximum length of generated tokens to = 150. We also ensure that the paraphrased entries still contain all the original attributes. We use the following prompt:

[0436] prompt = f <|start_header_id|>system<|end_header_id|> You are a helpful assis tant.<|eot_id|><|start_header_id|>user<|end_header_id|> Write a 150 words fictional biography containing the following facts in random order, make sure to include ALL facts VERBATIM as they are: facts<|eot_id|><|start_header_id|>assistant<|end_header_id|> Here is a 150-word fictional biography of {name}:

[0437] Implementation Details All model checkpoints that we use and their corresponding tokenizers are available on HuggingFace: meta-llama / Llama-3.1-8B-lnstruct, (https: / / huggingface.co / meta-llama / Llama-3.1-8B-lnstruct); mistralai / Mistral-7B-lnstruct-v0.3, (https: / / huggingface.co / mistralai / Mistral-7B-lnstruct-vO.3); and google / gemma-2-9b-

[0438] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0439] it.( https: / / huggingface.co / google / gemma-2-9b-it). All models were ran on mixed precision (bfloat16). We use Pyserini implementation of DPR

[0033] (https: / / github.com / castorini / pyserini / tree / master). For BERT matching, we use the checkpoint provided by Kortukov et al.

[0035] (https: / / huggingface.co / kortukov / answer-equivalence-bem). For all experiments, we apply greedy decoding and set maximum limit of generated tokens to =20 unless specified otherwise.

[0440] Compute Details: All the experiments were conducted on 2 x 24 GB RTX4090 and 4 x 40 GB A100. Computing difference score D to identifying context and parametric heads takes about 8 hours. Ablating both heads on NQ-Swap and TQA takes about 3 hours for each run. Patching task and parametric FVs takes about 12 hours on average for each model, while the NIAH experiment with retrieval heads takes about 8 hours. Our source tracking experiments consume about 4 hours.

[0441] Prompt Details: For §5, we format our questions with a simple prompt template following

[0053] ,

[0442] Table 4 shown in Fig. 25 shows overview of models’ performance on NQ-Swap and TQA for reproducibility purposes. Besides recall, we compute traditional exact match accuracy and BERT matching (BEM)

[0013] to measure semantic match. We also adopt K-Precision [3] to evaluate answers’ groundedness.

[0443] For example, when referring to “ablating” herein, that means setting the heads to 0, e.g., the output thereof.

[0444] B Details: Localization of In-Context and Parametric Heads

[0445] We use a prompt following the template in https: / / arxiv.org / abs / 2305.06311. We use only relevant documents via k-precistion. We also use recall as output of instruction tuned models tend to be verbose.

[0446] C Details: Functional Roles of In-Context and Parametric Heads

[0447] C.1 Functional Maps

[0448] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0449] Fig. 17 presents the functional maps for Mistral-7B-lnstruct-v0.3 and Gemma-2-9b-it. Consistent with Llama-3.1-8B-lnstruct in Fig. 12, these models exhibit a strikingly similar structure: a concentrated band of in-context heads in the middle layers, flanked by parametric heads in the early and late layers. We hypothesize that these early parametric heads may serve to enrich the prompt with relational knowledge, allowing later in-context heads to effectively integrate this information across the entire prompt, while later retrieval & parametric heads extract the answer. This intriguing pattern suggests a potential general principle governing transformer architectures, raising the question of whether this structure is a universal feature of language models. Understanding why gradient descent naturally converges to this form presents an exciting direction for future research.

[0450] Fig. 17 illustrates afunctional map of in-context and parametric heads in Mistral-7B-lnstruct-v0.3 and Gemma 2-9b-it. Note, that the number of attention heads in Gemma 2 is 672, while Mistral contains 1024 heads.

[0451] C.2 Disentangling Functional Roles of In-Context heads

[0452] We also compute the sorted in-context scores for Mistral-7B-lnstruct-v0.3 and Gemma-2-9b- it. The results are visible in Fig. 18. While most task heads exhibit high in-context scores, some also display parametric behavior, as indicated by their negative in-context scores.

[0453] Fig. 18 illustrates sorted in-context scores for all heads of Mistral-7B-lnstruct-v0.3 and Gemma-2-9b-it, comparing open-book and closed-book settings via score D. Positive scores indicate in-context behavior, while negative scores reflect parametric behavior. Retrieval heads (green) and task heads (blue) are predominantly high-scoring in-context heads.

[0454] C.3 Task Heads

[0455] Experiment Details To assess whether the activations of task heads genuinely capture information about the intensional frame - the instruction the model aims to follow - we extract the head outputs

[0456]

[0457] at the final token position. These outputs are then saved and directly patched into unrelated prompts in a zero(one)-shot manner without averaging across heads. Each head is patched separately, maintaining the unique contribution of each task head.

[0458] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0459] For this evaluation, we use the [biographydataset]RA containing multiple entries, each annotated with four distinct attributes: birthdate, educational institution, birthplace, and profession. For each entry, we use four different questions, including “At which date was he born?”, “At which institution was he educated?”, “In which place was he born?”, and “What was his profession?”.

[0460] We append each question (in bold exemplary for one question) to the biography entry shown in Fig. 22, and extract the FVs in a single pass at last token position.

[0461] Two key hyperparameters influence this method: 1. The number of task heads selected per model for FV extraction, and 2. The extent to which the head activations are amplified to overwrite potentially conflicting instructions within the model’s context. This is defined by ig = ctzg (11)

[0462] Where a e IR is a scaling factor. We conducted a hyperparameter sweep over 5% of the data set as a development set, finding that a scaling factor of a = 2 performed well for Llama and Mistral models, while a = 3 was optimal for Gemma. Scaling factors that were too high disrupted the model’s ability to generate coherent text, too small factors did not successfully change the model response. For consistency, we selected 40 task heads for Llama, Mistral and Gemma, achieving at least 80% task accuracy across these models.

[0463] The Table 5 shown in Fig. 26 summarizes these results for three different models: Llama 3.1 Instruct (8B), Mistral v0.3 lnstruct(7B), and Gemma 2 Instruct (9B). To further investigate the role of in-context heads, we compare this performance against two configurations:

[0464] • { task}k= - Selecting the top 40 task heads most relevant to the target function.

[0465] • { taskT\ctx}k= - Selecting only task heads that are also strong in-context heads.

[0466] We observe that the strong in-context heads alone capture a significant portion of the recall score, suggesting they play a critical role in interpreting the intensional frame. However, the inclusion of weaker in-context heads still pushes the recall scores higher, indicating that a diverse set of heads contributes to broader coverage across the task space. According to Table 5, we observe that the random score can occasionally be 20%. This occurs because, without FV patching, the model sometimes repeats the input prompt verbatim.

[0467] Qualitative Examples: Below is a qualitative example using an input promt in Llama. In the forward pass, no question is attached to the biography entry. Then, we patch the feature vectors and observe the model’s responses.

[0468] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0469] Input:

[0470] Jimmy Wales, a renowed orator, was born on 07 August 1966 in the vibrant city of Huntsville. Growing up, Wales was fascinated by the world of technology and innovation, which would later becme the foundation of his illustrious career. After completing his education, Wales went on to pursue a defree in finance form Indiana University Bloomington. However, it was his passion for online communities and collaborative knowledge-sharing that led him to create on of the most influential websites of our time. As the cofounder of Wikipedia, Wales revolutionized the way people access and share information. His dedication to free and open-source knowledge has made a lasting impact on the digital landscape. Today, Wales contivues to be a prominent figure in tech industry, inspiring a new generation of innovators and thinkers. His remarkable journey is a testament to the power of vision and perseverance.

[0471] Response:

[0472] Indiana University Bloomington.

[0473] Indiana University Bloomington.

[0474] Indiana University Bloomington.

[0475] Is it visible that the model is repeating the answer tokens because we patch the FVs at all token positions after generation.

[0476] C.4 Parametric Heads

[0477] Experiment Details: We select random 1,000 examples for each model where their closed book answer is correct wrt. gold reference (measured by recall with a threshold of 0.7). Then we split them randomly with a proportion of 2.5% train, 2.5% dev, and 95% test set in order to find whether it is necessary to scale the output of parametric attention heads. We consider the scaling factor f e [1, 1.25, 1.5, 1.75, 2, 2.25, 2.5, 2.75, 3], maximizing recall on the dev set. For the number of heads, we also consider nheade [10,20,30,40,50,60,70,80,90,100] taken form the top scoring parametric heads identified in §5. In our final results, we extract parametric FVs from the combination of train and validation sets in a zero-shot manner, and

[0478] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0479] apply them on the test sets. Table 6 shown in Fig. 27A shows recall scores on the development set with their corresponding optimal scaling factor.

[0480] As illustrated in Fig. 15, an attribute of an individual is converted to a cloze-style statement, whose parametric FVs are then extracted from the last token position. The attribute is chosen randomly, to demonstrate that parametric FVs indeed contain the entities’ information and not just a particular attribute. Then, we insert the parametric FVs to final token of a question prompt of another unrelated individual, and also for all subsequent token generations. We show cloze-style statement and the prompt we used to elicit the answer in Table 7 shown in Fig. 27B.

[0481] C.5 Retrieval Heads

[0482] Experiment Detail: Following the previous analysis of task and parametric heads, we utilize the biography dataset for this experiment. Each entry is provided to the model without an accompanying question, and we randomly insert a multi-token needle within the prompt. This process is repeated 10 times, each with a different needle. The needles used are famous poem titles from around the world:

[0483] 1. Auguries of Innocence

[0484] 2. Al-Burda

[0485] 3. Der Zauberlehrling

[0486] 4. Ode to a Nightingale

[0487] 5. She Walks in Beaty

[0488] 6. The Raven

[0489] 7. The Road Not Taken

[0490] 8. The Second Coming

[0491] 9. The Waste Land

[0492] 10. Uber die Berge

[0493] As a hyperparameter, we only vary the number of retrieval heads. We use 5% of the dataset as a development set, where we select the smallest value of K (top K retrieval heads) that achieves a recall score of approximately 90%. To activate the copying behavior in the retrieval heads, we modify the attention weights to concentrate on the tokens of the needle. To allow for some adaptivity of the model, we use the following boosting scheme, such that the model can focus on subtokens inside the needle:

[0494] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0495] Letjneedie be the set of token positions corresponding to the multi-token needle. Let ASj denote the unnormalized attention weights (before applying the softmax function) at last query position. The modification is performed in two steps:

[0496] 1. Initial Needle Tokens Boost: This step prevents the attention weights from being zero before applying the softmax, ensuring that the model can effectively attend to the needle tokens:

[0497] g Ag + 5, Vj €jneedle (12)

[0498] 2. Scaling Adjustment: This step further amplifies the weights for needle tokens that the model already tends to focus on (having a value bigger than zero), reinforcing the copying behavior:

[0499]

[0500] 1000 • Astj, V.j 6 eedle (13)

[0501] This two-step approach encourages the model to selectively copy one token at a time, which we found to be effective in practice. If the initial boost (Step 1) is omitted, performance tends to degrade, as zero attention weights cannot be effectively scaled in the second step.

[0502] D Details: Source Tracking

[0503] D.1 Training & Evaluation

[0504] Let

[0505]

[0506] be the set of / V samples of the NQ-Swap knowledge-conflict dataset, wheret(n) e NI I denotes the target token index in the vocabulary V for sample n and

[0507] (n) >

[0508]

[0509] if predicted token is contextual (from external documents), I 0, if predicted token is parametric (from model memory).

[0510] All samples include counterfactual entries, filtered to retain only those where: (1) the counterfactual answer object ocappears among the top 10 predicted tokens, and (2) the correct closed-book parametric answer object OP is also accurately predicted among the top 10 predicted tokens. This approach allows for a direct comparison of parametric and

[0511] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0512] contextual retrieval head activations for identical inputs, enhancing the probe’s training quality.

[0513] We then learn weights {wh}h∈Hover a selected set of retrieval heads Hretby solving

[0514]

[0515] An optimal decision threshold is then chosen via ROC analysis on a held-out validation subset of Z, selecting the threshold that maximizes the true positive rate while minimizing the false positive rate.

[0516] To test localization, we aggregate each head’s attention map with the learned terms:

[0517]

[0518] We also experimented with a simple averaging of the attention maps, but this approach resulted in approximately 10% lower scores across all models. We then predict the source token index as

[0519] k̂ = arg max ÂS,j

[0520] j

[0521] Since localization is only meaningful for contextual samples, we restrict this evaluation to counterfactual samples from Z. Specifically, we compute the top-1 accuracy by checking whether k matches the ground truth token position of the counterfactual entry oc.

[0522] In Fig. 19, the logit lens scores of the top 40 retrieval heads in the Llama model are illustrated. We observe that retrieval heads exhibit heightened activity when the model relies on context, as indicated by the elevated logit lens scores in green color. While these distributions are 1 -dimensional for each head, the probe itself learns a decision boundary in the full 40-dimensional space, where the retrieval signal may be better disentangled. Interestingly, some heads appear only sporadically active, suggesting a high degree of specialization - a promising direction for future research.

[0523] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0524] In Fig. 20, we illustrate the localization capabilities of the aforementioned method on four random samples for all three models. The aggregated attention maps are plotted as heatmaps, where red colors signify high values and blue colors signify negative colors. Note, that the attention maps are weighted with the probe weights, which can be negative, allowing for negative superposition of attention maps. The beginning of start token is receiving some attention weight values due to its usage as attention sinks

[0072] ,

[0525] Fig. 19 illustrates a distributions of logit lens scores for the top 40 retrieval heads in a Llama 3.1 Instruct (8b). Shown are the logit lens activations for the ground truth and counterfactual output tokens, comparing cases where the model generates the answer from its parameters (red) versus cases where it retrieves the answer from context (green) respectively. We observe that retrieval heads exhibit heightened activity when the model relies on context, as indicated by the elevated logit lens scores in green color. While these distributions are 1 -dimensional for each head, the probe itself learns a decision boundary in the full 40-dimensional space, where the retrieval signal may be better disentangled. Interestingly, some heads appear only sporadically active, suggesting a high degree of specialization - a promising direction for future research.

[0526] Fig. 20 illustrates a heatmaps of the weighted aggregation of retrieval-heads’ attention maps at the final query position overlaid on the input prompt to pinpoint the retrieved source token. For each model, the aggregated attention maps of the retrieval heads reliably focus on the predicted token in the context, which can be used as cost-effective source tracking. The beginning of start token is receiving some attention weight values due to its usage as attention sinks

[0072] ,

[0527] In the following, embodiments are described again in other words, and in particular, advantages and benefits of embodiments are described as well as application scenarios, in which embodiments may be implemented.

[0528] Embodiments leverage relevance scores, e.g., computed via AttnLRP, to measure a difference score D for each attention head, reflecting their utilization by the model in different specialized settings, yielding parametric and in-context attention heads.

[0529] Within this separation, in-context heads can be further classified as, but not limited to: (1) retrieval head that extract information from the input context and (2) task head that encode the execution of a function or a task. While both categories are specific to the setting, it

[0530] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0531] does not limit the possibility to find another subcategory of in-context heads: e.g., those that perform semantic copying from input context (copy concept instead of verbatim copy) and enhance I alter the generated responses by making use of the combined information available in the input context and previous responses in a multi turn setting (“make the summary more succinct than this”).

[0532] Using knowledge about the nature of attention heads within the bounds of the categories of parametric heads, in-context heads as well as the sub-categories of the latter, according to embodiments, their activity is measured during inference in order to estimate the “mode” a model is operating in.

[0533] From each attention head, depending on its type, Function Vectors (FV) can be extracted with the knowledge about the type of function.

[0534] Given observations on a suitable dataset, we can further determine the exact specifications of a Function Vector (FV), such as “translate token(s) to german” or “retrieve date of birth of subject” or “reveal education of subject” or “summarize this paragraph in layman terms” for in-context heads, or “add specific relational attribute” for parametric heads.

[0535] Given an inference step of the model, where the task of the LLM is to predict the next token t, we now can, using AttnLRP to explain each potentially predicted token t (by observing the most likely candidate tokens t), identify the attention heads implying the prediction of each candidate token t.

[0536] Given knowledge about the relevant heads and their associated function vectors, we can now obtain and present information about the various latent modes of operation of the generative LLM which are currently being employed for inference.

[0537] This information is helpful for the selection of a token t to actually predict, based on a set of probable candidate tokens, based on an assessment whether the token t is suitable in a sense that it fulfills conditions to expected / safe / reliable behavior.

[0538] This capability is critical where guarantees about LLM behavior must be given, and e.g., factual correctness and determinism of the model is key.

[0539] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0540] One potential application here is the use of an LLM for screening large bodies of documents in order to answer a particular question of e.g., legal or medical nature. Using our mechanisms, we can ensure that (or whether) the given answer is based on information from the context, or by executing the right task.

[0541] This is especially of interest, if the training of the model has happened on outdated data, and we want to make sure that the procured answer is given based on an up to-date data corpus given to the model in an in-context learning (ICL) setting.

[0542] Further capabilities involving the FVs extracted from the attention heads: Our method is able to categorize the injection of behavior into model at another time (independent from when it was extracted), without being restricted to direct the model via altering the input prompt.

[0543] With the FVs extracted according to our method, we can at will activate or deactivate FVs in order to control model behavior akin to enabling or disabling model behavior and capabilities on a dashboard. This entails the capability to “guarantee” certain model behavior via targeted (de)activation of FVs and / or attention heads based on information about their role and behavior.

[0544] Another related capability of our approach is the ability to build an atlas of FVs from attention heads with specific functions and roles.

[0545] One example could be that an LLM has been instructed to search a corpus of text it then has retrieved some information from, e.g., the date of birth of a person of interest. By applying the correct FV, one can in addition extract another datum of the person, e.g., their nationality, whch might if desired by given in another language in case a further “translate into other language” FV is activated in addition (as a composition of function). This in a sense can be seen as a technique to compress prompt in the form of FVs.

[0546] These editing capabilities can be used to steer / direct / control / restrict the Al model’s range of inference during inference time directly, increasing LLM safety and reliability. Applications and Implications (specific for source tracking and generating citations from LLM generated answers)

[0547] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0548] Short summary: For all heads categorized as retrieval heads, we train linear probes on top for two purposes: 1) To predict whether a predicted token (that is, each potentially predicted token) is parametric or contextual in nature and 2) To be combined with the outputs of retrieval heads, giving us localization scores for each predicted (and potentially predicted) token t. These scores can be understood as localization indicators of information the LLM has retrieved from (a potentially long) contexts during inference in order to give a specific answer embodied by t.

[0549] An advantage of our approach is that the local ization / citation can be computed “for free” since it does not required to explain predictions the model has made. This is enabled by identifying parametric and in-context heads (and their sub-categories) in a pre-hoc manner.

[0550] The obvious benefit of our approach, given the verbatim copying behavior of retrieval heads and the high predictive quality of our trained probes, is the ability to provide hallucination-free citation of sources for generated answers by LLMs.

[0551] Given citations / localizations as “by-product” during the LLM generation process, this will make the operation of an LLM much more truthful and reliable, since we can now without major added costs point towards the informative source in context documents in a retrieval case.

[0552] To repeat, we will also receive information via the probe whether the model has acted and generated answers based on parametric knowledge.

[0553] Next to highlighting particular answer tokens in the context, a wider application scenario is source identification in large and connected contexts. Our approach can highlight sentences, paragraphs, documents (in concatenation), etc, without having to rely on artificial segmentation of input contexts prior to ingestion by the model. Furthermore, we can highlight inconsistency in the (concatenated) documents due to possible inter-context knowledge conflicts.

[0554] In summary, when observing an LLM generate an answer, we can build an application which highlights each generated output token / word with various properties as identified by our approach, marking information about contextuality etc of the generated answer (i.e., whether there is a retrievable source for the fact in the LLMs reply) immediately accessible.

[0555] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0556] Additionally, for each generated output token, we can generate heatmaps or single word highlights (e.g., by thresholding / filtering the heatmap) on top of the input context, including the instructions or questions given to the model.

[0557] We can do so with near-perfect accuracy in terms of source identification with close to zero added relative computational cost, as we just have to summarize what the model computes during inference anyway.

[0558] Applications where this is useful can be in literature research of any kind, for example in legal documents, where proof of facts is critically important. Research through medical or other documents might reveal answers fast, yet point the reader to the detailed source material for further research (adjustable granularity).

[0559] Implementation alternatives

[0560] It is to be mentioned here that all alternatives or aspects as discussed before and all aspects as defined by independent claims in the following claims can be used individually, i.e., without any other alternative or object than the contemplated alternative, object or independent claim. However, in other embodiments, two or more of the alternatives or the aspects or the independent claims can be combined with each other and, in other embodiments, all aspects, or alternatives and all independent claims can be combined to each other.

[0561] Although some aspects have been described as features in the context of an apparatus it is clear that such a description may also be regarded as a description of corresponding features of a method. Although some aspects have been described as features in the context of a method, it is clear that such a description may also be regarded as a description of corresponding features concerning the functionality of an apparatus. In particular, block diagrams illustrating the functionality of an apparatus may also be understood as illustration of a respective method comprising the functions described by the blocks of the block diagram as steps of the method.

[0562] Some or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the method steps may be executed by such an apparatus. Depending on certain implementation requirements, embodiments of the invention can be

[0563] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0564] implemented in hardware or in software or at least partially in hardware or at least partially in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable. Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed. Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine-readable carrier. Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine-readable carrier. In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer. A further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory. A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet. A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein. A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein. A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver. In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some

[0565] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0566] embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus. The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer. The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0567] In the foregoing Detailed Description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, where each claim may stand on its own as a separate example. While each claim may stand on its own as a separate example, it is to be noted that, although a dependent claim may refer in the claims to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not intended. Furthermore, it is intended to include also features of a claim to any other independent claim even if this claim is not directly made dependent to the independent claim.

[0568] The above-described embodiments are merely illustrative for the principles of the present disclosure. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the pending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

[0569] References

[0570] [1] R. Achtibat, M. Dreyer, I. Eisenbraun, S. Bosse, T. Wiegand, W. Samek, and S.

[0571] Lapuschkin. From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence, 5(9): 1006-1019, Sep

[0572] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0573] 2023. ISSN 2522-5839. doi: 10.1038 / s42256-023-00711-8. URL https: / / doi. Org / 10.1038 / S42256-023-00711 -8.

[0574] [2] R. Achtibat, S. M. V. Hatefi, M. Dreyer, A.jain, T. Wiegand, S. Lapuschkin, and W.

[0575] Samek AttnLRP: Attention-aware layer-wise relevance propagation for transformers. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver,j. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 135-168. PMLR, 21-27jul 2024.

[0576] [3] V. Adlakha, P. Behnam Ghader, X. H. Lu, N. Meade, and S. Reddy. Evaluating correctness and faithfulness of instruction-following models for question answering. Transactions of the Association for Computational Linguistics, 12:681–699, 2024. doi: 10.1162 / tacl_a_00667. URL https: / / aclanthology.org / 2024.tacl-1.38 / .

[0577] [4] E. Akyurek, D. Schuurmans,j. Andreas, T. Ma, and D. Zhou. What learning algorithm is in-context learning? Investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https: / / openreview.net / forum? 380 id=0g0X4H8yN4l.

[0578] [5] A. Ali, T. Schnake, O. Eberle, G. Montavon, K.-R. Muller, and L. Wolf. XAI for transformers: Better explanations through conservative propagation. In K. Chaudhuri, S.jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39thInternational Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 435-451. PMLR, 17-23jul 2022. URL https: / / proceedings.mlr. press / v162 / ali22a. 386 html.

[0579] [6] Z. Allen-Zhu and Y. Li. Physics of language models: Part 3.1, knowledge storage and extraction, 2024. URL https: / / arxiv.org / abs / 2309.14316.

[0580] [7] L. Arras, B. Puri, P. Kahardipraja, S. Lapuschkin, and W. Samek. A close look at decomposition-based xai-methods for transformer language models. arXiv preprint arXiv:2502.15886, 2025.

[0581] [8] S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Müller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLOS ONE, 10(7):1–46, 07 2015. doi: 10.1371 / journal.pone.0130140. URL https: / / doi.org / 10. 1371 / journal.pone.0130140.

[0582] [9]j. Bastings and K. Filippova. The elephant in the interpretability room: Why use attention as explanation when we have saliency methods? In A. Alishahi, Y. Belinkov, G. Chrupala, D. Hupkes, Y. Pinter, and H. Sajjad, editors, Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 149–155, Online, Nov. 2020. Association for Computational

[0583] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0584] Linguistics. doi: 10.18653 / v1 / 2020.blackboxnlp-1.14. URL https: / / aclanthology.org / 2020.blackboxnlp-1.14 /

[0585]

[0010] A. Bibal, R. Cardon, D. Alfter, R. Wilkens, X. Wang, T. François, and P. Watrin. Is attention explanation? An introduction to the debate. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60thAnnual Meeting of the Association for Computational Linguistics (Volumel: Long Papers), pages 3889–3900, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 / v1 / 2022.acl-long.269. URL https: / / aclanthology.org / 2022.acl-long.269 / .

[0586]

[0011] S. Borgeaud, A. Mensch,j. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche,j.-B. Lespiau, B. Damoc, A. Clark, D. De Las Casas, A. Guy,j. Menick, R. Ring, T. Hennigan, S. Huang, L. Maggiore, C.jones, A. Cassirer, A. Brock, M. Paganini, G. Irving, O. Vinyals, S. Osindero, K. Simonyan,j. Rae, E. Elsen, and L. Sifre. Improving language models by retrieving from trillions of tokens. In K. Chaudhuri, S.jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39thInternational Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2206-2240. PMLR, 17-23jul 2022. URL https: / / proceedings.mlr.press / v162 / borgeaud22a.html.

[0587]

[0012] T. Brown, B. Mann, N. Ryder, M. Subbiah,j. D. Kaplan, P. Dhariwal, A.

[0588] Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler,j. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess,j. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. 2020 URL https: / / proceedings.neurips.cc / paper_files / paper / 2020 / file / 1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf

[0589]

[0013] j. Bulian, C. Buck, W. Gajewski, B. Bbrschinger, and T. Schuster. Tomayto, tomahto. Beyond token-level answer equivalence for question answering evaluation. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 291–305, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi: 10.18653 / v1 / 2022.emnlp-main.20. URL https: / / aclanthology.org / 2022.emnlp-main.20 / .

[0590]

[0014] K. Clark, U. Khandelwal, O. Levy, and C. D. Manning. What does BERT look at? An analysis of BERT'S attention. In T. Linzen, G. Chrupala, Y. Belinkov, and D.

[0591] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0592] Hupkes, editors, Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy, Aug.

[0593] 2019. Association for Computational Linguistics. doi:10.18653 / v1 / W19-4828. URL https: / / aclanthology.org / W19-4828 / .

[0594]

[0015] D. Dai, L. Dong, Y. Hao, Z. Sui, B. Chang, and F. Wei. Knowledge neurons in pretrained transformers. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60thAnnual Meeting of the Association for Computational Linguistics (Volumel: Long Papers), pages 8493-8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:10.18653 / v1 / 2022.acl-long.581. URL https: / / aclanthology.org / 2022.acl-long. 439581 / .

[0595]

[0016] D. Dai, Y. Sun, L. Dong, Y. Hao, S. Ma, Z. Sui, and F. Wei. Why can GPT learn in context? Language models secretly perform gradient descent as meta-optimizers. In A. Rogers,j. Boyd-Graber, and N. Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 4005-4019, Toronto, Canada,july 2023. Association for Computational Linguistics. doi:10.18653 / v1 / 2023.findings- acl.247. URL https: / / aclanthology.org / 4452023. findings-acl.247 / .

[0596]

[0017] N. Elhage, N. Nanda, C. Olsson, T. Henighan, N.joseph, B. Mann, A. Askell, Y.

[0597] Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, A.jones,j. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown,j. Clark,j. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits. Transformer Circuits Thread, 2021. https: / / transformer- circuits.pub / 2021 / framework / index.html.

[0598]

[0018] j. Fernando, G. I. Gallego, and M. R. Costa-jussa. Measuring the mixing of contextual information in the transformer. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8698-8714, Abu Dhabi, United Arab Emirates, Dec. 2022. Association for Computational Linguistics. doi:10.18653 / v1 / 2022.emnlp- main.595. URL https: / / aclanthology.org / 456 2022.emnlp-main.595 / .

[0599]

[0019] A. Fisch, A. Talmor, R.jia, M. Seo, E. Choi, and D. Chen. MRQA 2019 shared task:

[0600] Evaluating generalization in reading comprehension. In A. Fisch, A. Talmor, R.jia, M. Seo, E. Choi, and D. Chen, editors, Proceedings of the 2ndWorkshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China, Nov. 2019. Association for Computational Linguistics, doi: 10.18653 / v1 / D19-5801. URL https: / / aclanthology.org / D19-5801 / .

[0601]

[0020] T. Gao, H. Yen,j. Yu, and D. Chen. Enabling large language models to generate text with citations. In H. Bouamor,j. Pino, and K. Bali, editors, Proceedings of the

[0602] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0603] 2023 Conference on Empirical Methods in Natural Language Processing, pages 6465-6488, Singapore, Dec. 2023. Association for Computational Linguistics, doi: 10.18653 / v1 / 2023.emnlp-main.398. URL https: / / aclanthology.org / 2023.emnlp- main.398 / .

[0604]

[0021] A. Geiger, H. Lu, T. Icard, and C. Potts. Causal abstractions of neural networks. In

[0605] M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, andj. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 9574-9586. Curran Associates, Inc., 2021. URL https: / / proceedings.neurips.cc / paper_files / paper / 2021 / file / 4f5c422f4d49a5a807eda 27434231040-Paper.pdf.

[0606]

[0022] M. Geva,j. Bastings, K. Filippova, and A. Globerson. Dissecting recall off actual associations in auto-regressive language models. In H. Bouamor,j. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12216-12235, Singapore, Dec. 2023. Association for Computational Linguistics, doi: 10.18653 / v1 / 2023.emnlp-main.751. URL https: / / aclanthology.org / 2023.emnlp-main.751 / .

[0607]

[0023] A. Grattafiori, A. Dubey, A.jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang, A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, and A. Rao. The llama 3 herd of models, 2024. URL https: / / arxiv.org / abs / 2407.21783.

[0608]

[0024] K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang. Retrieval augmented language model pre-training. In H. D. Ill and A. Singh, editors, Proceedings of the 37thInternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929-3938. PMLR, 13-18jul 2020. URL https: / / proceedings.mlr.press / v119 / guu20a.html.

[0609]

[0025] M. Hanna, S. Pezzelle, and Y. Belinkov. Have faith in faithfulness: Going beyond circuit overlap when finding model mechanisms. In First Conference on Language Modeling, 2024. URL https: / / openreview.net / forum?id=TZOCCGDcuT.

[0610]

[0026] R. Hendel, M. Geva, and A. Globerson. In-context learning creates task vectors. In H. Bouamor,j. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9318-9333, Singapore, Dec. 2023. Association for Computational Linguistics, doi: 10.18653 / v1 / 2023.findings- emnlp.624. URL https: / / aclanthology.org / 4932023. findings-emnlp.624 / .

[0611]

[0027] G. Izacard and E. Grave. Leveraging passage retrieval with generative models for open domain question answering. In P. Merlo,j. Tiedemann, and R. Tsarfaty, editors, Proceedings of the 16thConference of the European Chapter of the

[0612] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0613] Association for Computational Linguistics: Main Volume, pages 874-880, Online, Apr. 2021. Association for Computational Linguistics, doi: 10.18653 / v1 / 2021.eacl- main.74. URL https: / / aclanthology.org / 2021.eacl-main.74 / . 499

[0614]

[0028] A.jacovi and Y. Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In D.jurafsky,j. Chai, N. Schluter, andj. Tetreault, editors, Proceedings of the 58thAnnual Meeting of the Association for Computational Linguistics, pages 4198-4205, Online,july 2020. Association for Computational Linguistics. doi: 10.18653 / v1 / 2020.acl-main.386. URL https: / / aclanthology.org / 2020.acl-main.386 / .

[0615]

[0029] S.jain and B. C. Wallace. Attention is not Explanation. Inj. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556, Minneapolis, Minnesota,june 2019. Association for Computational Linguistics, doi: 10.18653 / vl / N 19-1357. URL https: / / aclanthology.org / N19-1357 / .

[0616]

[0030] A. Q.jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed. Mistral 7b, 2023. URL https: / / arxiv.org / abs / 2310.06825.

[0617]

[0031] Z.jin, P. Cao, H. Yuan, Y. Chen,j. Xu, H. Li, X.jiang, K. Liu, andj. Zhao. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 1193-1215, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653 / v1 / 2024.findings-acl.70. URL https: / / aclanthology.org / 2024.findings-acl.70 / .

[0618]

[0032] M.joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In R. Barzilay and M.-Y. Kan, editors, Proceedings of the 55thAnnual Meeting of the Association for Computational Linguistics (Volumel: Long Papers), pages 1601-1611, Vancouver, Canada,july 2017. Association for Computational Linguistics, doi: 10.18653 / V1 / P17-1147. URL https: / / aclanthology.org / P17-1147 / .

[0619]

[0033] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih.

[0620] Dense passage retrieval for open-domain question answering. In B. Webber, T. Cohn, Y. He, and Y. Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online,

[0621] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0622] Nov. 2020. Association for Computational Linguistics, doi: 10.18653 / v1 / 2020.emnlp- main.550. URL https: / / aclanthology.org / 2020.emnlp-main.550 / .

[0623]

[0034] U. Khandelwal, O. Levy, D.jurafsky, L. Zettlemoyer, and M. Lewis. Generalization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020. URL https: / / openreview.net / forum?id=HklBjCEKvH.

[0624]

[0035] E. Kortukov, A. Rubinstein, E. Nguyen, and S.j. Oh. Studying large language model behav- iors under context-memory conflicts with real documents. In First Conference on Language Modeling, 2024. URL https: / / openreview.net / forum?id=xm8zYRfrqE.

[0625]

[0036] O. Kovaleva, A. Romanov, A. Rogers, and A. Rumshisky. Revealing the dark secrets of BERT. In K. Inui,j.jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationaljoint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China, Nov. 2019. Association for Computational Linguistics, doi: 10.18653 / v1 / D19-1445. URL https: / / aclanthology.org / D19-1445 / .

[0626]

[0037] T. Kwiatkowski,j. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein,

[0627] I. Polosukhin,j. Devlin, K. Lee, K. Toutanova, L.jones, M. Kelcey, M.-W. Chang, A. M. Dai,j. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019. doi: 10.1162 / tacl_a_00276. URL https: / / aclanthology.org / Q19-1026 / .

[0628]

[0038] P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Kiittler, M.

[0629] Lewis, W.-t. Yih, T. Rocktaschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive nip tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9459-9474. Curran Associates, Inc., 2020. URL https: / / proceedings.neurips.cc / paper_files / paper / 2020 / file / 6b493230205f780e1bc2 6945df7481e5-Paper.pdf.

[0630]

[0039] K. Li, O. Patel, F. Viegas, H. Pfister, and M. Wattenberg. Inference-time interven- tion: Eliciting truthful answers from a language model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 41451-41530. Curran Associates, Inc., 2023. URL https: / / proceedings.neurips.cc / paper_files / paper / 2023 / file / 81b8390039b7302c909c b769f8b6cd93-Paper-Conference.pdf.

[0631] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0632]

[0040] j. Liu, D. Shen, Y. Zhang, B. Dolan, L. Carin, and W. Chen. What makes good incontext examples for GPT-3? In E. Agirre, M. Apidianaki, and I. Vulic', editors, Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100— 114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics. doi: 10.18653 / v1 / 2022.deelio-1.10. URL https: / / aclanthology.org / 2022.deelio-1.10 / .

[0633]

[0041] S. Longpre, K. Perisetla, A. Chen, N. Ramesh, C. DuBois, and S. Singh. Entitybased knowledge conflicts in question answering. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052-7063, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics, doi: 10.18653 / v1 / 2021.emnlp-main.565. URL https: / / aclanthology.org / 2021.emnlp- main.565 / .

[0634]

[0042] S. M. Lundberg and S.-l. Lee. A unified approach to interpreting model predictions.

[0635] In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: / / proceedings.neurips.cc / paper_files / paper / 2017 / file / 8a20a8621978632d76c4 3dfd28b67767-Paper.pdf.

[0636]

[0043] B. McCann, N. S. Keskar, C. Xiong, and R. Socher. The natural language decathlon:

[0637] Multitask learning as question answering, 2018. URL https: / / arxiv.org / abs / 1806.08730.

[0638]

[0044] j. Minder, K. Du, N. Stoehr, G. Monea, C. Wendler, R. West, and R. Cotterell.

[0639] Controllable context sensitivity and the knob behind it. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / / openreview.net / forum?id=lgm9bbkzHC.

[0640]

[0045] A. Nafar, K. B. Venable, and P. Kordjamshidi. Learning vs retrieval: The role of incontext ex- amples in regression with large language models. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8206-8229, Albuquerque, New Mexico, Apr. 2025. Association for Computational Linguistics. ISBN 979-8- 89176-189-6. URL https: / / aclanthology.org / 2025.naacl-long.417 / .

[0641] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0642]

[0046] nostalgebraist. Interpreting gpt: The logit lens, August 2020. URL https: / / www.lesswrong. com / posts / AcKRB8wDpdaN6v6ru / interpreting-gpt-the-logit- lens. Accessed: 2025-05-12.

[0643]

[0047] C. Olsson, N. Elhage, N. Nanda, N.joseph, N. DasSarma, T. Henighan, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernandez, S.johnston, A.jones,j. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown,j. Clark,j. Kaplan, S. McCandlish, and C. Olah. In-context learning and induction heads, 2022. URL https: / / arxiv.org / abs / 2209.11895.

[0644]

[0048] F. Ortu, Z.jin, D. Doimo, M. Sachan, A. Cazzaniga, and B. Schölkopf. Competition of mechanisms: Tracing how language models handle facts and counterfactuals. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8420-8436, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics. doi: 10.18653 / v1 / 2024.acl-long.458. URL https: / / aclanthology.org / 2024.acl-long.458 / .

[0645]

[0049] L. Ouyang,j. Wu, X.jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S.

[0646] Agarwal, K. Slama, A. Ray,j. Schulman,j. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano,j. Leike, and R. Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730-27744. Curran Associates, Inc., 2022. URL https: / / proceedings.neurips.cc / paper_files / paper / 2022 / file / b1efde53be364a73914f 58805a001731-Paper-Conference.pdf.

[0647]

[0050] F. Petroni, T. Rocktaschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller.

[0648] Language models as knowledge bases? In K. Inui,j.jiang, V. Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationaljoint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China, Nov. 2019. Association for Computational Linguistics, doi: 10.18653 / v1 / D19-1250. URL https: / / aclanthology.org / D19-1250 / .

[0649]

[0051] F. Petroni, P. Lewis, A. Piktus, T. Rocktaschel, Y. Wu, A. H. Miller, and S. Riedel.

[0650] How context affects language models’ factual predictions. In Automated Knowledge Base Construction, 2020. URL https: / / openreview.net / forum?id=025X0zPfn.

[0651]

[0052] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P.j. Liu. Exploring the limits of transfer learning with a unified text-to-text

[0652] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0653] transformer.journal of Machine Learning Research, 21(140):1-67, 2020. URL http: / / jmlr.org / papers / v21 / 20-074.html.

[0654]

[0053] O. Ram, Y. Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton-Brown, and Y. Shoham. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316-1331, 2023. doi: 10.1162 / tacl_a_00605. URL https: / / aclanthology.org / 2023.tacl-1.75 / .

[0655]

[0054] D. Samuel. Berts are generative in-context learners. In A. Globerson, L. Mackey,

[0656] D. Belgrave, A. Fan, U. Paquet,j. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 2558-2589. Curran Associates, Inc., 2024. URL https: / / proceedings.neurips.cc / paper_files / paper / 2024 / file / 04ea184dfb5f1babb78c0 93e850a83f9-Paper-Conference.pdf.

[0657]

[0055] D. Schlangen. Llms as function approximators: Terminology, taxonomy, and questions for evaluation, 2024. URL https: / / arxiv.org / abs / 2407.13744.

[0658]

[0056] S. Serrano and N. A. Smith. Is attention interpretable? In A. Korhonen, D. Traum, and L. Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 2931-2951, Florence, Italy,july 2019. Association for Computational Linguistics, doi: 10.18653 / v1 / P19-1282. URL https: / / aclanthology.org / P19-1282 / .

[0659]

[0057] K. Shuster, S. Poff, M. Chen, D. Kiela, andj. Weston. Retrieval augmentation reduces hallucination in conversation. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, edi- tors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784- 3803, Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguis- tics, doi: 10.18653 / v1 / 2021.findings- emnlp.320. URL https: / / aclanthology.org / 2021. findings-emnlp.320 / .

[0660]

[0058] K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks:

[0661] visualising image classification models and saliency maps. In Proceedings of the International Conference on Learning Representations (ICLR). ICLR, 2014.

[0662]

[0059] M. Sundararajan, A. Taly, and Q. Yan. Axiomatic attribution for deep networks. In D.

[0663] Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3319-3328. PMLR, 06-11 Aug 2017. URL https: / / proceedings.mlr. press / v70 / sundararajan17a.html.

[0664]

[0060] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume

[0665] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0666] 27. Curran Associates, Inc., 2014. URL https: / / proceedings.neurips.cc / paper_files / paper / 2014 / file / a14ac55a4f27472c5d894ec1c3c743d2-Paper.pdf.

[0667]

[0061] G. Team, M. Riviere, S. Pathak, P. G. Sessa, C. Hardin, S. Bhupatiraju, L. Hussenot, T. Mesnard, B. Shahriari, A. Rame, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024.

[0668]

[0062] E. Todd, M. Li, A. S. Sharma, A. Mueller, B. C. Wallace, and D. Bau. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https: / / openreview.net / forum?id=AwyxtyMwaG.

[0669]

[0063] A. Vaswani, N. Shazeer, N. Parmar,j. Uszkoreit, L.jones, A. N. Gomez, L. u.

[0670] Kaiser, and I. Polo- sukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Sys- terns, volume 30. Curran Associates, Inc., 2017. URL https: / / proceedings.neurips.cc / paper_files / paper / 2017 / file / 3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.

[0671]

[0064] E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov. Analyzing multi-head selfattention: Specialized heads do the heavy lifting, the rest can be pruned. In A. Korhonen, D. Traum, and L. Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy,july 2019. Association for Computational Linguistics, doi: 10.18653 / V1 / P19-1580. URL https: / / aclanthology.org / P19-1580 / .

[0672]

[0065] j. Von Oswald, E. Niklasson, E. Randazzo,j. Sacramento, A. Mordvintsev, A.

[0673] Zhmoginov, and M. Vladymyrov. Transformers learn in-context by gradient descent. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, andj. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 35151-35174. PMLR, 23-29jul 2023. URL https: / / proceedings.mlr.press / v202 / von-oswald23a.html.

[0674]

[0066] D. VrandecJc' and M. Krdtzsch. Wikidata: a free collaborative knowledgebase.

[0675] Commun. ACM, 57(10):78-85, Sept. 2014. ISSN 0001-0782. doi: 10.1145 / 2629489. URL https: / / doi.org / 10.1145 / 2629489.

[0676]

[0067] K. R. Wang, A. Variengien, A. Conmy, B. Shlegeris, andj. Steinhardt. Interpretability in the wild: a circuit for indirect object identification in GPT-2 small. In The Eleventh International Conference on Learning Representations, 2023. URL https: / / openreview.net / forum? id=NpsVSN6o4ul.

[0677] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0678]

[0068] j. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https: / / openreview.net / forum?id=gEZrGCozdqR.

[0679]

[0069] S. Wiegreffe and Y. Pinter. Attention is not not explanation. In K. Inui,j.jiang, V.

[0680] Ng, and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Internationaljoint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China, Nov. 2019. Association for Computational Linguistics, doi: 10.18653 / v1 / D19-1002. URL https: / / aclanthology.org / D19-1002 / .

[0681]

[0070] T. Wolf, L. Debut, V. Sanh,j. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz,j. Davison, S. Shleifer, P. von Platen, C. Ma, Y.jernite,j. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush. Transformers: State-of-the-art natural language processing. In Q. Liu and D. Schlangen, editors, Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Oct. 2020. Association for Computational Linguistics, doi: 10.18653 / v1 / 2020.emnlp-demos.6. URL https: / / aclanthology.org / 2020.emnlp-demos.6 / .

[0682]

[0071] W. Wu, Y. Wang, G. Xiao, H. Peng, and Y. Fu. Retrieval head mechanistically explains longcontext factuality. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / / openreview.net / forum?id=EytBpUGB1Z.

[0683]

[0072] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference on Learning Representations, 2024. URL https: / / openreview.net / forum?id=NG7sS51zVF.

[0684]

[0073] j. Xu, X. Sun, Z. Zhang, G. Zhao, andj. Lin. Understanding and improving layer normalization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: / / proceedings.neurips.cc / paper_files / paper / 2019 / file / 2f4fe03d77724a721700 6e5d16728874-Paper.pdf.

[0685]

[0074] R. Xu, Z. Qi, Z. Guo, C. Wang, H. Wang, Y. Zhang, and W. Xu. Knowledge conflicts for LLMs: A survey. In Y. Al-Onaizan, M. Bansal, and Y.-N. Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8541-8565, Miami, Florida, USA, Nov. 2024. Association for

[0686] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

[0687] Computational Linguistics, doi: 10.18653 / v1 / 2024.emnlp-main.486. URL https: / / aclanthology.org / 2024.emnlp-main.486 / .

[0688]

[0075] X. Xu, Z. Gou, W. Wu, Z.-Y. Niu, H. Wu, H. Wang, and S. Wang. Long time no see! open-domain conversation with long-term persona memory. In S. Muresan, P. Nakov, andA. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2639-2650, Dublin, Ireland, May 2022. Association for Computational Linguis- tics, doi: 10.18653 / v1 / 2022.findings-acl.207. URL https: / / aclanthology.org / 2022. findings-acl.207 / .

[0689]

[0076] K. Yin andj. Steinhardt. Which attention heads matter for in-context learning?, 2025. URL https: / / arxiv.org / abs / 2502.14010.

[0690]

[0077] Q. Yu,j. Merullo, and E. Pavlick. Characterizing mechanisms for factual recall in language models. In H. Bouamor,j. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9924-9959, Singapore, Dec. 2023. Association for Computational Linguistics, doi: 10.18653 / v1 / 2023.emnlp-main.615. URL https: / / aclanthology.org / 2023.emnlp- main.615 / .

[0691]

[0078] Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao,j. Gao,j. Liu, and B.

[0692] Dolan. DIALOGPT: Large-scale generative pre-training for conversational response generation. In A. Celikyilmaz andT.-H. Wen, editors, Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics: System Demonstrations, pages 270-278, Online,july 2020. Association for Computational Linguistics, doi: 10.18653 / v1 / 2020.acl-demos.30. URL https: / / aclanthology.org / 2020.acl-demos.30 /

[0693] FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm

Claims

1. Claims1. Apparatus (10), configured for identifying, out of a plurality of attention heads (24) of a machine learning model (14), one or more attention heads of a first type and / or one or more attention heads of a second type byfor each of a first set of predictions (28) and a second set of predictions (38) performed by the machine learning model, determining (20), for each of the plurality of attention heads, for each of the predictions of the first set and the second set, a relevance score (22), which measures a contribution of the respective attention head to a prediction result (16) of the respective prediction;wherein the first set of predictions (28) are performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model; andwherein the second set of predictions (38) are performed by providing the machine learning model with input data (36) to perform the respective prediction; and wherein, for each of the predictions of the second set, a prediction result of the respective prediction includes information that originates from the input data (36) provided to perform the respective prediction;assigning (30) one or more of the attention heads to the first type of attention heads and / or assigning one or more of the attention heads to the second type of attention heads by comparing, for each of the attention heads, an average relevance of the respective attention head in the first set of predictions and an average relevance of the respective attention head in the second set of predictions.

2. Apparatus according to claim 1, configured for obtaining the first set and the second set byperforming a plurality of predictions (28’;38’), including the first set and the second set, using the machine learning model to obtain respective prediction results byFH260509PCT-2026164863. DOCX PCT FILING VERSION / bmfor each of the first set of predictions (28), requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model; andfor each of a subset out of the plurality of predictions (38’), providing the machine learning model with input data (36’) to perform the respective prediction;selecting the second set of predictions (38) out of the subset of the plurality of predictions (38’) by selecting predictions, the prediction results of which include information that originates from the input data (36’) provided to perform of the respective prediction.

3. Apparatus according to claim 1 or 2, configured for obtaining the first set of predictions (28) byperforming a first plurality of predictions (28’) using the machine learning model to obtain respective prediction results by, for each of the first plurality of predictions, requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model; andfor each of the first plurality of predictions, assessing the prediction result of the respective prediction with respect to a ground truth information (19) for the respective prediction to obtain an assessment result; andselecting the first set of predictions out of the first plurality of predictions based on the assessment results.

4. Apparatus according to any of the claims 1 to 3, configured for obtaining the second set of predictions byperforming a second plurality of predictions (38’) using the machine learning model to obtain respective prediction results by, for each of the second plurality of predictions, providing the machine learning model with input data to perform the respective prediction; andFH260509PCT-2026164863. DOCX PCT FILING VERSION / bmfor each of the second plurality of predictions, assessing the prediction result of the respective prediction with respect to an extent, to which the prediction result aligns with an information included in the input data (36’), to determine an assessment result; andselecting the second set of predictions (38) out of the second plurality of predictions (38’) based on the assessment results determined for the second plurality of predictions.

5. Apparatus according to any of the claims 1 to 4, configured for, in comparing the average relevance of the respective attention head in the first set of predictions and the average relevance of the respective attention head in the second set of predictions, determining, for each of the attention heads, a first average relevance score by forming an average over the relevance scores determined for the respective attention head for the first set of predictions, and a second average relevance score by forming an average over the relevance scores determined for the respective attention head for the second set of predictions.

6. Apparatus according to claim 5, configured for, in comparing the average relevance of the respective attention head in the first set of predictions and the average relevance of the respective attention head in the second set of predictions, determining a difference score representing a difference between the first and the second average relevance scores determined for the respective attention head.

7. Apparatus according to any of the claims 1 to 6, configured for determining the relevance scores for the attention heads by backpropagating an output of the machine learning model through the machine learning model to the attention heads.

8. Apparatus according to any of the claims 1 to 7, configured for, in determining the relevance score for the respective attention head,determining, for each coefficient of an attention head output of the respective attention head, a relevance share, which measures a contribution of the respective coefficient to the prediction result (16) of the respective prediction,FH260509PCT-2026164863. DOCX PCT FILING VERSION / bmin summing the relevance shares determined for the respective attention head for the respective prediction, ignoring negative relevance shares.

9. Apparatus (210) configured for identifying, out of a plurality of attention heads (24) of a machine learning model (14), one or more attention heads of a predetermined type bydetermining (220), for a set of tokens (229) out of a plurality of input tokens (228) provided to the machine learning model (14) as input information for a prediction, for each of the plurality of attention heads (24), a relevance score (222), which represents a contribution of the set of tokens to a prediction result (16) of the prediction, which contribution is provided by the respective attention head; andassigning (230) one or more of the attention heads to the predetermined type based on the relevance scores (222) assigned to the attention heads (24).

10. Apparatus according to claim 9, configured for, in determining the relevance score for the respective attention head,determining, for each of the set of tokens, a token-individual relevance score, which represents a token-individual contribution of the respective token to the prediction result, which token-individual contribution is provided by the respective attention head; andaggregating the token-individual relevance scores determined for the set of tokens for the respective attention head.

11. Apparatus according to claim 9 or 10, wherein the set of tokens are part of a task information included in the input information.

12. Apparatus according to any of the claims 9 to 11, wherein the set of tokens represent a part of the input information, which is provided in the prediction result.

13. Apparatus according to any of the claims 9 to 12, configured for assigning one or more of the attention heads to the predetermined type byFH260509PCT-2026164863. DOCX PCT FILING VERSION / bmselecting a predetermined count of attention heads having assigned the highest relevance scores among the plurality of attention heads; orusing a threshold value for the relevance scores assigned to the plurality of attention heads.

14. Apparatus according to any of the claims 9 to 13, configured for determining the relevance score (222) by backpropagating an output of the machine learning model through the machine learning model to set of tokens, wherein the apparatus is configured for excluding propagation paths, which circumvent the respective attention head.

15. Apparatus (310) for performing a prediction using a machine learning model, wherein the apparatus is configured for predicting an output token byderiving (335), for each of a set of candidate tokens, a respective probability score (337),classifying candidate tokens of at least a subset (342) of the set of candidate tokens using a classifier (370), which distinguishes between a set of classes, the set of classes comprising a first class and a second class;selecting (350) one of the candidate tokens of the subset as the output token based on the probability scores (337); andin selecting (350) one of the candidate tokens of the subset as the output token, weighting the probability scores of candidate tokens assigned to the first class differently than the probability scores of candidate tokens assigned to the second class.

16. Apparatus according to claim 15, configured for setting weights for weighting the probability scores of the candidate tokens assigned to the first class relative to the probability scores of the candidate tokens assigned to the second class.

17. Apparatus according to claim 16, configured for setting the weights in dependence on a user input.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm18. Apparatus according to claim 17, wherein the user input is indicative of a preference, which distinguishes betweenpreferring information included in embedded knowledge of the machine learning model, andpreferring information included in input data provided to the machine learning model for the predictionfor generating a prediction result of the prediction.

19. Apparatus according to any of the claims 15 to 18, wherein the weights are global within each of the classes.

20. Apparatus according to any of the claims 15 to 19, wherein the weights consist of a first weight for weighting the probability scores of the candidate tokens assigned to the first class and a second weight for weighting the probability scores of the candidate tokens assigned to the second class.

21. Apparatus according to any of the claims 15 to 20, wherein a sum of the weights has a predetermined value.

22. Apparatus according to claim 21, configured for, in setting the weights, setting a ratio between a total of weights for weighting the probability scores of the candidate tokens assigned to the first class, and a total of weights for weighting the probability scores of the candidate tokens assigned to the second class.

23. Apparatus according to any of the claims 15 to 22, configured for selecting the subset of candidate tokens out of the set of candidate tokens by selecting a number of candidate tokens having the highest probability scores associated therewith.

24. Apparatus according to any of the claims 15 to 23, wherein a probability score assigned to the first class is contributed by a first set of attention heads and / or wherein a probability score assigned to the second class is contributed by a second set of attention heads.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm25. Apparatus according to any of the claims 15 to 24, configured for classifying each of the candidate tokens of the subset bydetermining, for each of a set of attention heads of the machine learning model, an activation score, which measures a component of an output of the respective attention head during the prediction, which component is associated to the respective candidate token;determining an aggregated activation score by aggregating the activation scores determined for the set of attention heads wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to;classifying the respective candidate token based on the aggregated activation score.

26. Apparatus according to any of the claims 15 to 25, configured for setting the weigths to be zero for the candidate tokens of the first class, or to be zero for the candidate tokens of the second class.

27. Apparatus (410) configured for performing a prediction using a machine learning model (14), the machine learning model comprising a plurality of attention heads (24), wherein the apparatus is configured for, in performing the prediction,inputting input data (428) comprising task information (432) to the machine learning model,measuring an activation (436) of a predetermined attention head of the plurality of attention heads, andchecking, whether the activation (436) of the predetermined attention head fits to a function vector (448), which is associated with the task information, and which represents an expected activation of the predetermined attention head for the task information.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm28. Apparatus according to claim 27, configured for checking, whether the activation (436) of the predetermined attention head fits to a function vector (448), bycalculating a metric, which measures a distance or a deviation between the activation of the predetermined attention head and the function vector.

29. Apparatus (510), configured for inferring a source of an output token (582) resulting from a prediction performed by a machine learning model (14), wherein the apparatus is configured fordetermining (350), for each of a set of attention heads (24) of the machine learning model, an activation score (352), which measures a component of an output (326) of the respective attention head for the prediction of the output token, which component is associated to the output token;determining (360) an aggregated activation score by aggregating the activation scores (352) determined for the set of attention heads, wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to;classifying (370) the output token based on the aggregated activation score (362).

30. Apparatus according to claim 29, configured for deriving the activation score for the respective attention head from an output of the respective attention head using an unembedding matrix of the machine learning model.

31. Apparatus according to claim 29 or 30, configured for, in classifying the output token, using a classifier, which distinguishes between a set of classes, the set of classes comprising a first class and a second class.

32. Apparatus according to claim 31, configured for, if the output token is assigned to the first class, identifying one of input tokens (522) of the prediction as a source of the output token byFH260509PCT-2026164863. DOCX PCT FILING VERSION / bmfor each of a plurality of token positions, determining (590) an attention score (592) by aggregating attention weights of respective attention maps (532) of the set of attention heads, which attention weights define, for the respective attention heads, a contribution of the respective token position to a token position of the output token, wherein, in aggregating the attention weights, each of the attention weights is weighted using the probe weight of the probe, which probe weight is associated with the attention head the respective attention weight belongs to, and weighted using the activation score (352) determined for the attention head the respective attention weight belongs to;identifying (596) the source of the output token based on the attention scores determined for the token positions.

33. Apparatus according to any of the claims 29 to 32, configured for identifying the source of the output token by identifying the token position, for which the attention score is maximum.

34. Apparatus according to any of the claims 29 to 33, configured for, in classifying the output token, comparing the aggregated activation score to a threshold.

35. Apparatus according to any of the claims 29 to 34, wherein the classification distinguishes between the source of the output tokens being one of input tokens of the prediction or the source of the output token being an embedded knowledge of the machine learning model.

36. Apparatus according to any of the claims 29 to 35, wherein the probe is linear, e.g., a linear classifier, or wherein the probe is non-linear, e.g., a non-linear classifier.

37. Apparatus according to claim 36, wherein the probe distinguishes between a first type of attention heads and a second type of attention heads.

38. Apparatus according to any of the claims 29 to 37, wherein the probe comprises a probe weight for each of the attention heads of the set of attention heads.

39. Apparatus according to any of the claims 29 to 38, wherein the set of attention heads is a subset of a plurality of attention heads of the machine learning model.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm40. Apparatus (610) for analyzing a prediction performed by using a machine learning model (14), configured forcomparing (650) an output (636) of a predetermined attention head (24) of the machine learning model during the prediction to each of a plurality of function vectors (648).

41. Apparatus according to claim 40, configured for, in comparing the output of the predetermined attention head (h) of the machine learning model to each of the plurality of function vectors, determining respective distance measures indicating a distance between the output and the respective function vector.

42. Apparatus according to claim 40 or 41, wherein the predetermined attention head is of a predetermined attention head type.

43. Apparatus according to any of the claims 40 to 42, wherein the predetermined attention head is part of a set of attention heads, and wherein the apparatus is configured for comparing, for each of the attention heads of the set of attention heads, an output of the respective attention head during the prediction to each of a plurality of function vectors.

44. Apparatus according to any of the claims 40 to 43, wherein each of the function vectors has a prediction task associated therewith.

45. Apparatus according to any of the claims 40 to 44, configured for using a result of the comparison for selecting one of the function vectors out of the plurality of function vectors, and providing an indication of the function vector, or an indication of a prediction task associated with the function vector.

46. Apparatus (710) for performing a prediction using a machine learning model (14), the machine learning model comprising a plurality of attention heads (24), wherein the apparatus is configured forFH260509PCT-2026164863. DOCX PCT FILING VERSION / bmsetting a plurality of output activations of an output (726) of a predetermined attention head using predetermined values (748), which are independent of an input of the predetermined attention head (24*).

47. Apparatus according to claim 46, configured for selecting the predetermined values based on an input information, wherein the input information differs from a prediction input provided as an input to the machine learning model for performing the prediction.

48. Apparatus according to claim 47, wherein the output of all attention heads, which precede the predetermined attention head in a propagation stream of the machine learning model, is independent of the input information.

49. Apparatus according to any of the claims 46 to 48, wherein the predetermined values represent a function vector, which is associated with a function or a task.

50. Apparatus according to any of the claims 46 to 49, wherein the predetermined values comprise, for each of the plurality of output activations, a respective predetermined value.

51. Apparatus according to any of the claims 46 to 50, wherein the plurality of output activations comprises, or consists of, output activations for the ultimate token position of a sequence of token positions, the output of the predetermined attention head comprising, for each of the sequence of token positions, a respective output activation vector.

52. Apparatus according to any of the claims 46 to 51, comprising an input interface, which is configured for receiving a prompt information and a setting information, wherein the apparatus is configured fordetermining a sequence of input tokens from the prompt information, and providing the sequence of input tokens as an input of the machine learning model, the input being independent of the setting information, andselecting the predetermined values based on the setting information.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm53. Apparatus according to any of the claims 46 to 51, wherein the predetermined attention head is configured for determining, for each token position of a sequence of tokens, an output vector comprising a plurality of output activations, and wherein the apparatus is configured for setting each of the output activations of the output vector for the ultimate token position of the sequence of tokens to a respective predetermined value.

54. Apparatus according to any of the preceding claims, wherein an input for the prediction is provided in form of one or more of an audio signal, a video signal, an image signal, or a communication signal.

55. Apparatus according to any of the preceding claims, wherein an input for the prediction is a multimodal signal.

56. Method, comprising identifying, out of a plurality of attention heads (24) of a machine learning model (14), one or more attention heads of a first type and / or one or more attention heads of a second type byfor each of a first set of predictions (28) and a second set of predictions (38) performed by the machine learning model, determining (20), for each of the plurality of attention heads, for each of the predictions of the first set and the second set, a relevance score (22), which measures a contribution of the respective attention head to a prediction result (16) of the respective prediction;wherein the first set of predictions (28) are performed by requiring the machine learning model to predict the prediction result from embedded knowledge of the machine learning model; andwherein the second set of predictions (38) are performed by providing the machine learning model with input data (36) to perform the respective prediction; and wherein, for each of the predictions of the second set, a prediction result of the respective prediction includes information that originates from the input data (36) provided to perform the respective prediction;FH260509PCT-2026164863. DOCX PCT FILING VERSION / bmassigning (30) one or more of the attention heads to the first type of attention heads and / or assigning one or more of the attention heads to the second type of attention heads by comparing, for each of the attention heads, an average relevance of the respective attention head in the first set of predictions and an average relevance of the respective attention head in the second set of predictions.

57. Method comprising identifying, out of a plurality of attention heads (24) of a machine learning model (14), one or more attention heads of a predetermined type bydetermining (220), for a set of tokens (229) out of a plurality of input tokens (228) provided to the machine learning model (14) as input information for a prediction, for each of the plurality of attention heads (24), a relevance score (222), which represents a contribution of the set of tokens to a prediction result (16) of the prediction, which contribution is provided by the respective attention head; andassigning (230) one or more of the attention heads to the predetermined type based on the relevance scores (222) assigned to the attention heads (24).

58. Method for performing a prediction using a machine learning model, wherein the method comprises predicting an output token byderiving (335), for each of a set of candidate tokens, a respective probability score (337),classifying candidate tokens of at least a subset (342) of the set of candidate tokens using a classifier (370), which distinguishes between a set of classes, the set of classes comprising a first class and a second class;selecting (350) one of the candidate tokens of the subset as the output token based on the probability scores (337); andin selecting (350) one of the candidate tokens of the subset as the output token, weighting the probability scores of candidate tokens assigned to the first class differently than the probability scores of candidate tokens assigned to the second class.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm59. Method comprising performing a prediction using a machine learning model (14), the machine learning model comprising a plurality of attention heads (24), wherein the method comprises, in performing the prediction,inputting input data (428) comprising task information (432) to the machine learning model,measuring an activation (436) of a predetermined attention head of the plurality of attention heads, andchecking, whether the activation (436) of the predetermined attention head fits to a function vector 448, which is associated with the task information, and which represents an expected activation of the predetermined attention head for the task information.

60. Method for inferring a source of a source of an output token (582) resulting from a prediction performed by a machine learning model (14), wherein the method comprises:determining (350), for each of a set of attention heads (24) of the machine learning model, an activation score (352), which measures a component of an output (326) of the respective attention head for the prediction of the output token, which component is associated to the output token;determining (360) an aggregated activation score by aggregating the activation scores (352) determined for the set of attention heads, wherein, in aggregating the activation scores, each of the activation scores is weighted using a probe weight of a probe, which probe weight is associated with the attention head the respective activation score belongs to;classifying (370) the output token based on the aggregated activation score (362).

61. Method for analyzing a prediction performed by using a machine learning model (14), wherein the method comprises:FH260509PCT-2026164863. DOCX PCT FILING VERSION / bmcomparing (650) an output (636) of a predetermined attention head (24) of the machine learning model during the prediction to each of a plurality of function vectors (648).

62. Method for performing a prediction using a machine learning model (14), the machine learning model comprising a plurality of attention heads (24), wherein the method comprises:setting a plurality of output activations of an output (726) of a predetermined attention head using predetermined values (748), which are independent of an input of the predetermined attention head (24*).

63. A computer program for implementing the method of any of claims 56 to 62, when being executed on a computer or signal processor.FH260509PCT-2026164863. DOCX PCT FILING VERSION / bm