Large language model in-context learning defence methods and systems

Abstract

A large language model in-context learning defence method comprises: receiving a poisoned query comprising a query task and poisoned demonstrations, the poisoned demonstrations comprising query task specific examples and corresponding example outputs, at least one of the poisoned demonstrations comprising a query task specific poisoned example input including an attack trigger and a corresponding attack target example output; generating defensive demonstrations each comprising a task specific defensive example input and a corresponding task specific defensive example output; generating a large language model prompt comprising the query task, the poisoned demonstrations, and the defensive demonstrations; inputting the large language model prompt into a large language model; receiving a large language model response; and generating a query response based on the large language model response.

Description

[0001] LARGE LANGUAGE MODEL IN-CONTEXT LEARNING DEFENCE METHODS AND SYSTEMS

[0002] TECHNICAL FIELD

[0003] The present disclosure relates to large language models (LLMs) and in particular to in-context learning (ICL) in LLMs.

[0004] BACKGROUND

[0005] In-context learning (ICL) has proven to be highly effective in large language models (LLMs) due to its adaptability and parameter-free nature, allowing models to perform various tasks with only a few demonstrations. However, this success also introduces vulnerabilities, particularly the risk of backdoor attacks. These attacks enable adversaries to manipulate LLM behaviour by poisoning only a few ICL demonstrations. This poses significant security concerns, as these attacks can affect models without needing access to the training data or model parameters. Despite the recognition of these risks, there is still a lack of deep understanding of the attack mechanisms and defences against such threats.

[0006] SUMMARY

[0007] According to a first aspect of the present disclosure a large language model in-context learning defence method is provided. The method comprises: receiving a poisoned query comprising a query task and poisoned demonstrations, the poisoned demonstrations comprising query task specific examples and corresponding example outputs, at least one of the poisoned demonstrations comprising a query task specific poisoned example input including an attack trigger and a corresponding attack target example output; generating defensive demonstrations each comprising a task specific defensive example input and a corresponding task specific defensive example output; generating a large language model prompt comprising the query task, the poisoned demonstrations, and the defensive demonstrations; inputting the large language model prompt into a large language model; receiving a large language modelresponse; and generating a query response based on the large language model response.

[0008] The present disclosure introduces a dual-learning hypothesis, which suggests that LLMs simultaneously learn both the task-related latent concepts and attack-related latent concepts within poisoned ICL demonstrations. This hypothesis provides a new perspective on the vulnerability of ICL models to backdoor attacks. Moreover, the paper presents a formalization of the attack mechanism through the concept preference ratio, which is a critical factor influencing the attack success By adjusting this ratio, the present disclosure proposes a defence mechanism that dynamically alters the preference between task and attack-related concepts.

[0009] In an embodiment, generating defensive demonstrations comprises selecting the defensive demonstrations from a set of stored clean demonstrations.

[0010] In an embodiment, selecting the defensive demonstrations from the set of stored clean demonstrations comprises calculating a similarity with the poisoned demonstrations and selecting the defensive demonstrations based on the calculated similarity.

[0011] In an embodiment, calculating the similarity with the poisoned demonstrations comprises calculating a similarity between embeddings of examples in the clean demonstrations and an embedding of the poisoned demonstrations, wherein the embeddings are extracted from representation layers of the large language model.

[0012] In an embodiment, the similarity is a cosine similarity.

[0013] In an embodiment, the embeddings of the examples in the clean demonstrations are pre-calculated.

[0014] In an embodiment, selecting the defensive demonstrations from the set of stored clean demonstrations comprises calculating a model confidence for demonstrations of the set of stored clean demonstrations and selecting defensive demonstrations based on the calculated model confidence.In an embodiment, calculating the model confidence for demonstrations of the set of stored clean demonstrations comprises using the large language model conditioned on the poisoned demonstrations to calculate prediction probabilities for the demonstrations of the set of clean demonstrations.

[0015] According to a second aspect of the present disclosure a non-transitory computer readable medium carrying computer executable instructions which when executed on a processor cause the processor to carry out a method configured to carry out a method as set out above is provided.

[0016] According to a third aspect of the present disclosure, a system for large language model in-context learning defence is provided. The system comprises: a processor and a data storage device storing computer program instructions operable to cause the processor to: receive a poisoned query comprising a query task and a poisoned demonstrations, the poisoned demonstrations comprising query task specific examples and corresponding example outputs, at least one of the poisoned demonstrations comprising a task specific poisoned example input including an attack trigger and a corresponding attack target example output; generate defensive demonstrations each comprising a query task specific defensive example input and a corresponding defensive example output; generate a large language model prompt comprising the query task, the poisoned demonstration, and the defensive demonstrations; input the large language model prompt into a large language model; receive a large language model response; and generate a query response based on the large language model response.

[0017] In an embodiment, the system further comprises storage for a set of stored clean demonstrations and wherein the data storage device further stores computer program instructions operable to cause the processor to generate the defensive demonstrations by selecting the defensive demonstrations from the set of stored clean demonstrations.

[0018] In an embodiment, selecting the demonstrations from the set of clean demonstrations comprises calculating a similarity with the poisoned demonstrations and selecting the defensive demonstrations based on the calculated similarity.In an embodiment, calculating the similarity with the poisoned demonstrations comprises calculating a similarity between embeddings of examples in the clean demonstrations and an embedding of the poisoned demonstrations, wherein the embeddings are extracted from representation layers of the large language model.

[0019] In an embodiment, the similarity is a cosine similarity.

[0020] In an embodiment, the system further comprises storage for the embeddings of the examples in the clean demonstrations.

[0021] In an embodiment, the data storage device further stores computer program instructions operable to cause the processor to determine the embeddings of the examples in the clean demonstrations and to store the embeddings in the storage for the embeddings of the examples in the clean demonstrations.

[0022] In an embodiment, selecting the demonstrations from the set of clean demonstrations comprises calculating a model confidence for demonstrations of the set of clean demonstrations and selecting defensive demonstrations based on the calculated model confidence.

[0023] In an embodiment, calculating the model confidence for demonstrations of the set of clean demonstrations comprises using the large language model conditioned on the poisoned demonstrations to calculate prediction probabilities for the demonstrations of the set of clean demonstrations.

[0024] BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In the following, embodiments of the present invention will be described as non-limiting examples with reference to the accompanying drawings in which:

[0026] FIG.1 is a block diagram showing an LLM in-context learning defence system according to an embodiment of the present invention;FIG.2 is a flow chart showing a method of LLM in-context learning defence according to an embodiment of the present invention;

[0027] FIG.3 is a table showing results of different defence methods against ICLAttack in EleutherAI’s GPT models;

[0028] FIG.4 is a table showing results of experiments on closed-source models; and

[0029] FIG.5 is a graph showing the relationship between the number of examples added in demonstration defence and its defensive effectiveness.

[0030] DETAILED DESCRIPTION

[0031] The present disclosure provides a latent concept defence (LCD) mechanism. LCD works by adjusting the likelihood ratio between the task-related latent concepts and the attack-related latent concepts. During the ICL phase, the model is encouraged to select clean demonstrations based on confidence scores and similarity to previously observed tasks, thereby reducing the model’s susceptibility to backdoor manipulation. The attack success is tightly controlled through a concept preference ratio, which represents the model's bias toward either performing the original task or the attack task. The defence method dynamically introduces new, clean demonstrations that reinforce the task-related concepts, which counteracts the poisoned data's influence.

[0032] In-context learning provides demonstration to LLMs as conditions, enabling the model to handle new tasks without adjusting its parameters. Formally, let M be a LLM, the inference process in ICL can be written as

[0033] argmaxPM(y | S,x), (1 )

[0034]

[0035] y

[0036] where S = {x_i, y_i}^n_{i=1} is the demonstration consisting of n examples of task-specific inputs and corresponding outputs y_i. Given a new user input x, the goal of ICL is to generate the correct ground-truth output y_gt.Backdoor attacks on ICL aim to embed backdoors into LLMs via ICL. In ICL backdoor attacks, the attacker chooses a trigger t and a backdoor target yt. To poison the ICL, the attacker injects m poison examples into demonstration S. The poisoned demonstration Stis mathematically expressed as

[0037] S

[0038]

[0039] t = U {Xj,yt}Y=1,(2)

[0040] where x denotes an original input x modified to include the trigger t. The goal of the backdoor attacks is to produce the normal ground-truth output ygtwhen the input without the trigger, yet output the backdoor target ytwhen the input with the trigger. Formally, the attack aims to maximize

[0041] m

[0042]

[0043] ax[PM(y5t| St,x) + PM(ytI St,x)]. (3)

[0044] The newly generated tokens are conditionally independent of previous tokens. A continuous high-dimensional latent concept exists, acting as an approximate sufficient statistic for the posterior information derived from the previous prompt, thereby influencing the probability distribution of the newly generated tokens. In the case of incontext learning, this latent concept represents the task-related information from the provided demonstration.

[0045] LLMs are able to encode task-relevant information from demonstration into continuous high-dimensional latent concept variables 0:

[0046] P

[0047]

[0048] M(y \ S,x) - f0PM(y | H,x)PM(H, S,x)d(). (4)

[0049] For ICL backdoor attacks, due to the significant differences between the objectives from clean and poisoned examples, we infer that the latent concepts from poisoned demonstrations can be considered discrete rather than continuous. Building on this perspective, a dual-learning hypothesis is proposed, stated as follows.

[0050] LLMs can simultaneously learn both a task latent concept 0Tand an attack latent concept 02from the poisoned demonstration:

[0051]

[0052] pM(y lX) ~ PM (y lx> 0i )PM(. Ix) + PfitCy Ix> ®2 ^PM(. ^2 1 St> •*) (5)

[0053] Based on this dual-learning hypothesis, more precise definitions for each component involved are now given.

[0054] PMy I x,0i ) and PM(y I x, 02) are defined as the task conditional distribution and attack conditional distribution, respectively. They represent the output distribution under ideal conditions, one focusing on correctly performing the task (0 and the other carrying out the backdoor attack (02).

[0055] PM(0tI St,x)~ and PM(02I St,x) are also defined as the task posterior distribution and attack posterior distribution, capturing the extent to which the model learns or activates each latent concept from the ICL input (S.tand x).

[0056] For backdoor tasks, the probabilities that the model outputs either the correct task result or the attack target are primarily focussed on.

[0057] the normalized probabilities of PM(ygt| St,x) and PM(ygt| St, x) are utilised as the output probabilities. Specifically, the attack success probability can be defined as:

[0058] 6“ (vI c - _pM(yt\St>x) _

[0059] pM<ytI st,x) -PM(^t|^x)+PM(yt|5#,x) (6)

[0060]

[0061] According to the backdoor attack objective, the attacker wants the model to produce the ground-truth output when relying on the task latent concept and produce the attack target when relying on the attack latent concept. This gives rise to an assumption that when backdoor attacks achieve both high clean accuracy and high attack success and trigger does not affect the prediction of task latent concept. The conditional distribution can be assumed as

[0062] Pn(ygt | X, 0! ) = 1, PM(yt| X,02) = 1. (7)

[0063]

[0064] Under these conditions, the attack success probability can be rewritten as:

[0065] _ % ) _ 1 PiAyt I st,x~) = PM(0I |5t, x)+pM(' 02|5t, -?)P(8)

[0066] M(

[0067]

[0068] 02|‘S't»

[0069] Building on this assumption and leveraging Jensen's inequality, the following result is derived.

[0070] When PM(0i \ St,x^ and PM(02\ St,x^) are independent, the upper bound of the attack success probability is

[0071] f’wCyt I st,x ) < (9)

[0072]

[0073] The upper bound for attack success probability is determined by the concept pM(8il5e )

[0074] preference ratio, where a higher Concept Preference Ratio results in a lowerPM(02I^)

[0075] upper bound for the attack success probability. The concept preference is the latent concept posterior distribution of poisoned demonstration.

[0076] According to the above theorem, increasing the concept preference ratio can lower the upper bound of the attack success probability, thereby achieving a defensive effect.

[0077] In what follows, the factors that influence the concept preference ratio are further analysed. By applying Bayes' theorem, the following lemma is obtained.

[0078] The concept preference has a positive relationship with two components: the model's prior over the latent concept and the likelihood of each example in the demonstration being generated under the latent concept.

[0079] PM(0 15) is determined by the probability of each example in the demonstrations executing the latent concept and the model's marginal distribution over the latent concept:PM(Q i ^) oc pM(e) nti^(yi l O) ( )

[0080] By incorporating the above lemma regarding the concept preference into the concept preference ratio, the following theorem is established.

[0081] The Concept Preference Ratio has a positive relationship between task prior weight, poisoned impact factor, and clean impact factor:

[0082] PlK(91|c>fr )K9], )\ _ / PM{ygt\X> Qj )\ (1 1 ) PM(02I^) fyfef wtlw \pM(yat\x, e2) j '1'

[0083]

[0084] Task Poisoned ' '

[0085]

[0086] The task prior weight determined by the nature of the task and attack scenario,

[0087] PM( 2)

[0088] (P ( v I 9

[0089] , i-.'a1, influenced by the likelihood of poisoned

[0090]

[0091] pM\,yt\x, Q2) /

[0092] fpM(y

[0093] les and the clean impact factor —at\ a px> 9i )\ ex mn

[0094] is dominated by the likelihood

[0095]

[0096] of clean examples.

[0097] The ask prior weightPm^^, it is determined by the task and attack scenario. For an LLM, when the task and the attack's trigger and target are fixed, the task prior weight remains constant. The poisoned impact factor ’ g1> 'sdetermined by the

[0098]

[0099] poisoned examples. Since the exact poisoned examples cannot be known, it is not ( M. Vork possible to modify this factor. However, for the clean impact factor(Q)i y Y\n,

[0100]

[0101] clean demonstrations can be obtained from a database, allowing adjustments to be made in this aspect.

[0102] According to Equation (11), increasing the likelihood ratio

[0103]

[0104] can enhance the influence of the task latent concept on the output distribution while reducing the influence of the attack latent concept, defending against ICL backdoor attack. For the first part of the formulation, it is determined by the task and the attack scenario. Thesecond part is determined by the poisoned demonstrations in the demonstrations set, which cannot be known in the defence process. For the third part, it is determined by the clean demonstrations. Clean demonstrations can be obtained from databases or hand-crafted, so this is the most easily manipulated part.

[0105] In the following description, an ICL backdoor defence is described, that adjusts the concept preference ratio by adding extra clean examples from datasets that either have high confidence in the correct target or are similar to the poisoned demonstration.

[0106] It is observed that three defence processes that can increase the concept preference ratio and thereby reduce the attack success probability:

[0107] The concept preference ratio can be increased by adjusting the task prior weight, poisoned impact factor, and clean impact factor. This reduces the attack success upper bound, thereby defending against ICL backdoor attacks. However, in practical defence, the task prior weight is determined by the task and the attack scenario, while the poisoned impact factor is determined by the poisoned examples, both of which cannot be modified. Therefore, only the clean impact factor can be adjusted, as they can be easily obtained from the dataset.

[0108] Three defence processes can increase the clean impact factor:

[0109] 1. Increasing the number of clear examples

[0110] For clean impact factor, because clean examples are typically easier to obtain from dataset, we can add clean examples to the demonstrations to adjust the clean impact factor.

[0111] According to the assumption for task latent concept and attack latent concept, it canpM(yatlx>

[0112] be inferred that — — g- > l. Therefore, increasing?!, i.e., adding more cleanPM ygt\x> O2)

[0113] examples, can increase the clean impact factor.

[0114] 2. Increasing the similarity of clean examples to the attack trigger.Decreasing

[0115]

[0116] PM(ygt I x,02), i-®-> reducing the probability of the ground-truth output under the attack latent concept, can increase the clean impact factor. When the clean example contain content that is similar to the trigger, the attack latent concept may be activated, leading to a decrease in the probability of predicting the ground-truth. It is essential to include clean examples with higher semantic similarity to the attack trigger. When the clean examples contain content that is similar to the trigger, attack latent concept may be activated, increasing the probability of predicting attack output ytand reducing the probability of predicting task output ygt. Therefore, clean examples with higher semantic similarity to the attack trigger should be included.

[0117] 3. Increasing the probability of LLMs accurately predicting the clean examples.

[0118] Increasing PM(

[0119]

[0120] ygt| x, 0! ), i.e., raising the probability of the ground-truth output under the clean latent concept, can increase the clean impact factor. This indicates that we should select clean examples that have a high probability of accurate output.

[0121] Based on observation 1, adding more clean examples to poisoned demonstrations can reduce the upper bound of attack success probability, thus reduce the attack success rate. Therefore, a defence method against ICL backdoor attacks by combining a defensive demonstration Sdconsisting k clean examples selected from dataset T> - xi,yi)=1with the poisoned demonstration Stis proposed. To make the defensive demonstration more effective, following observation 2 and observation 3, similarity selection and confidence selection are proposed, k / 2 clean examples are selected through similarity selection and confidence, respectively, and concatenated to form the final defensive demonstration:

[0122] Sd= Sd* + Sdc. (12)

[0123] Based on the observation 2, clean examples with high semantic similarity to the attack trigger should be selected. However, in practical backdoor defence scenarios, the attack trigger is not known. To address this limitation, it is assumed that poisoned demonstrations also contain significant semantic information about the trigger. Thus,the selection criterion is extended to the semantic similarity between examples and poisoned demonstration. The semantic similarity is calculated by the cosine similarity between clean example embeddings and poisoned demonstration embeddings. The similarity defensive demonstration is mathematically expressed as:

[0124] = argtop^ / 2y )£l,cos(e(%i);e(5t)). (13)

[0125]

[0126] where e(-) represents the embedding of LLMs.

[0127] According to the observation 3, clean examples with highly accurate prediction probabilities under the task latent concept can be utilized for defence. Based on the goal of a backdoor attack, the poisoned demonstration can achieve high clean accuracy when the input is without triggers. Thereby, the poisoned demonstration is utilised instead of the task latent concept and the confidence defensive demonstration can be represented as:

[0128] S

[0129]

[0130] dc= | Xi, St). (14)

[0131] By combining similarity selection and confidence selection, the above approach effectively reduces the attack success probability by introducing semantically relevant and high-confidence clean examples. The method ensures a balanced trade-off between task relevance and defence robustness.

[0132] Similarity Selection and Confidence Selection are ideal methods for selecting defensive demonstrations. However, in practical backdoor defence scenarios, two issues are encountered: 1 ) The trigger is not known during the defence process, thus the similarity between examples and the trigger cannot be directly calculated in Similarity Selection; and 2) a clean demonstration cannot be obtained during the defence process, making it impossible to replace the task latent concept in Confidence Selection.

[0133] Therefore, a method is proposed which extends Similarity Selection and Confidence Selection to practical defence scenarios. On one hand, for Similarity Selection, it isassumed that poisoned demonstrations also contain significant semantic information about the trigger. Thus, the selection criterion is extended to the semantic similarity between examples and poisoned demonstrations. On the other hand, for Confidence Selection, based on the backdoor attack goal it is assumed that poisoned demonstrations can still represent the task latent concept when the input does not contain the trigger. Finally, the defensive demonstrations selected by these two methods for defence are combined. The defence demonstration can be represented as:

[0134] Sd= arg t

[0135]

[0136] op^..y.)EBcos(e(xi), e(5.t)) + argto^x. y. K1)PMQyi| (15)

[0137] FIG.1 is a block diagram showing an LLM in-context learning defence system according to an embodiment of the present invention. The LLM in-context learning defence system 100 is a computer system with memory that stores computer program modules which implement LLM in-context learning defence methods according to embodiments of the present invention.

[0138] As shown in FIG.1, the LLM in-context learning defence system 100 is coupled to an LLM server 140. The LLM server 140 implements a large language model 142. In some embodiments, the large language model 142 may be executed locally within the LLM in-context learning defence system 100 or remotely on a separate computing server. The presence of the LLM server 140 is implementation-dependent.

[0139] The LLM in-context learning defence system 100 comprises a processor 110, a working memory 112, an input interface 114, an output interface 116, program storage 120, and data storage 130. The processor 110 may be implemented as one or more central processing unit (CPU) chips. The program storage 120 is a non-volatile storage device such as a hard disk drive which stores computer program modules. The computer program modules are loaded into the working memory 112 for execution by the processor 110. The input interface 114 is an interface which allows data to be received by the LLM in-context learning defence system 100, for example LLM queries to be processed by the LLM in-context learning defence system 100. The inputinterface 114 may be a wireless network interface such as a Wi-Fi or Bluetooth interface, alternatively it may be a wired interface. The output interface 116 is an interface which allows the LLM in-context learning defence system 100 to output such as query responses. The output interface 116 may also allow the LLM in-context learning defence system 100 to send prompts to the LLM server 140 and the input interface 114 may allow the LLM in-context learning defence system 100 to receive prompt responses from the LLM server 140.

[0140] The program storage 120 stores a defensive demonstration generation module 122, an LLM interface module 124, and a prompt generation module 126. The computer program modules cause the processor 110 to execute various LLM in-context learning defence processing which is described in more detail below. The program storage 120 may be referred to in some contexts as computer readable storage media and / or non-transitory computer readable media. As depicted in FIG.1, the computer program modules are distinct modules which perform respective functions implemented by the LLM in-context learning defence system 100. It will be appreciated that the boundaries between these modules are exemplary only, and that alternative embodiments may merge modules or impose an alternative decomposition of functionality of modules. For example, the modules discussed herein may be decomposed into sub-modules to be executed as multiple computer processes, and, optionally, on multiple computers. Moreover, alternative embodiments may combine multiple instances of a particular module or sub-module. It will also be appreciated that, while a software implementation of the computer program modules is described herein, these may alternatively be implemented as one or more hardware modules (such as field-programmable gate array(s) or application-specific integrated circuit(s)) comprising circuitry which implements equivalent functionality to that implemented in software.

[0141] Although the LLM in-context learning defence system 100 is described with reference to a computer, it should be appreciated that the LLM in-context learning defence system 100 may be formed by two or more computers in communication with each other that collaborate to perform a task. For example, but not by way of limitation, an application may be partitioned in such a way as to permit concurrent and / or parallel processing of the instructions of the application. Alternatively, the data processed by the application may be partitioned in such a way as to permit concurrent and / or parallelprocessing of different portions of a data set by the two or more computers. In an embodiment, virtualization software may be employed by the LLM in-context learning defence system 100 to provide the functionality of a number of servers that is not directly bound to the number of computers in the LLM in-context learning defence system 100. In an embodiment, the functionality disclosed above may be provided by executing the application and / or applications in a cloud computing environment. Cloud computing may comprise providing computing services via a network connection using dynamically scalable computing resources. A cloud computing environment may be established by an enterprise and / or may be hired on an as-needed basis from a third-party provider.

[0142] The data storage 130 stores clean demonstrations 132 and LLM embeddings for the clean demonstrations 134. The clean demonstrations 132 may be sourced from existing relevant task datasets that have not been attacked, or from manually curated examples. The LLM embeddings for the clean demonstrations may be pre-calculated by the LLM in-context learning defence system 100 or may be calculated during execution of LLM in-context learning defence methods.

[0143] FIG.2 is a flow chart showing a method of LLM in-context learning defence according to an embodiment of the present invention. The method 200 shown in FIG.2 is carried out by the LLM in-context learning defence system 100 shown in FIG.1.

[0144] In step 202, the LLM in-context learning defence system 100 receives an in-context learning query. The poisoned query comprises a query task and poisoned demonstrations. The poisoned demonstrations comprise query task specific examples and corresponding example outputs. The poisoned demonstrations include one or more query task specific poisoned example and a corresponding output. The query task specific poisoned example includes an attack trigger which generates a corresponding attack target output. In addition to the poisoned example(s), the poisoned demonstrations may include one or more clean examples and corresponding clean outputs.

[0145] In step 204, the processor 110 of the LLM in-context learning defence system 100 executes the defensive demonstration generation module 122 to generate defensivedemonstrations. The defensive demonstrations each comprise a task specific defensive example input and a corresponding task specific defensive example output. The defensive demonstrations may be selected from the clean demonstrations 132 stored in the data storage 130. As described above, the selection may be based on similarity selection or confidence selection. The defensive demonstrations may include demonstrations selected based on similarity selection and demonstrations selected based on confidence selection. The selection based on similarity selection may make use of the LLM embeddings for clean demonstrations 134 stored in the data storage 130. In some embodiments, the LLM embeddings for clean demonstrations 134 may be pre-calculated using embedding layers of the large language model 142 implemented on the LLM server 140.

[0146] In step 206, the processor 110 of the LLM in-context learning defence system 100 executes the LLM interface module 124 to generate an LLM prompt which comprises the query task, the poisoned demonstrations and the defensive demonstrations.

[0147] In step 208, the LLM in-context learning defence system 100 inputs the LLM prompt into the large language model 142 implemented on the LLM server 140.

[0148] The large language model 142 then executes the LLM prompt and generates an LLM response. In step 210, the LLM in-context learning defence system 100 receives the LLM response.

[0149] In step 212, the LLM in-context learning defence system 100 generates a query response using the LLM response.

[0150] Following step 212, the in-context learning query response is provided to a user that submitted the in-context learning query.

[0151] In the following, applications of the LLM in-context learning defence method described above to tasks and defence against backdoor tasks are described. Experiments were conducted on classification and generative tasks. For the classification task, the attack target is to misclassify the text into a specific category. For the generative task, the attack target is sentiment steering and targeted refusal proposed in BackdoorLLM (Li,Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. BackdoorLLM: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint, 2024). In particular, sentiment steering manipulates LLMs to generate negative sentiment, while targeted refusal forces the LLM to generate a refusal response (e.g., “I am sorry...”). In this disclosure, ICLAttack is selected as the ICL backdoor attack method, and a backdoor attack is conducted by adding attack triggers to some examples with the attack target following Zhao, S., Jia, M., Tuan, L. A., Pan, F., and Wen, J. Universal vulnerabilities in large language models: Backdoor attacks for in-context learning. arXiv preprint, 2024.

[0152] Since ICL backdoor attacks do not modify the training data or model parameters, they can only be defended against inference-time defences. The proposed defence is compared with two commonly used inference time backdoor defence methods, ONION (Qi, F., Chen, Y., Li, M., Yao, Y., Liu, Z., and Sun, M. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint, 2020.) and Back-Translation (Qi, F., Li, M., Chen, Y., Zhang, Z., Liu, Z., Wang, Y., and Sun, M. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint, 2021).

[0153] Following BackdoorLLM benchmark (Li, Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. BackdoorLLM: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint, 2024), for classification tasks, we utilize SST-2 (Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, 2013) and AG’s News (Zhang, X., Zhao, J., and LeCun, Y. Character-level convolutional networks for text classification. In Advances in neural information processing systems, 2015.); for generative tasks, we use instruction datasets including Standford Alpaca (Taori, R., Gulrajani, I., Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An instruction-following llama model, 2023) and AdvBench (Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredriksen, M. Universal and transferable adversarial attacks on aligned language models. arXiv preprint, 2023).In the experiments, ICL backdoor attacks and defences for a range of open-sourced LLMs, including EleutherAI’s GPT models (GPT-NEO-1.3B, GPT-NEO-2.7B (Black, S., Leo, G., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021), and GPT-J-6B (Wang, B. and Komatsuzaki, A. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, https: / / github.com / kingoflolz / mesh-transformer-jax, May 2021). In addition, two closed-source black-box models (GPT-3.5 (Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022) and GPT-4o (Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint, 2023)) were also evaluated.

[0154] FIG.3 is a table showing results of different defence methods against ICLAttack in EleutherAI’s GPT models.

[0155] For the misclassification target, following the setting in Zhao, S., Jia, M., Tuan, L. A., Pan, F., and Wen, J. Universal vulnerabilities in large language models: Backdoor attacks for in-context learning. arXiv preprint, 2024, Clean Accuracy (CA) and Attack Success Rate (ASR) are utilised to evaluate defence methods. CA refers to the classification accuracy of the model on clean inputs, while ASR calculates the percentage of non-target label test samples with triggers that are predicted as the target label. As the setting in Li, Y., Huang, H., Zhao, Y., Ma, X., and Sun, J. BackdoorLLM: A comprehensive benchmark for backdoor attacks on large language models. arXiv preprint, 2024, the ASR in generative tasks represents the percentage of LLM’s responses that contain the attack target. The ASR is evaluated with the trigger (ASRw / t) and without the trigger (ASRw / o). For ASR, the lower values indicate the better defence ( 1 ); for CA, the higher values indicate the better original task performance preservation ( T ).

[0156] Similarity selection and confidence selection require information that is unavailable in practical defence scenarios. Specifically, similarity selection relies on attack triggers,while confidence selection requires clean demonstrations. Similarity selection and confidence selection are extended to practical defence scenarios the methods of the present disclosure are indicated as ICLShield. As the result shown in FIG.3, selecting examples with the highest semantic similarity to the attack trigger or the highest accurate prediction probability conditioned on clean demonstrations for defence can achieve outstanding defensive performance.

[0157] This further validates the effectiveness of the conclusion that increasing PM^ygt I x, 0i) and reducing PMygt I x,0x) can lowers the attack success bound, thereby defending against ICL backdoor attack. Furthermore, by comparing ICLShield with the ideal similarity selection and confidence selection, we find that despite differences between the practical selection criteria and theoretical results, ICLShield still achieves excellent defensive performance. This demonstrates the effectiveness of ICLShield in practical defence scenarios.

[0158] As shown in FIG.3, ONION and Back-Translation fail to achieve effective defence performance. For ONION, this may be because ICLAttack does not utilize special characters as attack trigger. Since the attack trigger is a word or phrase, it may not significantly affect the sentence’s perplexity, making it difficult for ONION to identify and remove the trigger in token level. For Back-Translation, although translation between languages changes vocabulary or word order to eliminate the trigger in sentence-level, it does not alter the semantic information. However, the attack latent concept learned from poisoned demonstrations may generalize to similar semantics of the attack trigger. Therefore, Back-Translation cannot effectively defend against ICLAttack. In contrast, the method of the present disclosure does not rely on eliminating the attack trigger but instead defends by reducing the attack success bound. As the result, the method of the present disclosure achieves significantly better defensive performance compared to ONION and Back-Translation.

[0159] FIG.4 is a table showing results of experiments on closed-source models. GPT-3.5 and GPT-4o models were accessed via commercial APIs and experiments were conduct on the SST-2 dataset with the misclassification target. Note that, for these closed-source models, the output probabilities cannot be accessed. Therefore, thedemonstrations selected on open-source models are transferred to closed-source models for defence. Due to the excellent alignment capabilities of GPT-3.5 and GPT-4o, the attack effectiveness of ICLAttack is limited. However, the demonstration defence method still achieves better defence performance than ONION and Back-Translation, reducing the ASR to less than half of its original value.

[0160] FIG.5 is a graph showing the relationship between the number of examples added in demonstration defence and its defensive effectiveness.

[0161] It is observed that as the number of examples increases, the ASR exhibits a downward trend. This indicates that as more defensive examples are added to the demonstrations, the output distribution of LLMs gradually shifts toward the clean distribution, leading to improved defensive effectiveness. Notably, the ASR drops sharply when the sixth defence example is added. This may be because, as the model’s distribution gradually shifts, the probability of predicting the ground truth label for most test data exceeds the probability of predicting the target label, thereby successfully achieving defence.

[0162] It can be concluded that randomly selected examples can also be used for defence, which aligns with the observation in motivation 1. However, for the same number of defensive demonstrations, the defence results of random selection are not as effective as those of confidence selection or similarity selection. This indicates that using examples more aligned with motivation 2 or motivation 3 enables more efficient defence, achieving excellent results with fewer additional examples.

[0163] Whilst the foregoing description has described exemplary embodiments, it will be understood by those skilled in the art that many variations of the embodiments can be made within the scope and spirit of the present invention.

Claims

CLAIMS1. A large language model in-context learning defence method comprising:receiving a poisoned query comprising a query task and poisoned demonstrations, the poisoned demonstrations comprising query task specific examples and corresponding example outputs, at least one of the poisoned demonstrations comprising a query task specific poisoned example input including an attack trigger and a corresponding attack target example output;generating defensive demonstrations each comprising a task specific defensive example input and a corresponding task specific defensive example output;generating a large language model prompt comprising the query task, the poisoned demonstrations, and the defensive demonstrations;inputting the large language model prompt into a large language model; receiving a large language model response; andgenerating a query response based on the large language model response.

2. The method according to claim 1, wherein generating defensive demonstrations comprises selecting the defensive demonstrations from a set of stored clean demonstrations.

3. The method according to claim 2, wherein selecting the defensive demonstrations from the set of stored clean demonstrations comprises calculating a similarity with the poisoned demonstrations and selecting the defensive demonstrations based on the calculated similarity.

4. The method according to claim 3, wherein calculating the similarity with the poisoned demonstrations comprises calculating a similarity between embeddings of examples in the clean demonstrations and an embedding of the poisoned demonstrations, wherein the embeddings are extracted from representation layers of the large language model.

5. The method according to claim 4, wherein the similarity is a cosine similarity.

6. The method according to claim 4 or claim 5, wherein the embeddings of the examples in the clean demonstrations are pre-calculated.

7. The method according to any one of claims 2 to 6, wherein selecting the defensive demonstrations from the set of stored clean demonstrations comprises calculating a model confidence for demonstrations of the set of stored clean demonstrations and selecting defensive demonstrations based on the calculated model confidence.

8. The method according to claim 7, wherein calculating the model confidence for demonstrations of the set of stored clean demonstrations comprises using the large language model conditioned on the poisoned demonstrations to calculate prediction probabilities for the demonstrations of the set of clean demonstrations.

9. A computer readable medium storing processor executable instructions which when executed on a processor cause the processor to carry out a method according to any one of claims 1 to 8.

10. A system for large language model in-context learning defence, the system comprising: a processor and a data storage device storing computer program instructions operable to cause the processor to:receive a poisoned query comprising a query task and poisoned demonstrations, the poisoned demonstrations comprising query task specific examples and corresponding example outputs, at least one of the poisoned demonstrations comprising a task specific poisoned example input including an attack trigger and a corresponding attack target example output;generate defensive demonstrations each comprising a query task specific defensive example input and a corresponding defensive example output;generate a large language model prompt comprising the query task, the poisoned demonstration, and the defensive demonstrations;input the large language model prompt into a large language model; receive a large language model response; andgenerate a query response based on the large language model response.

11. The system according to claim 10, further comprising storage for a set of stored clean demonstrations and wherein the data storage device further stores computer program instructions operable to cause the processor to generate the defensive demonstrations by selecting the defensive demonstrations from the set of stored clean demonstrations12. The system according to claim 11, wherein selecting the demonstrations from the set of clean demonstrations comprises calculating a similarity with the poisoned demonstrations and selecting the defensive demonstrations based on the calculated similarity.

13. The system according to claim 12, wherein calculating the similarity with the poisoned demonstrations comprises calculating a similarity between embeddings of examples in the clean demonstrations and embeddings of the poisoned demonstrations, wherein the embeddings are extracted from representation layers of the large language model.

14. The system according to claim 13, wherein the similarity is a cosine similarity.

15. The system according to claim 13 or 14, further comprising storage for the embeddings of the examples in the clean demonstrations.

16. The system according to claim 15, wherein the data storage device further stores computer program instructions operable to cause the processor to determine the embeddings of the examples in the clean demonstrations and to store the embeddings in the storage for the embeddings of the examples in the clean demonstration.

17. The system according to any one of claims 11 to 16, wherein selecting the demonstrations from the set of clean demonstrations comprises calculating a model confidence for demonstrations of the set of clean demonstrations and selecting defensive demonstrations based on the calculated model confidence.

18. The system according to claim 17, wherein calculating the model confidence for demonstrations of the set of clean demonstrations comprises using the large language model conditioned on the poisoned demonstrations to calculate prediction probabilities for the demonstrations of the set of clean demonstrations.

Images