Jailbreak attack-based large model test method and system

Abstract

The present invention belongs to the technical field of security of large language models. Disclosed are a jailbreak attack-based large model test method and system. The method comprises: constructing an example pool and an example selection model; training in the example pool the example selection model; using the trained example selection model to select an example from the example pool for an input instruction; and, on the basis of the input instruction and the selected example, attacking a target large model to obtain a jailbreak attack test result of the target large model. By enhancing the effectiveness and efficiency of jailbreak attacks, the present invention more effectively tests large models.

Description

A Large-Scale Model Detection Method and System Based on Jailbreak Attacks Technical Field

[0001] This invention belongs to the field of large language model security technology, and in particular relates to a method and system for detecting large models based on jailbreak attacks. Background Technology

[0002] Large language models are increasingly used due to their advanced capabilities, but they also bring significant security risks, including bias, discrimination, illegal activities, and ethical issues. Security tuning, particularly through Human Feedback Reinforcement Learning (RLHF), offers a viable approach to enhancing the security of these models. However, RLHF still faces challenges such as target management errors and inaccurate generalization, which can lead to various security vulnerabilities. Jailbreak attacks utilize role-playing, intent obfuscation, and other methods to specialize the input of designed prompts with target questions into large models, inducing them to generate sensitive content. In information security, a red team is a team simulating real attackers whose task is to test the security of target systems through various attack methods to discover potential vulnerabilities and weaknesses. Currently, jailbreak attacks have become a primary method for red teams to identify and exploit the vulnerabilities and weaknesses of these large language models.

[0003] Researchers discovered significant differences in Attack Success Rate (ASR) across different example selection strategies in context learning, as shown in Figure 1. To verify the impact of different example combinations on contextual jailbreak performance, a set of validation experiments was conducted. The experiments randomly selected ten test data points from the AdvBench dataset and simultaneously selected eight from 128 candidate examples to verify their effectiveness and average time when attacking Llama2. Example selection strategies included Random, Diversity, Similarity, and Greedy Search. Two findings emerged from the experiments shown in Figure 1:

[0004] Finding 1: There exists an optimal combination of examples of a certain length that achieves a high attack success rate (ASR). Randomly selected examples failed to break through any of the 10 questions. In contrast, the greedy search method achieved a 20% attack success rate in 10 iterations, while the relevance and diversity strategies each achieved a 10% attack success rate. Therefore, for the context learning capabilities of large models, there exists an optimal combination of examples that can provide explicit answers to harmful questions and generate harmful content.

[0005] Finding 2: The heuristic greedy search method has several drawbacks, including slow search speed, high computational cost, and difficulty in scaling applications. As shown in Figure 1, although heuristic greedy search achieves a higher attack success rate, it takes approximately twice as long as other methods to find the optimal combination of examples within 10 iterations. If more iterations are performed to achieve an even better attack success rate, the time cost will increase further, making large-scale, automated security vulnerability and risk discovery challenging using context learning.

[0006] In summary, while leveraging the contextual learning (ICL) capabilities of large language models offers a more feasible and efficient approach to jailbreak hint design, significant challenges remain in terms of convergence speed, computational cost, and scalability. Developing an optimized few-shot selection and ranking method that ensures high attack effectiveness and efficient automatic generation of optimal results is crucial for enhancing the performance of contextual learning in jailbreak attack scenarios. This approach is also important for identifying novel paradigmatic security vulnerabilities in contextual learning. Addressing this pressing and challenging problem is vital for the advancement of this field. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides a method and system for detecting large models based on jailbreak attacks, which enhances the effectiveness and efficiency of jailbreak attacks and enables more effective detection of large models.

[0008] To achieve the above objectives, the technical solution of the present invention includes the following:

[0009] A method for detecting large models based on jailbreak attacks, the method comprising:

[0010] Construct an example pool and an example selection model; wherein each example is a data pair of harmful instructions and responses, and the example selection model is constructed using a transformer-based neural network model, eliminating the LM-HEAD layer of the neural network model and introducing two linear layers and a Select-Head layer after the last transformer MLP layer;

[0011] The example selection model is trained on the example pool;

[0012] Use the trained example selection model as the input instruction to select examples from the example pool;

[0013] The target large model is attacked based on the input instructions and the selected example to obtain the jailbreak attack detection results of the target large model.

[0014] Further, training the example selection model on the example pool includes:

[0015] Obtain the input text x, initialize the strategy π and experience cache of the example selection model. And setting the exploration probability ε and batch size in the greedy strategy.

[0016] Determine if the random number generated by the random function rand() is less than the exploration probability ε;

[0017] If the random number is less than the exploration probability ε, the Action() mapping function is used to map the random number to action a in the action space. If the random number is not less than the exploration probability ε, action a is selected according to the current strategy π.

[0018] Perform the chosen action a to obtain a new state x′ and a reward r;

[0019] Store the experience (x, a, r, x′) in the experience cache. middle

[0020] If experience cache If the size reaches or exceeds the minimum batch size, then the empirical cache will be used. Medium sampling Examples are provided, and a reward is calculated for each transformation in the minimum batch. i ;

[0021] Based on reward i Update the parameters of the selected example model.

[0022] Further, the reward is calculated. i The process includes:

[0023] Calculate the cross-entropy loss Loss for each transition in the minimum batch;

[0024] Calculate the entropy H(π) of strategy π;

[0025] The reward is obtained based on the cross-entropy loss Loss and the entropy H(π) of the policy π. i .

[0026] Furthermore, the cross-entropy loss in, Indicates that given input x i and a series of known input-output pairs {(x1,y1),...,(x k ,y k In the case of )}, the example selects the model output. The probability of L is given by L, where L represents the total number of output categories or the sequence length.

[0027] Furthermore, the entropy of the policy π is H(π) = -∑x∈Xπ(a|x)logπ(a|x); where π(a|x) is the probability of choosing action a given input x under policy π, and X represents the input set.

[0028] A large-scale model detection system based on jailbreak attacks, the system comprising:

[0029] A training module is used to construct an example pool and an example selection model; wherein each example is a data pair of harmful instructions and responses, and the example selection model is constructed using a transformer-based neural network model, eliminating the LM-HEAD layer of the neural network model and introducing two linear layers and a Select-Head layer after the last transformer MLP layer; the example selection model is trained on the example pool;

[0030] The detection module is used to select examples from the example pool using a trained example selection model as input instructions; and to attack the target large model based on the input instructions and the selected examples to obtain jailbreak attack detection results for the target large model.

[0031] Further, training the example selection model on the example pool includes:

[0032] Obtain the input text x, initialize the strategy π and experience cache of the example selection model. And setting the exploration probability ε and batch size in the greedy strategy.

[0033] Determine if the random number generated by the random function rand() is less than the exploration probability ε;

[0034] If the random number is less than the exploration probability ε, the Action() mapping function is used to map the random number to action a in the action space. If the random number is not less than the exploration probability ε, action a is selected according to the current strategy π.

[0035] Perform the chosen action a to obtain a new state x′ and a reward r;

[0036] Store the experience (x, a, r, x′) in the experience cache. middle

[0037] If experience cache If the size reaches or exceeds the minimum batch size, then the empirical cache will be used. Medium sampling Examples are provided, and a reward is calculated for each transformation in the minimum batch. i ;

[0038] Based on reward i Update the parameters of the selected example model.

[0039] Further, the reward is calculated. i The process includes:

[0040] Calculate the cross-entropy loss Loss for each transition in the minimum batch;

[0041] Calculate the entropy H(π) of strategy π;

[0042] The reward is obtained based on the cross-entropy loss Loss and the entropy H(π) of the policy π. i .

[0043] An electronic device, characterized in that the electronic device comprises: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the large model detection method based on jailbreak attacks described above.

[0044] A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the large model detection method based on jailbreak attacks described above.

[0045] Compared with the prior art, the present invention has at least the following beneficial effects.

[0046] To address the limitations of existing methods, this invention introduces a reinforcement learning method to optimize the selection of a small number of instances, thereby improving the effectiveness, efficiency, and scalability of jailbreak LLMs.

[0047] This invention treats instance selection as an optimization problem, aiming to maximize the probability of the target model generating a harmful response. It modifies the GPT-2 model by adding a new layer for instance selection and trains it using a reinforcement learning strategy based on a cross-entropy loss reward function. To accelerate model convergence, an empirical replay mechanism is employed, where high-impact attack samples are selected from a buffer with a certain probability during training. To prevent the model from getting stuck in local optima during instance selection, an entropy penalty and random sampling strategy are designed to diversify the model's sampling range, thereby ensuring the convergence and high attack capability of the instance selection model.

[0048] The jailbreak attack method proposed in this invention has achieved excellent attack effects and performance through comparative testing on various large language models, demonstrating a powerful jailbreak attack capability against large language models. Attached Figure Description

[0049] Figure 1 is an example strategy analysis diagram.

[0050] Figure 2 is a flowchart of a large model detection method based on jailbreak attacks. Detailed Implementation

[0051] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0052] The present invention provides an efficient few-shot LLM jailbreak attack method based on reinforcement learning. This method extends the GPT-2 architecture by adding an example selection module and employs strategies such as experience replay and entropy penalty to accelerate convergence and avoid local optima.

[0053] Specifically, this invention formalizes the example selection framework as an optimization problem, the objective of which is to maximize the probability that the target model will generate a harmful response. The problem is defined as follows:

[0054] Context-learning jailbreak example selection is defined as an optimal solution problem. Specifically, it involves solving a function f that automatically selects examples from a pool D for any given malicious instruction x. i Select k optimal example combinations {(x1,y1),...,(x k ,y k As shown in formula (1).

[0055] Then, select the k best examples and the harmful instructions x i The contextual hints are constructed and input into the target model M to maximize the harmful response y generated by model M. i The probability of jailbreaking is determined by the probability of the jailbreak. The specific formal representation is shown in formula (2).

[0056] Figure 2 is a flowchart of an efficient few-shot LLM jailbreak attack method driven by reinforcement learning, which includes two phases: training and detection. The training phase mainly consists of four components: constructing the example pool, designing the example selection model, formulating the reward function, and implementing reinforcement training. The detection phase focuses on constructing the optimal example combination using the trained example selection model. This stimulates the context learning ability of large models, thereby inducing the generation of harmful risks in these models.

[0057] I. Training Phase.

[0058] 1. Example pool construction.

[0059] Based on a large model with weak security alignment that generates malicious responses, this invention constructs a malicious instruction and response {(x1,y1),...,(x k ,yk )} dataset, numbered from 1 to k (0 < k ≤ pool size). This supports the example selection model to select optimized context examples by number. In this paper, a few-shot method is used to construct the example pool D as experimental data.

[0060] 2. Design of the example selection model.

[0061] In this paper, a large model is adopted as the example selection model, denoted as E LLM , which is a Transformer-based neural network model with powerful language feature analysis and understanding capabilities. E LLM can effectively explore the relationship between context examples and harmful instructions, making it a suitable choice for the example selection model. To make the E LLM model adapt to the task of selecting examples, the present invention makes corresponding adjustments to its structure. The present invention eliminates the original LM-HEAD layer of E LLM ] and introduces two linear layers and a Select-Head layer after the MLP layer of the last Transformer layer. This enables the present invention to convert the output of E LLM into the probability of selecting examples, as shown in Figure 2. In this modified architecture, E is mainly responsible for extracting semantic features from the input, while the linear layers select the most appropriate examples based on these features. Since E LLM already has excellent feature extraction capabilities, only training the linear layers can achieve the desired effect, thereby reducing the overall training cost. The example selection model will generate k digital strings according to the input harmful instruction x LLM , where each character N represents the Nth example in the example pool D. The specific formal representation is shown in formula (3): Selection Model(x i ) = (N1,N2,N3,...,N i ) (3) k ) (3)

[0062] Then, according to the characters, K corresponding example pairs are selected from the example pool, as shown in formula (4). The selected example pairs are combined with the harmful instruction to construct a prompt input to the target model to satisfy formula (2) and maximize the probability that the target model outputs the harmful response A. D(N1,…,N k )={(x1,y1),…,(x k ,y k )} (4)

[0063] 3. Formulation of the reward function.

[0064] According to formula (2), the goal of optimizing the sample combination of the selected models is to maximize the probability that the target model outputs a harmful response A. This paper uses the probability of the target model generating a deterministic response (Sure, Step) to represent the probability of a harmful response, and uses cross-entropy loss as one of the rewards. Meanwhile, considering the specificity of the downstream task of this invention, this paper designs a reward function consisting of two parts: cross-entropy loss with the expected output and entropy penalty. The cross-entropy loss provides the attack model with a clear objective and guides the convergence direction, while the entropy penalty helps the attack model escape from local optima and converge to the optimal result. It can be formally expressed as: H(π)=-∑x∈Xπ(a|x)logπ(a|x) (6) Reward i =-Loss+β·H(π) (7)

[0065] The specific meaning of each symbol here is as follows:

[0066] Loss: This stands for cross-entropy loss function, used to quantify the difference between model predictions and actual results.

[0067] This is the logarithm of the conditional probability, representing the probability given input x. i and a series of known input-output pairs {(x1,y1),...,(x k ,y k In the case of )}, the model output The probability of L. Here, L represents the total number of output categories or the length of the sequence considered.

[0068] H(π): This represents the entropy of policy π, used to measure the randomness or uncertainty of the policy.

[0069] π(a|x): This is the probability of choosing action a given input x under policy π.

[0070] X: Represents the set of all possible inputs.

[0071] Reward: This is the total reward calculated, combining cross-entropy loss and entropy penalty.

[0072] -Loss: The negative sign indicates a desire to minimize the loss.

[0073] β·H(π): This is the entropy penalty term, multiplied by the coefficient β. The coefficient β controls the weight of the entropy penalty in the total reward, balancing exploration and exploitation, and encouraging the policy to explore more possible actions.

[0074] 4. Implementation of intensive training.

[0075] The problem is defined as a search optimization problem in reinforcement learning, using the classic policy gradient (PG) algorithm to maximize the probability of the optimal choice example, as in Algorithm 1. Despite employing these advanced training techniques, this invention notes that the algorithm may stagnate at local optima during execution. To address this, this invention introduces a dynamic combination of stochastic and deterministic policies with an initial probability of γ. This policy involves the model randomly selecting actions in the initial phase, with the probability gradually decreasing over time, eventually stabilizing on the most strategic action choice. This approach helps avoid premature convergence to local maxima, thus preserving the potential to discover better policies. Furthermore, recognizing the ephemeral nature of key training samples in traditional reinforcement learning models, this invention employs an optimization mechanism involving the replay of strategic experiences. This involves retaining key training samples in a dedicated buffer and periodically reusing these key samples, enhancing the model's robustness and convergence speed. This systematic reuse of key experiences prevents their premature obsolescence and significantly improves the effectiveness of the training process.

[0076] Based on the above algorithm 1, the reinforcement training process of the present invention includes the following steps 4.1 to 4.2.

[0077] Step 4.1: Obtain the input text x, and initialize the policy π and experience cache of the GPT-2 model. And setting the exploration probability ε and batch size in the greedy strategy.

[0078] Step 4.2: For each training round, perform the operations from Step 4.2.1 to Step 4.2.5.

[0079] Step 4.2.1: Select an action.

[0080] If the random number generated by the random function satisfies the constraint rand()<ε, then the Action() mapping function is used to map the random number generated by the random function rand() to action a in the action space; otherwise, action a is selected according to the current policy π(·|x).

[0081] Step 4.2.2: Perform the action and observe.

[0082] Perform the chosen action a to obtain a new state x′ and a reward r.

[0083] Step 4.2.3: Store the experience (x, a, r, x′) in the experience cache. middle.

[0084] Step 4.2.4: Experience replay.

[0085] If experience cache If the size reaches or exceeds the minimum batch size, then the empirical cache will be used. Medium sampling Examples are provided, and a reward is calculated for each transformation in the minimum batch. i .

[0086] Step 4.2.5: Update the policy model using the policy gradient method. The update formula is as follows:

[0087] Where η represents the learning rate. This indicates the calculation of divergence.

[0088] II. Testing Phase.

[0089] After constructing the optimal example combination using the trained example selection model during the inference phase, the input text and the optimal example combination are fed into the large model. The details are as follows: the input text and the optimal example combination are concatenated into a single long text, which is then sent to the large model service. The large model service uses GPU analysis to generate the corresponding text response content. The generated text response content is then checked for any risky content using manual or automated detection tools (such as the Moderation API provided by OpenAI).

[0090] In summary, based on the significant variations in performance during context learning across different examples, researchers discovered the impact of example length and search strategies on jailbreak effectiveness. Accordingly, this invention models example selection as an optimization problem and employs experience replay and entropy penalty strategies to achieve high attack efficiency and effectiveness.

[0091] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for detecting large models based on jailbreak attacks, characterized in that, The method includes: Construct an example pool and an example selection model; wherein each example is a data pair of harmful instructions and responses, and the example selection model is constructed using a transformer-based neural network model, eliminating the LM-HEAD layer of the neural network model, and introducing two linear layers and a Select-Head layer after the last transformer MLP layer; The example selection model is trained on the example pool; Use the trained example selection model as the input instruction to select examples from the example pool; The target large model is attacked based on the input instructions and the selected example to obtain the jailbreak attack detection results of the target large model.

2. The method according to claim 1, characterized in that, Training the example selection model on the example pool includes: Obtain the input text x, initialize the strategy π and experience cache of the example selection model. And setting the exploration probability ∈ and batch size in the greedy strategy. Determine if the random number generated by the random function rand() is less than the exploration probability ∈; If the random number is less than the exploration probability ∈, the Action() mapping function is used to map the random number to action a in the action space. If the random number is not less than the exploration probability ∈, action a is selected according to the current strategy π. Perform the chosen action a to obtain a new state x′ and a reward r; Store the experience (x, a, r, x′) in the experience cache. middle If experience cache If the size reaches or exceeds the minimum batch size, then the empirical cache will be used. Medium sampling Examples are provided, and a reward is calculated for each transformation in the minimum batch. i ; Based on reward i Update the parameters of the selected example model.

3. The method according to claim 2, characterized in that, Calculate the reward i The process includes: Calculate the cross-entropy loss Loss for each transition in the minimum batch; Calculate the entropy H(π) of strategy π; The reward is obtained based on the cross-entropy loss Loss and the entropy H(π) of the policy π. i .

4. The method according to claim 3, characterized in that, The cross-entropy loss in, Indicates that given input x i and a series of known input-output pairs {(x1,y1),...,(x k ,y k In the case of )}, the example selects the model output. The probability of L is given by L, where L represents the total number of output categories or the sequence length.

5. The method according to claim 3, characterized in that, The entropy of the policy π is H(π) = -∑x∈Xπ(a|x)logπ(a|x); where π(a|x) is the probability of choosing action a given input x under policy π, and X represents the input set.

6. A large-scale model detection system based on jailbreak attacks, characterized in that, The system includes: A training module is used to construct an example pool and an example selection model; wherein each example is a data pair of harmful instructions and responses, and the example selection model is constructed using a transformer-based neural network model, eliminating the LM-HEAD layer of the neural network model and introducing two linear layers and a Select-Head layer after the last transformer MLP layer; the example selection model is trained on the example pool; The detection module is used to select examples from the example pool using a trained example selection model as input instructions; and to attack the target large model based on the input instructions and the selected examples to obtain jailbreak attack detection results for the target large model.

7. The system according to claim 6, characterized in that, Training the example selection model on the example pool includes: Obtain the input text x, initialize the strategy π and experience cache of the example selection model. And setting the exploration probability ∈ and batch size in the greedy strategy. Determine if the random number generated by the random function rand() is less than the exploration probability ∈; If the random number is less than the exploration probability ∈, the Action() mapping function is used to map the random number to action a in the action space. If the random number is not less than the exploration probability ∈, action a is selected according to the current strategy π. Perform the chosen action a to obtain a new state x′ and a reward r; Store the experience (x, a, r, x′) in the experience cache. middle If experience cache If the size reaches or exceeds the minimum batch size, then the empirical cache will be used. Medium sampling Examples are provided, and a reward is calculated for each transformation in the minimum batch. i ; Based on reward i Update the parameters of the selected example model.

8. The system according to claim 7, characterized in that, Calculate the reward i The process includes: Calculate the cross-entropy loss Loss for each transition in the minimum batch; Calculate the entropy H(π) of strategy π; The reward is obtained based on the cross-entropy loss Loss and the entropy H(π) of the policy π. i .

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the large model detection method based on jailbreak attacks as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the large model detection method based on jailbreak attacks as described in any one of claims 1 to 5.

Images