LLM Jailbreak Detection Using Token Probability Shifts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) are vulnerable to 'jailbreak' attacks where attackers manipulate prompts to bypass safety measures, leading to the generation of malicious or illegal content, which can compromise user safety and system integrity.
Innovation Solution
A method is employed to detect jailbreak attempts by analyzing shifts in probability-based metrics, such as perplexity, across tokens in input or output data to identify sudden changes indicative of malicious content, using a sliding window approach to assess the probability of each token given its context.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LLMs are trained on large diverse datasets to improve versatility and capability, then the model can perform a wide range of tasks, but the model becomes vulnerable to jailbreak attacks that exploit the training data to generate malicious content
Solution Approach 1:
The system performs preliminary detection of jailbreak attempts by analyzing probability-based metrics (perplexity, entropy) of tokens before the LLM generates malicious content. This early detection allows the system to prevent harmful output without requiring the model to be retrained, thus maintaining versatility while addressing security vulnerabilities.
Solution Approach 2:
An intermediary detection layer is introduced between the LLM and the output. This layer analyzes the probability metrics of tokens and acts as a gatekeeper, blocking potentially malicious content while allowing legitimate tasks to proceed. This mediator approach preserves the model's capabilities while filtering harmful outputs.
2Object-affected harmful factors
If safety measures are added to moderate generated content, then harmful output is reduced, but the system complexity increases
Solution Approach 1:
The LLM itself is used to calculate the probability-based metrics (perplexity, entropy) of its own output tokens. This self-service approach eliminates the need for separate complex analysis systems, as the model leverages its own internal probability distributions to detect anomalies, thereby reducing overall system complexity while maintaining safety.
Solution Approach 2:
The system changes the parameter space by analyzing probability metrics (perplexity, entropy) of tokens rather than using complex content analysis. This parameter-based approach simplifies the detection mechanism compared to traditional content filtering, reducing system complexity while effectively identifying malicious content through statistical anomalies.
3Measurement precision
If probability-based metrics are calculated for each token to detect jailbreak attempts, then detection accuracy improves, but computational cost increases
Solution Approach 1:
The system calculates probability-based metrics for all tokens (excessive action) to ensure comprehensive detection coverage. While this increases computational cost, the use of efficient probability calculations and the model's own internal mechanisms keeps the overhead manageable, achieving high detection accuracy without proportionally high computational expense.
Data Source
AI summary
A computer-implemented method is provided that detects jailbreak attempts against generative models, which may involve a shift between benign and malicious content. The method includes determining a probability-based metric for each of a plurality of tokens in a target text using a language model, the probability-based metric being based on a probability at least one preceding token. The probability-based metrics are processed to identify a subset of the plurality of tokens having a change in the probability-based metric with respect to others of the plurality of the tokens not within the subset of the plurality of tokens. A jailbreak attempt in the target text is detected in response to identifying the change in the probability-based metric in the subset of the plurality of tokens.