Generative LLM Attention Analysis for Poisoned Token Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative large language models (LLMs) are vulnerable to poisoning, where malicious actors fine-tune the models to strongly associate specific input tokens with unintended output tokens, leading to the generation of harmful or offensive content, and current defense mechanisms require resources like additional unpoisoned LLMs or clean training data that are often unavailable in real-world scenarios.
Innovation Solution
The method involves analyzing the attention layers of the LLM to identify potentially poisoned input tokens by calculating trigger scores, creating a sanitized input prompt without flagged tokens, and generating a new output using the sanitized prompt to mitigate the effects of poisoning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If attention layer analysis is used to detect poisoned tokens, then detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the detection process into distinct stages: extracting attention scores from intermediate layers, calculating trigger scores for each token, comparing against thresholds, and generating sanitized outputs. This segmentation allows the system to manage computational complexity by processing different aspects of token detection separately rather than simultaneously.
Solution Approach 2:
The patent performs preliminary actions by extracting and analyzing attention scores during the forward propagation phase before final output generation. Trigger scores are calculated in advance for all tokens, allowing the system to identify poisoned tokens before they affect the final output, thereby improving detection accuracy without adding significant computational overhead to the critical path.
2Reliability
If trigger score calculation is performed for all tokens, then detection completeness is improved, but processing time increases
Solution Approach 1:
The patent applies local quality by focusing computational resources on tokens that exhibit suspicious characteristics in the attention scores. Rather than uniformly processing all tokens with equal depth, the system calculates trigger scores for all tokens but can prioritize or deepen analysis for tokens with higher suspicion scores, thereby maintaining detection completeness while optimizing processing time.
Solution Approach 2:
The patent changes parameters dynamically by adjusting detection thresholds and analysis depth based on the calculated trigger scores. Tokens with moderate trigger scores may use simplified detection criteria, while tokens with extreme scores receive more rigorous analysis, balancing detection completeness with processing efficiency.
3Measurement precision
If sanitized output is generated by removing flagged tokens, then output accuracy is improved, but information loss increases
Solution Approach 1:
The patent extracts and removes only the specifically identified poisoned tokens from the input sequence while preserving all other tokens. This selective extraction approach ensures that harmful content is eliminated to improve output accuracy, while the majority of legitimate information in the input remains intact and is processed normally through the model.
Solution Approach 2:
The patent creates a sanitized copy of the input sequence for processing, rather than modifying the original input. The sanitized version contains removed or replaced poisoned tokens, while the original input is preserved. This allows the system to generate accurate outputs from the sanitized copy while maintaining the complete original information for potential review or analysis.
Data Source
AI summary
An initial output that was generated by a generative large language model (generative LLM) LLM in response to processing an initial input prompt is obtained. Attention scores for the initial input prompt is extracted based on an attention layer of the generative LLM. A trigger score for a particular initial input token of the initial input prompt is developed based on the attention scores. That the trigger score meets a trigger flag condition is determined. A sanitized input prompt that does not include the particular initial input token is created based on the determining. The generative LLM is prompted with the sanitized input prompt.