LLM Prompt Injection Detection via Trusted Input Influence Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Model (LLM)-integrated applications are vulnerable to prompt injection attacks, which allow attackers to manipulate the program flow by embedding malicious instructions within the input prompt, posing a critical security risk due to the lack of effective protection mechanisms.
Innovation Solution
A method to detect prompt injections by calculating a trust score based on the influence of the trusted part of the prompt on the model output using attention values or SHAP/LIME saliency methods, identifying prompt injections when the trust score falls below a predefined threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If LLM-integrated applications process untrusted input data directly without verification, then the application can operate with high speed and simplicity, but the system becomes vulnerable to prompt injection attacks that can manipulate program flow
Solution Approach 1:
The system performs preliminary analysis of the model output by calculating attention values and trust scores before the output is executed. This advance verification checks the relationship between trusted instructions and generated content, detecting potential prompt injections before they can manipulate program flow, thus preventing security issues while maintaining operational speed
2Reliability
If the system implements comprehensive security checks on all model outputs, then the system can detect prompt injection attacks, but the processing time and computational complexity increase
Solution Approach 1:
Instead of uniformly verifying all tokens in the model output, the system focuses verification efforts on specific high-risk tokens identified through attention value analysis. The trust score calculation concentrates on evaluating the relationship between trusted instructions and critical portions of the generated output, performing localized security checks where they are most needed rather than exhaustive verification of entire outputs
3Measurement precision
If the system uses detailed attention value analysis to calculate trust scores, then the detection accuracy for prompt injections improves, but the computational complexity and resource consumption increase
Solution Approach 1:
The system calculates trust scores using a selective subset of attention values rather than analyzing all attention mechanisms in the transformer model. By focusing on attention values that directly relate the trusted instructions to the generated output tokens, the system achieves sufficient detection accuracy without the excessive computational burden of comprehensive attention analysis across all model layers and heads
Data Source
AI summary
Method for recognizing manipulation attacks in the form of prompt injections on large-language model-integrated applications which generate a prompt in order to pass this as input to the LLM, which generates a model output based on the prompt, the prompt comprising a trusted part which contains the instructions defined by the LLM-integrated application, over which an attacker by definition has no influence, and wherein the prompt consists of an untrusted part comprising the data to be processed, over which the attacker by definition has full control, characterized in that a prompt injection is present if the trust score of the trusted part on the model output is below a threshold, wherein the trust score of the trusted part for the model output is calculated using an input feature saliency method.