Prompt Injection Detection Using Trusted Prompt Saliency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Model (LLM)-integrated applications are vulnerable to prompt injection attacks, which manipulate the program flow by embedding malicious instructions within the input prompt, posing a critical security risk due to insufficient syntactic separation and current defense mechanisms being inadequate.
Innovation Solution
A method to detect prompt injections by calculating a trust score based on the influence of a trusted part of the prompt on the model output using attention values or SHAP/LIME, identifying prompt injections when the trust score falls below a threshold, and employing a neural network classifier for enhanced detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If prompt injection detection is implemented using attention values or SHAP/LIME, then detection accuracy improves, but computational complexity and processing time increase
Solution Approach 1:
The prompt is segmented into trusted and untrusted parts, allowing selective analysis of token influence. This segmentation enables the use of attention values or SHAP/LIME specifically on relevant portions of the prompt, reducing overall computational complexity while maintaining detection accuracy.
Solution Approach 2:
The system performs preliminary calculation of attention values or SHAP/LIME values during the prompt processing stage, before final output generation. This preliminary action allows detection metrics to be computed in advance, improving detection accuracy without adding significant overhead to the main generation process.
2Reliability
If threshold adjustment is used to reduce false positives/negatives, then detection reliability improves, but system configuration complexity increases
Solution Approach 1:
The threshold parameter is made adjustable and dynamic rather than fixed. Different thresholds can be configured for different application scenarios, trusted/untrusted prompt ratios, or risk levels. This dynamic configuration improves detection reliability by allowing optimization for specific use cases without hardcoding complex detection logic.
Solution Approach 2:
The system allows modification of the trust score threshold parameter to balance false positives and false negatives. By providing a configurable parameter rather than a fixed value, the system can adapt to different security requirements and operational contexts, improving reliability without requiring complex rule-based systems.
3Adaptability or versatility
If local LLMs are used to calculate attention values when API access is limited, then system autonomy improves, but computational resource requirements increase
Solution Approach 1:
A local LLM acts as an intermediary component that can independently calculate attention values or SHAP/LIME values when external API access is unavailable. This intermediary capability provides system autonomy for detection operations without requiring constant external service dependency.
Solution Approach 2:
Instead of relying on external APIs to provide attention values, the system implements a local copy of the necessary computational functionality using a local LLM. This local copy enables autonomous operation and reduces dependency on external services, though it requires local computational resources.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Method for recognizing manipulation attacks in the form of prompt injections on large-language model-integrated applications which generate a prompt in order to pass this as input to the LLM, which generates a model output based on the prompt, the prompt comprising a trusted part which contains the instructions defined by the LLM-integrated application, over which an attacker by definition has no influence, and wherein the prompt consists of an untrusted part comprising the data to be processed, over which the attacker by definition has full control, characterized in that a prompt injection is present if the trust score of the trusted part on the model output is below a threshold, wherein the trust score of the trusted part for the model output is calculated using an input feature saliency method.