Prompt Injection Detection for RAG-Augmented LLM Prompts
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Indirect prompt injection attacks occur when data sources accessed by retrieval-augmented generation systems are compromised, leading to malicious instructions being inadvertently added to prompts for large language models, which can cause the models to perform unauthorized actions or leak sensitive data.
Innovation Solution
A malicious instructions detection model is introduced to preprocess augmented prompts by tokenizing sentences and applying natural language processing, generating confidence scores for each sentence, and blocking prompts with scores above a threshold, while providing alerts to users about the malicious content.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If retrieval-augmented generation systems access external data sources to improve prompt quality, then the quality and relevance of generated responses is improved, but the system becomes vulnerable to indirect prompt injection attacks from compromised data sources
Solution Approach 1:
A malicious instructions detection model is introduced as an intermediary component between the RAG system and the LLM. This detector preprocesses augmented prompts by tokenizing them into sentences, applying NLP preprocessing, and generating confidence scores for each sentence to identify malicious content before it reaches the LLM, thus resolving the vulnerability while preserving data augmentation benefits
2Reliability
If the system blocks prompts from external data sources to prevent malicious injections, then security against indirect prompt injection is improved, but the ability to retrieve useful external information is reduced
Solution Approach 1:
The detection model applies localized analysis to individual sentences within augmented prompts rather than blocking entire prompts. By tokenizing prompts into sentences and generating confidence scores for each sentence independently, the system can identify and block only malicious sentences while allowing benign informative content to pass through to the LLM
3Measurement precision
If a detection model analyzes each sentence in augmented prompts to identify malicious instructions, then detection accuracy is improved, but the processing time and computational complexity increase
Solution Approach 1:
The detection model segments augmented prompts into individual sentences for analysis. By tokenizing prompts and processing sentences independently rather than analyzing entire prompts as single units, the system achieves better detection accuracy through granular sentence-level confidence scoring while reducing overall computational complexity through divide-and-conquer processing
Data Source
AI summary
A malicious instructions detection model (“detector”) intercepts augmented prompts destined for a large language model (“LLM”). Each augmented prompt was augmented with data from potentially compromised data sources susceptible to indirect prompt injection attacks. The detector tokenizes/preprocesses sentences in the augmented prompts and is invoked on the tokenized/preprocessed sentences to obtain confidence scores that each sentence comprises malicious instructions. If one or more of the confidence scores is above a threshold, the detector blocks the augmented prompt and generates an alert indicating the blocking and the malicious instructions. Otherwise, the detector communicates the augmented prompt to its intended LLM.