LLM Prompt Segmentation for Prompt Injection Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) are vulnerable to prompt injection attacks, where malicious users bypass instructions by phrasing queries in various ways, making it challenging to detect such attacks automatically due to the vast number of queries and diverse phrasing possibilities, and human review is infeasible.
Innovation Solution
Implement an LLM firewall with a malicious prompt detector that segments user prompts into vectors, scores them against stored vectors, and sets a prompt injection signal based on similarity scores to block malicious prompts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If instructions are added to the LLM to self-control which questions to answer, then security against malicious queries is improved, but vulnerability to prompt injection attacks increases
Solution Approach 1:
The patent introduces an intermediary malicious prompt detector between the user prompt and the LLM. This detector segments the user prompt into multiple segments, encodes each segment into a vector, compares the vectors against a database of malicious prompt vectors, and determines whether to block the prompt before it reaches the LLM. This intermediary layer prevents prompt injection attacks from bypassing the LLM's self-control instructions.
2Measurement precision
If human review of individual user queries is performed, then detection accuracy of prompt injection attacks is improved, but productivity decreases
Solution Approach 1:
The patent replaces the mechanical system of human review with an automated malicious prompt detector that uses vector encoding and comparison. The detector segments user prompts, encodes each segment into a vector representation, compares these vectors against a database of malicious prompt vectors using cosine similarity, and automatically blocks malicious prompts. This substitution maintains high detection accuracy while enabling the system to process large volumes of queries without human intervention.
3Device complexity
If prompt injection detection is performed using traditional methods, then implementation simplicity is improved, but detection capability against diverse attack phrases decreases
Solution Approach 1:
The patent transforms the detection approach by changing the parameter representation from raw text to vector embeddings. Each prompt segment is encoded into a vector that captures its semantic meaning, and the system compares these vectors against a database of malicious prompt vectors. This parameter transformation enables the system to detect diverse attack phrases including those with different wording, spelling variations, and obfuscation techniques, while maintaining a relatively simple implementation through the use of pre-trained encoding models and cosine similarity comparison.
Data Source
AI summary
A method includes receiving, at a server from a user device, a user prompt to a large language model (LLM). The user prompt is segmented to generate a set of user segments. An encoding model generates the set of user segments into a set of user vectors. The method further includes scoring each user vector of the set of user vectors based on a comparison between the user vector and a set of stored vectors in a vector store to generate a set of user vector scores, detecting whether the user prompt is malicious according to the set of user vector scores, and setting a prompt injection signal based on whether the user prompt is detected as malicious according to the set of user vector scores.