LLM Prompt Injection Detection via Prohibited Request Insertion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face challenges in automatically detecting and blocking prompt injection attacks, which attempt to bypass security features by phrasing queries in various ways, making human review and pre-emptive blocking infeasible.
Innovation Solution
The method involves inserting a system prohibited request into the LLM query, which requests the LLM to include a prohibited response in its output. This allows the system to detect prohibited responses and set a prompt injection signal, enabling automated blocking of such attacks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human review of individual user queries is performed, then detection accuracy of prompt injection attacks is improved, but processing speed and scalability deteriorate
Solution Approach 1:
The patent introduces an intermediary detection system that acts as a mediator between user queries and the LLM. This system inserts prohibited requests into queries, processes them through the LLM, and analyzes responses to detect prompt injection attacks automatically, eliminating the need for human review while maintaining detection accuracy.
Solution Approach 2:
The system implements self-service detection by having the LLM process modified queries with embedded prohibited requests and automatically analyzing its own responses for signs of prompt injection. This self-detection mechanism replaces human reviewers while maintaining high detection accuracy through automated response analysis.
2Reliability
If pre-emptive blocking of queries is implemented, then security is improved, but false positive rate increases
Solution Approach 1:
The system performs preliminary action by inserting prohibited requests into queries before they reach the LLM. This allows the system to test the LLM's response to prohibited content in advance and detect prompt injection attempts before they can cause harm, while maintaining security without excessive blocking.
Solution Approach 2:
The system uses feedback by analyzing the LLM's responses to modified queries containing prohibited requests. The response analysis provides feedback on whether prompt injection occurred, allowing the system to make informed blocking decisions rather than using rigid pre-emptive blocking rules that cause false positives.
3Productivity
If automated detection systems are implemented, then processing speed is improved, but detection precision deteriorates
Solution Approach 1:
The automated detection system uses an intermediary approach by inserting standardized prohibited requests as mediators between user input and LLM processing. This intermediary mechanism enables automated analysis of LLM responses to detect prompt injection with high precision while maintaining fast processing speeds through systematic response evaluation.
Data Source
AI summary
A method includes receiving, at a server from a user device, a user query to a large language model (LLM), creating an LLM query from the user query, inserting a system prohibited request into the LLM query to generate a revised LLM query, and sending the revised LLM query to the LLM. The method further includes receiving, from the LLM, a first LLM response to the LLM query, testing the first LLM response to detect whether a prohibited response to the system prohibited request is included in the first LLM response, and setting a prompt injection signal based on whether the prohibited response to the system prohibited request is included in the first LLM response.