Prompt Segmentation for Generative Model Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large Language Models (LLMs) are vulnerable to 'jailbreaking' attacks where malicious instructions are injected into the prompt, bypassing safety measures and leading to unintended and potentially harmful outputs.
Innovation Solution
The method involves generating a prompt that includes a first prompt section from input data, accompanied by instructions that inform the generative model of the boundary of the unsafe part of the prompt, and third instructions to ignore any instructions present in that part, thereby preventing malicious instructions from influencing the model's output.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If input data is directly used to generate prompts for the generative model, then task processing efficiency is improved, but the model becomes vulnerable to malicious instructions injected into the input data
Solution Approach 1:
The prompt is divided into distinct sections: a first prompt section containing transformed input data and a second prompt section containing trusted instructions. This segmentation allows the model to process efficient input data while maintaining security boundaries that prevent malicious instruction injection.
Solution Approach 2:
A data transformation process acts as an intermediary between raw input data and the generative model. The transformation (encoding, interleaving special characters, or delimiting) modifies the input data structure so that malicious instructions are neutralized while preserving task-relevant information.
2Reliability
If transformations are applied to input data to prevent attacks, then model security is improved, but prompt processing complexity increases
Solution Approach 1:
The input data undergoes parameter changes through transformation processes such as encoding, interleaving special characters, or adding delimiters. These transformations modify the data structure to prevent malicious instruction execution while maintaining the essential task information.
Solution Approach 2:
Data transformation is performed in advance before the prompt is submitted to the generative model. This preliminary action prepares the input data in a secure format, eliminating the need for complex runtime security checks during model processing.
3Object-affected harmful factors
If instructions are added to inform the model of prompt boundaries and ignore malicious instructions, then attack resistance is improved, but prompt length and processing overhead increase
Solution Approach 1:
The prompt includes preliminary instructions that counteract potential malicious instructions. The second prompt section explicitly tells the model to ignore instructions from the first section, creating a preemptive defense against prompt injection attacks before they can take effect.
Data Source
AI summary
A computer-implemented method is provided that prevents prompt injection attacks against generative models. Input data is received, and a first prompt section is generated from the input data. First, second and third instructions are received, which respectively instruct the generative model to carry out a task based on the first prompt section, inform the generative model of a boundary of the first prompt section, and instruct the generative model to ignore any instructions in present in the first prompt section. A prompt for the generative model is generated from the first prompt section and the first instructions, second instructions and third instructions.