Privilege Segmented LLM Prompts for Injection Defense

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Generative large language models (LLMs) are vulnerable to instruction injection attacks, where malicious end-users can access confidential business information and perform unintended tasks by providing malicious data instructions within instruction prompts.

Innovation Solution

A privilege-based segmented instruction prompt generation system that divides prompts into three segments: a trusted segment with the highest privilege level, a program segment with the second highest level, and a data segment with the lowest level. This system uses boundary tags to enclose each segment and defines privilege levels to prevent conflicts and identify potential attacks.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If generative LLM uses data instructions from end-users to perform tasks, then the system becomes more versatile and user-friendly, but the system becomes vulnerable to instruction injection attacks that can access confidential information

Engineering Contradiction:

Improveability to perform user-defined tasksVSAvoidvulnerability to instruction injection attacks

Core Design Contradiction:

Adaptability or versatilityVSObject-affected harmful factors

Solution Approach 1:

The instruction prompt is divided into three distinct segments: a trusted segment containing trusted instructions, a program segment containing program instructions, and a data segment containing data instructions from end-users. Each segment is enclosed by boundary tags and assigned a specific privilege level. This segmentation isolates malicious data instructions from trusted instructions, preventing injection attacks while preserving the system's ability to execute user-defined tasks.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Different segments of the instruction prompt are assigned different privilege levels: the trusted segment has the highest privilege level, the program segment has the second highest privilege level, and the data segment has the lowest privilege level. This local differentiation of quality (privilege levels) ensures that data instructions cannot override or modify trusted instructions, thereby preventing harmful attacks while maintaining system versatility.

Inventive Principle:

Principle #3Local quality

2Adaptability or versatility

If the system allows data instructions to modify program instructions for flexibility, then the system becomes more adaptable, but the integrity of program instructions is compromised

Engineering Contradiction:

Improveflexibility in task executionVSAvoidintegrity of program instructions

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The program segment is assigned a specific privilege level (second highest) that is higher than the data segment but lower than the trusted segment. This local quality assignment creates a controlled hierarchy where data instructions can influence program execution within defined boundaries but cannot compromise the integrity of program instructions or trusted instructions, thus maintaining both flexibility and reliability.

Inventive Principle:

Principle #3Local quality

3Device complexity

If the system uses a single unified instruction prompt without segmentation, then the system structure is simple, but the system cannot prevent instruction conflicts or attacks

Engineering Contradiction:

Improvesimplicity of prompt structureVSAvoidability to prevent instruction conflicts

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The instruction prompt is segmented into three distinct parts with clear boundary tags, creating a structured format that is both simple to implement and effective at preventing instruction conflicts. Each segment serves a specific purpose and has defined privilege levels, making the system reliable without excessive complexity.

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS20250077916A1Systems and methods for generating privilege based segmented instruction prompts for a generative large language model

Publication Date: 2025.03.06 SALESFORCE INC

AI summary

A method and system for generating a privilege based segmented instruction prompt has been developed. Trusted instructions defining the trusted instructions as having a first privilege level, program instructions as having a second privilege level, and data instructions as having a third privilege level are received. The program instructions to implement tasks associated with the data instructions are received. The data instructions are received. The generated privilege based segmented instruction prompt includes the trusted instructions, the program instructions, and the data instructions. The privilege based segmented instruction prompt enables a generative LLM to determine whether the privilege based segmented instruction prompt is an instruction injection attack based on whether there is a conflict between the trusted instructions, the program instructions, and the data instructions in violation of the first, second, and third privilege levels.