AI Prompt Filtering Using Trusted and Untrusted Token Tagging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative Pre-trained Transformers (GPT) models are vulnerable to malicious user input that can generate offensive, adversarial, or untruthful outputs due to misalignment with user intentions, necessitating mitigation techniques to ensure safe and aligned text generation.
Innovation Solution
Implementing a classifier to detect and remove untrusted instructions, coupled with reinforcement learning (RL) to penalize the model for following untrusted inputs, and using token tagging and separate input sequences to ensure only trusted instructions are processed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the GPT accepts all user input text, then the model is highly versatile and responsive to user needs, but it becomes vulnerable to prompt injection attacks and generates harmful outputs
Solution Approach 1:
The patent segments user input into trusted and untrusted instruction components using a classifier. The classifier divides the input text into separate token sets, allowing the system to process only trusted instructions while filtering out potentially harmful untrusted instructions. This segmentation resolves the contradiction by maintaining versatility for trusted inputs while blocking harmful untrusted inputs.
Solution Approach 2:
The patent introduces a classifier as an intermediary component between user input and the GPT model. This classifier acts as a mediator that evaluates input text, tags instructions as trusted or untrusted, and determines which instructions should be processed. The intermediary resolves the contradiction by filtering harmful content before it reaches the versatile GPT model.
2Object-affected harmful factors
If the system filters all user input to prevent harmful outputs, then safety is improved, but the model loses the ability to process legitimate user instructions
Solution Approach 1:
The patent applies local quality by differentiating between trusted and untrusted instructions within the user input. Instead of uniformly filtering all input, the system assigns different trust qualities to different parts of the input based on classifier evaluation. This allows legitimate trusted instructions to pass through while blocking untrusted harmful instructions, resolving the contradiction between safety and functionality.
Solution Approach 2:
The patent changes the parameter of instruction trustworthiness by using a classifier to evaluate and tag each instruction as trusted or untrusted. This parameter change enables dynamic filtering based on trust level rather than static blocking of all user input. The GPT model processes instructions based on their trust parameter, maintaining safety while preserving legitimate functionality.
3Object-affected harmful factors
If the system uses a classifier to detect and remove untrusted instructions, then prompt injection risk is reduced, but system complexity increases
Solution Approach 1:
The patent introduces a classifier intermediary that adds a layer of security between user input and the GPT model. While this increases system complexity, it effectively reduces command injection risk by filtering untrusted instructions. The classifier uses reinforcement learning to improve its detection accuracy over time, making the added complexity worthwhile for security.
Solution Approach 2:
The patent implements feedback mechanisms where the classifier is trained using reinforcement learning on examples of trusted and untrusted instructions. The system learns from feedback about which instructions are harmful and adjusts its filtering behavior accordingly. This feedback loop reduces command injection risk while managing system complexity through adaptive learning rather than hard-coded rules.
Data Source
AI summary
A system for use with an artificial intelligence (AI) model configured to accept text input, such as generative pre-trained transformer (GPT), that detects and tags trusted instructions and nontrusted instructions of an input provided by a user responsive to an AI model prompt. The system uses reinforcement learning (RL) and a set of rules to remove the untrusted instructions from the input and provide only trusted instructions to the AI model. The input is represented as tokens, wherein the trusted instructions and the untrusted instructions are represented using incompatible token sets.