Prompt Sequence Screening for Generative AI Attack Prevention
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing generative AI systems are vulnerable to malicious prompt injection attacks where malicious actions are concealed across multiple seemingly innocuous prompts, making it difficult to detect and prevent such threats.
Innovation Solution
A device identifies sequences of related prompts, performs individual and collective maliciousness assessments, and prevents the input of potentially malicious prompts to the generative model, using machine learning models to detect and block malicious sequences before processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If individual prompt assessment is performed, then detection precision for overtly malicious prompts is improved, but detection precision for concealed malicious sequences deteriorates
Solution Approach 1:
The system segments the detection process into two distinct levels: individual prompt assessment and collective sequence assessment. Each prompt is first evaluated independently for overtly malicious content, then the entire sequence is evaluated together to detect concealed malicious patterns that span multiple prompts. This segmentation allows the system to leverage both individual and collective analysis strengths.
Solution Approach 2:
The invention adds a temporal and contextual dimension to prompt detection by analyzing prompts not just in isolation but as part of a sequence over time. The collective maliciousness assessment considers the relationships, patterns, and cumulative effects across multiple prompts, transforming the detection from a single-point assessment to a multi-dimensional temporal analysis.
2Measurement precision
If collective maliciousness assessment of prompt sequences is performed, then detection precision for concealed malicious sequences is improved, but device complexity increases
Solution Approach 1:
The system segments the complex collective assessment task into manageable components: identifying related prompts within a sequence, evaluating their combined maliciousness, and making prevention decisions. This segmentation breaks down the complexity into discrete, processable steps that can be implemented systematically.
Solution Approach 2:
The patent introduces an intermediary assessment layer between individual prompt evaluation and final prevention decisions. The collective maliciousness assessment acts as a mediator that synthesizes information from multiple prompts and applies contextual understanding, reducing the complexity burden on any single component while improving overall detection accuracy.
3Reliability
If multiple assessment levels are implemented, then security reliability is improved, but processing time increases
Solution Approach 1:
The system performs preliminary individual maliciousness assessments on each prompt as they arrive, preparing evaluation results in advance. This preliminary action allows the collective sequence assessment to build upon pre-computed individual evaluations, reducing redundant processing and minimizing the time penalty of multi-level assessment.
Solution Approach 2:
The patent maintains continuous monitoring and assessment of prompt sequences, where individual assessments feed continuously into collective assessments. This continuous action ensures that security evaluation is ongoing without interruption, and the system can make real-time prevention decisions based on accumulated evidence from the sequence.
Data Source
AI summary
In one implementation, a device identifies a sequence of related prompts for input to a generative model. The device makes individual maliciousness assessments of those prompts in the sequence of related prompts. The device makes a collective maliciousness assessment of the sequence of related prompts. The device prevents at least a portion of the sequence of related prompts from being input to the generative model, when any of the individual maliciousness assessments or the collective maliciousness assessment indicates a malicious request to the generative model.