Detection-Based LLM Mediation for Unauthorized Tool Actions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative AI models are vulnerable to manipulation by threat actors, leading to compromised system integrity and unauthorized actions, such as data exfiltration, through indirect prompt injection attacks.
Innovation Solution
A threat detection system using a detection-based large generative model processes communications between applications and LLMs to identify and prevent anomalous outputs, safeguarding against compromised models and unauthorized tool invocations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a detection-based large generative model is implemented to classify LLM outputs, then system security and reliability are improved, but device complexity and computational resources increase
Solution Approach 1:
A detection-based large generative model is introduced as an intermediary component between the application and the LLM. This mediator classifies LLM outputs to identify anomalous content, thereby improving system security without requiring fundamental changes to the existing LLM architecture. The detection model acts as a buffer that filters and validates outputs before they are executed.
Solution Approach 2:
The security detection function is segmented into a separate, specialized model rather than being integrated directly into the LLM's core architecture. This allows the detection-based large generative model to operate independently, classifying outputs in parallel without interfering with the LLM's primary generation tasks, thus managing complexity through functional separation.
2Measurement precision
If comprehensive output classification is performed on all LLM outputs, then detection precision is improved, but processing time and productivity are reduced
Solution Approach 1:
The detection-based large generative model performs classification on all LLM outputs to ensure comprehensive security coverage, but the system is designed to handle this processing efficiently by leveraging the model's parallel processing capabilities. The detection model prioritizes security thoroughness while optimizing processing speed through its architectural design.
Solution Approach 2:
The detection model provides real-time feedback on LLM output classification, enabling the system to immediately identify and respond to anomalous content. This feedback mechanism allows for continuous monitoring and rapid detection of security threats without requiring manual intervention, thereby minimizing the time loss despite comprehensive classification.
3Reliability
If the detection model processes only initial application requests without external content, then the model is protected from indirect prompt injection attacks, but the ability to detect anomalies in subsequent requests is limited
Solution Approach 1:
The system applies preliminary protective measures by restricting the detection-based large generative model from processing external content that could contain indirect prompt injection attacks. The model is trained and configured using only initial application requests, preventing malicious content from compromising the detection model's integrity before it can perform its security function.
Solution Approach 2:
The detection model serves as an intermediary that processes only sanitized, initial application requests rather than all incoming content including external sources. This intermediary approach protects the detection model from direct exposure to malicious inputs while still enabling it to learn and detect anomalous patterns in legitimate request structures.
Data Source
AI summary
This disclosure relates to utilizing a threat detection system to detect anomalous actions provided by a compromised large generative language model (LLM). For instance, the threat detection system utilizes a detection-based large generative model to process select communication between an application system and the LLM and determine when the LLM may have been potentially compromised. In various implementations, utilizing the detection-based large generative model, the threat detection system determines when an LLM is improperly instructing an application system to invoke tools to perform unapproved actions. Furthermore, when an LLM becomes compromised, the threat detection system intelligently safeguards the detection-based large generative model against similar threats that seek to evade detection or compromise the detection-based large generative model.