Activation Analysis for Policy-Based Multimodal LLM Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) are vulnerable to cyberattacks that manipulate their behavior, leading to undesired outputs such as data leakage, unauthorized access, and the solicitation of restricted information, due to the lack of clear policies on acceptable and unacceptable behaviors.
Innovation Solution
Implement a policy-driven approach using a classifier trained on intermediate results from LLMs to identify and prevent undesired behaviors by taking remedial actions like blocking or modifying prompts, and disconnecting devices or accounts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a classifier is trained using intermediate results to map to undesired model behavior, then the reliability of detecting malicious prompts is improved, but the device complexity increases due to the need for additional monitoring components
Solution Approach 1:
The patent introduces a classifier as an intermediary component that analyzes intermediate results (activations) from the GenAI model's processing pipeline. This classifier acts as a mediator between the input prompt and the model's output, detecting undesired behavior patterns without requiring modification of the core GenAI model. The classifier is trained on intermediate results to identify malicious intent, thereby improving detection reliability while maintaining a modular architecture that adds complexity only where needed for security monitoring.
2Measurement precision
If hooks are applied to layers to capture activation values, then the measurement precision of intermediate results is improved, but the device complexity increases due to additional monitoring infrastructure
Solution Approach 1:
The patent applies segmentation by dividing the GenAI model into discrete layers and applying hooks to specific layers where intermediate results can be captured. Rather than monitoring the entire model uniformly, the approach segments the monitoring process to focus on critical activation points within the transformer architecture. This allows precise capture of activation values at strategically chosen layers, improving measurement precision while reducing the overall complexity compared to comprehensive full-model monitoring.
3Reliability
If a proxy of the GenAI model is used to capture intermediate results, then the reliability of behavior analysis is improved, but the loss of energy increases due to running additional model instances
Solution Approach 1:
The patent employs a proxy (quantized version) of the GenAI model as a copy that replicates the behavior of the full model for monitoring purposes. This copied model processes prompts in parallel to capture intermediate results and detect undesired behavior. The quantized proxy requires fewer computational resources than a full-precision model, thereby reducing energy loss while maintaining sufficient reliability for behavior analysis. The proxy serves as a lightweight duplicate that enables robust monitoring without proportionally increasing energy consumption.
Data Source
AI summary
An analysis engine receives data characterizing a multimodal prompt for ingestion by a generative artificial intelligence (GenAI) model. The multimodal prompt is processed and fed into a plurality of layers from which an intermediate result of the GenAI model or a proxy of the GenAI model is obtained. The analysis engine, using a classifier and the intermediate result, determines whether the prompt elicits undesired behavior by the GenAI model. Data characterizing the determination is provided to a consuming application or process. Related apparatus, systems, techniques and articles are also described.