LLM Tool Invocation Detection Against Indirect Prompt Injection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Generative AI models are vulnerable to indirect prompt injection attacks, leading to compromised integrity and unauthorized actions by threat actors, compromising data security and violating responsible AI considerations.

Innovation Solution

A threat detection system utilizing a detection-based large generative model processes select communications between applications and LLMs to identify and prevent anomalous outputs, safeguarding against indirect prompt injection attacks and unauthorized tool invocations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If generative AI models are deployed to execute tasks, then task execution capabilities are improved, but vulnerability to indirect prompt injection attacks increases

Engineering Contradiction:

Improvetask execution capabilitiesVSAvoidvulnerability to indirect prompt injection attacks

Core Design Contradiction:

Adaptability or versatilityVSObject-affected harmful factors

Solution Approach 1:

A detection-based large generative model is introduced as an intermediary between the application and the task-executing LLM. This mediator analyzes LLM outputs before execution, detecting anomalous tool invocations and preventing indirect prompt injection attacks while allowing legitimate tasks to proceed

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The detection model performs preliminary analysis of LLM outputs before they are executed by the application. By checking for anomalies in advance, the system prevents malicious actions from being carried out while maintaining the full capabilities of the LLM for legitimate tasks

Inventive Principle:

Principle #10Preliminary action

2Measurement precision

If LLM outputs are monitored and analyzed, then detection accuracy is improved, but system complexity increases

Engineering Contradiction:

Improveanomaly detection accuracyVSAvoidsystem complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The detection-based large generative model serves multiple functions: it detects anomalous tool invocations, prevents indirect prompt injection attacks, and maintains system integrity. This multi-functional approach achieves high detection accuracy without requiring multiple separate systems

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The detection model uses the same large generative model architecture as the task-executing LLM, allowing it to understand and analyze LLM outputs effectively. The model essentially analyzes itself, leveraging its own capabilities to detect anomalies without requiring entirely different analysis tools

Inventive Principle:

Principle #25Self-service

Data Source

PatentUS20260003961A1Determining anomalous tool invocations by applications using large generative models

Publication Date: 2026.01.01 MICROSOFT TECHNOLOGY LICENSING LLC

AI summary

This disclosure relates to utilizing a threat detection system to detect anomalous actions provided by a compromised large generative language model (LLM). For instance, the threat detection system utilizes a detection-based large generative model to process select communication between an application system and the LLM and determine when the LLM may have been potentially compromised. In various implementations, utilizing the detection-based large generative model, the threat detection system determines when an LLM is improperly instructing an application system to invoke tools to perform unapproved actions. Furthermore, when an LLM becomes compromised, the threat detection system intelligently safeguards the detection-based large generative model against similar threats that seek to evade detection or compromise the detection-based large generative model.