Inference-Time Backdoor Detection via Activation Logging
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current adversarial backdoor attack detection technologies are ineffective at inference time, particularly for models employed in transfer learning or obtained from model marketplaces, as they require access to training data, are computationally expensive, and lack resilience to distribution differences in inference time data.
Innovation Solution
A system that records predictions and activation values generated by a trained model during inference requests and employs a model at inference time to detect backdoor trigger requests based on these values, without accessing training data, using components like log and analysis components to automatically detect adversarial attacks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If training data is accessed and analyzed for backdoor detection, then detection capability is improved, but applicability to transfer learning and model marketplace scenarios deteriorates
Solution Approach 1:
Instead of analyzing training data to detect backdoors (traditional approach), the patent inverts the approach by analyzing inference-time predictions and activation values. The system records predictions from the trained model and corresponding activation values, then analyzes these inference-time outputs to detect backdoor triggers without needing access to training data.
Solution Approach 2:
The patent introduces an intermediary detection model that is trained on clean inference data to recognize normal activation patterns. This intermediary model then evaluates new predictions against the learned normal patterns, serving as a mediator between the suspicious model outputs and the detection decision, enabling backdoor detection without direct access to original training data.
2Measurement precision
If complex analysis methods are used for backdoor detection, then detection accuracy is improved, but computational cost deteriorates
Solution Approach 1:
Instead of performing comprehensive analysis of all model inputs and internal states, the patent applies partial action by focusing analysis only on specific activation values from key layers and predictions. The system selectively monitors critical activation patterns that are most indicative of backdoor triggers, rather than analyzing the entire model state, reducing computational overhead while maintaining detection accuracy.
3Measurement precision
If detection methods are designed for specific data distributions, then detection precision is improved, but resilience to distribution differences deteriorates
Solution Approach 1:
The detection system is designed with universal applicability by training the detection model on diverse clean inference data from multiple sources and distributions. The detection model learns distribution-agnostic patterns of normal behavior that can generalize across different data distributions, enabling the system to maintain detection precision while being resilient to distribution differences in deployment scenarios.
Data Source
AI summary
Systems, computer-implemented methods, and computer program products that can facilitate detection of an adversarial backdoor attack on a trained model at inference time are provided. According to an embodiment, a system can comprise a memory that stores computer executable components and a processor that executes the computer executable components stored in the memory. The computer executable components can comprise a log component that records predictions and corresponding activation values generated by a trained model based on inference requests. The computer executable components can further comprise an analysis component that employs a model at an inference time to detect a backdoor trigger request based on the predictions and the corresponding activation values. In some embodiments, the log component records the predictions and the corresponding activation values from one or more layers of the trained model.