Provenance-Based Poison Attack Detection in ML Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning (ML) systems are vulnerable to poisoning attacks, where adversaries manipulate training data to compromise model accuracy, and existing detection methods are often reactive and inefficient, especially in online learning environments where dataset shifts occur frequently.
Innovation Solution
A provenance-based defense method that utilizes data provenance to identify and remove poisoned data points by segmenting datasets based on provenance signatures, training both complete and filtered models, and comparing their performance to determine and mitigate poisoning attacks, even in fully untrusted data environments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional reactive detection methods are used, then detection capability is achieved, but detection timing is delayed until after poisoning occurs
Solution Approach 1:
The patent applies preliminary action by establishing provenance tracking infrastructure before poisoning attacks occur. Data provenance information is collected and stored alongside training data from the outset, enabling proactive identification of poisoned samples through provenance analysis rather than waiting for reactive detection after model degradation is observed.
2Measurement precision
If comprehensive data verification is performed, then detection accuracy is improved, but computational overhead increases
Solution Approach 1:
The patent extracts and utilizes pre-existing provenance information that is already associated with training data samples. Rather than performing comprehensive verification of all data characteristics, the system selectively uses provenance metadata (such as data source, collection timestamp, and processing history) to efficiently identify poisoned samples, reducing computational overhead while maintaining detection accuracy.
3Reliability
If trusted test sets are required for detection, then detection reliability is improved, but system applicability is reduced
Solution Approach 1:
The patent enables the system to self-verify training data quality using its own provenance information without requiring external trusted test sets. By analyzing provenance characteristics of training samples and comparing them against expected patterns, the system autonomously identifies poisoned data, making it applicable to scenarios where trusted test data is unavailable while maintaining detection reliability.
Data Source
AI summary
Computer-implemented methods, program products, and systems for provenance-based defense against poison attacks are disclosed. In one approach, a method includes: receiving observations and corresponding provenance data from data sources; determining whether the observations are poisoned based on the corresponding provenance data; and removing the poisoned observation(s) from a final training dataset used to train a final prediction model. Another implementation involves provenance-based defense against poison attacks in a fully untrusted data environment. Untrusted data points are grouped according to provenance signature, and the groups are used to train learning algorithms and generate complete and filtered prediction models. The results of applying the prediction models to an evaluation dataset are compared, and poisoned data points identified where the performance of the filtered prediction model exceeds the performance of the complete prediction model. Poisoned data points are removed from the set to generate a final prediction model.