Active Learning Filter for Poisoning Attack Mitigation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Active learning systems are vulnerable to poisoning attacks when relying on untrusted annotators, as they assume annotators are trusted and do not adequately filter new training samples, leading to potential manipulation of the ground truth.
Innovation Solution
The system assesses the risk of new labeled samples by determining reference model risk scores and overall risk scores, and uses probing mechanisms to verify annotator trustworthiness through vertical and horizontal probing, adjusting trust scores and filtering out potentially poisonous samples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the system accepts all labeled samples from annotators to improve model accuracy, then the model performance increases, but the system becomes vulnerable to poisoning attacks from untrusted annotators
Solution Approach 1:
The system performs preliminary risk assessment and trust evaluation on annotators before accepting their labeled samples. By calculating risk scores based on annotator history, label quality, and sample characteristics in advance, the system prevents potentially harmful samples from being added to the training set, thus protecting against poisoning attacks while maintaining model accuracy
Solution Approach 2:
The system introduces an intermediary risk assessment module that acts as a mediator between annotators and the model training process. This module evaluates each labeled sample through multiple criteria (annotator trust score, sample risk score, label quality) and selectively accepts or rejects samples, thereby filtering out poisonous inputs while preserving beneficial training data
2Reliability
If the system implements comprehensive risk assessment and probing mechanisms to filter harmful samples, then the system becomes resistant to poisoning attacks, but the system complexity increases
Solution Approach 1:
The system segments the risk assessment process into distinct modular components: annotator evaluation module, sample risk assessment module, label quality verification module, and probing mechanism. Each module handles a specific aspect of the filtering process independently, making the complex system more manageable and maintainable while achieving comprehensive protection against poisoning attacks
Solution Approach 2:
The system dynamically adjusts assessment parameters such as risk thresholds, trust score weights, and probing intensity based on the observed data quality and annotator behavior. This adaptive approach allows the system to maintain high security against poisoning attacks while reducing unnecessary complexity when the risk level is low
3Measurement precision
If the system uses multiple reference models to assess risk scores for labeled samples, then the filtering accuracy improves, but the computational cost increases
Solution Approach 1:
The system employs a tiered reference model approach where not all reference models are applied to every sample. Instead, it uses a subset of models based on the sample's characteristics and the annotator's trust level, performing more comprehensive assessments only when necessary. This partial application of full assessment reduces computational cost while maintaining sufficient risk assessment accuracy
Data Source
AI summary
A computing method receives a labeled sample from an annotator. The method may determine a plurality of reference model risk scores for the first labeled sample, where each reference model risk score corresponds to an amount of risk associated with adding the first labeled sample to a respective reference model of a plurality of reference models. The method may determine an overall risk score for the first labeled sample based on the plurality of reference model risk scores. The method may further determine a probe for confirmation of the first labeled sample and a trust score for the annotator by sending the probe to one or more annotators. In response to determining a trust score for the annotator the method may add the labeled sample to a ground truth or reject the labeled sample.