Neural Network Activation Clustering for Poisoning Attack Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods fail to effectively detect and prevent backdoor poisoning attacks on machine learning models, especially when legitimate training data is unavailable, as they either require extensive retraining or large amounts of legitimate samples, making them infeasible for complex neural networks.
Innovation Solution
A method and system that utilize activation clustering to detect poisonous data points in a training set by recording neuronal activations of a neural network, applying dimensionality reduction, and clustering algorithms to identify and isolate poisonous data, allowing for the detection and removal of backdoors without relying on extensive retraining or large amounts of legitimate data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional methods are used to detect poisoning attacks, then detection capability is provided, but extensive retraining or large amounts of legitimate samples are required
Solution Approach 1:
The patent extracts and analyzes neuronal activation patterns from the trained neural network as a separate diagnostic layer. By taking out the activation patterns from the normal training process and analyzing them independently through clustering, the method detects poisonous data without requiring additional legitimate training data or retraining the model.
Solution Approach 2:
The patent introduces activation clustering as an intermediary detection mechanism between the neural network and the poisoning attack. Instead of directly analyzing model weights or requiring clean data for comparison, the clustering of activation patterns serves as a mediator that reveals anomalies caused by poisonous training samples.
2Reliability
If conventional methods are used to detect poisoning attacks, then detection capability is provided, but extensive retraining is required
Solution Approach 1:
The patent performs preliminary analysis of activation patterns during or after the single training process. By clustering activation patterns beforehand and establishing baseline clusters, the system can detect poisonous data in new batches without requiring time-consuming retraining cycles.
Solution Approach 2:
The method extracts activation patterns as a separate analytical component that can be studied independently of the training process itself. This extraction allows detection through pattern analysis rather than through iterative retraining, significantly reducing the time required.
3Measurement precision
If activation clustering is used to detect poisonous data, then detection accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent segments the detection process into distinct phases: training the neural network once, extracting activation patterns, clustering them by class, and then comparing new activations against established clusters. This segmentation allows the computationally intensive clustering to be performed once per class rather than continuously, reducing overall complexity while maintaining high detection accuracy.
Data Source
AI summary
One embodiment provides a method comprising receiving a training set comprising a plurality of data points, where a neural network is trained as a classifier based on the training set. The method further comprises, for each data point of the training set, classifying the data point with one of a plurality of classification labels using the trained neural network, and recording neuronal activations of a portion of the trained neural network in response to the data point. The method further comprises, for each classification label that a portion of the training set has been classified with, clustering a portion of all recorded neuronal activations that are in response to the portion of the training set, and detecting one or more poisonous data points in the portion of the training set based on the clustering.