Neural Network Internal-State Anomaly Detection for ML Attacks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for protecting Machine Learning (ML) models from attacks require large amounts of data, sophisticated models, and increased computing resources, and are limited in addressing unknown or new threats, especially when detecting anomalies that do not appear at the pixel level.
Innovation Solution
Anomaly detection is performed by examining the internal state of neural network layers rather than input or output data, using a classifier to segregate normal and anomalous data, and taking protective actions if an attack is detected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional anomaly detection methods (pixel-level analysis) are used, then the system can detect obvious anomalies, but it cannot detect complex anomalies that do not manifest at the pixel level (such as contextual anomalies like three eyes or two noses)
Solution Approach 1:
The patent transitions from analyzing data at the pixel level (input dimension) to analyzing data at the feature representation level (internal state dimension). By examining the internal activations of the neural network model, the system gains access to a higher-dimensional space where contextual patterns are explicitly represented, enabling detection of anomalies that are invisible at the pixel level.
Solution Approach 2:
The patent introduces the internal state of the neural network model as an intermediary between the input data and the anomaly detection process. This intermediary (the model's hidden layer activations) serves as a bridge that translates complex input patterns into detectable feature representations, allowing the detection system to understand contextual meaning without requiring domain-specific knowledge.
2Measurement precision
If sophisticated models with deep contextual understanding are built to detect complex anomalies, then detection accuracy improves, but the requirement for data preparation, feature engineering, and computing resources increases considerably
Solution Approach 1:
The patent makes the target model serve itself by using its own internal state for anomaly detection. The model's hidden layer activations, which are already computed during normal operation, are directly utilized for detection purposes. This eliminates the need for separate feature engineering and reduces computational overhead, as the model's own processing is leveraged for both prediction and anomaly detection.
Solution Approach 2:
The patent makes the neural network model multi-functional by using it for both its primary task (e.g., image classification) and anomaly detection. The same model that processes input data for its intended purpose also provides anomaly detection capabilities through its internal state, eliminating the need for separate detection models and reducing overall system complexity.
3Reliability
If adversarial training is used to protect against known attacks, then the model can recognize and categorize labeled anomalous data, but it requires continuous maintenance and cannot address unknown attacks without additional labeled data sets
Solution Approach 1:
The patent inverts the traditional approach by not trying to teach the model what anomalies look like through labeled training data. Instead, it uses the model's own internal behavior as the reference for normality and detects deviations from this behavior. This unsupervised approach naturally adapts to unknown attack patterns without requiring prior knowledge or retraining.
Solution Approach 2:
The patent changes the parameter being monitored from input data characteristics to internal model state characteristics. By monitoring the activations and gradients within the model's hidden layers, the system detects anomalies based on behavioral changes rather than input pattern recognition, enabling adaptation to novel threats without additional training data.
4Measurement precision
If large amounts of training data and sophisticated models are used to build a model of 'what is normal', then detection capability improves, but computing resources and preprocessing requirements increase considerably
Solution Approach 1:
The patent leverages the computational work already performed by the target model during its normal operation. The same forward propagation computations used for prediction also generate the internal state representations needed for anomaly detection, eliminating redundant computations and reducing overall energy consumption.
Solution Approach 2:
The patent extracts anomaly detection capability from the model's internal processing without adding external detection infrastructure. By utilizing the existing hidden layer activations and gradients that are already computed during model inference, the system obtains detection signals without requiring additional data processing or computational resources.
Data Source
AI summary
Disclosed implementations include a method of detecting attacks on Machine Learning (ML) models by applying the concept of anomaly detection based on the internal state of the model being protected. Instead of looking at the input or output data directly, disclosed implementation look at the internal state of the hidden layers of a neural network of the model after processing of data. By examining how different layers within a neural network model are behaving an inference can be made as to whether the data that produced the observed state is anomalous (and thus possibly part of an attack on the model).