Detection of adversarial attacks

Abstract

A method and system for detecting an adversarial attack of an artificial intelligence computational model, comprising the steps: receiving, by a processor, a series of input queries to the computational model; calculating, by the processor, gradient vectors between pairs of successive input queries, wherein the gradient vectors represent the variation between the pairs of successive input queries; determining, by the processor, angles between pairs of successive gradient vectors, wherein the angles represent the similarity between successive variations between pairs of successive input queries; classifying, by a trained artificial neural network, the series of input queries based on the distribution of angles between gradient vectors; and outputting, by the artificial neural network, a determination that the series of input queries are indicative of an attempted adversarial attack on the computational model.

Description

[0001] DETECTION OF ADVERSARIAL ATTACKS

[0002] Field of the Invention

[0003] The present invention relates to the protection of artificial intelligence (Al) models from adversarial attacks. More specifically, it relates to a security mechanism, or framework, for recognising, detecting and identifying attempts to attack Al models.

[0004] Background of the Invention

[0005] The use of artificial intelligence (Al) has become widespread in recent years and is now widely used across most fields of technology. Al models are computational structures that analyse data in order to output to predictions or classifications. There are many different types of Al model, suited to different applications.

[0006] Nefarious attacks on computational systems are common. Adversarial attacks on Al models attempt to fool Al models into making incorrect predictions or classifications and can jeopardise the integrity of the models.

[0007] Traditional approaches to detecting adversarial attacks generally focus on analysing the attacks after they have occurred, i.e. where the adversarial examples have already been generated. There are several defences against adversarial attacks, but they can be broken by more advanced and adaptive attack strategies. The 'white-box' scenario assumes the adversary has access to the model parameters, architecture and gradient. This scenario has been investigated thoroughly to provide a worst-case vulnerability analysis. The 'black-box' scenario corresponds to an adversary with access that is limited to the input / output of the model. In this case, the adversary can generate malicious noise without having access to the model's architecture. This is practical in most real-world situations, especially in the widely used ML as a Service (MLaaS) setting, where the model is hosted in remote servers. In such a setting, the adversary generates adversarial perturbation only by iteratively querying the model and accordingly analysing the output. Many query-based black box attacks have been proposed, showing high efficiency in fooling ML models.

[0008] While existing adversarial defence techniques mainly focus on detecting adversarial perturbation applied to the input example, a few schemes have been introduced to detect an attempt of the attack. Li et al. (2022) (Blacklight) detects adversarial attack attempts by monitoring the similarity between input examples. Blacklight shows good detection performance against query-based black-box attacks. However, it is not robust enough to avoid false positive problems to detect adaptive attacks, andrequires a high computational overhead. Stateful Detection (Chen, Carlini, and Wagner 2020) also monitors input queries.

[0009] It is an aim of the present invention to eliminate, or at least mitigate, deficiencies of the prior art. Summary of the Invention

[0010] Accordingly, there is provided a method for detecting an adversarial attack of an artificial intelligence computational model, comprising the steps: receiving, by a processor, a series of input queries to the computational model; pre-processing, by the processor, the series of input queries to filter out benign input queries; calculating, by the processor, gradient vectors between pairs of successive input queries, wherein the gradient vectors represent the variation between the pairs of successive input queries; determining, by the processor, angles between pairs of successive gradient vectors, wherein the angles represent the similarity between successive variations between pairs of successive input queries; classifying, by a trained artificial neural network, the series of input queries based on the distribution of angles between gradient vectors; and outputting, by the artificial neural network, a determination that the series of input queries are indicative of an attempted adversarial attack on the computational model.

[0011] Preferably, the step of pre-processing comprises detecting irregular batch attacks by removing benign queries injected between malicious queries. This may comprise converting each input query into a binary image. This may comprise rescaling each input query into 32 x 32 image space, wherein each rescaled query is converted into a binary image using Canny edge detection.

[0012] Preferably, the step of pre-processing comprises compressing each row of the binary image into a 4 byte representation by encoding the binary pixel values as bits. The compressed representation of each row may form part of a 1 x 128 vector representing a transformed input query. Each transformed input query may be stored by pushing the transformed input queries into an n depth FIFO buffer. The n depth FIFO buffer maintains a sequence of n most recent transformed queries for subsequent gradient vector calculation.

[0013] The step of pre-processing may comprise calculating a similarity value between an input query and previously stored input queries, and comparing the similarity value to a pre-determined threshold. The similarity value may be calculated by detecting binary pixel-wise overlaps between the input query and previously stored input queries.Preferably, the step of determining the angles between successive vector gradients comprises measuring the cosine similarity between successive gradient vectors in hyper-dimensional space. This may comprise generating a histogram of the cosine similarities, extracting a feature vector from the histogram, and inputting the feature vector into the trained artificial neural network.

[0014] The method may further comprise outputting, by the artificial neural network, a classification of the attempted adversarial attack. Input queries to the computational model may be pre-processed to filter out benign input queries.

[0015] The method preferably further comprises receiving a series of inputs by the processor occurs concurrently with outputting a determination by the artificial neural network that a series of input queries indicate an adversarial attack, such that the attack detection occurs in real-time.

[0016] According to a second aspect of the invention, there is provided a method of detecting a malignant series of inputs to a deep neural network, comprising receiving a series of input queries to the neural network, pre-processing, by the processor, the series of input queries to filter out benign input queries, for each triplet of successive input queries, calculating a cosine similarity, wherein the cosine similarity is a measurement of the similarity between a first variation from a first input query to a second input query and a second variation from the second input query to a third input query; analysing the distribution of calculated cosine similarities to determine a distribution pattern indicative of a malignant series of inputs.

[0017] Preferably, the step of pre-processing comprises detecting irregular batch attacks by removing benign queries injected between malicious queries. The step of pre-processing may comprise wherein the step of pre-processing comprises converting each input query into a binary image. The step of converting may comprise rescaling each input query into 32 x 32 image space, wherein each rescaled query is converted into a binary image using Canny edge detection.

[0018] The step of pre-processing may comprise compressing each row of the binary image into a 4-byte representation by encoding the binary pixel values as bits. The compressed representation of each row may form part of a 1 x 128 vector representing a transformed input query. Each transformed input query may be stored by pushing the transformed input queries into an n-depth FIFO buffer. The n-depth FIFO buffer may maintain a sequence of n most recent transformed queries for subsequent gradient vector calculation.The step of pre-processing may comprise calculating a similarity value between an input query and previously stored input queries, and comparing the similarity value to a pre-determined threshold. The similarity value may be calculated by detecting binary pixel-wise overlaps between the input query and previously stored input queries.

[0019] The malignant series of inputs may be indicative of an adversarial attack against the deep neural network, and preferably different types of attack are indicated by different distributions of cosine similarities.

[0020] The method optionally further comprises modelling the distribution of the calculated cosine similarities between variations in sequential query inputs to the computational model as a histogram. It may further comprise extracting, from a distribution of a predefined number of cosine similarities, a features vector, and inputting the features vector to a classifier configured to detect a malignant series of inputs.

[0021] Preferably, the method further comprises outputting a determination that the series of input queries are indicative of an attempted adversarial attack against the deep neural network.

[0022] According to a third aspect of the invention, there is provided a system for detecting an adversarial attack of an artificial intelligence computational model, comprising a processor and means for executing a trained classifier, wherein the processor is configured to: receive a series of input queries to the computational model; calculate gradient vectors between pairs of successive input queries, wherein the gradient vectors represent the variation between pairs of successive input queries; determining angles between pairs of successive gradient vectors, wherein the angles represent the similarity between successive variations between pairs of successive input queries, wherein the means for executing a trained classifier are configured to: receive the angles and classify the series of input queries based on the distribution of angles between gradient vectors; and output a determination that the series of input queries are indicative of an adversarial attack on the computational model.

[0023] According to a fourth aspect of the invention, there is provided a system for detecting a malignant series of inputs to a deep neural network, comprising a processor configured to receive a series of input queries to the neural network, for each triplet of successive input queries, calculate a cosine similarity, wherein the cosine similarity is a measurement of the similarity between a variation from a first input query to a second input query and a variation from the second input query to a third input query; and analyse the distribution of calculated cosine similarities to determine a distribution pattern indicative of a malignant series of inputs.Preferably, the processor is further configured to execute a trained neural network, wherein the trained neural network is configured to analyse the distribution of calculated cosine similarities and output a determination that the series of inputs are a malicious attack on the deep neural network. The framework of the present invention provides attack classification capability as well as robust attack detection performance. It addresses the issue of adversarial attacks under a black box setting by continuously monitoring the stream of inputs (queries) being made to an Al model to detect the process of an adversary probing the system to construct an adversarial query, so as to detect and identify potential attempts to attack an Al model in real-time (i.e. while the attack is 'in-progress'). The framework described herein detects if an adversarial noise instance is being generated. Rather than know techniques that detect adversarial noise generation by observing the input space, the approach of the present invention learns adversarial patterns in the input update similarity space.

[0024] The approach described herein uses advanced pattern recognition to detect adaptive attacks. It learns from varying attack strategies and updates its model accordingly, making it robust against attempts to evade detection. As will be described, the a 'Delta Similarity' (DS) metric more efficiently indicates or 'captures' adversarial behaviour. When applying the framework of the invention, an adversary who hides the malicious iterations within benign or random queries to evade detection needs an excessively high number of queries (which can be easily limited in a practical setting) in order to evade detection and bypass the defence provided by the invention.

[0025] The analysis metric used (similarity in the hyper-dimensional space) makes the system of the described invention more precise in distinguishing positives (attacks) from benign queries.

[0026] Moreover, the present framework results in negligible overhead in terms of computation, requiring analysing around 200 input queries (compared to the average number of queries necessary for an attack of over 10,000 queries).

[0027] Brief description of the figures

[0028] Figure 1 is diagram of the difference in benign and malign queries;

[0029] Figure 2 is an illustration comparing, for three difference computational models, a benign query sequence and six known adversarial query sequences;

[0030] Figure 3 is an illustration of attack detection according to an embodiment of the invention;

[0031] Figure 4 is a diagram of a pre-processing methodology to screen benign queries;

[0032] Figure 5 shows a distribution and histogram representation of the Delta Similarity metric;Figure 6 shows a series of confusion matrices of GWAD attack classification performances over validation histogram of Delta Similarity feature sets:

[0033] Figure 7 is a series of graphs showing the number of queries required to achieve successful attacks as benign example injection rate grows for HSJA, NES and Sign-Opt, respectively, and GWAD attack detection accuracy;

[0034] Figure 8 shows ASR of adaptive attacks and detection accuracy of GWAD as the upper bound rp of p variation in random distributions grows: (a) Adaptive-HSJA; (b) Adaptive-NES

[0035] Figure 9 illustrates pre-process performance over irregular batch attacks based on HSJA with various batch sizes.

[0036] Detailed description

[0037] Overview

[0038] The detection framework described herein utilises Gradient Vector Analysis to analyse the gradient vectors of successive queries to the protected model. In Al, a gradient vector represents the direction and rate of change that minimizes or maximizes the model's loss function. By examining the angles between these vectors, the proposed system can distinguish between benign queries and potential adversarial activity.

[0039] The Artificial Neural Network (ANN) model according to the detection framework of the invention is trained to recognize the difference between benign and malicious query patterns based on the distribution of the angles between the vectors in a high-dimensional space. A benign query would typically cause gradient changes in predictable patterns, whereas an adversarial attack would exhibit irregular or unusual gradient patterns.

[0040] Attackers may try to camouflage their malicious queries with benign ones. The proposed approach uses advanced pattern recognition to detect these adaptive attacks. It learns from varying attack strategies and updates its model accordingly, making it robust against attempts to evade detection. The approach described herein shifts the emphasis from defending against pre-generated attacks to actively identifying ongoing adversarial perturbation generation in real time. Underpinning the approach is the assumption that query-based attack methods commonly utilise a variation of zeroth-order optimisation to estimate gradients and, therefore, exhibit unique patterns in their attack processes, which are highly distinguishable from benign queries. The zeroth-order optimisation estimates a gradient using multiple pairs of input queries and output observations. In this process,intermediate states are incrementally explored towards the generation of efficient adversarial perturbation. In contrast, benign queries to the model do not have the same pattern.

[0041] Figure 1 illustrates the assumption, or intuition, that query-based attack methods commonly utilise a variation of zeroth-order optimisation to estimate gradients and, therefore, exhibit unique patterns in their attack processes, which could be highly distinguishable from benign queries if the right metrics are used for monitoring.

[0042] To capture this behavioural difference, an indicator, Delta Similarity (DS), represents the relationship between examples in the sequence of queries through time. DS measures the similarity between input updates, rather than analysing the input examples themselves. This captures better the attack's behaviour regardless of the type of input examples and models.

[0043] DS is used to analyse the updates through time in a sequence of queries instead of focusing on the input examples. As will be illustrated, DS is a reliable indicator of query-based adversarial attack behaviour. A simple ANN based on DS and shows high accuracy in adversarial attack detection, and is dataset- and model-agnostic.

[0044] Delta Similarity (DS)

[0045] Query-based black-box adversarial attack methods commonly carry out zeroth-order optimisations to estimate gradients, which consumes the majority of their attack queries. In the zeroth-order optimisation processes, query-based attack methods continuously update input examples and observe corresponding responses from the target model. However, the framework of the present invention is based in part on the assumption that such updates are unlikely to exist in benign queries. Rather, each attack method exhibits a specific pattern in how it applies updates, allowing it to be distinguished from benign queries via a classifier. Delta Similarity (DS) quantifies the relationship between variations in the input examples to a Deep Neural Network (DNN).

[0046] To distinguish between benign and malicious behaviour, the framework of the present invention analyses the differences between queries at the input space. DS quantifies such behaviour, and is defined as:Definition 1. (Delta Similarity) Let

[0047] {. / ■.. 2 i

[0048]

[0049] be three queries to a model

[0050] F. Let δ be the variation between two queries,

[0051] which we refer to as the update: is calculated by:

[0052]

[0053] We define DS as the cosine similarity between updates δi−1 and δi; DS is expressed by:

[0054]

[0055] In query-based black-box attack settings, the iteration is a process to estimate a gradient which is used to optimise the current intermediate adversarial example Xt towards maximizing the loss function of the victim Xt model. Hence,

[0056]

[0057] is not updated to until the gradient estimation step is completed. We note a set of queries to a target model in a gradient estimation step can be defined as follows:

[0058] {

[0059]

[0060] xt+ u{, Xt-h Ut+ 1, Xt + ■?4+2,.... Xt + (3> where p. is the random updates, which consist of d-dimensional random vectors sampled from a normal distribution AuO. cr2) and n indicates the last query to the target model in the iteration. The variations 6 between examples are found by applying Equation 1 to the examples as: d

[0061]

[0062] j {Xt+ Ut:!) •- (; L + *. t / j (4)

[0063] As shown in Equation 4, the variation

[0064]

[0065] and are expressed as a linear operation between random vectors

[0066]

[0067] uj,;and ■ Due to the concentration of measure phenomenon (Ladoux 2001), these random vectors are expected to be orthogonal in the hyperdimensional space and to have equal length with a very high probability (Bandeira 2020). An empirical study to verify this hypothesis is discussed below.

[0068] An empirical analysis of DS has been conducted with the various sequences of queries from benign examples as well as known black-box attacks. The analysis was carried out in the three settings, and each setting has a dedicated image classification dataset along with a target model. Three standard image classification datasets are utilised for the analysis: CIFAR-10, Tiny-lmageNet and ImageNet. Three DNN models are also chosen for the target models: ResNet-18, Efficientnet, and VGG-16 respectively. The results are shown in Figure 2.

[0069] For the DS distribution of the benign queries, DS was calculated by injecting 5K benign examples from the test split of a dedicated dataset into the model. In contrast, the DS distribution of attack queries isacquired by performing a query-based atack over an arbitrary example from the dataset with the same 5K query budget. The DS distributions ploted in Figure 2 suggest that the query-based black-box adversarial atack methods leave unique traces in the DS space during their atack processes. While the DS of benign examples are distributed with a high variance, the DS distribution of the atacks have one or more primary components. For example, NES (Ilyas et al. 2018) exhibit a strong DS component at -0.7071, and HSJA (Chen, Jordan, and Wainwright 2020) shows most of DS at -0.5 and ±1.0. The paterns of adversarial atacks' DS distributions are highly distinguishable from the ones in a sequence of queries of benign examples. This supports the assumption that these distributions are amenable to classification.

[0070] Defence Framework

[0071] As discussed above, the behaviour of query-based black-box adversarial atack methods can be characterised by the paterns in DS distributions. Herein described is a framework to identify ongoing adversarial generation atempts and detect adversarial examples in the making. The detection framework is illustrated in Figure 3; it consists of continuously monitoring the DS of input queries and analysing the DS distribution using an ML model to detect if the protected system is under atack. Optional input Pre-processing: Figure 4 provides an illustrative overview of the pre-processing to filter input queries that will be analysed according to the present invention ('GWAD queue' queries). The pre-processing screens out unsuspicious queries which are deliberately injected by the irregular batch atacks. It represents an image x with the 128-Byte vector. Image x is first converted to 32 x 32 grayscale image, then transformed into a binary image by Canny edge detection algorithm (Canny 1986). Each pixel of the binary image is represented with bit 0 or 1, and finally forms a 128-Byte vector. The similarity between two images is scored by the rate of mismatching pixels which is simply calculated with the bit-wise xor of represented vectors and the number of bit 1 in the result, as described in Equation 5 below. The pre-processing pushes extracted vector into the n-depth FIFO, and runs the algorithm described above to detect binary pixel-wise overlaps between input image and those in the FIFO. The pre-processing passes queries which have sufficient overlap to undergo Delta Similarity analysis (discussed below) process whilst rejects others.

[0072] Features for Attack Classification: The simplicity of DS computation allows the framework of the present invention to continuously calculate DS at every query. Moreover, as discussed above, DS distributions are highly distinguishable. Therefore, a set of calculated DS is directly captured and a histogram representation is applied to generate a formalised feature to address the atack classification problem as illustrated in Figure 5 (" Histogram of DS (HoDS)").For every triplet of queries, a DS metric is calculated. A histogram of the last 256 DS is generated, which is empirically chosen for the best prediction. DS is bounded by -1.0 < DS < 1.0. The HoDS has 200 equally spaced bins over this range, with an extra bin for DS = 1.0. The strengths of the histogram vary over the sequence of queries, hence, a local contrast normalisation is necessary. Min-max normalisation is adopted for this purpose. Finally, 1 x 201 HoDS feature vector is generated.

[0073] Attack Classifier: In the proposed framework, the extracted HoDS features vector is fed to a classifier to detect attack queries. A neural network is trained to classify attack queries. The network consists of six fully-connected hidden layers with ReLU activation function. The classification probabilities are provided by a Log-Softmax output layer. The network architecture is intuitively chosen in order to evaluate the effectiveness of HoDS features for the query-based black-box adversarial attack classification problem. The network is trained for 100 epochs with Stochastic Gradient Descent (SGD) optimizer and a batch size of 128.

[0074] Once an attack or an attempted attack is detected, the provider of the attacked model is notified immediately. The provider of the model may choose to limit the number of queries for one or more specific clients and / or for a specific service, for a predetermined amount of time.

[0075] Experiments

[0076] To evaluate the performance of the proposed framework, experiments were conducted in three phases: i) the generalisation capacity between different image classification problems and also their dedicated DNN models; ii) the attack detection performance including the false positive rate under various practical use cases; iii) the robustness to possible adaptive attack scenarios.

[0077] Experimental Setup: the detection framework of the invention described was applied to the scenarios where DNN models for image classification problems are under attack from query-based black-box attack methods. To fully explore the effectiveness of the framework in attack detection, the experiments were conducted with a wide range of input examples and DNN model architectures. To evaluate the effectiveness of the detection framework, two standard image classification datasets were adopted: CIFAR-10 (Krizhevsky and Hinton 2009) and ImageNet (Deng et al.2009). Three practical uses cases where also considered and applied representative datasets as: Hollywood Heads Dataset (Vu, Osokin, and Laptev 2015), FLIR ADAS (FLIR 2024), BIRDSAI (Bondi et al. 2020). DNN models utilised in the experiments are also listed in Table 1.Dataset Model Accuracy Usage

[0078] CIFAR-10 ResNet-18 93.07% Train / Test

[0079] MobileNet-V2 93.90% Evaluation

[0080] VGG-16 78.98% Train / Test

[0081] ImageNet

[0082] EflicientNet-b3 83.58% Evaluation

[0083] Table 1: DNN Models’ accuracy. Some DNNs are used

[0084] to train GWAD while others are used for evaluation

[0085] Attack Method | ASR

[0086]

[0087] NES(Ilyas et al. 2018) 98.40%

[0088] SimBA(Guo et al. 2019) 97.60%

[0089] HSJA(Chen, Jordan, and Wainwright 2020) 99.50%

[0090] Sign-Opt(Cheng et al. 2020) 99.30%

[0091] BA(Brendel, Rauber, and Bethge 2018) 50.90%

[0092] Sign-Flip(Chen et al. 2020) 95.70%

[0093]

[0094] Table 2: Query-based Black-box Adversarial Attack

[0095] Methods and Baseline ASR (untargeted) on MobileNet- V2 trained on CIFAR-10 dataset

[0096] Atack detection performance was evaluated while the six known query-based black-box atack method listed in Table 2 atacked target models. For the scenario of atacks on CIFAR-1O and ImageNet image classification tasks, 200 images were randomly selected from the test split, and atack methods were configured to conduct each atack with the 5259 query budget, which allows the framework to monitor 5K queries during an atack. The ratio of perturbation added by the atack methods to achieve a successful atack is calculated as p =

[0097]

[0098] 11.27 ~~ ct.’ / 11 H 2, where x is an adversarial example computed by the atack (Park, McLaughlin, and Miller 2024). The atack is limited with p = 0.1 and this is the perturbation budget of the atack. Any successful atacks which exceed the budgets were considered as a failed atack. The atacks were generally configured according to the settings described in their original papers. However, some atack methods were further optimised if necessary.

[0099] Generalisation Property Evaluation: The detection framework's performance was evaluated across different datasets, i.e., to determine that, if the framework's classifier had been trained on DS distribution of one dataset, whether it will generalise to detect atacks on other datasets. HoDS features were extracted to train the framework's classifier on 2 datasets: CIFAR-10 and ImageNet. For simplicity, these trained classifiers are referred to as DF-CIFAR10 and DF-ImageNet.

[0100] DF-CIFAR10 and DF-ImageNet were evaluated based on settings described in table 1, i.e., DF-CIFAR10 and DF-ImageNet were evaluated, each, on atack detection to protect victim models inferring both CIFAR-10 and Imagenet. The experimental results are displayed in Figure 6, which is a confusion matrix of the detection framework (DF) atack classification performances over validation HoDS feature sets: (a) and (b) show DF-CIFAR10 performance over CIFAR-10, and ImageNet, respectively; (c) and (d) show DF-ImageNet performance over CIFAR-10, and ImageNet, respectively. The results show highgeneralisation capacity across the datasets. For example, DF-CIFAR10 classifies CIFAR-10 validation feature set with 98.22% accuracy, but also achieves 97.32% classification accuracy over ImageNet validation set. Similarly, DF-ImageNet, achieves 98.94% classification accuracy on ImageNet and achieves 86.36% classification accuracy on CIFAR-10.

[0101] Sensitivity Evaluation: The detection framework's attack detection performance was evaluated by conducting each of the six query-based black-box attacks against target models: EfficientNet for ImageNet and Mobile Net- 2 for CIFAR-10 classification task. In every instance, each attack method selected a random example from the dataset, and performed the untargeted attack until the query budget is reached. The detection framework monitored the sequence of the attack queries and extracted HoDS at 500 random check points per attack. The test results for all attacks are quantified in table 3. It can be seen that while the framework generally performs high classification accuracy, it fails to classify attack queries from Sign-Opt attack on ImageNet task, and SimBA attack on CIFAR-10 task. The common primary components, which exist in DS distributions of different attack methods, are considered to cause the degraded classification performance of the framework. For example, SimBA and Sign-Flip attacks on CIFAR-10 task exhibit the common primary DS components at 0 and -1. In contrast, the detection framework performs near perfect attack detection performance. We see that the framework correctly detects over 99.42% attack queries. Moreover, the framework detects all the attack queries from many attacks such as HSJA, NES and SimBA attacks on ImageNet, and NES, SimBA, Sign-Opt and Sign-Flip attacks on CIFAR-10.

[0102]

[0103] ImageNet | CIFAR-10

[0104] Predict Detect Predict Detect

[0105] HSJA 90.71% 100.00% 98.12% 99.99%

[0106] NES 100.00% 100.00% 100.00% 100.00%

[0107] SimBA 100.00% 100.00%’ 5.60% 100.00%

[0108] Sign-Opt 62.53% 99.92% 98.34% 100.00%

[0109] Sign-Flip 93.68% 99.42% 99.26% 100.00%

[0110] BA 88.17% 99.70% 99.52% 99.56%

[0111]

[0112] Table 3: GWAD sensitivity to the sequences of 100K attack queries against EfficientNet trained on ImageNet and

[0113] MobileNet-v2 trained on CIFAR-10.

[0114] Sequence Queries FPR

[0115] MNIST 100000 00.00%

[0116] CIFAR-10 100000 00.00%

[0117] ImageNet 100000 00.03%

[0118] FLIR ADAS RGB 10061 00.00%

[0119] BIRDS Al Real 40404 01.29%

[0120] Table 4: GWAD False Positive on the sequence of queries

[0121] with benign examples which represent practical use casesSpecificity Evaluation: In a further experiment, the detection framework was evaluated over the sequence of queries with various benign examples which represent practical use cases, and the false positive rate of the framework was measured. Six publicly available datasets which represent specific real-world scenarios were used.

[0122] A first test was conducted with image classification tasks, such as MINST, CIFAR-10 and ImageNet, which likely show very low similarity between their images. Each task was shuffled and generates 100K images sequence which were monitored by the framework. The framework does not react to the sequence of images from MNIST and CIFAR-10, whereas 0.03% of ImageNet images in the sequence triggers false alarm of the detection framework.

[0123] Urban Scene images FLIR ADAS dataset (FLIR 2024) is specifically designed for autonomous vehicles. It consists of urban scene images in both RBG and IR spaces. As images are continuously captured while a vehicle is moving in urban area, FLIR ADAS dataset includes series of images which have relatively high similarity between them. GWAD monitors the test sequence and achieved 0.0% false positive rate (FPR).

[0124] Bird's eye images BIRDSAI dataset (Bondi et al. 2020) is an infra-red image dataset specifically designed for surveillance systems with the Aerial Intelligence. The dataset provides 40404 real night-time images of animals and humans captured from airborne vehicles. Series of Images captured especially from a high altitude show high similarity, and 1.29% of images in the sequence triggered false alarms from the framework.

[0125] The framework was compared with Blacklight (Li et al. 2022) using a Hollywood Heads dataset which represents the sequence of queries with single class examples. Along with an ordinary split with shuffled examples, the dataset especially provides a small subset which includes human head examples annotated from the sequential movie frames as per the image below, showing low similarity images (top row); medium similarity images (middle row); and high similarity images (bottom row).

[0126]

[0127] ■■■■■■■■

[0128]

[0129] Therefore, the similarity between neighbouring images is high. As shown in Table 5, the detection framework shows much less FPR compared to Blacklight on the sequence of highly similar single class images. GWAD achieves 17.99% FPR, whereas Blacklight performs 62.32% FPR. (Blacklight operates by first running a quantisation process on a new query to convert into a small vector space, wherededicate S x 32 bytes hash values are computed. It then compares the hash values to the dictionary where the previous hash values are stored to with their corresponding query ID. Blacklight uses the matched-up hash values to check the similarity between queries, and pushes others into the dictionary for later use. If a query ID in the dictionary has matched up hash values more than a configured threshold, Blacklight indicates that the current query is from the attack.)

[0130] Image Queries FPR (%)

[0131] Similarity GWAD Blacklight(Li et al. 2022)

[0132] Low 9742 00.00 05.33

[0133] Medium 9510 00.00 03.05

[0134]

[0135] High 1425 17.99 62.32

[0136] Table 5: False Positive comparison: GWAD and Blacklight(Li et al. 2022) monitor the sequence of similar images from Hollywood head dataset

[0137] Adaptive Attacks

[0138] The framework was also evaluated under strong adversaries to investigate its limits. One strong requirement on the defence methods is to withstand adaptive attacks with full knowledge of the defence strategy, i.e., the DS-based approach and the framework implementation. With this consideration, two adaptive attacks were proposed: i) Benign example injection, and ii) Varying parameters of random distribution in the zeroth-order optimisation.

[0139] Benign Example Injection: An attacker with prior knowledge of the framework may inject benign examples between malicious (attack) queries to influence the DS distribution and evade detection. To evaluate the robustness of the framework under this scenario, the rate of benign example injections was used and the rate was varied incrementally to explore the limits of our defence. CIFAR-10 was used, and adaptive attacks were implemented while varying the rates of benign example injections until the attack totally evades detection. Figure 7 (d) displays the evolution of framework attack detection accuracy in this setting. Interestingly, the framework shows the persistent attack detection performance in the range of the rate of benign example injections (rb) < 150% achieving perfect or near perfect detection accuracy against all the attack methods, and gradually degraded. This means that the attacker needs over 2.5x the number of queries to start possibly evading detection.

[0140] Figure 7 (a), (b) and (c) shows the average number of queries to achieve successful attacks. The required number of queries to generate successful attacks increases linearly with rb. For example, Sign-Opt requires 2221 queries at rb = 0%, and it is increased to 7095 queries at rb = 250%. (An attack being successfully generated in Figure 7 (a), (b) and (c) does not mean that it evaded detection).Moving target attack: Here, an atacker that implements a "moving-target" strategy to evade detection (while still generating adversarial examples while querying the victim model) was considered. The idea is to manipulate the parameters of the noise distribution from which the atack samples the noise. Noise vectors utused in zeroth-order optimisation need to be the zero-mean random distributions with a common variance such as N (0, 1) to guarantee acceptable quality of atacks (QoA). However, one adaptive atack, with full knowledge of the proposed detection scheme, may introduce a variation in the random distributions by varying p or o2, with no serious consideration of QoA. To simulate such adaptive atacks, HSJA and NES performed the adaptive atacks to MobileNet-V2 trained on CIFAR-10 dataset.

[0141] Atacks were initially conducted with varying variance of random distributions. In this setting, generated random noises are scaled by a factor a, where 0 < a < ra. While the atack performance of NES gradually degraded, the detection framework maintains the atack detection performance showing 100% accuracy against varying variance adaptive atacks across all the variations in o2. Also considered was the scenario where the adversary varies to the mean p of random noise vectors. The atack first finds the range of pixel intensity for an input x as s = max(x) - min(x), and sets the bounds of variation as a ratio r_μ of s. We also display the impacts on ASR caused by the adaptive atack setting. The experimental results are displayed in Figure 8. The ASR of the atacks are gradually decreased as the rate of variation grows. The adaptive atacks with HSJA and NES show only 61.54% and 10.2% ASRs respectively at r = 0.30. In contrast, the detection framework maintains the robustness in binary classification performance until r = 0.24 achieving 100% accuracy against both atacks.

[0142] Enhancement for Batch attack: As presented in Figure 8, the framework is capable of detecting irregular batch atacks, which inject a set of benign queries between atack queries, when rb < 150%. However, this limitation can be easily addressed by integrating a simple pre-processing to screen out non-suspicious queries, as discussed above. For this purpose, input pre-processing was adopted, which transforms incoming query into 1 x 128 vector e0through the steps: i) rescale query to 32 x 32 image space, ii) convert to binary image with Canny edge detection (Canny 1986), iii) compress each row to 4-Byte by representing binary pixels to bits. Finally transformed query is pushed into n-depth FIFO (n previous queries). The similarity between input query eo and queries e, in the FIFO is calculated as:

[0143] f 1 if < 0

[0144] mi = I ' (5)

[0145]

[0146] I 0. otherwise,

[0147] where i = {1, 2,.., n}, ® is bit-wise xor, is the number of bit 1, and 9 is a predefined threshold.The performance of the pre-processing is shown in Figure 9. Irregular batch attacks based on HSJA conduct 100 attacks on CIFAR-10 Mobile-Net with various batch sizes. Regardless batch sizes, the benign queries are perfectly screened out at 100% accuracy, and over 99.5% of attack queries are fed into GWAD for the precise attack detection. Although FIFO depth is set to n = 100 in this experiment, it is believed that the depth is not limited.

Claims

Claims1. A method for detecting an adversarial attack of an artificial intelligence computational model, comprising the steps:receiving, by a processor, a series of input queries to the computational model;pre-processing, by the processor, the series of input queries to filter out benign input queries; calculating, by the processor, gradient vectors between pairs of successive input queries, wherein the gradient vectors represent the variation between the pairs of successive input queries;determining, by the processor, angles between pairs of successive gradient vectors, wherein the angles represent the similarity between successive variations between pairs of successive input queries;classifying, by a trained artificial neural network, the series of input queries based on the distribution of angles between gradient vectors; andoutputting, by the artificial neural network, a determination that the series of input queries are indicative of an attempted adversarial attack on the computational model.

2. The method of claim 1, wherein the step of pre-processing comprises detecting irregular batch attacks by removing benign queries injected between malicious queries.

3. The method of claim 1 or claim 2, wherein the step of pre-processing comprises converting each input query into a binary image.

4. The method of claim 3, wherein the step of converting comprises rescaling each input query into 32 x 32 image space, wherein each rescaled query is converted into a binary image using Canny edge detection.

5. The method of claim 3 or claim 4, wherein the step of pre-processing comprises compressing each row of the binary image into a 4-byte representation by encoding the binary pixel values as bits.

6. The method of claim 5, wherein the compressed representation of each row forms part of a 1 x 128 vector representing a transformed input query.

7. The method of claim 6, wherein each transformed input query is stored by pushing the transformed input queries into an n-depth FIFO buffer.

8. The method of claim 7, wherein the n-depth FIFO buffer maintains a sequence of n most recent transformed queries for subsequent gradient vector calculation.

9. The method of any of claim 8, wherein the step of pre-processing comprises calculating a similarity value between an input query and previously stored input queries, and comparing the similarity value to a pre-determined threshold.

10. The method of claim 9, wherein the similarity value is calculated by detecting binary pixel-wise overlaps between the input query and previously stored input queries.

11. The method of claim any preceding claim, wherein the step of determining the angles between successive vector gradients comprises measuring the cosine similarity between successive gradient vectors in hyper-dimensional space.

12. The method of claim 11, further comprisinggenerating a histogram of the cosine similarities,extracting a feature vector from the histogram, andinputting the feature vector into the trained artificial neural network.

13. The method of any preceding claim, further comprising outputting, by the artificial neural network, a classification of the attempted adversarial attack.

14. The method of any preceding claim, wherein receiving a series of inputs by the processor occurs concurrently with outputting a determination by the artificial neural network that a series of input queries indicate an adversarial attack, such that the attack detection occurs in real-time.

15. A method of detecting a malignant series of inputs to a deep neural network, comprising: receiving a series of input queries to the neural network,pre-processing, by the processor, the series of input queries to filter out benign input queries; for each triplet of successive input queries, calculating a cosine similarity, wherein the cosine similarity is a measurement of the similarity between a first variation from a first input query to a second input query and a second variation from the second input query to a third input query;analysing the distribution of calculated cosine similarities to determine a distribution pattern indicative of a malignant series of inputs.

16. The method of claim 15, wherein the step of pre-processing comprises detecting irregular batch attacks by removing benign queries injected between malicious queries.

17. The method of claim 15 or claim 16, wherein the step of pre-processing comprises converting each input query into a binary image.

18. The method of claim 17, wherein the step of converting comprises rescaling each input query into 32 x 32 image space, wherein each rescaled query is converted into a binary image using Canny edge detection.

19. The method of claim 17 or claim 18, wherein the step of pre-processing comprises compressing each row of the binary image into a 4-byte representation by encoding the binary pixel values as bits.

20. The method of claim 19, wherein the compressed representation of each row forms part of a 1 x 128 vector representing a transformed input query.

21. The method of claim 20, wherein each transformed input query is stored by pushing the transformed input queries into an n-depth FIFO buffer.

22. The method of claim 21, wherein the n-depth FIFO buffer maintains a sequence of n most recent transformed queries for subsequent gradient vector calculation.

23. The method of any of claim 22, wherein the step of pre-processing comprises calculating a similarity value between an input query and previously stored input queries, and comparing the similarity value to a pre-determined threshold.

24. The method of claim 23, wherein the similarity value is calculated by detecting binary pixelwise overlaps between the input query and previously stored input queries.

25. The method of claim any preceding claim, wherein a malignant series of inputs is indicative of an adversarial attack against the deep neural network, and wherein different types of attack are indicated by different distributions of cosine similarities.

26. The method of claim 25, further comprising modelling the distribution of the calculated cosine similarities between variations in sequential query inputs to the computational model as a histogram.

27. The method of claim 25 or 26, further comprising extracting, from a distribution of a predefined number of cosine similarities, a features vector, and inputting the features vector to a classifier configured to detect a malignant series of inputs.

28. The method any of claims 25 to 27, further comprising outputting a determination that the series of input queries are indicative of an attempted adversarial attack against the deep neural network.

29. A computer readable medium configured to store executable instructions which, when executed by a processor, carry out the method of any of claims 1 to 28.

30. A system for detecting an adversarial attack of an artificial intelligence computational model, comprising a processor and means for executing a trained classifier,wherein the processor is configured to:receive a series of input queries to the computational model;pre-process the series of input queries to filter out benign input queries; calculate gradient vectors between pairs of successive input queries, wherein the gradient vectors represent the variation between pairs of successive input queries;determining angles between pairs of successive gradient vectors, wherein the angles represent the similarity between successive variations between pairs of successive input queries,wherein the means for executing a trained classifier are configured to: receive the angles and classify the series of input queries based on the distribution of angles between gradient vectors; andoutput a determination that the series of input queries are indicative of an adversarial attack on the computational model.

31. A system for detecting a malignant series of inputs to a deep neural network, comprising a processor configured toreceive a series of input queries to the neural network,pre-process the series of input queries to filter out benign input queries;for each triplet of successive input queries, calculate a cosine similarity, wherein the cosine similarity is a measurement of the similarity between a variation from a first input query to a second input query and a variation from the second input query to a third input query; and analyse the distribution of calculated cosine similarities to determine a distribution pattern indicative of a malignant series of inputs.

32. The system of claim 31, wherein the processor is further configured to execute a trained neural network, wherein the trained neural network is configured to analyse the distribution of calculated cosine similarities and output a determination that the series of inputs are a malicious attack on the deep neural network.

33. The system of claim 30 or 31, configured to carry out the method of any one of claims 1 to 28.

Images