Computer-implemented method and determination device for determining whether a sample of interest was used to train a specific image captioning model

Abstract

A computer-implemented method for determining whether a sample of interest was used to train a specific image captioning model.

Description

[0001] COMPUTER-IMPLEMENTED METHOD AND DETERMINATION DEVICE FOR DETERMINING WHETHER A SAMPLE OF INTEREST WAS USED TO TRAIN A SPECIFIC IMAGE CAPTIONING MODEL

[0002] TECHNICAL FIELD

[0003] The present invention is directed at a computer-implemented method for determining whether a sample of interest was used to train a specific image captioning model. The present invention is further directed at a corresponding determination device.

[0004] BACKGROUND ART

[0005] Image captioning models have become essential digital task assistance tools which can provide a text caption to an input image and / or video. The training of such models relies heavily on the collection of vast amounts of data, which may include copyright protected or sensitive information (such as of personally identifiable information, PH). It is therefore desirable to provide a tool for reliably determining whether a sample of interest was used to train a specific image captioning model.

[0006] An objective of the present invention is the provision of an improved determination whether a sample of interest was used to train a specific image captioning model.

[0007] SUMMARY OF THE INVENTION

[0008] According to a first aspect, a computer-implemented method for determining whether a sample of interest was used to train a specific image captioning model is provided. The method comprises: (51) inputting a selected sample into the specific image captioning model to generate, by the specific image captioning model, a generated caption describing the selected sample;

[0009] (52) using a pretrained embedding model, encoding the generated caption into a feature vector including m embedding features, m being an integer and m>2;

[0010] (53) repeating the steps S1 and S2 n times, n being an integer and n>2, such as to obtain n generated captions and n feature vectors for the same selected sample;

[0011] (54) calculating a variance vector having m components, each component representing a variance of an i-th feature of the m features across the n feature vectors, i being an integer and 1<i<m;

[0012] (55) calculating a mean variance which is a mean of the m components of the variance vector; wherein either

[0013] (i) the sample of interest is used as the selected sample in steps S1 to S5 and wherein the method further includes a step S6 of determining whether the sample of interest was used to train the specific image captioning model based on the calculated mean variance; and / or

[0014] (ii) the steps S1 to S5 are performed for multiple non-members forming the selected sample and for multiple unknown members forming the selected sample, wherein the non-members are samples for which it is known that they were not used to train the specific image captioning model and wherein the unknown members are samples having unknown membership, wherein the method further includes:

[0015] (57) determining a member threshold value from a meannoof the mean variances MVnoof the multiple non-members;

[0016] (58) comparing the mean variance of each unknown sample with the member threshold value;

[0017] (59) determining whether each unknown sample is a pseudo-member based on the result of the comparison in step S8;

[0018] (510) training a classifier based on training data including:

[0019] the non-members and / or features of the non-members determined using an image encoder, as well as their corresponding label indicating their non-membership, and the pseudo-members and / or features of the pseudo-members determined using the image encoder, as well as their corresponding label indicating their pseudomembership;

[0020] wherein the classifier is for determining whether the sample of interest input into the classifier was used to train the specific image captioning model; and

[0021] (511 ) inputting the sample of interest into the classifier to determine whether the sample of interest was used to train the specific image captioning model.

[0022] Recent studies have proposed MIAs (membership interference attacks) that do not rely on shadow models. Specifically, Ko et al. (“Practical membership interference attacks against large-scale multimodal models: a pilot study” by M. Ko, M. Jin, C. Wang and R. Jia, in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2003) introduced the cosine similarity (CS) attack (CSA) and the weakly supervised attack (WSA). These attacks target large-scale multi-modal contrastive language-image pretraining (CLIP) models that utilize text-image data pairs. Both methods use a CS metric to measure the similarity between the embedding vectors of images and text from the target CLIP model. The second method, WSA, combines ground-truth data about non-members with a pseudo-member set, which consists of samples flagged as likely members based on their CS values. A classifier is then trained on this combined dataset to distinguish members from non-members.

[0023] Herein, two MIAs targeting image captioning models are introduced: the means-of-variance (MV) threshold attack (MVTA) and the confidence-based weakly supervised attack (C-WSA). In contrast to the techniques of Ko et al., we tailor our methods specifically for image captioning models and enhance them with a new metric, the MV metric, which measures the variance between vectors. The methods described herein use only images as input, operating under the realistic assumption that an adversary lacks access to ground-truth reference text (i.e., captions) and instead uses captionsgenerated by the target image captioning model. Additionally, a confidence threshold training method is integrated that leverages the pseudo-dataset’s confidence to improve the classifier’s performance.

[0024] To summarize, the main contributions of the solution as claimed are as follows:

[0025] New MIAs targeting image captioning models: The MVTA and C-WSA are introduced for the image captioning domain, two new attacks in which the input consists solely of images. The means-of-variance metric: A quantitative metric is introduced which is designed to assess the variance between vectors in a multi-dimensional space. While we used it for texts, this metric is general and can be used in various fields that utilize embedding vectors.

[0026] Innovative training of the pseudo-member set: A new step is introduced in the process of training the attack model, a confidence-based training step that utilizes a pseudo-member set. Real-world application: By evaluating the methods described herein on Hugging Face models, the practicality and effectiveness of said methods in real-world scenarios is demonstrated.

[0027] The method of the first aspect allows accurately determining whether the sample of interest was used to train the specific image captioning model. This can give the owner of the sample of interest a warranty that his sample was not misused by unallowably training the specific image captioning model therewith. The privacy and secrecy of the sample of interest can thereby be ensured, and violations of privacy and intellectual property can be detected. The method of the first aspect is computer-implemented, meaning that is can conveniently be performed in an automatized manner.

[0028] As used herein, the expression “computer-implemented method” indicates that the method is realized using a computer, a computer network or another programmable apparatus, wherein the method is in particular implemented by executing a computer program on such a computer, computer network or other programmable apparatus.

[0029] The specific image captioning model can be a machine learning system designed to generate detailed and contextually accurate descriptions for images input into the specific image captioning model. This model typically employs an autoregressive approach, where it predicts the next word in a caption sequence based on the image and the previously generated words. Preferably, the entity training the specific image captioning is different to the entity owning the sample of interest. In particular, the entity owning the sample of interest does not know which samples of interest the specific image captioning model was trained on.

[0030] The sample of interest can include an image or a video of any format and type. The sample of interest is preferably of the same type as the samples used to train the specific image captioning model. As used herein, the term “sample” designates documents of the same format as the sample of interest.The method of the first aspect can include two alternatives or options, labelled (i) and (ii), which both rely on method steps S1 to S5. The two alternatives (i) and (ii) can also be performed in combination, for example by considering both outcoming results. The two alternatives (i) and (ii) are described separately in the following, after a description of the common part S1 to S5.

[0031] The common part S1 - S5 is performed on the selected sample. The selected sample is either the sample of interest in alternative (i) or a non-member and an unknown member in alternative (ii), as will be described in detail in the following. The selected sample is input into the specific image captioning model (the one that is being tested). Accordingly, the specific image captioning model provides a generated caption as an output, which describes the selected sample (step S1 ).

[0032] With a readily available pretrained embedding model (which is a text encoder), the generated caption is uniquely encoded into a feature vector including m embedding features (step S2). The m embedding features numerically represent the generated caption. The m embedding features form the components of the feature vector. For example, a text encoder based on Contrastive Language-Image Pretraining (CLIP)’s ViT-B / 32 Transformer architecture can be used, which outputs embeddings with a dimensionality m of 512. In particular, m is greater than 200, more particularly greater than 400.

[0033] The steps S1 and S2 are repeated n times (step S3). In particular, n is an integer of at least 2. Through sensitivity analysis, it was determined that a good n is 40 to ensure accurate and balanced results when determining whether the sample of interest was used to train the specific image captioning model. In particular, n is between 20 and 60. The steps S1 and S2 are repeated n times for the same selected sample. Since the specific image captioning model is a machine learning model, a different output (i.e. a different generated caption) is expected for each of the n iterations. Accordingly, n different generated captions are obtained and correspondingly, n different feature vectors are obtained, for the same selected sample. The n feature vectors can be represented as follows, with fij designating a feature f forming the i-th component of the feature vector with j being iteration count:

[0034]

[0035] In step S4, the variance vector is calculated, each component of the variance vector being the variance of all i-th features of each of the n feature vectors, wherein i is an integer that is between 1 and m. For example, the i-th component of the variance vector is the variance of the i-th feature of all feature vectors. Accordingly, the variance vector is a vector having m components (namely thesame number of components as the feature vectors). The variance vector is associated to the selected sample. Using the same notations as above, the variance vector is as given in equation 2, wherein Vi is the variance over fii , fa, fa, ... , fin.

[0036] (equation 2)

[0037]

[0038] In step S5, a mean variance MV, which is a mean of the m components of the variance vector (i.e. ofvi, V2, V3,...,V4), is calculated.

[0039] Alternative (i) includes a step S6 in addition to steps S1 - S5 being performed on the sample of interest instead of the selected sample. In step, S6, based on the value of the mean variance calculated for the sample of interest, it is determined whether the sample of interest was used to train the specific image captioning model.

[0040] Alternative (ii) includes steps S7 to S11 in addition to steps S1 - S5 (and instead of step S6 of alternative (i)). In alternative (ii), the common steps S1 - S5 are performed multiple times, namely on multiple non-members and on multiple unknown members, respectively forming the selected sample. Accordingly, for each of the non-members and for each of the unknown members, n embedding vectors are determined, from which a mean variance is calculated (for each of the non-members and for each of the unknown members).

[0041] The non-members are samples for which it is known that they were not used to train the specific image captioning model. For example, the non-members are generated at the moment of testing whether the sample of interest was used to train the specific image captioning model (i.e. when testing the membership of the sample of interest), in particular using an artificial intelligence (Al) for generating the non-member samples or by capturing photos / videos.

[0042] “Membership” here designates that a sample was used to train the specific image captioning model. “Unknown membership” here designates that it is not known whether a sample was used to train the specific image captioning model. The unknown members are samples of unknown membership. In other words, the unknown members are samples for which it is unknown whether they were used to train the specific image captioning model.

[0043] In a step S7, the mean po of all mean variances (i.e. the result of step S5) belonging to the multiple non-members is calculated. From this mean po, a member threshold value can be determined, forexample using a predetermine equation, or as a percentage of the mean o, as will be detailed in the following.

[0044] By comparing the mean variance (resulting from step S5) of each of the unknown samples with the member threshold value (step S8), it is determined whether each unknown sample is a pseudomember (step S9). A pseudo-member is in particular an unknown sample for which it is determined that there is an increased probability that said unknown sample is a member, i.e that it. was used to train the specific image captioning model. This increased probability is determined based on the result of the comparison in step S8. The pseudo-member set in particular includes a mix of likely members and potential false positives.

[0045] In step S10, a classifier is trained. The classifier is in particular a machine learning algorithm used to assign a class label to input data. The classifier can be trained using training data, which includes:

[0046] the non-members and / or features of the non-members, as well as a corresponding label indicating non-membership; and

[0047] the pseudo-members and / or features of the pseudo-members, as well as a corresponding label indicating pseudo-membership.

[0048] The features of the non-members can be determined using an image encoder, which provides numerical features to describe the non-member input into the image encoder. The features of the pseudo-members can be determined using the same or a different image encoder as for the non-members, said image encoder providing numerical features to describe the pseudo-member input into the image encoder. The purpose of the trained classifier is to determine, upon inputting the sample of interest into the classifier, whether it was used to train the specific image captioning model or not. The classifier may output a binary output indicating membership or non-membership.

[0049] In a step S11 , the sample of interest is input into the classifier, which determines whether the sample of interest was used to train the specific image captioning model or not.

[0050] According to an embodiment, in option (i), it is determined that the sample of interest was used to train the specific image captioning model if the calculated mean variance of the sample of interest is smaller than a predefined threshold.

[0051] In particular, the predefined threshold is determined based on an analysis of how membership of a sample of interest reflects into its mean variance. The value of the predefined threshold can be selected in accordance with the use-case for which it is important to know the membership of the sample of interest, and on how accurate the result should be.

[0052] According to a further embodiment, in option (ii), the member threshold value r is calculated as:T = no(MVno) - Aono, whereinno(MVno) is the meannoof the mean variances MVno of the multiple non-members, A is a constant, and Ono is the standard deviation of the mean variances MVno of the multiple non-members.

[0053] According to a further embodiment, in option (ii), it is determined that one of the unknown samples is a pseudo-member only if the mean variance of said unknown sample is smaller than the member threshold value r.

[0054] In particular, if an unknown sample’s mean variance is substantially lower thanno, specifically by more than Aono, it is classified as a pseudo-member.

[0055] According to a further embodiment, in option (ii), the classifier determines whether the sample of interest was used to train the specific image captioning model by determining features of the sample of interest using the image encoder and determining whether the sample of interest was used to train the specific image captioning model based on the determined features of the sample of interest.

[0056] The features of the sample of interest can be determined using the image encoder, which provides numerical features describing the sample of interest.

[0057] According to a further embodiment, in option (ii), the classifier is trained in two stages including a warm-up training phase and a subsequent training phase, wherein in the warm-up training phase, the classifier is trained using the standard training process to establish a baseline prediction tendency, and wherein in the subsequent training phase, training continues using only the pseudomembers for which the classifier indicates a confidence level above a predetermined confidence threshold.

[0058] In the training data, samples labelled with 0 are definitively non-members, whereas those labelled with 1 may not actually be members. This distinction guides the training strategy for the classifier, which incorporates a confidence threshold. Initially, the attack model undergoes a "warm-up” phase (warm-up training phase), training without interruptions to establish baseline prediction tendencies. This phase lasts for a predefined ratio of epochs, referred to as "warm-epoch.” Then, before each training batch, the membership confidence of the samples is evaluated. For those identified as pseudo-members (b = 1), training continues only if their membership confidence exceeds a predetermined threshold. This approach refines the training process, focusing primarily on samples most likely to be members, thus enhancing the accuracy and effectiveness of the classifier.

[0059] According to a second aspect, a computer program product stored on a machine-readable media and comprising machine readable instructions for executing the method according to the first aspect or any embodiment thereof is provided.All features described in view of the first aspect or any embodiment thereof also hold for the computer program product of the second aspect.

[0060] According to a third aspect, a determination device for determining whether a sample of interest was used to train a specific image captioning model is provided, the determination device being configured to perform the following steps:

[0061] (51) inputting a selected sample into the specific image captioning model to generate, by the specific image captioning model, a generated caption describing the selected sample;

[0062] (52) using a pretrained embedding model, encoding the generated caption into a feature vector including m embedding features, m being an integer and m>2;

[0063] (53) repeating the steps (S1) and (S2) n times, n being an integer and n>2, such as to obtain n generated captions and n feature vectors for the same selected sample;

[0064] (54) calculating a variance vector having m components, each component representing a variance of an i-th feature of the m features across the n feature vectors, i being an integer and 1<i<m;

[0065] (55) calculating a mean variance which is a mean of the m components of the variance vector; wherein either

[0066] (i) the sample of interest is used as the selected sample in steps (S1) to (S5) and wherein the method further includes a step (S6) of determining whether the sample of interest was used to train the specific image captioning model based on the calculated mean variance; and / or

[0067] (ii) the steps (S1) to (S5) are performed for multiple non-members forming the selected sample and for multiple unknown members forming the selected sample, wherein the non-members are samples for which it is known that they were not used to train the specific image captioning model and wherein the unknown members are samples having unknown membership, wherein the method further includes:

[0068] (57) determining a member threshold value from a meannoof the mean variances MVnoof the multiple non-members;

[0069] (58) comparing the mean variance of each unknown sample with the member threshold value;

[0070] (59) determining whether each unknown sample is a pseudo-member based on the result of the comparison in step (S8);

[0071] (S10) training a classifier based on training data including:

[0072] the non-members and / or features of the non-members determined using an image encoder, as well as their corresponding label indicating their non-membership, and the pseudo-members and / or features of the pseudo-members determined using the image encoder, as well as their corresponding label indicating their pseudomembership;wherein the classifier is for determining whether the sample of interest input into the classifier was used to train the specific image captioning model; and

[0073] (S11) inputting the sample of interest into the classifier to determine whether the sample of interest was used to train the specific image captioning model.

[0074] All features described in view of the first aspect or any embodiment thereof also hold for the determination device of the third aspect.

[0075] According to an embodiment, the determination device is configured to perform the method steps of the method according to the first aspect or any embodiment thereof.

[0076] The present invention will be described more fully hereinafter with reference to the accompanying figures in which like numerals represent like elements throughout the different figures, and in which prominent aspects and features of the invention are illustrated.

[0077] BRIEF DESCRIPTION OF THE FIGURES

[0078] Fig. 1 shows a computer-implemented method for determining whether a sample of interest was used to train a specific image captioning model;

[0079] Fig. 2 shows an example of steps S1 to S5 of the method of Fig. 1; and

[0080] Fig. 3 shows an example of steps S7 to S11 of the method of Fig. 1.

[0081] DETAILED DESCRIPTION

[0082] In the invention as claimed, two MIAs targeting specific image captioning models are described: a means-of-variance threshold attack (M VTA), corresponding to the variant (i), and a confidence-based weakly supervised attack (C-WSA), corresponding to variant (ii). The goal is to determine whether a given sample of interest is a member of the specific image captioning model’s training set or not.

[0083] The described methodology is designed to leverage the use of variance in the outputs of a target image captioning model (specific image captioning model) as a means of inferring infer membership information. The pipeline of the methodology consists of three main stages: text generation and multimodal feature encoding, variance metric computation, and application of proposed attacks. First, the target model generates text captions from images. These captions and images are then encoded into feature embeddings, using pre-trained text and image encoders, respectively. The outputs are then analyzed to determine membership, aiming to ascertain whether a new image was part of the target model’s training dataset. Next, a variance-based metric, the MV (mean variance) metric, is computed to quantify the variance among text embeddings generated for each image. This metric serves as the basis for identifying membership by capturing the relationship between variance and membership. Finally, two attacks are proposed based on the MV metric: the MVTA and C-WSA. The MVTA operates in a zero-knowledge setting by exploiting the reduced variance of text embeddingsfor training set images. The C-WSA, designed for a partial-knowledge setting, combines non-member ground truth with probabilistic pseudo-member labels and employs a confidence threshold to train an attack classifier model. The stages of this methodology are described below.

[0084] Fig. 1 shows both the MVTA (as variant (i)) and the C-WSA (as variant (ii)), as well as a common anterior part including steps S1 to S5. The common part (steps S1 to S5) is described first, under consideration of Fig. 1 and 2.

[0085] Typically, image captioning models use two text generation decoding strategies to generate concise descriptions: beam search and top-p sampling. Beam search is deterministic, consistently generating the same outputs for a given input by keeping the best candidates when choosing the next word. In contrast, top-p sampling, also known as nucleus sampling, is non-deterministic and introduces diversity by selecting from a probabilistic "nucleus” of options. When trained on paired data of the prompt and image, an image generation model generates similar images from the same prompt for two different random seeds. Similarly, in the presented scenario, it is highly probable that when using the top-p sampling, two output captions generated by the specific image captioning model will be similar for the same image. Based on this, it is assumed that the variance among the n captions generated for an image will be lower if the target model has been trained on that image.

[0086] In step S1 of Fig. 1, the selected sample 1, here an image of a dog jumping through water puddles, is input into the specific image captioning model 2 (see Fig. 2). Accordingly, the specific image captioning model 2 outputs a generated caption 3a describing the selected sample, which here reads “a dirty white dog is running through water”. The generated caption 3a is then input into a readily available pretrained embedding model 4, here the BERT (Bidirectional Encoder Representations from Transformers) model, which generates a feature vector 5 corresponding to the generated caption 3a and has the following format:

[0087] / 31 ■

[0088] \fmJ

[0089] As can be seen, the feature vector has m embedding features as components. This corresponds to step S2 of Fig. 1.

[0090] In step S3, steps S1 and S2 are repeated n times. In each instance, the same selected sample 1 is input into the specific image captioning model 2, resulting in slightly different generated captions 3 each time. For example, a subsequently generated caption 3b can read “a white dog is running through the water with a black leash”. This provides for n different feature vectors 5, as shown in Fig.

[0091] 2.In step S4, a variance vector 6 is calculated, wherein the i-th component Vi of the variance vector 6 is calculated as the variance of all i-th embedding features of the n feature vectors 5. For example, the variance component vi is calculated as the variance of fn, fi2, fis, ... , fin, the variance component Vi is calculated as the variance of fii , fi2, fa, ..., fin, etc. In step S5, a mean variance MV is calculated as the mean of all components Vi of the variance vector 6.

[0092] In the following, the variants (i) and (ii) are described, which respectively correspond to method steps S6 and S7 to S11 of Fig. 1.

[0093] Variant (i):

[0094] In variant (i), which is the MVTA method, the attacker has no prior knowledge of the distribution of the data used to train the target model (specific image captioning model), making this approach challenging but realistic in restricted data access scenarios. This corresponds to a “zero knowledge setting”.

[0095] In step S6, if the calculated mean variance MV for an image x (which corresponds to the sample of interest which is used as the selected sample 1) falls below a predetermined threshold r, x is classified as a member of the training set:

[0096] Predic 4t.ed mem ,bers ,hi.p = I (Member , if ,MV C rr(equat .i.on 3)

[0097] tNon — member Otherwise

[0098] The threshold r is determined based on empirical analysis aimed at optimizing detection accuracy. This approach adapts confidence or loss-based thresholds for data membership to multi-modal image captioning.

[0099] Variant (ii):

[0100] In variant (ii), which is the C-WSA method, the attacker has partial-knowledge. In such partialknowledge setting, the attacker has access to a publicly available dataset of images, Daii, approximating the data distribution used to train the target model. The attacker also has access to a non-member dataset Dno, while the membership status of Daii is unknown. The non-member dataset can be collected from content published after the model’s release or synthesized by imagegenerating Al models, such as publicly available diffusion models.

[0101] The C-WSA introduces an innovative adaptation in the partial-knowledge setting by utilizing the MV metric to enhance the model’s ability to distinguish between member and non-member data. An overview of the attack is provided in Fig. 3. The described attack integrates ground-truth information about non-members, along with probabilistic assessments of membership. In the described attackpipeline, we begin by querying the target model with images from the known non-member dataset (Dno) to generate n captions per image. In other words, the steps S1 to S5 are performed for each of the images (samples) of the non-members, which form the selected samples. Given the availability of public or synthetic datasets, an adversary can readily access or create non-members. Then, the MV metric is calculated for the non-member dataset in order to establish a baseline distribution of variance scores among non-members, which is hypothesized to follow a Gaussian distribution. The MV scores for non-members are denoted as follows: MVno = {MV (ftext(Gen(x))) | x e Dno}, where ftext(Gen(x)) refers to the text embeddings extracted by the text encoder ftext from the captions generated by the target model for each image x, and MV is the function that performs the metric on the vectors.

[0102] Using MVno, we calculate the meannoand standard deviation onoof the MV scores:no= mean(MVno), and ono= std(MVno). These parameters (noand ono) are crucial for identifying potential members by comparing their MV scores to the non-member distribution baseline mentioned above. Intuitively, if a sample’s MV is substantially lower thanno, specifically by more than Aonofor a given constant A, it suggests that the sample is likely a member. Accordingly, a member threshold value r is determined in step S7, wherein r=no- Aono.

[0103] For the unknown member dataset Daii, which comprises potential members, the steps S1 to S5 are also performed for each unknown member, which form the selected samples. The MV scores of the unknown member are determined using the text embeddings (generated captions) generated by the target model. Samples in Daii whose MV scores significantly deviate from this non-member baseline are identified as potential members, i.e. as pseudo-members. Specifically, each non-member’s MV is compared with the member threshold value r (step S8). If a non-member’s MV score exceeds r, it is classified as a pseudo-member (step S9). The corresponding subset of these identified pseudomember samples (DP-mem), serves as a pseudo-member set. This set comprises a mix of likely members and potential false positives and can be represented by DP-mem = {x | MV (ftext(Gen(x))) < r, x e Dan}.

[0104] With the Dp-mem and Dnoin place, in step S10 it is proceeded to create the training data set Dtrain, comprising image features from Dno u (union) DP-mem with their associated binary labels (L=no being the label for non-members and L=p-mem being the label for pseudo-members):

[0105] Dtrain — {fimg(x) | X G Dno U Dp-mem, b — 1 [X G Dp-mem]},

[0106] where fimg(x) represents the image features extracted by the image encoder fimgfor each image x, and b is a binary indicator with 1 representing a potential member and 0 representing a non-member.

[0107] Finally, in step S11, using Dtrain, the classifier model 7 is trained so that it is capable of determining the membership status of any target sample by evaluating the image features of the sample of interest 8.In Daii, samples labelled with 0 are definitively non-members, whereas those labelled with 1 may not actually be members. This distinction guides the training strategy for the classifier model 7 which incorporates a confidence threshold. Initially, the classifier 7 undergoes a "warm-up” phase, training without interruptions to establish baseline prediction tendencies. This phase lasts for a predefined ratio of epochs, referred to as "warm-epoch.” Then, before each training batch, the membership confidence of the samples is evaluated. For those identified as pseudo-members (b = 1), training continues only if their membership confidence exceeds a predetermined threshold. Low confidence pseudo-members Dioware excluded. This approach refines the training process, focusing primarily on samples most likely to be members, thus enhancing the accuracy and effectiveness of the classifier 7.

[0108] The above disclosed subject-matter is to be considered illustrative, and not restrictive, and serves to provide a better understanding of the invention defined by the independent claims.

[0109] REFERENCE NUMERALS

[0110] 1 selected sample

[0111] 2 specific image captioning model

[0112] 3 generated caption

[0113] 4 pretrained embedding model

[0114] 5 feature vector

[0115] 6 variance vector

[0116] 7 classifier

[0117] 8 sample of interest

Claims

CLAIMS1. A computer-implemented method for determining whether a sample of interest (8) was used to train a specific image captioning model (2), the method comprising:51 inputting a selected sample (1) into the specific image captioning model (2) to generate, by the specific image captioning model (2), a generated caption (3) describing the selected sample (1); 52 using a pretrained embedding model (4), encoding the generated caption (3) into a feature vector (5) including m embedding features, m being an integer and m>2;53 repeating the steps S1 and S2 n times, n being an integer and n>2, such as to obtain n generated captions (3) and n feature vectors (5) for the same selected sample (1);54 calculating a variance vector (6) having m components, each component representing a variance of an i-th feature of the m features across the n feature vectors (5), i being an integer and 1<i<m;55 calculating a mean variance which is a mean of the m components of the variance vector (6); wherein either(i) the sample of interest (8) is used as the selected sample (1) in steps S1 to S5 and wherein the method further includes a step S6 of determining whether the sample of interest (8) was used to train the specific image captioning model (2) based on the calculated mean variance; and / or(ii) the steps S1 to S5 are performed for multiple non-members forming the selected sample (1) and for multiple unknown members forming the selected sample (1), wherein the non-members are samples for which it is known that they were not used to train the specific image captioning model (2) and wherein the unknown members are samples having unknown membership, wherein the method further includes:57 determining a member threshold value (r) from a meannoof the mean variances MVnoof the multiple non-members;58 comparing the mean variance of each unknown sample with the member threshold value (r);59 determining whether each unknown sample is a pseudo-member based on the result of the comparison in step S8;S10 training a classifier (7) based on training data including:the non-members and / or features of the non-members determined using an image encoder, as well as their corresponding label indicating their non-membership, and the pseudo-members and / or features of the pseudo-members determined using the image encoder, as well as their corresponding label indicating their pseudomembership;wherein the classifier (7) is for determining whether the sample of interest (8) input into the classifier (7) was used to train the specific image captioning model (2); andS11 inputting the sample of interest (8) into the classifier (7) to determine whether the sample of interest (8) was used to train the specific image captioning model (2).

2. The method according to claim 1, wherein in option (i), it is determined that the sample of interest (8) was used to train the specific image captioning model (2) if the calculated mean variance of the sample of interest (8) is smaller than a predefined threshold.

3. The method according to claim 1 or 2, wherein in option (ii), the member threshold value (r) is calculated as:T = no(MVno) - Aono, whereinno(MVno) is the meannoof the mean variances MVno of the multiple non-members, A is a constant, and Ono is the standard deviation of the mean variances MVno of the multiple non-members.

4. The method according to any one of claims 1 to 3, wherein in option (ii), it is determined that one of the unknown samples is a pseudo-member only if the mean variance of said unknown sample is smaller than the member threshold value (r).

5. The method according to any one of claims 1 to 4, wherein in option (ii), the classifier (7) determines whether the sample of interest (8) was used to train the specific image captioning model (2) by determining features of the sample of interest (8) using the image encoder and determining whether the sample of interest (8) was used to train the specific image captioning model (2) based on the determined features of the sample of interest (8).

6. The method according to any one of claims 1 to 5, wherein in option (ii), the classifier (7) is trained in two stages including a warm-up training phase and a subsequent training phase, wherein in the warm-up training phase, the classifier (7) is trained using the standard training process to establish a baseline prediction tendency, and wherein in the subsequent training phase, training continues using only the pseudo-members for which the classifier (7) indicates a confidence level above a predetermined confidence threshold.

7. A computer program product stored on a machine-readable media and comprising machine readable instructions for executing the method according to any one of claim 1 to 6.

8. A determination device for determining whether a sample of interest (8) was used to train a specific image captioning model (2), the determination device being configured to perform the following steps:S1 inputting a selected sample (1) into the specific image captioning model (2) to generate, by the specific image captioning model (2), a generated caption (3) describing the selected sample (1);52 using a pretrained embedding model (4), encoding the generated caption (3) into a feature vector (5) including m embedding features, m being an integer and m>2;53 repeating the steps S1 and S2 n times, n being an integer and n>2, such as to obtain n generated captions (3) and n feature vectors (5) for the same selected sample (1);54 calculating a variance vector (6) having m components, each component representing a variance of an i-th feature of the m features across the n feature vectors (5), i being an integer and 1<i<m;55 calculating a mean variance which is a mean of the m components of the variance vector (6); wherein either(i) the sample of interest (8) is used as the selected sample (1) in steps S1 to S5 and wherein the method further includes a step S6 of determining whether the sample of interest (8) was used to train the specific image captioning model (2) based on the calculated mean variance; and / or(ii) the steps S1 to S5 are performed for multiple non-members forming the selected sample (1) and for multiple unknown members forming the selected sample (1), wherein the non-members are samples for which it is known that they were not used to train the specific image captioning model (2) and wherein the unknown members are samples having unknown membership, wherein the method further includes:57 determining a member threshold value (r) from a meannoof the mean variances MVnoof the multiple non-members;58 comparing the mean variance of each unknown sample with the member threshold value (r);59 determining whether each unknown sample is a pseudo-member based on the result of the comparison in step S8;510 training a classifier (7) based on training data including:the non-members and / or features of the non-members determined using an image encoder, as well as their corresponding label indicating their non-membership, and the pseudo-members and / or features of the pseudo-members determined using the image encoder, as well as their corresponding label indicating their pseudomembership;wherein the classifier (7) is for determining whether the sample of interest (8) input into the classifier (7) was used to train the specific image captioning model (2); and 511 inputting the sample of interest (8) into the classifier (7) to determine whether the sample of interest (8) was used to train the specific image captioning mode (2)1.

9. The determination device according to claim 8, which is configured to perform the method steps of any one of claims 1 to 6.

Images