Image classification device, image classification system, image classification method, and image classification program

Abstract

This image classification device comprises: a prompt acquisition unit (11) that, on the basis of a source text prompt, acquires a target text prompt including information relating to an adversarial attack; and a control unit (15) that embeds the class name of various classes, from among a class set which includes a plurality of classes, into the target text prompt, inputs a target image to be classified and the target text prompt, in which the class name has been embedded, into a machine learning model that includes a text encoder, image encoder, and distance calculation module and that has completed text and image contrast learning, acquires, from the machine learning model, a distance index that is an index representing the distance between the target image and each class of the class set, and controls, on the basis of the distance index, the display of an image classification result indicating a class to which the subject of the target image is predicted to belong.

Description

Image Classification Device, Image Classification System, Image Classification Method, and Image Classification Program

[0001] The present disclosure relates to image classification technology.

[0002] In recent years, research and development have been conducted on vision and language models (VLMs), which are deep learning models that handle different modalities such as vision and language. For example, Non-Patent Document 1 discloses a VLM called CLIP (Contrastive Language-Image Pre-training). CLIP is a model that optimizes an image encoder and a language encoder through contrastive learning using a huge amount of paired data in which images collected from the web and captions are paired.

[0003] Radford, Alec, et al. “Learning transferable visual models from natural language supervision.” International conference on machine learning. PMLR, 2021.

[0004] Conventional vision and language models that take images as input targets, including the technology of Non-Patent Document 1, have a problem of being vulnerable to adversarial attacks that apply many small changes to pixel values in images.

[0005] The present disclosure has been made to solve such problems, and an object thereof is to provide an image classification technology that is robust against adversarial attacks.

[0006] One aspect of an image classification device according to an embodiment of the present disclosure includes: a prompt acquisition unit that acquires a target text prompt containing information about a hostile attack based on a source text prompt; a control unit that embeds the class names of each class from a class set comprising multiple classes into the target text prompt, inputs the target image to be classified and the target text prompt with the embedded class names into a machine learning model equipped with a text encoder, an image encoder, and a distance calculation module, which has been trained to compare text and images, obtains a distance index from the machine learning model which is an index representing the distance between the target image and each class in the class set, and controls the display of an image classification result indicating the class to which the subject of the target image is predicted to belong based on the distance index.

[0007] According to the image classification device of the embodiment of this disclosure, image classification can be performed robustly against adversarial attacks.

[0008] This figure shows an example configuration of an image classification device according to Embodiment 1. This figure shows the structure and operation of a machine learning model. This figure shows an example configuration of an image classification device according to Embodiment 1. This figure shows an example configuration of an image classification device according to Embodiment 2. This figure shows cross-entropy by cosine similarity. This figure shows cross-entropy by dot product. This figure shows an example configuration of an image classification system according to Embodiment 3. This figure shows an example hardware configuration of an image classification device or image classification system. This figure shows an example hardware configuration of an image classification device or image classification system. This is a flowchart showing an image classification method.

[0009] Various embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, identical or similar parts are denoted by the same or similar reference numerals, and redundant descriptions of such parts will be omitted. In this disclosure, the term "or" is used in the sense of an inclusive OR unless otherwise stated.

[0010] Embodiment 1. <Configuration> Referring to Figure 1, the image classification device 10 according to Embodiment 1 of this disclosure will be described. The image classification device 10 is a device for performing zero-shot image classification. That is, the image classification device 10 is a device for performing image classification without learning which of multiple classes the subject represented in the target image (hereinafter sometimes simply referred to as "image") belongs to. Furthermore, the image classification device 10 is a device for performing image classification while suppressing the effects of adversarial attacks, even if the target image has been processed by an adversarial attack.

[0011] An adversarial attack refers to an attack intentionally designed to cause a trained model to misclassify. Examples of adversarial attacks include FGSM (Fast Gradient Sign Method), PGD (Projected Gradient Descent), or DeepFool. The image classification techniques disclosed herein are intended to provide resilience against adversarial attacks, although implementations of the image classification techniques disclosed herein may suppress misclassification of images with simply random noise.

[0012] To enable robust image classification against adversarial attacks, the image classification device 10 includes a prompt coupling unit 11 and a control unit 15, as shown in Figure 1. The language encoder 12, image encoder 13, and distance index calculation module 14 are pre-trained machine learning models that have undergone comparative learning and are referenced by the control unit 15 of the image classification device 10. The machine learning models may be stored in the memory of the image classification device 10 or on an external server device. If the machine learning models are stored on an external server device, the image classification device 10 accesses the external server device via a communication interface (not shown) to reference the machine learning models. In Figure 1, reference numerals are not used to denote data or information that are the processing targets or processing results of the image classification device 10. The functional units of the image classification device 10 and the machine learning models referenced by the image classification device 10 will be described in more detail below.

[0013] (Machine Learning Model) The machine learning model is a pre-trained deep learning model that has undergone comparative learning to align multiple pairs of images and captions. The machine learning model comprises a language encoder 12, an image encoder 13, and a distance metric calculation module 14. The language encoder 12 converts the input natural language text into language features and outputs the language features. The language features are represented as multidimensional numerical language feature vectors so that they can be compared with image features. The image encoder 13 converts the input image into image features and outputs the image features. The image features are represented as numerical image feature vectors with the same number of dimensions as the language feature vectors so that they can be compared with language features. During pre-training, the distance metric calculation module 14 is trained to minimize the distance between the language features and image features of each pair in the feature space. During inference, the distance metric calculation module 14 calculates the distance between the image features of the target image to be classified and each feature from among the multiple language features of the multiple classes provided, and outputs the calculated distance. Note that the term "similarity" is sometimes used instead of "distance." There is a relationship between distance and similarity: the closer the distance, the higher the similarity, and the farther the distance, the lower the similarity.

[0014] As such a pre-trained machine learning model, CLIP, as taught in Non-Patent Document 1, may be used after being transferred. In CLIP, a Transformer is used as the language encoder, and ResNet-50 or Vision Transformer is used as the image encoder. The feature space is called the multimodal embedding space.

[0015] (Prompt coupling unit) The prompt coupling unit 11 is a functional unit that acquires a target text prompt containing information about a hostile attack based on a source text prompt. The prompt coupling unit 11 is an example of a prompt acquisition unit in this disclosure. The source text prompt is input, for example, by a user of the image classification device 10 via an input device (not shown).

[0016] As an example, the source text prompt includes a first source text prompt containing an image style and a second source text prompt containing a description of the adversarial attack. The prompt merging unit 11 obtains the target text prompt by merging the first source text prompt and the second source text prompt. An example of an image style includes a photograph or a painting.

[0017] For example, as shown in Figure 1, there are two source text prompts: text prompt A, which reads "A photo of a [class]", and text prompt B, which reads "even with some added noise". The prompt merging unit 11 merges text prompts A and B to obtain text prompt C, which reads "A photo of a [class], even with some added noise", as the target text prompt. The prompt merging unit 11 may also merge text prompts by simply adding another text prompt (in this case, text prompt B) after one text prompt (in this case, text prompt A). The prompt merging unit 11 may also insert a comma between the text prompts to be merged. The prompt merging unit 11 may also merge text prompt A after text prompt B. The prompt merging unit 11 may insert a period at the end of the prompt obtained by merging in this way. The prompt merging unit 11 may also merge text prompts by adding another text prompt immediately before a period in one of the text prompts.

[0018] The reason why text prompt A includes the image style is that machine learning models often perform pre-training using captions that include image styles when performing comparative learning. If it is known that the machine learning model performed comparative learning using only class names without including image styles, it may be appropriate not to include the image style in text prompt A during inference.

[0019] In the above description, the prompt merging unit 11 described an example in which the source text prompt includes a specific first source text prompt and a second source text prompt. However, the source text prompt may include three or more specific source text prompts. For example, as shown in Figure 3, the source text prompt may include text prompt D in addition to the text prompts A and B described above. As shown in Figure 3, text prompt D, which says "that belongs to [superclass]", contains a description of the superclass to which the class belongs. The prompt merging unit 11 may combine these text prompts A, B, and D to obtain text prompt E, which says "A photo of a [class] that belongs to [superclass], even with some added noise", as the target text prompt.

[0020] (Control Unit 15) The control unit 15 controls the entire image classification device 10 from one side.

[0021] Furthermore, the control unit 15 embeds the class names of each class from a class set comprising multiple classes into the target text prompt acquired by the prompt acquisition unit. This point will be explained with reference to Figure 2. As shown in Figure 2, a class set consisting of N classes such as bird, cat, ..., and truck is prepared. The class set is prepared in advance by, for example, the user of the image classification device 10, and the control unit 15 acquires the class set via an input / output interface (not shown). The control unit 15 embeds the class names of each class from the class set into the text prompt C, which is the target text prompt. Specifically, for example, the control unit 15 deletes [class] contained in the text prompt C and places the class names of each class in the deleted location. The control unit 15 then inputs the text prompt C, in which the class names have been embedded, into the language encoder 12 of the machine learning model.

[0022] If the target text prompt includes a description of a higher class, as in text prompt E, it is advisable to pre-classify multiple classes in a pre-prepared class set so that they belong to one of several higher classes. For example, the classes "bird" and "cat" can be classified into the higher class "animal," and the classes "truck" and "small car" can be classified into the higher class "mobility." The control unit 15 refers to the class set including such higher class classifications and embeds the class name and higher class name of each class in the class set into text prompt E. After the class name and higher class name are embedded in text prompt E, the control unit 15 inputs the text prompt E to the language encoder 12 of the machine learning model. By including the higher class name in the text prompt in this way, the probability of the image to be classified being misclassified into a class belonging to a different higher class than the higher class to which the correct class belongs is suppressed, thereby improving the performance of class classification.

[0023] The language encoder 12, upon receiving N text prompts C or E for each class, generates language features for each class. Since text prompts C or E contain descriptions of adversarial attacks, the language features also include features that reflect these descriptions.

[0024] Note that text prompt A or text prompt D may be prepared in advance as a template, rather than being obtained through user input.

[0025] Furthermore, the control unit 15 inputs the target image to be classified to the image encoder 13 of the machine learning model. The target image to be classified is prepared, for example, by the user of the image classification device 10, and the control unit 15 acquires the target image to be classified via an input / output interface (not shown). This target image to be classified may have been subjected to a hostile attack.

[0026] The image encoder 13, upon receiving the target image to be classified, generates image features of the target image. If the target image has been subjected to an adversarial attack, the generated image features will also include features that reflect the adversarial attack.

[0027] The distance index calculation module 14 calculates the distance between the image features of the target image to be classified, generated by the image encoder 13, and each of the N language features generated by the language encoder 12.

[0028] Furthermore, the control unit 15 obtains, on one side, the distance index calculated by the machine learning model distance index calculation module 14 from the machine learning model. The distance index is expressed as a probability distribution representing the distance between the target image to be classified and each class in the class set. Since the obtained distance index indicates the distance between the image features of the target image and the linguistic features that reflect the explanation of the adversarial attack, it is possible to suppress misclassification of the image due to the adversarial attack, even if the target image to be classified has been subjected to an adversarial attack.

[0029] The control unit 15 may perform display control to display on a display device (not shown) an image classification result showing multiple classes to which the subject of the target image is predicted to belong, based on distance indicators obtained from a machine learning model. Alternatively, the control unit 15 may input the distance indicators obtained from the machine learning model into the argmax function to obtain the predicted class with the highest probability, and perform display control to display on a display device (not shown) an image classification result showing a single class to which the subject of the target image is predicted to belong.

[0030] The image classification device 10 described above includes a prompt coupling unit 11, which acquires a target text prompt containing information about a hostile attack. Since the image classification device 10 performs image classification using such a target text prompt, it can perform robust image classification even if the image to be classified has been subjected to a hostile attack.

[0031] <Modification> Next, a modification of Embodiment 1 will be described. In Embodiment 1, the prompt merging unit, which acts as a prompt acquisition unit, acquired a target text prompt by merging multiple text prompts. Instead of such merging, the prompt acquisition unit may acquire the target text prompt using a large language model (LLM) 16. The LLM 16 is a type of generative AI (artificial intelligence) and is a neural network model constructed by performing deep learning based on a large dataset. The LLM 16 may be stored locally in the memory of the image classification device 10A, or it may be stored in a server device (not shown) outside the image classification device 10A.

[0032] To utilize such an LLM 16, the image classification device 10A is equipped with an LLM input / output unit 11A as a prompt acquisition unit. The LLM input / output unit 11A inputs a source text prompt to the LLM 16 that instructs the generation of a prompt for image classification that explains the effect of a particular adversarial attack on an image. An example of such a source text prompt is the text prompt, "Create a prompt for image classification that explains how PGD attacks affect images." The LLM input / output unit 11A obtains the answer to the source text prompt from the LLM 16. The source text prompt to be input to the LLM 16 is entered by the user of the image classification device via an input device (not shown). The above example of a source text prompt assumes that the adversarial attack is PGD, but by writing the name of DeepFool or other adversarial attack in the source text prompt instead of PGD, a source text prompt corresponding to any adversarial attack may be prepared. Furthermore, the source text prompt may include a command specifying the format of the output response. The response from LLM 16 is, for example, a text prompt such as "A photo of a [class] with minor digital artifacts". The LLM input / output unit 11A acquires such a response as the target text prompt. In this way, it becomes possible to automatically generate a detailed explanatory text about the attack using LLM, and to generate text prompts corresponding to any attack method.

[0033] Similar to Embodiment 1, the control unit 15 embeds a class name into the target text prompt acquired by the LLM input / output unit 11A, and inputs the target text prompt with the embedded class name to the language encoder 12.

[0034] An image classification device 10A equipped with such a modified LLM input / output unit 11A can, like the image classification device 10, robustly perform image classification even if the image to be classified is under hostile attack.

[0035] As described above, cosine similarity can be used as the distance index in the image classification device 10 or image classification device 10A, but the dot product may also be used. This point will be explained with reference to Figures 5A and 5B. Both Figures 5A and 5B are plots showing the change in the loss value of the cross-entropy loss function due to a PGD attack. Figure 5A shows the case when cosine similarity is used as the distance index, and Figure 5B shows the case when the dot product is used as the distance index. In both Figures 5A and 5B, the perturbation range ε ​​is 0.006, 0.018, 0.03, and 0.1, and the number of steps is 20.

[0036] As shown in Figure 5A, when the distance index is cosine similarity, the loss value increases as the number of steps increases, regardless of the value of the perturbation range ε. When the loss value is large, the norm (magnitude) of the gradient increases, making it easier for perturbations that cause large changes to occur. In other words, obtaining a large loss value means that the attacker of a PGD attack has obtained useful information. A PGD attack adds the largest perturbation obtained within the perturbation range ε ​​to an image by gradually changing the pixel values ​​against an image classifier that can infer correctly.

[0037] On the other hand, as shown in Figure 5B, when the distance index is the dot product, the loss value is kept small even as the number of steps increases, regardless of the value of the perturbation range ε. In Figure 5B, the loss value at any number of steps is almost the same as the loss value when the number of steps is 1. A small loss value means that the attacker of the PGD attack is not obtaining any useful information.

[0038] Cosine similarity is calculated by normalizing the dot product of the vector-represented language and image features by the product of the norms of both vectors. As explained above, by using the dot product as the distance metric, that is, by not normalizing the features from the image encoder and language encoder, a large value is input to the subsequent softmax function, which has the effect of suppressing the gradient of the loss used in PGD attacks. As a result, the accuracy of the PGD attack decreases.

[0039] Embodiment 2. <Image Classification System> Next, an image classification system 1 according to Embodiment 2 will be described with reference to Figure 6. The image classification system 1 comprises a plurality of image classification devices 10 or 10A according to Embodiment 1 (hereinafter, it will be described as image classification device 10, but it may also be image classification device 10A), as well as an average calculation unit 20 and a control unit 30. Each of the plurality of image classification devices 10-1 to 10-N is input with different text prompts 1 to N as target text prompts, after the class name or subclass name has been embedded, and each image classification device 10 performs image separation on the same image.

[0040] (Average Calculation Unit) The average calculation unit 20 acquires N distance indicators from multiple image classification devices 10-1 to 10-N and calculates the average of the acquired N distance indicators. By taking an ensemble average of the distance indicators, fluctuations in the distance indicators calculated by each image classification device 10 can be suppressed, thereby further improving the classification performance.

[0041] (Control Unit) The control unit 30 performs display control to display the class classification result on a display device (not shown) based on the result calculated by the average calculation unit 20.

[0042] <Hardware Configuration> Next, an example of the hardware configuration of the image classification device 10 will be described with reference to Figures 7A and 7B. Each function of the image classification device 10 is realized by a processing circuit. The processing circuit may be a dedicated processing circuit 100a as shown in Figure 7A, or a processor 100b that executes a program stored in memory 100c as shown in Figure 7B.

[0043] If the processing circuitry is a dedicated processing circuit 100a, the dedicated processing circuit 100a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), or a combination thereof. The functions of the image classification device 10 may be realized by multiple separate processing circuits, or the functions of the image classification device 10 may be realized by a single processing circuit.

[0044] When the processing circuitry is the processor 100b as a computer, the functions of the image classification device 10 are realized by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the memory 100c. The processor 100b realizes the functions of the image classification device 10 by reading and executing the programs stored in the memory 100c. Here, examples of the memory 100c include non-volatile or volatile semiconductor memories such as RAM (random access memory), ROM (read-only memory), flash memory, EPROM (erasable programmable read only memory), EEPROM (electrically erasable programmable read-only memory), etc., as well as magnetic disks, flexible disks, optical disks, compact disks, mini disks, and DVDs.

[0045] Note that a part of the functions of the image classification device 10 may be realized by dedicated hardware, and the other functions may be realized by software or firmware. Thus, the processing circuitry can realize the functions of the image classification device 10 by hardware, software, firmware, or a combination thereof.

[0046] Also, the average calculation unit 20 and the control unit 30 included in the image classification system 1 may also be realized by a processing circuitry as shown in FIG. 7A or FIG. 7B.

[0047] <Operation> Next, referring to FIG. 8, the operation of the image classification device 10 will be described.

[0048] (Step ST1) In Step ST1, the prompt acquisition unit (prompt combination unit 11 or LLM input / output unit 11A) acquires a target text prompt based on the source text prompt. When the prompt acquisition unit is the prompt combination unit 11, on one hand, the prompt combination unit 11 combines the first source text prompt and the second source text prompt to acquire a target text prompt. The prompt combination unit 11 may add more specific source text prompts to the combination of the first source text prompt and the second source text prompt.

[0049] When the prompt acquisition unit is the LLM input / output unit 11A, the LLM input / output unit 11A inputs the source text prompt to the LLM 16 and acquires the response from the LLM 16 as the target text prompt.

[0050] (Step ST2) In Step ST2, the control unit 15 embeds the class name of each class in the class set including a plurality of classes into the target text prompt acquired by the prompt acquisition unit. The class name of the upper class may be embedded.

[0051] (Step ST3) In Step ST3, the control unit 15 inputs the target text prompt embedded with a plurality of class names and the target image to be classified into a pre-trained machine learning model in which contrastive learning has been performed.

[0052] (Step ST4) In Step ST4, the control unit 15 acquires a distance metric representing the distance between the target image to be classified and each class from such a machine learning model.

[0053] (Step ST5) In Step ST5, the control unit 15 performs display control to display the classification result of the target image to be classified on the display device based on the distance metric acquired from the machine learning model.

[0054] In the case of processing by the image classification system 1, each image classification device performs steps ST1 to ST4, and in a step (not shown) following step ST4, the average calculation unit 20 calculates the average of the distance indices obtained from each image classification device.

[0055] In step ST5, which follows this step (not shown), the control unit 30 performs display control to display the classification result of the target image to be classified on the display device based on the calculated average.

[0056] <Note> Some aspects of the various embodiments described above are summarized below.

[0057] (Note 1) The image classification device according to Note 1 includes: a prompt acquisition unit (11) that acquires a target text prompt containing information about a hostile attack based on a source text prompt; a unit (15) that embeds the class names of each class from a class set comprising multiple classes into the target text prompt, inputs the target image to be classified and the target text prompt with the embedded class names into a machine learning model equipped with a text encoder, an image encoder, and a distance calculation module, and which has been trained to compare text and images; obtains a distance index from the machine learning model which is an index representing the distance between the target image and each class in the class set; and controls the display of an image classification result indicating the class to which the subject of the target image is predicted to belong based on the distance index.

[0058] (Note 2) The image classification device according to Note 2 is the image classification device described in Note 1, wherein the source text prompt includes a first source text prompt including the style of the image and a second source text prompt including a description of a hostile attack, and the text prompt acquisition unit includes a prompt combining unit that acquires the target text prompt by combining the first source text prompt and the second source text prompt.

[0059] (Note 3) The image classification device according to Note 3 is the image classification device described in Note 2, wherein the source text prompt further includes a third source text prompt that includes a higher class to which the class belongs, and the prompt coupling unit obtains the target text prompt by coupling the first source text prompt, the second source text prompt, and the third source text prompt.

[0060] (Appendix 4) The image classification device according to Appendix 4 is the image classification device described in Appendix 1, wherein the source text prompt includes a text prompt that instructs the generation of a prompt for image classification that describes the effect of a particular adversarial attack on an image, and the prompt acquisition unit inputs the source text prompt to a large-scale language model and acquires the response obtained from the large-scale language model as the target text prompt.

[0061] (Note 5) The image classification device according to Note 5 is an image classification device described in any one of Notes 1 to 4, wherein the distance index is the dot product of the language feature vector output from the text encoder and the image feature vector output from the image encoder.

[0062] (Note 6) The image classification system according to Note 6 comprises a plurality of image classification devices described in any one of Notes 1 to 5, and an average calculation unit (20) that calculates the average of a plurality of distance indicators output from the plurality of image classification devices.

[0063] (Note 7) The image classification method according to Note 7 is an image classification method performed by an image classification device comprising a prompt acquisition unit (11) and a control unit (15), comprising: a step (ST1) in which the prompt acquisition unit (11) acquires a target text prompt containing information about a hostile attack based on a source text prompt; a step (ST2) in which the control unit (15) embeds the class names of each class from a class set comprising multiple classes into the target text prompt; a step (ST3) in which the target image to be classified and the target text prompt with the embedded class names are input to a machine learning model comprising a text encoder, an image encoder, and a distance calculation module, and which has been trained to compare text and images; a step (ST4) in which a distance index, which is an index representing the distance between the target image and each class in the class set, is obtained from the machine learning model; and a step (ST5) in which the display of an image classification result indicating the class to which the subject of the target image is predicted to belong is controlled based on the distance index.

[0064] (Note 8) The image classification program according to Note 8 causes the computer to perform the following functions: a function to obtain a target text prompt containing information about adversarial attacks based on a source text prompt; a function to embed the class names of each class from a class set comprising multiple classes into the target text prompt; a function to input the target image to be classified and the target text prompt with the embedded class names into a machine learning model equipped with a text encoder, an image encoder, and a distance calculation module, and trained to compare text and images; a function to obtain a distance index from the machine learning model, which is an index representing the distance between the target image and each class in the class set; and a function to control the display of the image classification result, which indicates the class to which the subject of the target image is predicted to belong, based on the distance index.

[0065] Furthermore, it is possible to combine embodiments, or to modify or omit each embodiment as appropriate.

[0066] The image classification technology disclosed herein can be used as an image classification device that performs image classification with zero shots.

[0067] 1 Image classification system, 10 (10A, 10-1 to 10-N) Image classification device, 11 Prompt coupling unit (prompt acquisition unit), 11A LLM input / output unit (prompt acquisition unit), 12 Language encoder, 13 Image encoder, 14 Distance index calculation module, 15 Control unit, 20 Average calculation unit, 30 Control unit, 100a Processing circuit, 100b Processor, 100c Memory.

Claims

1. An image classification device comprising: a prompt acquisition unit that acquires a target text prompt containing information about a hostile attack based on a source text prompt; a prompt acquisition unit that embeds the class names of each class from a class set comprising multiple classes into the target text prompt; inputs the target image to be classified and the target text prompt with the embedded class names into a machine learning model equipped with a text encoder, an image encoder, and a distance calculation module, which has been trained to compare text and images; obtains a distance index from the machine learning model which is an index representing the distance between the target image and each class in the class set; and controls the display of an image classification result indicating the class to which the subject of the target image is predicted to belong based on the distance index.

2. The image classification apparatus according to claim 1, wherein the source text prompt includes a first source text prompt including an image style and a second source text prompt including a description of a hostile attack, and the prompt acquisition unit includes a prompt combining unit that acquires the target text prompt by combining the first source text prompt and the second source text prompt.

3. The image classification apparatus according to claim 2, wherein the source text prompt further includes a third source text prompt that includes a higher class to which the class belongs, and the prompt coupling unit obtains the target text prompt by coupling the first source text prompt, the second source text prompt, and the third source text prompt.

4. The image classification apparatus according to claim 1, wherein the source text prompt includes a text prompt that instructs the generation of a prompt for image classification that describes the effect of a particular adversarial attack on an image, and the prompt acquisition unit inputs the source text prompt to a large language model and acquires the response obtained from the large language model as the target text prompt.

5. The image classification device according to any one of claims 1 to 4, wherein the distance index is the dot product of the language feature vector output from the text encoder and the image feature vector output from the image encoder.

6. An image classification system comprising: a plurality of image classification devices according to any one of claims 1 to 5; and an average calculation unit that calculates the average of a plurality of distance indicators output from the plurality of image classification devices.

7. An image classification method performed by an image classification device comprising a prompt acquisition unit and a control unit, comprising: the steps of: the prompt acquisition unit acquiring a target text prompt containing information about a hostile attack based on a source text prompt; the control unit embedding the class names of each class from a class set comprising multiple classes into the target text prompt; inputting the target image to be classified and the target text prompt with the embedded class names into a machine learning model comprising a text encoder, an image encoder, and a distance calculation module, which has been trained to compare text and images; obtaining a distance index from the machine learning model which is an index representing the distance between the target image and each class in the class set; and controlling the display of an image classification result indicating the class to which the subject of the target image is predicted to belong, based on the distance index.

8. An image classification program that causes a computer to perform the following functions: a function to obtain a target text prompt containing information about a hostile attack based on a source text prompt; a function to embed the class names of each class from a class set comprising multiple classes into the target text prompt; a function to input the target image to be classified and the target text prompt with the embedded class names into a machine learning model equipped with a text encoder, an image encoder, and a distance calculation module, and trained to compare text and images; a function to obtain a distance index from the machine learning model which is an index representing the distance between the target image and each class in the class set; and a function to control the display of an image classification result indicating the class to which the subject of the target image is predicted to belong, based on the distance index.

Images