Learning device, classification device, learning method, and learning program

Abstract

This learning device comprises: a sorting unit that sorts a plurality of images into a plurality of types of negative examples and positive examples on the basis of labels attached to images of interest included among the plurality of images and labels attached to each of the plurality of images; a re-sorting unit that re-sorts the plurality of images sorted by the sorting unit into a plurality of types of negative examples and positive examples, including semihard negative examples, by using the degree of similarity between feature amounts of the images of interest and feature amounts of the positive examples together with the degree of similarity between the feature amounts of the images of interest and feature amounts of the negative examples; and a learning unit that, by using the plurality of images re-sorted by the re-sorting unit, trains a machine learning model such that (i) the degree of similarity between the feature amounts of the images of interest and the feature amounts of the positive examples increases, (ii) the degree of similarity between the feature amounts of the images of interest and the feature amounts of the negative examples decreases, and (iii) the degree of similarity between the feature amounts of the images of interest and the feature amounts of the semihard negative examples is more emphasized than the degree of similarity between the feature amounts of the images of interest and the feature amounts of the negative examples other than the semihard negative examples. With this machine learning model, users such as medical professionals can be assisted in making decisions.

Description

Learning device, classification device, learning method, and learning program

[0001] The present disclosure relates to a learning device, a classification device, a learning method, and a learning program.

[0002] Techniques for training a machine learning model are known. For example, Patent Document 1 discloses a method for training a machine learning model including a classifier for an entire class by determining weighted input points of a contribution set that stores values for predicting true positives and true negatives.

[0003] In addition, techniques for training a machine learning model by contrastive learning are known. In contrastive learning, the machine learning model is trained so that the inner product between the feature vector of an anchor, which is the target image, and the feature vector of a positive example becomes large, and the inner product between the feature vector of the anchor and the feature vector of a negative example becomes small.

[0004] Japanese Patent Application Laid-Open No. 2023-058029

[0005] In contrastive learning, the degree of similarity between the features of the anchor and the negative examples may vary for each negative example. There may be negative examples that have some common points with the features of the anchor, and there may also be negative examples that have almost no common points with the features of the anchor. If contrastive learning is performed by uniformly defining a plurality of samples with different degrees of similarity to the anchor as negative examples, it may not be possible to appropriately advance the machine learning and increase the inference accuracy of the machine learning model.

[0006] The present disclosure has been made in view of the above problems, and an exemplary object thereof is to provide a technique for training a machine learning model so that the inference accuracy becomes higher.

[0007] A learning device according to an exemplary aspect of the present disclosure includes a sorting unit that sorts a plurality of images into a plurality of types of negative examples and positive examples based on labels attached to a target image included in the plurality of images and labels attached to each of the plurality of images, and a similarity between a feature amount of the target image and a feature amount of the positive example, and a similarity between the feature amount of the target image and the feature amount of the negative example. A re-sorting unit that re-sorts the plurality of images sorted by the sorting unit into a plurality of types of negative examples including medium negative examples and positive examples, and (i) the similarity between the feature amount of the target image and the feature amount of the positive example becomes large, (ii) the similarity between the feature amount of the target image and the feature amount of the negative example becomes small, and (iii) the similarity between the feature amount of the target image and the feature amount of the medium negative example is more important than the similarity between the feature amount of the target image and the feature amount of the negative example other than the medium negative example. A learning unit that learns a machine learning model so as to be emphasized.

[0008] A learning method according to an exemplary aspect of the present disclosure includes a sorting process in which at least one processor sorts a plurality of images into a plurality of types of negative examples and positive examples based on labels attached to a target image included in the plurality of images and labels attached to each of the plurality of images, and the at least one processor uses a similarity between a feature amount of the target image and a feature amount of the positive example, and a similarity between the feature amount of the target image and the feature amount of the negative example. A re-sorting process for re-sorting the plurality of images sorted by the sorting process into a plurality of types of negative examples including medium negative examples and positive examples, and (i) the similarity between the feature amount of the target image and the feature amount of the positive example becomes large, (ii) the similarity between the feature amount of the target image and the feature amount of the negative example becomes small, and (iii) the similarity between the feature amount of the target image and the feature amount of the medium negative example is more important than the similarity between the feature amount of the target image and the feature amount of the negative example other than the medium negative example. A learning process for learning a machine learning model so as to be emphasized.

[0009] A learning program relating to an illustrative aspect of this disclosure is a program for causing a computer to function as a learning device, comprising: a selection means for the computer to select a plurality of images into a plurality of negative examples and positive examples based on labels attached to a particular image contained in a plurality of images and labels attached to each of the plurality of images; and a system that uses the similarity between the feature quantities of the particular image and the feature quantities of the positive examples, and the similarity between the feature quantities of the particular image and the feature quantities of the negative examples to classify the plurality of images selected by the selection means into medium negative examples. The system functions as a re-selection means for re-selecting multiple types of negative and positive examples, including the above, and a learning means for training a machine learning model using the multiple images re-selected by the re-selection means, such that (i) the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples increases, (ii) the similarity between the feature quantities of the image of interest and the feature quantities of the negative examples decreases, and (iii) the similarity between the feature quantities of the image of interest and the feature quantities of the medium negative examples is given more weight than the similarity between the feature quantities of the image of interest and the feature quantities of negative examples other than the medium negative examples.

[0010] One exemplary effect of this disclosure is that it can provide a technique for training machine learning models to improve inference accuracy.

[0011] This is a block diagram showing the configuration of the learning device related to this disclosure. This is a flowchart showing the flow of the learning method related to this disclosure. This is a block diagram showing the configuration of the information processing device related to this disclosure. This is a diagram showing an example of a class and subclass related to this disclosure. This is a diagram showing an example of a class, middle class, and subclass related to this disclosure. This is a diagram showing the configuration of the machine learning model related to this disclosure. This is a diagram showing a specific example of the re-selection process related to this disclosure. This is a diagram for explaining the learning process of the feature analysis model related to this disclosure. This is a block diagram showing the configuration of the information processing device related to this disclosure. This is a diagram showing a specific example of the re-selection process related to this disclosure. This is a block diagram showing the configuration of a computer that functions as a learning device and information processing device related to this disclosure.

[0012] The following are examples of embodiments of the present invention. However, the present invention is not limited to the exemplary embodiments shown below, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining some or all of the technologies (things or methods) employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. Furthermore, embodiments obtained by appropriately omitting some of the technologies employed in each of the exemplary embodiments shown below may also be included in the scope of the present invention. In addition, the effects mentioned in each of the exemplary embodiments shown below are examples of effects that can be expected in that exemplary embodiment and do not define the scope of the present invention. That is, embodiments that do not produce the effects mentioned in each of the exemplary embodiments shown below may also be included in the scope of the present invention.

[0013] [First Exemplary Embodiment] A first exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. This exemplary embodiment is the basic form for each of the exemplary embodiments described later. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur. Furthermore, each technology shown in the drawings referenced to explain this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems occur.

[0014] (Configuration of the Learning Device) The configuration of the learning device 1 will be explained with reference to Figure 1. Figure 1 is a block diagram showing the configuration of the learning device 1. As shown in Figure 1, the learning device 1 comprises a selection unit 11, a re-selection unit 12, and a learning unit 13. The selection unit 11 selects the multiple images into multiple types of negative examples and positive examples based on the labels attached to the image of interest included in the multiple images and the labels attached to each of the multiple images. The re-selection unit 12 uses the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples, and the similarity between the feature quantities of the image of interest and the feature quantities of the negative examples, to re-select the multiple images selected by the selection unit 11 into multiple types of negative examples, including medium negative examples, and positive examples. The learning unit 13 uses the multiple images re-selected by the re-selection unit 12 to train a machine learning model such that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the moderately negative examples is given more weight than the similarity between the features of the image of interest and the features of negative examples other than the moderately negative examples.

[0015] (Effects of the learning device) As described above, the learning device 1 employs a configuration that includes: a selection unit 11 that sorts the multiple images into multiple types of negative examples and positive examples based on the labels attached to the image of interest included in the multiple images and the labels attached to each of the multiple images; a re-selection unit 12 that sorts the multiple images sorted by the selection unit 11 into multiple types of negative examples, including medium negative examples, and positive examples, using the similarity between the features of the image of interest and the features of the positive examples, and the similarity between the features of the image of interest and the features of the negative examples; and a learning unit 13 that uses the multiple images sorted by the re-selection unit 12 to train a machine learning model such that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the negative examples decreases, and (iii) the similarity between the features of the features of the image of interest and the features of the medium negative examples is given more weight than the similarity between the features of the image of interest and the features of negative examples other than the medium negative examples. Therefore, the learning device 1 has the effect of training a machine learning model to achieve higher inference accuracy.

[0016] (Flow of the Learning Method) The flow of the learning method S1 will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the learning method S1. As shown in Figure 2, the learning method S1 includes a selection process S11, a re-selection process S12, and a learning process S13. In the selection process S11, at least one processor selects the multiple images into multiple types of negative examples and positive examples based on the labels attached to the image of interest included in the multiple images and the labels attached to each of the multiple images. In the re-selection process S12, at least one processor re-selects the multiple images selected in the selection process S11 into multiple types of negative examples, including medium negative examples, and positive examples, using the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples, and the similarity between the feature quantities of the image of interest and the feature quantities of the negative examples. In the learning process S13, at least one processor uses the multiple images re-selected by the re-selection process S12 to train a machine learning model such that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the moderately negative examples is given more weight than the similarity between the features of the image of interest and the features of negative examples other than the moderately negative examples.

[0017] (Effects of the learning method) As described above, in the learning method S1, at least one processor performs a selection process S11 in which it selects the multiple images into multiple types of negative examples and positive examples based on the labels attached to the image of interest included in the multiple images and the labels attached to each of the multiple images, and at least one processor uses the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples, and the similarity between the feature quantities of the image of interest and the feature quantities of the negative examples to classify the multiple images selected by the selection process S11 into multiple types of negative examples, including intermediate negative examples, and positive examples. The system employs a configuration that includes a re-selection process S12 for re-selecting images, and a learning process S13 in which at least one processor uses the multiple images re-selected by the re-selection process S12 to train a machine learning model such that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the negative examples decreases, and (iii) the similarity between the features of the image of interest and the features of the moderately negative examples is given more weight than the similarity between the features of the image of interest and the features of negative examples other than the moderately negative examples. As a result, the learning method S1 has the effect of training the machine learning model to achieve higher inference accuracy.

[0018] [Second Exemplary Embodiment] A second exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0019] <1. Configuration of Information Processing Device 1A> The Information Processing Device 1A according to this disclosure is a learning device for training a machine learning model for performing an image recognition task. The Information Processing Device 1A is a device that performs an image recognition task using a machine learning model generated by machine learning. An example of an image recognition task is a classification task that identifies which of the predefined classes an object included as a subject in an image belongs to. An example of an object included as a subject in an image is a sample cell. In this case, in the classification task, the image to be recognized is classified into classes according to whether the sample cell is benign or malignant, and further classified into subclasses according to the type of sample cell. The Information Processing Device 1A can be used, for example, in cytology in rapid on-site evaluation (ROSE). The results of the classification by the above machine learning model support decision-making in diagnosis by, for example, a physician.

[0020] The configuration of the information processing device 1A will be described with reference to Figure 3. Figure 3 is a block diagram showing the configuration of the information processing device 1A. The information processing device 1A includes a control unit 10A, a storage unit 20A, a communication unit 30A, an input unit 40A, and an output unit 50A.

[0021] (Communication Unit) The communication unit 30A communicates with external devices of the information processing device 1A via a communication line N. The specific configuration of the communication line N is not limited to this exemplary embodiment, but examples of communication line N include a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public network, a mobile data communication network, or a combination thereof. The communication unit 30A transmits data supplied from the control unit 10A to other devices and supplies data received from other devices to the control unit 10A.

[0022] (Input Unit) The input unit 40A is configured to receive input to the information processing device 1A, and may include, for example, an input device such as a keyboard, mouse, touch panel, camera, or microphone. Alternatively, the input unit 40A may be configured to receive data from the input device via an interface such as USB (Universal Serial Bus).

[0023] (Output Unit) The output unit 50A is configured to output from the information processing device 1A, and may include, for example, an output device such as a display, printer, touch panel, or speaker. The output unit 50A may also be configured to include an interface such as USB, and output data to the output device via this interface.

[0024] (Storage Unit) The storage unit 20A stores various types of information that the control unit 10A refers to. Examples of such information include the image set IS and the machine learning model LM1 used for machine learning training. When we say that the machine learning model LM1 is stored in the storage unit 20A, we mean that the parameters that define the machine learning model LM1 are stored in the storage unit 20A.

[0025] (Image Set) The image set IS is a collection of images used to train the machine learning model LM1. The images included in the image set IS include, for example, images with objects such as sample cells as subjects. Each image included in the image set IS is labeled according to the type of sample cells it contains. More specifically, each image included in the set IS is labeled with a label indicating the class and subclass to which it belongs. In other words, each image included in the image set IS belongs to one of several classes, and each of those classes is further classified into one of several subclasses. The above classes and subclasses are classified according to the characteristics of the subjects contained in the images.

[0026] Figure 4 shows an example of classes and subclasses. Figure 4 shows that sample cells included as subjects in an image are classified into classes depending on whether they are benign or malignant. Furthermore, Figure 4 shows that sample cells included as subjects in an image are classified into subclasses depending on the type of cell. In Figure 4, for example, the class "benign cells" is further classified into several subclasses: "EC (normal epithelial cells)", "IEC (inflammatory EC)", "M (macrophages)", "LC (lymphocytes)", and "WBC (white blood cells)".

[0027] Furthermore, the images represent multiple middle classes, each of which is a subclass, and may belong to any of the multiple middle classes that group one or more subclasses. An example of further classifying the example shown in Figure 4 into middle classes is shown in Figure 5. Figure 5 is a diagram showing an example of a class, middle class, and subclass. In Figure 5, for example, the class "benign cells" is classified into three middle classes: "normal cells," "normal cells with findings," and "other normal cells." Also in Figure 5, the subclass "EC normal epithelial cells" is grouped within the middle class "normal cells," and the subclass "IEC inflammatory EC" and "M macrophages" are grouped within the middle class "normal cells with findings."

[0028] (Machine Learning Model) The machine learning model LM1 is a model generated by machine learning, such as a neural network. As an example, the machine learning model LM1 includes a first group of layers that take an image as input and generate feature quantities of the image, and a second group of layers connected to the first group of layers that take the image feature quantities as input and classify the input image into a class or subclass.

[0029] Figure 6 shows an example of a machine learning model LM1. In the example in Figure 6, the machine learning model LM1 includes a feature analysis model LM11 and a classifier LM12. The feature analysis model LM11 is an encoder (first layer) that takes an input image as input and generates features of the input image. The feature analysis model LM11 is used as a pre-trained model for the classifier LM12. The feature analysis model LM11 outputs features (feature vectors) of the input image when an input image containing the subject is input.

[0030] The classifier LM12 is connected to the feature analysis model LM11 and is a Classifier (second layer) that takes the features of the input image output from the feature analysis model LM11 as input and classifies the input image into a class or subclass. The machine learning model LM1 outputs the class or subclass classified by the Classifier as the classification result.

[0031] (Control Unit) As shown in Figure 3, the control unit 10A includes a learning phase execution unit 110A and an estimation phase execution unit 120A. The learning phase execution unit 110A includes an acquisition unit 111A, a sorting unit 112A, a re-sorting unit 113A, and a learning unit 114A. The estimation phase execution unit 120A includes a classification unit 121A. The sorting unit 112A, the re-sorting unit 113A, the learning unit 114A, and the classification unit 121A are examples of sorting means, re-sorting means, learning means, and class classification means according to this disclosure.

[0032] (Acquisition Unit) The acquisition unit 111A acquires an image set IS. For example, the acquisition unit 111A acquires an image set IS by receiving an image set IS from another device via the communication unit 30A. Alternatively, the acquisition unit 111A may acquire an image set IS input to the input unit 40A. More specifically, the acquisition unit 111A acquires, for example, an image output by an imaging device that images cells collected using an endoscope, or an image of cells contained in said image. Examples of imaging devices include microscope cameras and virtual slide scanners. Alternatively, the acquisition unit 111A may acquire an image set IS by reading it from a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A).

[0033] (Selection Unit) The selection unit 112A selects an arbitrary anchor (image of interest) from the image set IS and identifies the sample type of each of the other images included in the image set IS based on the label attached to the selected anchor and the labels attached to the other images included in the image set IS. The sample type indicates the selection result for comparative learning of the sample images. Examples of sample types identified by the selection unit 112A include positive sample, first negative sample (Group 1 negative sample), and second negative sample (Group 2 negative sample).

[0034] (Positive Examples) A ​​positive example is an image belonging to the same subclass as the anchor. A positive example may also be a data-enhanced image obtained by data-enhancing the anchor. Examples of data-enhanced images include an image in which the anchor has been rotated, an image in which the subject contained in the anchor has been moved, an image in which the subject contained in the anchor has been enlarged or reduced, an image in which the anchor has been flipped in at least one direction (up / down or left / right), an image in which a part of the anchor has been cropped, and an image in which the hue, saturation, or brightness of the anchor has been changed.

[0035] (First negative example, second negative example) The first negative example is an image that belongs to a different subclass than the subclass to which the anchor belongs, and also belongs to the same class as the anchor. The second negative example is an image that belongs to a different class than the class to which the anchor belongs. It can also be said that the first negative example is a sample that is more difficult to distinguish from a positive example than the second negative example. In the following, when it is not necessary to distinguish between the first and second negative examples, they will simply be referred to as "negative examples".

[0036] As an example, the selection unit 112A selects images belonging to the same subclass as the subclass to which the anchor belongs as positive examples. The selection unit 112A also selects images belonging to a different subclass than the subclass to which the anchor belongs, but belonging to the same class as the class to which the anchor belongs, as first negative examples. The selection unit 112A also selects images belonging to a different class than the class to which the anchor belongs as second negative examples. The selection unit 112A may also select data-extended images obtained by data-extending the anchor as positive examples.

[0037] Furthermore, the selection unit 112A may further select the first negative examples into multiple sample types. For example, if the images are classified into classes and subclasses in addition to middle classes, the selection unit 112A may select images from the first negative examples that belong to the same middle class as the anchor as third negative examples, and images that belong to a different middle class than the anchor as fourth negative examples. (Re-selection unit) The re-selection unit 113A re-selects the multiple images selected into multiple sample types by the selection unit 112A based on the features of each image. As an example, the re-selection unit 113A re-selects the multiple images selected by the selection unit 112A using at least one of the similarity between the anchor's features and the positive examples' features, and the similarity between the anchor's features and the negative examples' features. Examples of sample types that the re-selection unit 113A re-selects include positive examples, weakly negative examples (hard negative samples), moderately negative examples (semihard negative samples), and strongly negative examples (easy negative samples). In other words, as an example, the re-selection unit 113A re-selects multiple images that have been selected as negative examples by the selection unit 112A into weakly negative examples, moderately negative examples, and strongly negative examples. It can also be said that weakly negative examples are negative examples that are difficult to distinguish from positive examples, and strongly negative examples are negative examples that are easy to distinguish from positive examples. Moderately negative examples are easier to distinguish than weakly negative examples, but more difficult to distinguish than strongly negative examples. Hereafter, when it is not necessary to distinguish between weakly negative examples, moderately negative examples, and strongly negative examples, they will simply be referred to as "negative examples."

[0038] More specifically, the re-selection unit 113A re-selects multiple images based, as an example, on at least one of the following: (i) the confidence interval and minimum value of the similarity between the anchor and a positive example, (ii) the confidence interval, minimum value and maximum value of the similarity between the anchor and a first negative example, and (iii) the confidence interval and maximum value of the similarity between the anchor and a second negative example. The first negative example and the second negative example are examples of "one type of negative example among multiple types of negative examples" relating to this disclosure. Here, the similarity value increases as the degree of similarity increases.

[0039] The similarity between the anchor feature and the sample (positive example / first negative example / second negative example) can be measured by, for example, the distance in a predetermined feature space (Euclidean distance, etc.). In this case, the re-selection unit 113A can also re-select multiple images based on at least one of the distance between the anchor and the positive example, and the distance between the anchor and the negative example in the predetermined feature space. In this case, the smaller the distance, the greater the similarity. As an example, the re-selection unit 113A re-selects multiple images based on a threshold determined by (i) at least one of the confidence interval and maximum value of the distance between the anchor and the positive example, (ii) at least one of the confidence interval, maximum value and minimum value of the distance between the anchor and the first negative example, and (iii) at least one of the confidence interval and minimum value of the distance between the anchor and the second negative example in the predetermined feature space.

[0040] However, the similarity between the anchor feature and the sample feature is not limited to the examples described above. The similarity may be, for example, the dot product of the feature vector representing the anchor feature and the feature vector representing the sample feature, or cosine similarity, etc.

[0041] Furthermore, if the first negative example is further sorted into multiple sample types, the re-sorting unit 113A may perform re-sorting based on the similarity between each sample type and the anchor. For example, if the first negative example is sorted into a third negative example and a fourth negative example, the re-sorting unit 113A may re-sort the multiple images selected by the sorting unit 112A into positive examples, weakly negative examples, moderately negative examples, and strongly negative examples based on at least one of the following: the similarity between the anchor feature and the positive example feature, the similarity between the anchor feature and the third negative example feature, the similarity between the anchor feature and the fourth negative example feature, and the similarity between the anchor feature and the second negative example feature. Details of the re-sorting process performed by the re-sorting unit 113A will be described later.

[0042] (Learning Unit) The learning unit 114A trains the feature analysis model LM11 using the multiple images re-selected by the re-selection unit 113A. That is, the learning unit 114A updates the parameters that define the feature analysis model LM11, which are stored in the storage unit 20A. At this time, as an example, the learning unit 114A trains the feature analysis model LM11 so that (i) the similarity between the anchor feature and the positive example feature increases, (ii) the similarity between the anchor feature and the negative example feature decreases, and (iii) the similarity between the anchor feature and the medium negative example feature is given more weight than the similarity between the anchor feature and the negative example features other than the medium negative example. More specifically, when the re-selection unit 113A re-selects negative examples into weakly negative examples, moderately negative examples, and strongly negative examples, the re-selection unit 113A defines a loss function that changes most sensitively to the similarity between the anchor feature and the feature of the moderately negative example, and trains the feature analysis model LM11. In other words, the learning unit 114A trains the feature analysis model LM11 to prioritize moderately negative examples by controlling the contribution to learning for each type of negative example. Details of the loss function will be described later.

[0043] Furthermore, the learning unit 114A uses the image set SI, in which each image is labeled, to train a machine learning model LM1, in which a classifier LM12 is connected to a feature analysis model LM11. In other words, the learning unit 114A updates the parameters that define the machine learning model LM1, which are stored in the memory unit 20A. Details of the training process performed by the learning unit 114A will be described later.

[0044] (Classification Unit) The classification unit 121A acquires a target image, which is an image containing a subject that is the object of recognition for the image recognition task, and classifies the input target image into a class or subclass by inputting the acquired target image to the machine learning model LM1. For example, the classification unit 121A acquires a target image by receiving it from another device via the communication unit 30A. Alternatively, the classification unit 121A may acquire a target image input to the input unit 40A. Furthermore, the classification unit 121A may acquire a target image by reading it from a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A).

[0045] Furthermore, the classification unit 121A outputs the classification results. For example, the classification unit 121A may output the classification results by writing them to a storage location specified by the user of the information processing device 1A (which may be a storage device within the information processing device 1A or a storage device outside the information processing device 1A). Alternatively, the classification unit 121A may transmit the classification results via the communication unit 30A, or it may output the classification results to an output device such as a display. Examples of output devices to which the classification results are output include, but are not limited to, display devices such as displays or projectors used in intraoperative rapid diagnosis.

[0046] <2. Examples of processing by the re-sorting unit> Specific examples of the re-sorting process performed by the re-sorting unit 113A will be explained. Processing examples 1 to 3 will be explained as specific examples of the re-sorting process performed by the re-sorting unit 113A.

[0047] (Processing example 1 by the re-selection unit) In this example, the re-selection unit 113A re-selects the first negative example and the second negative example into weak negative example, medium negative example, and strong negative example. First, the re-selection unit 113A calculates a first interval that indicates the range in which the similarity between the anchor feature and the feature of the first negative example falls within a predetermined threshold, and a second interval that indicates the range in which the confidence level of the similarity between the anchor feature and the feature of the positive example falls within a predetermined threshold. Next, the re-selection unit 113A re-selects, from the images selected as negative examples (first negative example or second negative example) by the selection unit 112A, those with similarity to the anchor that are (i) less than the lower limit of the first interval are re-selected into strong negative examples, (ii) those with similarity to the anchor that are greater than or equal to the lower limit of the first interval and less than the lower limit of the second interval are re-selected into medium negative examples, and (iii) those with similarity to the lower limit of the second interval are re-selected into weak negative examples.

[0048] Here, we will explain an example of processing when distance is used as the similarity measure. In this example, the re-selection unit 113A calculates the 95% confidence interval of the distance Dis_AN1 from the anchor to the first negative example in a predetermined feature space. Distance Dis_AN1 is calculated as an example by the following equation (1). In equation (1), N1 is the set of the first negative examples, and n∈N1 is the first negative example included in the set N1. Dis_AN1 ｎThis is the Euclidean distance between an anchor in a given feature space and the first negative example n.

[0049] Furthermore, the re-selection unit 113A calculates the 95% confidence interval for the distance Dis_AP from the anchor to a positive example. Dis_AP is calculated, for example, by the following equation (2). In equation (2), P is the set of positive examples, and p∈P are positive examples included in the set P. Dis_AP ｐ This is the Euclidean distance between an anchor and a positive example p in a given feature space.

[0050] The re-selection unit 113A calculates the upper limit of the 95% confidence interval for the distance between the anchor and the first negative example as threshold th1. The re-selection unit 113A also calculates the upper limit of the 95% confidence interval for the distance between the anchor and the positive example as threshold th2.

[0051] Figure 7 shows a specific example of the re-selection process related to Processing Example 1. In Figure 7, Image A11 shows an example of the distribution of anchors, positive examples, first negative examples (Group 1 negative sample), and second negative examples (Group 2 negative sample) in a predetermined feature space. Image A12 schematically shows the thresholds th1 and th2 calculated by the re-selection unit 113A in the distribution of Image A11.

[0052] The re-selection unit 113A uses thresholds th1 and th2 to re-select the first and second negative examples into weak negative examples, medium negative examples, and strong negative examples. As an example, the re-selection unit 113A selects negative examples whose distance from the anchor is greater than or equal to threshold th1 (i.e., whose similarity is less than or equal to the threshold) as strong negative examples. The re-selection unit 113A also selects negative examples whose distance from the anchor is greater than or equal to threshold th2 but less than threshold th1 as medium negative examples. Furthermore, it selects negative examples whose distance from the anchor is less than threshold th2 (i.e., whose similarity is greater than the threshold) as weak negative examples. Image A13 in Figure 7 shows the state after the negative examples have been re-selected into weak negative examples (hard negative sample), medium negative examples (semi-hard negative sample), and strong negative examples (easy negative sample) using thresholds th1 and th2.

[0053] (Example of processing by the re-selection unit 2) The re-selection unit 113A may also select images that have been selected as negative examples (first negative examples or second negative examples) by the selection unit 112A and whose similarity to the anchor is greater than a threshold as examples not to be used for training the feature analysis model LM11. Here, the threshold is, for example, a value determined by the similarity between the anchor's feature and the positive example's feature. More specifically, the re-selection unit 113A may, for example, select negative examples whose distance to the anchor is less than the lower limit of the 95% confidence interval of the distance from the anchor to the positive example (i.e., the similarity to the anchor is greater than the upper limit of the 95% confidence interval of the positive example) as examples not to be used for training the feature analysis model LM11. In this case, the re-selection unit 113A may, for example, select negative examples whose distance to the anchor is greater than or equal to the threshold th1 as strongly negative examples, and images whose distance is greater than or equal to the threshold th2 but less than the threshold th1 as weakly negative examples.

[0054] As another example, the re-selection unit 113A may, for instance, exclude from the training of the feature analysis model LM11 images that have been selected as first negative examples by the selection unit 112A if their distance to the anchor is greater than the threshold th1 (i.e., their similarity is less than the threshold). In other words, the re-selection unit 113A may select images that have been selected as first negative examples by the selection unit 112A if their distance to the anchor is greater than the threshold th1 as examples that will not be used to train the feature analysis model LM11.

[0055] Furthermore, the re-selection unit 113A may exclude from the training of the feature analysis model LM11 images that have been selected as second negative examples by the selection unit 112A and whose distance from the anchor is less than the threshold th1 (i.e., whose similarity is greater than the threshold). In other words, the re-selection unit 113A may select images that have been selected as second negative examples by the selection unit 112A and whose distance from the anchor is less than the threshold th1 as examples that will not be used to train the feature analysis model LM11.

[0056] (Processing Example 3 by Re-selection) The re-selection unit 113A may also perform sample re-selection by combining the above-described processing example 1 and processing example 2. Specifically, as an example, the re-selection unit 113A may select negative examples such as: threshold th1 being the upper limit of the 95% confidence interval of the distance between the anchor and the first negative example; threshold th2 being the upper limit of the 95% confidence interval of the distance between the anchor and the positive example; threshold th3 being the upper limit of the 99% confidence interval of the distance between the anchor and the positive example; strongly negative examples being those whose distance from the anchor is greater than threshold th1; moderately negative examples being those whose distance from the anchor is less than or equal to threshold th1 and greater than threshold th2; weakly negative examples being those whose distance from the anchor is less than or equal to threshold th2 and greater than threshold th3; and negative examples whose distance from the anchor is less than or equal to threshold th3 being those not used for training the feature analysis model LM11.

[0057] (Other examples of thresholds used for re-selection) The thresholds used by the re-selection unit 113A for re-selection are not limited to the examples described above. For example, in processing examples 1 to 3 described above, the upper limit of the 95% confidence interval of the distance between the anchor and the first negative example was used as the threshold th1. However, the threshold th1 may also be the maximum value of the distance between the anchor and the first negative example, or the minimum value of the distance between the anchor and the second negative example. Alternatively, the threshold th1 may be, for example, the lower limit of the 95% confidence interval of the distance from the anchor to the second negative example. Furthermore, the threshold th1 may be the average value of the upper limit of the 95% confidence interval of the distance between the anchor and the first negative example and the lower limit of the 95% confidence interval of the distance between the anchor and the second negative example.

[0058] Furthermore, in the processing examples 1 to 3 described above, the upper limit of the 95% confidence interval for the distance between the anchor and the positive example was used as the threshold th2. However, the threshold th2 may, for example, be the maximum value of the distance between the anchor and the positive example. Alternatively, the threshold th2 may be, for example, the lower limit of the 95% confidence interval for the distance between the anchor and the first negative example. Alternatively, the threshold th2 may be the average value of the upper limit of the 95% confidence interval for the distance between the anchor and the positive example and the lower limit of the 95% confidence interval for the distance between the anchor and the first negative example.

[0059] Furthermore, in the above-described processing example 3, the upper limit of the 97% confidence interval for the distance between the anchor and the positive example was used as the threshold th3. However, the threshold th3 may, for example, be the maximum value of the distance between the anchor and the positive example. Also, the threshold th3 may be, for example, the lower limit of the 95% confidence interval for the distance between the anchor and the first negative example. Furthermore, the threshold th2 may be the average value of the upper limit of the 95% confidence interval for the distance between the anchor and the positive example and the lower limit of the 95% confidence interval for the distance between the anchor and the first negative example.

[0060] (Example of re-selection process when selected into three or more types of negative examples) Furthermore, if the first negative example is further selected into multiple sample types, the re-selection unit 113A may perform re-selection based on the similarity between each sample type and the anchor. For example, if the first negative example is selected into a third negative example and a fourth negative example, the re-selection unit 113A may, as an example, set the upper limit of the 95% confidence interval of the distance from the anchor to the third negative example as a threshold th4, and select the third and fourth negative examples, whose distance to the anchor is less than or equal to the threshold th4, as first intermediate negative examples. Alternatively, the re-selection unit 113A may, as an example, select the third and fourth negative examples, whose distance to the anchor is greater than the threshold th4, as second intermediate negative examples. The threshold th4 is not limited to the upper limit of the 95% confidence interval for the distance between the anchor and the third negative example, but may be, for example, (i) the maximum distance from the anchor to the third negative example, (ii) the lower limit of the confidence interval for the distance from the anchor to the fourth negative example, (iii) the minimum distance from the anchor to the fourth negative example, (iv) the average of the upper limit of the 95% confidence interval for the distance from the anchor to the third negative example and the lower limit of the confidence interval for the distance from the anchor to the fourth negative example, etc.

[0061] Furthermore, the re-selection unit 113A may, as an example, use the upper limit of the 95% confidence interval for the distance from the anchor to the fourth negative example as threshold th5, select the fourth negative example whose distance to the anchor is greater than threshold th5 as a strongly negative example, and select the second negative example whose distance to the anchor is less than threshold th5 as a second moderately negative example. The threshold th5 is not limited to the upper limit of the 95% confidence interval for the distance from the anchor to the fourth negative example, but may be, for example, (i) the maximum value of the distance from the anchor to the fourth negative example, (ii) the lower limit of the confidence interval for the distance from the anchor to the second negative example, (iii) the minimum value of the distance from the anchor to the second negative example, (iv) the average value of the upper limit of the 95% confidence interval for the distance from the anchor to the fourth negative example and the lower limit of the confidence interval for the distance from the anchor to the second negative example, etc.

[0062] <3. Examples of processing by the learning unit> Next, we will explain the process by which the learning unit 114A (a) trains the feature analysis model LM11, and (b) trains the machine learning model LM1, in which the classifier LM12 is connected to the feature analysis model LM11.

[0063] (Example 1 of training the feature analysis model LM11) In this example, the learning unit 114A trains the feature analysis model LM11 using a loss function whose contribution changes depending on whether the image is a positive example, a weakly negative example, a moderately negative example, or a strongly negative example. As an example, the learning unit 114A trains the feature analysis model LM11 so that (i) the similarity between the anchor feature and the positive example feature (dot product of feature vectors, etc.) becomes large, (ii) the similarity between the anchor feature and the negative example feature becomes small, and (iii) the similarity between the anchor feature and the moderately negative example feature is given more weight than the similarity between the anchor feature and the negative example feature other than the moderately negative example.

[0064] As an example, the learning unit 114A trains the feature analysis model LM11 so that the value of the following loss function, equation (3), becomes small. I: The set of images belonging to the class to which the anchor belongs i: The anchor (the image of interest) z ｉ : Feature vector of anchor i P(i): Set of positive examples for anchor i p: Positive examples included in set P(i) zｐ : Feature vector E(i) of positive example p: Set E(i) of strong negative examples for anchor i; e: Strong negative example contained in set E(i); z ｅ : Feature vector W of strong negative example e ｅ : Weight coefficient W given to strong negative example e; S(i): Set S(i) of medium negative examples for anchor i; s: Medium negative example contained in set S(i); z ｓ : Feature vector W of medium negative example s ｓ : Weight coefficient W given to medium negative example s; H(i): Set H(i) of weak negative examples for anchor i; h: Weak negative example contained in set H(i); z ｈ : Feature vector W of weak negative example h ｈ : Weight coefficient W given to weak negative example h; ·: Inner product; τ: Temperature parameter (constant)

[0065] The temperature parameter τ is a constant, and as an example, it is a constant of 0.1 or less. Weight coefficient W ｅ , W ｓ , W ｈ are real numbers from 0 to 1. Weight coefficient W ｅ , W ｓ , W ｈ are, as an example, W ｓ = 0.6, W ｈ = 0.3, W ｅ = 0.1.

[0066] The learning unit 114A learns the feature analysis model LM11 so that the inner product between the feature vector of the anchor and the feature vector of the positive example becomes larger in order to reduce the value of the formula (3) which is a loss function. Also, the learning unit 114A learns the feature analysis model LM11 so that the inner product between the feature vector of the anchor and the feature vector of the negative example becomes smaller. That is, in the formula (3), the feature quantity is the feature vector, and the similarity between the feature vector of the anchor and the feature vectors of the positive example and each of the plurality of types of negative examples is based on the inner product between the feature vector of the anchor and the feature vectors of the positive example and each of the plurality of types of negative examples.

[0067] In the formula (3), the sum of the weight coefficient W ｈ given to the weak negative example and the weight coefficient W ｓ given to the medium negative example and the weight coefficient W ｎ given to the strong negative example is 1.0. Also, the weight coefficient Wｓ The weight coefficient W ｅ It is set to a larger value. Therefore, the value of equation (3) changes more depending on the inner product of the anchor feature vector and the feature vector of the moderately negative example than on the inner product of the anchor feature vector and the feature vector of the strongly negative example. In other words, the learning unit 114A is trained to move the moderately negative example further away from the anchor.

[0068] Furthermore, in equation (3), the weighting coefficient W ｓ and weight coefficient W ｈ This refers to the weight coefficient W ｅ It may be set to a larger value. In other words, the learning unit 114A may train the feature analysis model LM11 so that the change in similarity between the anchor feature and the feature of the moderately negative example, and the change in similarity between the anchor feature and the feature of the weakly negative example, are greater than the change in similarity between the anchor feature and the feature of the strongly negative example.

[0069] Furthermore, in equation (3), the weighting coefficient W ｓ The weight coefficient W ｈ It may be set to a larger value. In other words, the learning unit 114A may train the feature analysis model LM11 such that the change in similarity between the anchor feature and the moderately negative example feature is greater than the change in similarity between the anchor feature and the weakly negative example feature.

[0070] Figure 8 is a diagram illustrating the learning process of the feature analysis model LM11. In Figure 8, threshold th11 is the lower limit of the 95% confidence interval for the distance from the anchor to the moderately negative example, and threshold th21 is the upper limit of the 95% confidence interval for the distance from the anchor to the positive example. As an example, the learning unit 114A calculates the upper limit of the 95% confidence interval for the distance from the anchor to the positive example (threshold th21 in Figure 8) and the lower limit of the 95% confidence interval for the distance from the anchor to the moderately negative example (threshold th11 in Figure 8) during the learning process, and continues learning until a predetermined condition is met, at which point learning ends. As an example of a predetermined condition, the predetermined condition may be that the number of negative examples located inside threshold th21 is less than or equal to a predetermined percentage (e.g., less than 1% of the total number of negative examples). Furthermore, the predetermined conditions may include, for example, a condition where the distance between threshold th21 and threshold th11 is greater than or equal to a predetermined value (e.g., the distance from the anchor to threshold th11 is twice or more the distance from the anchor to threshold th21).

[0071] (Example 2 of training the feature analysis model LM11) As described above, the images included in the image set IS may be multiple middle classes, each of which is a subclass, and may belong to any of the multiple middle classes that group one or more subclasses. In this case, each image is selected as a positive example, a weakly negative example, a first moderately negative example, a second moderately negative example, or a strongly negative example, as described above.

[0072] As an example, the learning unit 114A trains the feature analysis model LM11 so that the similarity between the anchor feature and the first neutral example feature is greater than the similarity between the anchor feature and the second neutral example feature. As an example, the learning unit 114A trains the feature analysis model LM11 so that the value of the following loss function, equation (4), becomes small. I: The set of images belonging to the class to which the anchor belongs i: The anchor (the image of interest) z ｉ : Feature vector of anchor i P(i): Set of positive examples for anchor i p: Positive examples included in set P(i) z ｐ : Feature vector of positive example p E(i): Set of strongly negative examples for anchor i e: Strongly negative examples included in set E(i) z ｅ: Feature vector of strongly negative example e W ｅ : Weight coefficient assigned to strongly negative example e S1(i): Set of first moderately negative examples for anchor i s1: First moderately negative example included in set S1(i) z ｓ１ : Feature vector W of the first negative example s1 ｓ１ : Weight coefficient assigned to the first neutral example s1 S2(i): Set of second neutral examples for anchor i s2: Second neutral example included in set S2(i) z ｓ２ : Feature vector W of the second negative example s2 ｓ２ : Weight coefficient assigned to the second moderately negative example s2 H(i): Set of weakly negative examples for anchor i h: Weakly negative examples included in set H(i) z ｈ : Feature vector W of a weakly negative example h ｈ : Weighting coefficient assigned to the weakly negative example h : Inner product τ: Temperature parameter (constant)

[0073] The learning unit 114A trains the feature analysis model LM11 so that the dot product between the anchor feature vector and the feature vector of the first neutral example is small, in order to reduce the value of equation (4), which is the loss function. Furthermore, the learning unit 114A trains the feature analysis model LM11 so that the dot product between the anchor feature vector and the feature vector of the second neutral example is small. In addition, the learning unit 114A trains the feature analysis model LM11 so that the dot product between the anchor feature vector and the feature vector of the positive example is large. Furthermore, for strongly negative examples, the learning unit 114A trains the feature analysis model LM11 so that the dot product between the anchor feature vector and the feature vector of the strongly negative example is small.

[0074] In equation (4), the weighting coefficient W is assigned to the weakly negative example. ｈ And the weight coefficient W is assigned to the first negative example. ｓ１ And the weight coefficient W is assigned to the second negative example. ｓ２ And the weight coefficient W is assigned to strongly negative examples. ｅ１ The sum of these is 1.0. Also, the weight coefficient W ｓ１ and weight coefficient W ｓ２ The weight coefficient W ｅIt is set to a larger value. Therefore, the value of equation (4) changes more depending on the inner product of the anchor's feature vector and the feature vectors of the moderately negative examples (the first moderately negative example and the second moderately negative example) than on the inner product of the anchor's feature vector and the feature vector of the strongly negative example. In other words, the learning unit 114A is trained to move the first moderately negative example and the second moderately negative example further away from the anchor.

[0075] Furthermore, in equation (4), the weighting coefficient W ｓ１ The weight coefficient W ｓ２ It may be set to a larger value. In this case, the value of equation (4) changes more depending on the dot product between the anchor feature vector and the first neutral example's feature vector than on the dot product between the anchor feature vector and the second neutral example's feature vector. That is, the learning unit 114A is trained to move the first neutral example further away from the anchor.

[0076] (Example of the process for training the machine learning model LM1) The learning unit 114A also trains the machine learning model LM1 using multiple images and teacher labels corresponding to the results of classifying each of the multiple images into a class or subclass. Known methods are used for the method and loss function (second loss function) by the learning unit 114A for training the machine learning model LM1. One example is a method that minimizes the error between the output data from the machine learning model LM1 and the teacher labels using cross-entropy. The learning unit 114A mainly trains the classifier LM12, but may also train the feature analysis model LM11 to make fine adjustments as needed.

[0077] <4. Effects of Information Processing Device 1A> As shown in image A12 of Figure 7, there are cases where negative examples have a greater similarity to the anchor than positive examples. This could be because the negative example happens to have features similar to the positive example, or because of mislabeling. In addition, there are cases where doctors may have differing opinions, and the labels attached to the images are not always appropriate. When a machine learning model is trained using such a set of images, the negative examples as a whole may be mistakenly perceived as already sufficiently far from the anchor in the pre-training state, and as a result, the training may not proceed properly.

[0078] In contrast, with the information processing device 1A of this disclosure, even if the label information classifies an image as a negative example, if the training data actually contains an image whose similarity to the anchor is as high as that of a positive example, the re-selection unit 113A can re-select the image, thereby allowing the feature analysis model LM11 to be trained more appropriately.

[0079] Furthermore, according to the information processing device 1A, even if the training data includes images that are classified as strongly negative examples based on label information but are actually relatively close to the anchor (weakly negative examples in terms of distance from the anchor in the feature space), or images that are classified as weakly negative examples based on label information but are actually relatively far from the anchor (strongly negative examples in terms of distance from the anchor in the feature space), the re-selection unit 113A can re-select the images, thereby allowing the feature analysis model LM11 to be trained more appropriately. In addition, according to the information processing device 1A, even if the training data contains inaccurate label information, the re-selection unit 113A can re-select the images, thereby allowing the feature analysis model LM11 to be trained more appropriately.

[0080] Furthermore, according to the information processing device 1A of this disclosure, the feature analysis model LM11 is trained so that the similarity between the anchor feature and the weakly negative example feature is greater than the similarity between the anchor feature and the strongly negative example feature. This makes it possible to further improve the classification accuracy of the machine learning model LM1.

[0081] Furthermore, in the information processing device 1A according to this disclosure, the re-selection unit 113A re-selects the multiple images selected as negative examples by the selection unit 112A into weakly negative examples, moderately negative examples, and strongly negative examples, and the learning unit 114A trains the feature analysis model LM11 to prioritize moderately negative examples by controlling the contribution to learning for each type of negative example. Therefore, with the information processing device 1A, by dividing the negative examples into weakly negative examples, moderately negative examples, and strongly negative examples and performing learning, the feature analysis model LM11 can be trained to have a higher inference accuracy for the machine learning model LM1.

[0082] Furthermore, in the information processing device 1A relating to this disclosure, the re-selection unit 113A calculates a first interval that indicates the range in which the similarity between the anchor feature quantity and the feature quantity of the first negative example included in the multiple types of negative examples selected by the selection unit 112A falls within a predetermined threshold, and a second interval that indicates the range in which the confidence level of the similarity between the anchor feature quantity and the feature quantity of the positive example falls within a predetermined threshold. Among the images selected as negative examples by the selection unit 112A, those with similarity to the anchor that are (i) less than the lower limit of the first interval are re-selected as strongly negative examples, (ii) those with similarity to the anchor that are greater than or equal to the lower limit of the first interval and less than the lower limit of the second interval are re-selected as moderately negative examples, and (iii) those with similarity to the lower limit of the second interval are re-selected as weakly negative examples. Therefore, the information processing device 1A can train the feature analysis model LM11 to improve the inference accuracy of the machine learning model LM1 by re-selecting negative examples using the confidence interval of the similarity between the anchor and the first negative example and the confidence interval of the similarity between the anchor and the positive example.

[0083] Furthermore, in the information processing device 1A relating to this disclosure, the learning unit 114A is configured to train the feature analysis model LM11 such that the change in similarity between the anchor feature and the moderately negative example feature, and the change in similarity between the anchor feature and the weakly negative example feature, are greater than the change in similarity between the anchor feature and the strongly negative example feature. By giving more weight to moderately negative examples and weakly negative examples in the training than to strongly negative examples, the information processing device 1A can train the feature analysis model LM11 so that the inference accuracy of the machine learning model LM1 is higher.

[0084] Furthermore, in the information processing device 1A relating to this disclosure, the learning unit 114A is configured to train the feature analysis model LM11 such that the change in the similarity between the anchor feature and the feature of a moderately negative example is greater than the change in the similarity between the anchor feature and the feature of a weakly negative example.

[0085] Among the weakly negative examples, there is a possibility of mislabeling, or there may be cases where doctors disagree on the judgment, and the label may not always be appropriate. Therefore, if the model is trained to move weakly negative examples away from the anchor, the model may also learn to move samples with a high degree of commonality with the anchor away from the anchor, resulting in inappropriate learning. In contrast, the information processing device 1A according to this disclosure allows the feature analysis model LM11 to be trained in such a way that the inference accuracy of the machine learning model LM1 is higher, by giving more weight to moderately negative examples than weakly negative examples during training.

[0086] Furthermore, in the information processing device 1A relating to this disclosure, the re-selection unit 113A is configured to re-select multiple images based on at least one of the following: (i) the confidence interval and minimum value of the similarity between the anchor and a positive example, and (ii) the confidence interval, maximum value and minimum value of the similarity between the anchor and one of the multiple types of negative examples. Therefore, according to the information processing device 1A, by re-selecting images using the confidence interval and minimum value of the similarity between the anchor and a positive example, the confidence interval and maximum value of the similarity between the anchor and a negative example, etc., the feature analysis model LM11 can be trained to have a higher inference accuracy for the machine learning model LM1.

[0087] Furthermore, in the information processing device 1A according to this disclosure, the re-selection unit 113A is configured to select images selected as negative examples by the selection unit 112A, where the similarity to the anchor is greater than a threshold determined by the similarity between the anchor's feature quantity and the positive example's feature quantity, and these are selected as examples not to be used for training the feature analysis model LM11. Therefore, according to the information processing device 1A, by excluding negative examples with a similarity to the anchor greater than the threshold from training the feature analysis model LM11, the feature analysis model LM11 can be trained to have a higher inference accuracy than the machine learning model LM1.

[0088] Furthermore, the information processing device 1A according to this disclosure employs a configuration in which the subjects included in the images used for learning are sample cells, and the images are labeled according to the type of sample cells included in the images. Therefore, the information processing device 1A has the effect of improving the inference accuracy of the machine learning model LM1 that classifies the input images according to the type of sample cells.

[0089] Furthermore, the information processing device 1A according to this disclosure is configured to include a classification unit 121A that classifies an input image containing a subject into a class or subclass by inputting the image to a machine learning model LM1. Therefore, the information processing device 1A has the effect of being able to classify the class or subclass of an image containing a subject with greater accuracy by using the machine learning model LM1 generated by machine learning.

[0090] Furthermore, in the information processing device 1A according to this disclosure, the images included in the image set IS belong to one of several classes, and each of those classes is further classified into one of several subclasses. The labels attached to the images indicate the class and subclass to which the image belongs. The sorting unit 112A sorts images belonging to the subclass to which the anchor belongs as positive examples, images belonging to a subclass different from the subclass to which the anchor belongs, and also belonging to the class to which the anchor belongs, as first negative examples, and images belonging to a class different from the class to which the anchor belongs, as second negative examples. Therefore, with the information processing device 1A, by using images belonging to classes and subclasses, the feature analysis model LM11 can be trained to improve the inference accuracy of the machine learning model LM1.

[0091] Furthermore, in the information processing device 1A according to this disclosure, the images included in the image set IS are further divided into multiple middle classes, each of which is further classified into multiple middle classes, and each middle class is a group of one or more subclasses. The sorting unit 112A selects from the first negative examples images that belong to the same middle class as the middle class to which the anchor belongs as the third negative example, and selects from the first negative examples images that belong to a different middle class from the middle class to which the anchor belongs as the fourth negative example. The re-selection unit 113A re-selects the multiple images selected by the selection unit 112A into positive examples, weakly negative examples, first moderately negative examples, second semi-negative examples, and strongly negative examples based on at least one of the following: (i) the similarity between the anchor feature and the positive example feature, (ii) the similarity between the anchor feature and the third negative example feature, (iii) the similarity between the anchor feature and the fourth negative example feature, and (iv) the similarity between the anchor feature and the second negative example feature. Therefore, the information processing device 1A trains the feature analysis model LM11 by considering the first moderately negative example and the second moderately negative example, which are further selected from the moderately negative examples. This provides the effect of training the feature analysis model LM11 to have higher inference accuracy than the machine learning model LM1.

[0092] Furthermore, in the information processing device 1A related to this disclosure, the subject matter included in the images used for learning is sample cells, and the images are classified into classes according to whether the sample cells included in the images are benign or malignant, and further classified into subclasses according to the type of sample cells included in the images. Therefore, the information processing device 1A has the effect of improving the inference accuracy of the machine learning model LM1 that classifies the input images into classes indicating whether the sample cells are benign or malignant, and into subclasses according to the type of sample cells.

[0093] Furthermore, in the information processing device 1A according to this disclosure, the machine learning model LM1 includes a first group of layers that take an image as input and generate feature quantities of the image, and a second group of layers connected to the first group of layers that take the image feature quantities as input and classify the input image into a class or subclass. The learning unit 114A trains the machine learning model LM1 using a plurality of images and teacher labels corresponding to the results of classifying each of the plurality of images into a class or subclass. As a result, the information processing device 1A has the effect of being able to generate a machine learning model LM1 with higher inference accuracy.

[0094] [Third Exemplary Embodiment] A third exemplary embodiment, which is an example of an embodiment of the present invention, will be described in detail with reference to the drawings. Components having the same function as those described in the above-described exemplary embodiment are denoted by the same reference numerals, and their descriptions are omitted as appropriate. The scope of application of each technology adopted in this exemplary embodiment is not limited to this exemplary embodiment. That is, each technology adopted in this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise. Furthermore, each technology shown in the drawings referenced to describe this exemplary embodiment can also be adopted in other exemplary embodiments included in this disclosure, to the extent that no particular technical problems arise.

[0095] (Configuration of Information Processing Device 1B) The configuration of Information Processing Device 1B will be explained with reference to Figure 9. Figure 9 is a block diagram showing the configuration of Information Processing Device 1B. Information Processing Device 1B includes a re-selection unit 113B and a learning unit 114B in place of the re-selection unit 113A and learning unit 114A provided in Information Processing Device 1A. The re-selection unit 113B re-selects the samples selected by the selection unit 112A. The learning unit 114B trains the machine learning model LM1 using the multiple images re-selected by the re-selection unit 113B.

[0096] (Example of processing by the re-selection unit 1) Figure 10 is a diagram showing a specific example of the re-selection process performed by the re-selection unit 113B. In Figure 10, image A11 shows an example of the distribution of anchors, positive examples, first negative examples (Group 1 negative sample), and second negative examples (Group 2 negative sample) in a predetermined feature space. Image A12 schematically shows the thresholds th1 and th2 calculated by the re-selection unit 113B in the distribution of image A11. The method for calculating thresholds th1 and th2 is the same as the calculation method by the re-selection unit 113A described above, and a detailed explanation is omitted here.

[0097] The re-selection unit 113B uses thresholds th1 and th2 to re-select the first and second negative examples into weak negative examples, moderate negative examples, and strong negative examples. The re-selection process for negative examples performed by the re-selection unit 113B is the same as the re-selection process performed by the re-selection unit 113A described above. That is, as an example, the re-selection unit 113A selects negative examples whose distance from the anchor is greater than or equal to threshold th1 (i.e., whose similarity is less than or equal to the threshold) as strong negative examples. The re-selection unit 113A also selects negative examples whose distance from the anchor is greater than or equal to threshold th2 but less than threshold th1 as moderate negative examples. Furthermore, it selects negative examples whose distance from the anchor is less than threshold th2 (i.e., whose similarity is greater than the threshold) as weak negative examples.

[0098] Furthermore, in addition to the process of re-selecting negative examples, the re-selection unit 113B re-selects positive examples into easily positive samples and hard positive samples. In other words, the re-selection unit 113B uses the similarity between the anchor feature and the positive example feature to re-select the images selected as positive examples by the selection unit 112A into hard positive examples and hard positive examples. Hard positive examples can also be described as positive examples that are easy to distinguish from negative examples, while hard positive examples are positive examples that are difficult to distinguish from negative examples.

[0099] As an example, the re-selection unit 113B selects positive examples where the distance to the anchor is greater than or equal to the threshold th2 (i.e., the similarity is less than or equal to the threshold) as weak positive examples. The re-selection unit 113A selects negative examples where the distance to the anchor is less than the threshold th2 as strong positive examples. In other words, as an example, the re-selection unit 113B calculates a second interval that indicates the range in which the confidence level of the similarity between the anchor feature and the positive example feature falls within a predetermined threshold. The re-selection unit 113B also re-selects images selected as positive examples by the selection unit 112A that have a similarity to the anchor feature greater than or equal to the lower limit of the second interval as strong positive examples, and re-selects positive examples where the similarity to the anchor feature is less than the lower limit of the second interval as weak positive examples.

[0100] Image A33 in Figure 10 shows the state in which negative examples are re-selected into weakly negative examples, moderately negative examples, and strongly negative examples using thresholds th1 and th2, and positive examples are re-selected into weakly positive examples and strongly positive examples using threshold th2.

[0101] (Example of processing by the re-selection unit 2) Alternatively, the re-selection unit 113B may select positive examples whose distance from the anchor is greater than or equal to the threshold th2 as examples that will not be used for training the feature analysis model LM11.

[0102] (Example of processing by the learning unit) In this example, the learning unit 114B trains the feature analysis model LM11 using a loss function whose contribution changes depending on whether the image is a strongly positive example, a weakly positive example, a weakly negative example, a moderately negative example, or a strongly negative example. At this time, the learning unit 114A trains the feature analysis model LM11 to give more weight to the similarity between the anchor feature and the strongly positive example feature than to the similarity between the anchor feature and the weakly positive example feature.

[0103] As an example, the learning unit 114B trains the feature analysis model LM11 so that the value of the loss function shown in equation (5) becomes small. EP(i): Set of strongly positive examples for anchor i ep: Strongly positive examples included in set EP(i) z ｅｐ : Feature vector W of the strongly positive example ep ｅｐ : Weight coefficient assigned to strongly positive example ep HP(i): Set of weakly positive examples for anchor i hp: Weakly positive examples included in set HP(i) zｈｐ : Feature vector of weakly positive example hp W ｈｐ : Weight coefficient assigned to weak positive example hp

[0104] In equation (5), the set I, E(i), H(i), S(i), anchor i, moderately negative example s, strongly negative example e, weakly negative example h, and feature vector z ｉ , z ｅ , z ｓ , z ｈ Weight coefficient W ｅ , W ｓ , W ｈ The dot product and the temperature parameter τ are the same as those explained in equation (3) above, and their explanation is omitted here.

[0105] Weight coefficient W ｅ and W ｓ and W ｈ The sum of these is 1.0. Weight coefficient W ｅ and W ｓ and W ｈ For example, W ｓ >W ｈ >W ｅ A value that results in (for example, W) ｓ = 0.6, W ｈ = 0.3, W ｅ Set the weight coefficient W to 0.1. ｅｐ , W ｈｐ is a real number between 0 and 1, and the weight coefficient W ｅｐ and W ｈｐ The sum of these is 1.0. Weight coefficient W ｅｐ , W ｈｐ For example, W ｈｐ >W ｅｐ A value that results in (for example, W) ｈｐ = 0.8, W ｅｐ Set it to =0.2).

[0106] The learning unit 114B continues learning the feature analysis model LM11 until a predetermined condition is met, at which point it terminates. The predetermined condition is the same as the condition used by the learning unit 114A described above. For example, the condition may be that the number of negative examples located inside the threshold th21 (the upper limit of the 95% confidence interval for the distance from the anchor to a positive example) in Figure 8 is less than or equal to a predetermined percentage (less than 1% of the total negative examples, etc.). Alternatively, the predetermined condition may be that the distance between threshold th21 and threshold th11 (threshold th11 is the lower limit of the 95% confidence interval for the distance from the anchor to a moderately negative example) is greater than or equal to a predetermined value (for example, the distance from the anchor to threshold th11 is more than twice the distance from the anchor to threshold th21, etc.).

[0107] (Effects of Information Processing Device 1B) In the Information Processing Device 1B according to this disclosure, the re-selection unit 113B re-selects images selected as positive examples by the selection unit 112A into weak positive examples and strong positive examples using the similarity between the anchor feature and the positive example feature, and the learning unit 114B trains the machine learning model to give more weight to the similarity between the anchor feature and the strong positive example feature than to the similarity between the anchor feature and the weak positive example feature. Therefore, according to the Information Processing Device 1B, by dividing positive examples into weak positive examples and strong positive examples and performing training, the feature analysis model LM11 can be trained to have a higher inference accuracy for the machine learning model LM1.

[0108] Furthermore, in the information processing device 1B relating to this disclosure, the re-selection unit 113B calculates a second interval that indicates the range in which the confidence level of the similarity between the anchor feature and the positive example feature falls within a predetermined threshold. Among the images selected as positive examples by the selection unit 112A, those whose similarity to the anchor feature is equal to or greater than the lower limit of the second interval are re-selected as strong positive examples, and positive examples whose similarity to the anchor feature is less than the lower limit of the second interval are re-selected as weak positive examples. Therefore, according to the information processing device 1B, by re-selecting positive examples using the confidence interval of the similarity between the anchor and the positive example, the feature analysis model LM11 can be trained to improve the inference accuracy of the machine learning model LM1.

[0109] [Example of implementation by software] Some or all of the functions of the learning device 1 and the information processing devices 1A and 1B (hereinafter also referred to as "each of the above devices") may be implemented by hardware such as integrated circuits (IC chips) or by software.

[0110] In the latter case, each of the above devices is implemented, for example, by a computer that executes instructions for a program, which is software that realizes each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 11. Figure 11 is a block diagram showing the hardware configuration of computer C, which functions as each of the above devices.

[0111] Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P for operating Computer C as each of the above-mentioned devices. In Computer C, the processor C1 reads and executes the program P from memory C2, thereby realizing each of the above-mentioned devices.

[0112] For processor C1, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), TPU (Tensor Processing Unit), quantum processor, microcontroller, or a combination thereof can be used. For memory C2, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof can be used.

[0113] Furthermore, computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Computer C may also be equipped with a communication interface for sending and receiving data with other devices. Furthermore, computer C may also be equipped with an input / output interface for connecting input / output devices such as a keyboard, mouse, display, and printer.

[0114] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such recording medium M can include, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such recording medium M. Program P can also be transmitted via a transmission medium. Such transmission mediums can include, for example, a communication network or broadcast waves. Computer C can also acquire program P via such transmission medium.

[0115] Furthermore, each of the above functions of each of the above devices may be implemented by a single processor in a single computer, by multiple processors in a single computer working together, or by multiple processors in each of multiple computers working together. In addition, the programs for implementing each of the above functions in each of the above devices may be stored in a single memory in a single computer, distributed and stored in multiple memories in a single computer, or distributed and stored in multiple memories in each of multiple computers.

[0116] [Addendum A] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0117] (Appendix A1) A learning device comprising: a selection means for selecting a plurality of images into a plurality of types of negative examples and positive examples based on a label attached to a particular image included in a plurality of images and a label attached to each of the plurality of images; a re-selection means for re-selecting the plurality of images selected by the selection means into a plurality of types of negative examples, including a moderate negative example, and positive examples, using the similarity between the feature quantities of the particular image and the feature quantities of the positive examples, and the similarity between the feature quantities of the particular image and the feature quantities of the negative examples; and a learning means for training a machine learning model using the plurality of images re-selected by the re-selection means such that (i) the similarity between the feature quantities of the particular image and the feature quantities of the positive examples increases, (ii) the similarity between the feature quantities of the particular image and the feature quantities of the negative examples decreases, and (iii) the similarity between the feature quantities of the particular image and the feature quantities of the moderate negative examples is given more weight than the similarity between the feature quantities of the particular image and the feature quantities of negative examples other than the moderate negative examples.

[0118] (Appendix A2) The learning apparatus as described in Appendix A1, wherein the re-selection means re-selects the plurality of images selected as negative examples by the selection means into weakly negative examples, moderately negative examples, and strongly negative examples, and the learning means trains the machine learning model to give emphasis to moderately negative examples by controlling the contribution to learning for each type of negative example.

[0119] (Note A3) The learning device as described in Note A2, wherein the re-selection means calculates a first interval that indicates a range in which the similarity between the feature quantity of the image of interest and the feature quantity of a first negative example included in the plurality of negative examples selected by the selection means falls within a predetermined threshold, and a second interval that indicates a range in which the confidence level of the similarity between the feature quantity of the image of interest and the feature quantity of the positive example falls within a predetermined threshold, and among the images selected as negative examples by the selection means, those whose similarity to the image of interest is (i) less than the lower limit of the first interval are re-selected as strongly negative examples, (ii) those whose similarity is greater than or equal to the lower limit of the first interval and less than the lower limit of the second interval are re-selected as moderately negative examples, and (iii) those whose similarity is greater than or equal to the lower limit of the second interval are re-selected as weakly negative examples.

[0120] (Appendix A4) The learning device according to Appendix A2 or A3, wherein the learning means trains the machine learning model such that the change in the similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example, and the change in the similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example, are greater than the change in the similarity between the feature quantity of the image of interest and the feature quantity of the strongly negative example.

[0121] (Note A5) The learning device described in any one of Notes A2 to A4, wherein the learning means trains the machine learning model such that the change in the similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example is greater than the change in the similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example.

[0122] (Note A6) The learning device according to any one of Notes A1 to A5, wherein the re-selection means re-selects the images selected as positive examples by the selection means into weak positive examples and strong positive examples using the similarity between the features of the image of interest and the features of the positive examples, and the learning means trains a machine learning model to give more weight to the similarity between the features of the image of interest and the features of the strong positive examples than to the similarity between the features of the image of interest and the features of the weak positive examples.

[0123] (Note A7) The learning device as described in Note A6, wherein the re-selection means calculates a second interval that indicates the range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples falls within a predetermined threshold, and among the images selected as positive examples by the selection means, those whose similarity to the feature quantities of the image of interest is equal to or greater than the lower limit of the second interval are re-selected as strongly positive examples, and positive examples whose similarity to the feature quantities of the image of interest is less than the lower limit of the second interval are re-selected as weakly positive examples.

[0124] (Note A8) The learning device according to any one of Notes A1 to A7, wherein the re-selection means re-selects the plurality of images based on at least one of the following: (i) the confidence interval and minimum value of the similarity between the image of interest and the positive example, and (ii) the confidence interval, maximum value and minimum value of the similarity between the image of interest and one of the plurality of negative examples selected by the selection means.

[0125] (Note A9) The learning device according to any one of Notes A1 to A8, wherein the re-selection means selects from the images selected as negative examples by the selection means that the similarity to the image of interest is greater than a threshold determined by the similarity between the features of the image of interest and the features of the positive examples, and these are selected as examples not to be used for training the machine learning model.

[0126] (Note A10) The learning device according to any one of Notes A1 to A9, wherein the subject in the image is a sample cell, and the image is labeled according to the type of sample cell contained in the image.

[0127] (Appendix A11) A classification device further comprising a class classification means for classifying an image into a class or subclass by inputting an image including a subject to the machine learning model trained by the learning device described in any one of Appendices A1 to A10.

[0128] (Note A12) The learning device according to any one of Notes A1 to A10, wherein each of the plurality of images belongs to one of a plurality of classes and each of the plurality of classes is further classified into one of a plurality of subclasses, the label indicates the class and subclass to which each of the plurality of images belongs, the selection means selects an image belonging to the subclass to which the image of interest belongs as a positive example, an image belonging to a subclass different from the subclass to which the image of interest belongs and belonging to the class to which the image of interest belongs as a first negative example, and an image belonging to a class different from the class to which the image of interest belongs as a second negative example.

[0129] (Note A13) Each of the plurality of images further belongs to one of a plurality of middle classes, which are obtained by further classifying each of the plurality of classes into a plurality of middle classes, and which are groups of one or more of the subclasses, and the selection means sets an image from the first negative example that belongs to the same middle class as the middle class to which the image of interest belongs as a third negative example, and an image from the first negative example that belongs to a different middle class than the middle class to which the image of interest belongs as a fourth negative example, and the re-selection means, The learning device according to Appendix A12, wherein the plurality of images selected by the selection means are re-selected into the positive example, the weakly negative example, the first quasi-negative example, the second quasi-negative example, and the strongly negative example, based on at least one of the following: (i) the similarity between the feature quantity of the image of interest and the feature quantity of the positive example, (ii) the similarity between the feature quantity of the image of interest and the feature quantity of the third negative example, (iii) the similarity between the feature quantity of the image of interest and the feature quantity of the fourth negative example, and (iv) the similarity between the feature quantity of the image of interest and the feature quantity of the second negative example.

[0130] (Note A14) The learning device according to Note A12, wherein the subject of the image is a sample cell, and the image is classified into classes according to whether the sample cells contained in the image are benign or malignant, and further classified into subclasses according to the type of sample cell contained in the image.

[0131] (Note A15) The machine learning model comprises: a first group of layers that take an image as input and generate feature quantities of the image; and a second group of layers connected to the first group of layers that take the feature quantities of the image as input and classify the input image into the class or the subclass, wherein the learning means trains the machine learning model using a plurality of images and teacher labels corresponding to the result of classifying each of the plurality of images into the class or the subclass, as described in any one of Notes A12 to A14.

[0132] [Addendum B] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0133] (Note B1) A selection process in which at least one processor sorts the multiple images into multiple types of negative examples and positive examples based on the label attached to the image of interest included in the multiple images and the label attached to each of the multiple images; and a re-selection process in which the at least one processor sorts the multiple images sorted by the selection process into multiple types of negative examples, including a moderate negative example, and positive examples, using the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples, and the similarity between the feature quantities of the image of interest and the feature quantities of the negative examples. A learning method comprising: a learning process in which at least one processor trains a machine learning model using the plurality of images re-selected by the re-selection process such that (i) the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples increases, (ii) the similarity between the feature quantities of the image of interest and the feature quantities of the negative examples decreases, and (iii) the similarity between the feature quantities of the image of interest and the feature quantities of the medium negative examples is given more weight than the similarity between the feature quantities of the image of interest and the feature quantities of negative examples other than the medium negative examples.

[0134] (Appendix B2) The learning method according to Appendix B1, wherein in the re-selection process, the at least one processor re-selects the plurality of images selected as negative examples in the selection process into weakly negative examples, moderately negative examples, and strongly negative examples, and in the learning process, the at least one processor trains the machine learning model to give emphasis to moderately negative examples by controlling the contribution to learning for each type of negative example.

[0135] (Note B3) The learning method described in Note B2, wherein in the re-selection process, the at least one processor calculates a first interval indicating a range in which the similarity between the feature quantities of the image of interest and the feature quantities of the first negative example included in the plurality of types of negative examples selected in the selection process falls within a predetermined threshold, and a second interval indicating a range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive example falls within a predetermined threshold, and among the images selected as negative examples by the selection process, those whose similarity to the image of interest is (i) less than the lower limit of the first interval are re-selected as strongly negative examples, (ii) those whose similarity is greater than or equal to the lower limit of the first interval and less than the lower limit of the second interval are re-selected as moderately negative examples, and (iii) those whose similarity is greater than or equal to the lower limit of the second interval are re-selected as weakly negative examples.

[0136] (Appendix B4) The learning method according to Appendix B2 or B3, wherein in the learning process, the at least one processor trains the machine learning model such that the change in the similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example, and the change in the similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example, are greater than the change in the similarity between the feature quantity of the image of interest and the feature quantity of the strongly negative example.

[0137] (Appendix B5) The learning method according to any one of Appendices B2 to B4, wherein in the learning process, the at least one processor trains the machine learning model such that the change in similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example is greater than the change in similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example.

[0138] (Note B6) The learning method according to any one of Notes B1 to B5, wherein in the re-selection process, the at least one processor re-selects the images selected as positive examples in the selection process into weak positive examples and strong positive examples using the similarity between the features of the image of interest and the features of the positive examples, and in the learning process, the at least one processor trains a machine learning model to give more weight to the similarity between the features of the image of interest and the features of the strong positive examples than to the similarity between the features of the image of interest and the features of the weak positive examples.

[0139] (Note B7) The learning method described in Note B6, wherein in the re-selection process, at least one processor calculates a second interval indicating the range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples falls within a predetermined threshold, and among the images selected as positive examples by the selection process, those whose similarity to the feature quantities of the image of interest is equal to or greater than the lower limit of the second interval are re-selected as strong positive examples, and positive examples whose similarity to the feature quantities of the image of interest is less than the lower limit of the second interval are re-selected as weak positive examples.

[0140] (Note B8) The learning method according to any one of Notes B1 to B7, wherein in the re-selection process, the at least one processor re-selects the plurality of images based on at least one of the following: (i) the confidence interval and minimum value of the similarity between the image of interest and the positive example, and (ii) the confidence interval, maximum value and minimum value of the similarity between the image of interest and one of the plurality of negative examples selected in the selection process.

[0141] (Note B9) The learning method according to any one of Notes B1 to B8, wherein in the re-selection process, at least one processor selects from the images selected as negative examples by the selection process those whose similarity to the image of interest is greater than a threshold determined by the similarity between the features of the image of interest and the features of the positive examples, and these are selected as examples not to be used for training the machine learning model.

[0142] (Note B10) The learning method described in any one of Notes B1 to B9, wherein the subject of the image is a sample cell, and the image is labeled according to the type of sample cell contained in the image.

[0143] (Appendix B11) A classification method further comprising a class classification process in which at least one processor inputs an image including a subject to the machine learning model trained by the learning device described in any one of Appendix B1 to B10, thereby classifying the image into a class or a subclass.

[0144] (Appendix B12) The learning method according to any one of Appendix B1 to B10, wherein each of the plurality of images belongs to one of a plurality of classes and each of the plurality of classes is further classified into one of a plurality of subclasses, the label indicates the class and subclass to which each of the plurality of images belongs, and in the selection process, the at least one processor selects an image belonging to the subclass to which the image of interest belongs as a positive example, selects an image belonging to a subclass different from the subclass to which the image of interest belongs and also belongs to the class to which the image of interest belongs as a first negative example, and selects an image belonging to a class different from the class to which the image of interest belongs as a second negative example.

[0145] (Note B13) Each of the plurality of images further belongs to one of a plurality of middle classes, which are obtained by classifying each of the plurality of classes into a plurality of middle classes, and which are groups of one or more of the subclasses, and in the selection process, the at least one processor sets an image from the first negative example that belongs to the same middle class as the middle class to which the image of interest belongs as a third negative example, and an image from the first negative example that belongs to a different middle class than the middle class to which the image of interest belongs as a fourth negative example, and in the re-selection process, the at least one processor, The learning method described in Appendix B12, wherein the plurality of images selected by the selection process are re-selected into the positive example, the weak negative example, the first quasi-negative example, the second quasi-negative example, and the strongly negative example, based on at least one of the following: (i) the similarity between the feature quantity of the image of interest and the feature quantity of the positive example, (ii) the similarity between the feature quantity of the image of interest and the feature quantity of the third negative example, (iii) the similarity between the feature quantity of the image of interest and the feature quantity of the fourth negative example, and (iv) the similarity between the feature quantity of the image of interest and the feature quantity of the second negative example.

[0146] (Appendix B14) The learning method described in Appendix B12, wherein the subject included in the image is a sample cell, and the image is classified into classes according to whether the sample cells included in the image are benign or malignant, and further classified into subclasses according to the type of sample cells included in the image.

[0147] (Note B15) The learning method according to any one of Notes B12 to B14, wherein the machine learning model includes: a first group of layers that take an image as input and generate feature quantities of the image; and a second group of layers connected to the first group of layers that take the feature quantities of the image as input and classify the input image into the class or the subclass, and in the learning process, the at least one processor trains the machine learning model using a plurality of images and teacher labels corresponding to the result of classifying each of the plurality of images into the class or the subclass.

[0148] [Addendum C] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0149] (Note C1) A program that causes a computer to function as a learning device, comprising: a selection means for selecting a plurality of images into a plurality of types of negative examples and positive examples based on a label attached to a particular image included in a plurality of images and a label attached to each of the plurality of images; a re-selection means for re-selecting the plurality of images selected by the selection means into a plurality of types of negative examples, including a moderate negative example, and positive examples, using the similarity between the feature quantities of the particular image and the feature quantities of the positive examples, and the similarity between the feature quantities of the particular image and the feature quantities of the negative examples; and a learning means for training a machine learning model using the plurality of images re-selected by the re-selection means such that (i) the similarity between the feature quantities of the particular image and the feature quantities of the positive examples increases, (ii) the similarity between the feature quantities of the particular image and the feature quantities of the negative examples decreases, and (iii) the similarity between the feature quantities of the particular image and the feature quantities of the moderate negative examples is given more weight than the similarity between the feature quantities of the particular image and the feature quantities of negative examples other than the moderate negative examples. A learning program designed to function as such.

[0150] (Appendix C2) The learning program described in Appendix C1, wherein the re-selection means re-selects the plurality of images selected as negative examples by the selection means into weakly negative examples, moderately negative examples, and strongly negative examples, and the learning means trains the machine learning model to give emphasis to moderately negative examples by controlling the contribution to learning for each type of negative example.

[0151] (Note C3) The learning program described in Note C2, wherein the re-selection means calculates a first interval indicating a range in which the similarity between the feature quantity of the image of interest and the feature quantity of a first negative example included in the plurality of negative examples selected by the selection means falls within a predetermined threshold, and a second interval indicating a range in which the confidence level of the similarity between the feature quantity of the image of interest and the feature quantity of the positive example falls within a predetermined threshold, and among the images selected as negative examples by the selection means, (i) those with a similarity to the image of interest that is less than the lower limit of the first interval are re-selected as strongly negative examples, (ii) those with a similarity to the lower limit of the first interval and less than the lower limit of the second interval are re-selected as moderately negative examples, and (iii) those with a similarity to the lower limit of the second interval are re-selected as weakly negative examples.

[0152] (Appendix C4) The learning program described in Appendix C2 or C3, wherein the learning means trains the machine learning model such that the change in the similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example, and the change in the similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example, are greater than the change in the similarity between the feature quantity of the image of interest and the feature quantity of the strongly negative example.

[0153] (Note C5) The learning means is a learning program according to any one of Notes C2 to C4, which trains the machine learning model such that the change in similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example is greater than the change in similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example.

[0154] (Note C6) The learning program described in any one of Notes C1 to C5, wherein the re-selection means re-selects the images selected as positive examples by the selection means into weak positive examples and strong positive examples using the similarity between the features of the image of interest and the features of the positive examples, and the learning means trains a machine learning model to give more weight to the similarity between the features of the image of interest and the features of the strong positive examples than to the similarity between the features of the image of interest and the features of the weak positive examples.

[0155] (Note C7) The learning program described in Note C6, wherein the re-selection means calculates a second interval indicating the range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples falls within a predetermined threshold, and among the images selected as positive examples by the selection means, those whose similarity to the feature quantities of the image of interest is equal to or greater than the lower limit of the second interval are re-selected as strongly positive examples, and positive examples whose similarity to the feature quantities of the image of interest is less than the lower limit of the second interval are re-selected as weakly positive examples.

[0156] (Note C8) The learning program according to any one of Notes C1 to C7, wherein the re-selection means re-selects the plurality of images based on at least one of the following: (i) the confidence interval and minimum value of the similarity between the image of interest and the positive example, and (ii) the confidence interval, maximum value and minimum value of the similarity between the image of interest and one of the plurality of negative examples selected by the selection means.

[0157] (Note C9) The learning program described in any one of Notes C1 to C8, wherein the re-selection means selects from the images selected as negative examples by the selection means that the similarity to the image of interest is greater than a threshold determined by the similarity between the features of the image of interest and the features of the positive examples, and these are selected as examples not to be used for training the machine learning model.

[0158] (Note C10) The learning program described in any one of Notes C1 to C9, wherein the subject of the image is a sample cell, and the image is labeled according to the type of sample cell contained in the image.

[0159] (Appendix C11) A classification program that further functions as a class classification means for classifying images into classes or subclasses by inputting images including a subject into the machine learning model which has been trained by the learning program described in any one of Appendices C1 to C10.

[0160] (Note C12) The learning program according to any one of Notes C1 to C10, wherein each of the plurality of images belongs to one of a plurality of classes and each of the plurality of classes is further classified into one of a plurality of subclasses, the label indicates the class and subclass to which each of the plurality of images belongs, the selection means selects an image belonging to the subclass to which the image of interest belongs as a positive example, an image belonging to a subclass different from the subclass to which the image of interest belongs and belonging to the class to which the image of interest belongs as a first negative example, and an image belonging to a class different from the class to which the image of interest belongs as a second negative example.

[0161] (Note C13) Each of the plurality of images further belongs to one of a plurality of middle classes, which are obtained by further classifying each of the plurality of classes into a plurality of middle classes, and which are groups of one or more of the subclasses, and the selection means sets an image from the first negative example that belongs to the same middle class as the middle class to which the image of interest belongs as a third negative example, and an image from the first negative example that belongs to a different middle class than the middle class to which the image of interest belongs as a fourth negative example, and the re-selection means, The learning program described in Appendix C12, which re-selects the plurality of images selected by the selection means into the positive example, the weakly negative example, the first quasi-negative example, the second quasi-negative example, and the strongly negative example, based on at least one of the following: (i) the similarity between the feature quantity of the image of interest and the feature quantity of the positive example, (ii) the similarity between the feature quantity of the image of interest and the feature quantity of the third negative example, (iii) the similarity between the feature quantity of the image of interest and the feature quantity of the fourth negative example, and (iv) the similarity between the feature quantity of the image of interest and the feature quantity of the second negative example.

[0162] (Appendix C14) The learning program described in Appendix C12, wherein the subject of the image is a sample cell, and the image is classified into classes according to whether the sample cells contained in the image are benign or malignant, and further classified into subclasses according to the type of sample cell contained in the image.

[0163] (Note C15) The machine learning model comprises: a first group of layers that take an image as input and generate feature quantities of the image; and a second group of layers connected to the first group of layers that take the feature quantities of the image as input and classify the input image into the class or the subclass; and the learning means trains the machine learning model using a plurality of images and teacher labels corresponding to the result of classifying each of the plurality of images into the class or the subclass, as described in any one of Notes C12 to C14.

[0164] [Addendum D] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0165] (Note D1) The system comprises at least one processor, the at least one processor, a selection process for sorting a plurality of images into multiple types of negative examples and positive examples based on a label attached to a particular image included in the plurality of images and a label attached to each of the plurality of images, a re-selection process for sorting the plurality of images sorted by the selection process into multiple types of negative examples, including a medium negative example, and positive examples, using the similarity between the features of the particular image and the features of the positive examples, and the similarity between the features of the particular image and the features of the negative examples, and a learning process for training a machine learning model using the plurality of images sorted by the re-selection process, such that (i) the similarity between the features of the particular image and the features of the positive examples increases, (ii) the similarity between the features of the particular image and the features of the negative examples decreases, and (iii) the similarity between the features of the features of the particular image and the features of the medium negative examples is given more weight than the similarity between the features of the features of the particular image and the features of negative examples other than the medium negative examples. A learning device that performs this task.

[0166] The learning device may also include memory. Furthermore, the memory may store a program that causes at least one processor to execute each of the aforementioned processes.

[0167] (Note D2) The learning apparatus according to Note D1, wherein in the re-selection process, the at least one processor re-selects the plurality of images selected as negative examples in the selection process into weak negative examples, moderate negative examples, and strongly negative examples, and in the learning process, the at least one processor trains the machine learning model to give emphasis to moderate negative examples by controlling the contribution to learning for each type of negative example.

[0168] (Note D3) In the re-selection process, the at least one processor calculates a first interval indicating a range in which the similarity between the feature quantities of the image of interest and the feature quantities of the first negative example included in the plurality of types of negative examples selected in the selection process falls within a predetermined threshold, and a second interval indicating a range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive example falls within a predetermined threshold. Among the images selected as negative examples by the selection process, those whose similarity to the image of interest is (i) less than the lower limit of the first interval are re-selected as strongly negative examples, (ii) those whose similarity is greater than or equal to the lower limit of the first interval and less than the lower limit of the second interval are re-selected as moderately negative examples, and (iii) those whose similarity is greater than or equal to the lower limit of the second interval are re-selected as weakly negative examples, as described in Note D2.

[0169] (Appendix D4) The learning device according to Appendix D2 or D3, wherein in the learning process, the at least one processor trains the machine learning model such that the change in the similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example, and the change in the similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example, are greater than the change in the similarity between the feature quantity of the image of interest and the feature quantity of the strongly negative example.

[0170] (Note D5) The learning device according to any one of Notes D2 to D4, wherein in the learning process, the at least one processor trains the machine learning model such that the change in similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example is greater than the change in similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example.

[0171] (Note D6) The learning device according to any one of Notes D1 to D5, wherein in the re-selection process, the at least one processor re-selects the images selected as positive examples in the selection process into weak positive examples and strong positive examples using the similarity between the features of the image of interest and the features of the positive examples, and in the learning process, the at least one processor trains a machine learning model to give more weight to the similarity between the features of the image of interest and the features of the strong positive examples than to the similarity between the features of the image of interest and the features of the weak positive examples.

[0172] (Note D7) The learning device according to Note D6, wherein in the re-selection process, at least one processor calculates a second interval indicating a range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples falls within a predetermined threshold, and among the images selected as positive examples by the selection process, those whose similarity to the feature quantities of the image of interest is equal to or greater than the lower limit of the second interval are re-selected as strongly positive examples, and positive examples whose similarity to the feature quantities of the image of interest is less than the lower limit of the second interval are re-selected as weakly positive examples.

[0173] (Note D8) The learning device according to any one of Notes D1 to D7, wherein in the re-selection process, the at least one processor re-selects the plurality of images based on at least one of the following: (i) the confidence interval and minimum value of the similarity between the image of interest and the positive example, and (ii) the confidence interval, maximum value and minimum value of the similarity between the image of interest and one of the plurality of negative examples selected in the selection process.

[0174] (Note D9) The learning device according to any one of Notes D1 to D8, wherein in the re-selection process, at least one processor selects from the images selected as negative examples by the selection process those whose similarity to the image of interest is greater than a threshold determined by the similarity between the features of the image of interest and the features of the positive examples, and these are selected as examples not to be used for training the machine learning model.

[0175] (Note D10) The learning device according to any one of Notes D1 to D9, wherein the subject of the image is a sample cell, and the image is labeled according to the type of sample cell contained in the image.

[0176] (Appendix D11) A classification device that further performs a class classification process, in which at least one processor inputs an image including a subject to the machine learning model trained by the learning device described in any one of Appendices D1 to D10, thereby classifying the image into a class or a subclass.

[0177] (Note D12) The learning device according to any one of Notes D1 to D10, wherein each of the plurality of images belongs to one of a plurality of classes and each of the plurality of classes is further classified into one of a plurality of subclasses, the label indicates the class and subclass to which each of the plurality of images belongs, and in the selection process, the at least one processor selects an image belonging to the subclass to which the image of interest belongs as a positive example, an image belonging to a subclass different from the subclass to which the image of interest belongs and belonging to the class to which the image of interest belongs as a first negative example, and an image belonging to a class different from the class to which the image of interest belongs as a second negative example.

[0178] (Note D13) Each of the plurality of images further belongs to one of a plurality of middle classes, which are obtained by further classifying each of the plurality of classes into a plurality of middle classes, and which are groups of one or more of the subclasses, and in the selection process, the at least one processor sets an image from the first negative example that belongs to the same middle class as the middle class to which the image of interest belongs as a third negative example, and an image from the first negative example that belongs to a different middle class than the middle class to which the image of interest belongs as a fourth negative example, and in the re-selection process, the at least one processor, A learning device according to Appendix D12, which re-selects the plurality of images selected by the selection process into the positive example, the weakly negative example, the first quasi-negative example, the second quasi-negative example, and the strongly negative example, based on at least one of the following: (i) the similarity between the feature quantity of the image of interest and the feature quantity of the positive example, (ii) the similarity between the feature quantity of the image of interest and the feature quantity of the third negative example, (iii) the similarity between the feature quantity of the image of interest and the feature quantity of the fourth negative example, and (iv) the similarity between the feature quantity of the image of interest and the feature quantity of the second negative example.

[0179] (Note D14) The learning device according to Note D12, wherein the subject of the image is a sample cell, and the image is classified into classes according to whether the sample cells contained in the image are benign or malignant, and further classified into subclasses according to the type of sample cells contained in the image.

[0180] (Note D15) The learning device according to any one of Notes D12 to D14, wherein the machine learning model comprises: a first group of layers that take an image as input and generate feature quantities of the image; and a second group of layers connected to the first group of layers that take the feature quantities of the image as input and classify the input image into the class or the subclass, and in the learning process, the at least one processor trains the machine learning model using a plurality of images and teacher labels corresponding to the result of classifying each of the plurality of images into the class or the subclass.

[0181] [Addendum E] This disclosure includes the technologies described in the following addendums. However, the present invention is not limited to the technologies described in the following addendums, and various modifications are possible within the scope of the claims.

[0182] (Note E1) A program for causing a computer to function as a learning device, comprising: a selection process for the computer to sort a plurality of images into a plurality of types of negative examples and positive examples based on a label attached to a particular image included in a plurality of images and a label attached to each of the plurality of images; a re-selection process for sorting the plurality of images sorted by the selection process into a plurality of types of negative examples, including a medium negative example, and positive examples, using the similarity between the features of the particular image and the features of the positive examples, and the similarity between the features of the particular image and the features of the negative examples; and a learning process for training a machine learning model using the plurality of images sorted by the re-selection process, such that (i) the similarity between the features of the particular image and the features of the positive examples increases, (ii) the similarity between the features of the particular image and the features of the negative examples decreases, and (iii) the similarity between the features of the features of the particular image and the features of the medium negative examples is given more weight than the similarity between the features of the features of the particular image and the features of negative examples other than the medium negative examples. A non-temporary recording medium that records a learning program for executing [something].

[0183] 1 Learning device 1A, 1B Information processing device 11 Sorting unit 111A Acquisition unit 12, 113A, 113B Re-sorting unit 13, 114A, 114B Learning unit 112A Sorting unit 121A Classification unit

Claims

1. A learning device comprising: a selection means for selecting a plurality of images into a plurality of types of negative examples and positive examples based on a label attached to a particular image contained in a plurality of images and a label attached to each of the plurality of images; a re-selection means for re-selecting the plurality of images selected by the selection means into a plurality of types of negative examples, including a moderate negative example, and positive examples, using the similarity between the feature quantities of the particular image and the feature quantities of the positive examples, and the similarity between the feature quantities of the particular image and the feature quantities of the negative examples; and a learning means for training a machine learning model using the plurality of images re-selected by the re-selection means such that (i) the similarity between the feature quantities of the particular image and the feature quantities of the positive examples increases, (ii) the similarity between the feature quantities of the particular image and the feature quantities of the negative examples decreases, and (iii) the similarity between the feature quantities of the particular image and the feature quantities of the moderate negative examples is given more weight than the similarity between the feature quantities of the particular image and the feature quantities of negative examples other than the moderate negative examples.

2. The learning device according to claim 1, wherein the re-selection means re-selects the plurality of images selected as negative examples by the selection means into weak negative examples, moderate negative examples, and strong negative examples, and the learning means trains the machine learning model to give emphasis to moderate negative examples by controlling the contribution to learning for each type of negative example.

3. The learning device according to claim 2, wherein the re-selection means calculates a first interval indicating a range in which the similarity between the feature quantities of the image of interest and the feature quantities of a first negative example included in a plurality of types of negative examples selected by the selection means falls within a predetermined threshold, and a second interval indicating a range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive example falls within a predetermined threshold, and among the images selected as negative examples by the selection means, (i) those whose similarity to the image of interest is less than the lower limit of the first interval are re-selected as strongly negative examples, (ii) those whose similarity is greater than or equal to the lower limit of the first interval and less than the lower limit of the second interval are re-selected as moderately negative examples, and (iii) those whose similarity is greater than or equal to the lower limit of the second interval are re-selected as weakly negative examples.

4. The learning device according to claim 2 or 3, wherein the learning means trains the machine learning model such that the change in the similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example, and the change in the similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example, are greater than the change in the similarity between the feature quantity of the image of interest and the feature quantity of the strongly negative example.

5. The learning device according to any one of claims 2 to 4, wherein the learning means trains the machine learning model such that the change in similarity between the feature quantity of the image of interest and the feature quantity of the moderately negative example is greater than the change in similarity between the feature quantity of the image of interest and the feature quantity of the weakly negative example.

6. The learning device according to any one of claims 1 to 5, wherein the re-selection means re-selects the images selected as positive examples by the selection means into weak positive examples and strong positive examples using the similarity between the features of the image of interest and the features of the positive examples, and the learning means trains a machine learning model to give more weight to the similarity between the features of the image of interest and the features of the strong positive examples than to the similarity between the features of the image of interest and the features of the weak positive examples.

7. The learning device according to claim 6, wherein the re-selection means calculates a second interval indicating a range in which the confidence level of the similarity between the feature quantities of the image of interest and the feature quantities of the positive examples falls within a predetermined threshold, and re-selects images selected as positive examples by the selection means that have a similarity to the feature quantities of the image of interest that is equal to or greater than the lower limit of the second interval as strongly positive examples, and re-selects positive examples that have a similarity to the feature quantities of the image of interest that is less than the lower limit of the second interval as weakly positive examples.

8. The learning device according to any one of claims 1 to 7, wherein the re-selection means re-selects the plurality of images based on at least one of (i) the confidence interval and minimum value of the similarity between the image of interest and the positive example, and (ii) the confidence interval, maximum value and minimum value of the similarity between the image of interest and one of the plurality of negative examples selected by the selection means.

9. The learning device according to any one of claims 1 to 8, wherein the re-selection means selects from the images selected as negative examples by the selection means that the similarity to the image of interest is greater than a threshold determined by the similarity between the features of the image of interest and the features of the positive examples, and these are selected as examples not to be used for training the machine learning model.

10. The learning device according to any one of claims 1 to 9, wherein the subject included in the image is a sample cell, and the image is labeled according to the type of sample cell included in the image.

11. A classification device further comprising: a class classification means for classifying an image into a class or subclass by inputting an image including a subject to the machine learning model trained by the learning device according to any one of claims 1 to 10.

12. A selection process in which at least one processor sorts a plurality of images into a plurality of types of negative examples and positive examples based on a label attached to the image of interest contained in the plurality of images and a label attached to each of the plurality of images; a re-selection process in which the at least one processor sorts the plurality of images sorted by the selection process into a plurality of types of negative examples, including a medium negative example, and positive examples, using the similarity between the features of the image of interest and the features of the positive examples, and the similarity between the features of the image of interest and the features of the negative examples; and a learning process in which the at least one processor trains a machine learning model using the plurality of images sorted by the re-selection process such that (i) the similarity between the features of the image of interest and the features of the positive examples increases, (ii) the similarity between the features of the image of interest and the features of the negative examples decreases, and (iii) the similarity between the features of the features of the image of interest and the features of the medium negative examples is given more weight than the similarity between the features of the image of interest and the features of negative examples other than the medium negative examples. Learning methods that include this.

13. A program for causing a computer to function as a learning device, comprising: a selection means for selecting a plurality of images into a plurality of types of negative examples and positive examples based on a label attached to a particular image included in a plurality of images and a label attached to each of the plurality of images; a re-selection means for re-selecting the plurality of images selected by the selection means into a plurality of types of negative examples, including a moderate negative example, and positive examples, using the similarity between the feature quantities of the particular image and the feature quantities of the positive examples, and the similarity between the feature quantities of the particular image and the feature quantities of the negative examples; and a learning means for training a machine learning model using the plurality of images re-selected by the re-selection means such that (i) the similarity between the feature quantities of the particular image and the feature quantities of the positive examples increases, (ii) the similarity between the feature quantities of the particular image and the feature quantities of the negative examples decreases, and (iii) the similarity between the feature quantities of the particular image and the feature quantities of the moderate negative examples is given more weight than the similarity between the feature quantities of the particular image and the feature quantities of negative examples other than the moderate negative examples. A learning program designed to function as such.

Images