Image processing device and computer program

Abstract

This image processing device comprises a plurality of determination units (31, 32, 33) that perform image processing on a target image (Di) and an integration unit (34) that performs integration processing on the basis of output results from the determination units (31, 32, 33). Each of the determination units (31, 32, 33) uses a machine learning model (M1, M2, M3) to output a determination result (Ra, Rb, Rc) for the target image (Di) and an individual heat map (Ha, Hb, Hc) that shows reactivity to the pixels of the target image (Di) with respect to the determination process. The integration unit (34) generates an integrated determination result (Rt) for the target image (Di) on the basis of the plurality of determination results (Ra, Rb, Rc) and generates an integrated heat map (Ht) that shows reactivity to the pixels of the target image (Di) with respect to the integrated determination result (Rt) on the basis of the plurality of individual heat maps (Ha, Hb, Hc). The present invention thereby makes it possible for an ensemble learning model that is constructed from a plurality of machine learning models to achieve explainability / interpretability.

Description

Image Processing Apparatus and Computer Program

[0001] The present invention relates to ensemble learning that obtains highly accurate output using a plurality of machine learning models.

[0002] Conventionally, as machine learning models for image processing, an image recognition model that identifies an object depicted in an image and a region segmentation model that determines to which object class each pixel in the image belongs are known. In such machine learning models, in some cases, explainability is required to determine the superiority, inferiority, and characteristics of the machine learning model.

[0003] As a machine learning model having explainability, for example, Grad-CAM for image recognition is known to visualize the reaction positions of a deep learning model with a heatmap or the like. By using this, the reaction positions in the image can be known.

[0004] On the other hand, in a machine learning model, a technique called ensemble learning is used in which a plurality of machine learning models are combined to obtain more accurate output. An ensemble learning system is described in, for example, Patent Document 1.

[0005] Japanese Patent Application Laid-Open No. 2022-131558

[0006] However, in a machine learning model for image region segmentation, a general means for heatmapping the reaction positions of a deep learning model has not been established. Therefore, the machine learning model has no explainability, and it is difficult to determine the superiority, inferiority, and characteristics of the machine learning model.

[0007] The present invention has been made in view of such circumstances, and an object thereof is to provide a technique for obtaining explainability in an ensemble learning model composed of a plurality of machine learning models.

[0008] To solve the above problems, the first invention of the present application is an image processing apparatus comprising: a plurality of discrimination units that perform image processing on a target image to be processed; and an integration unit that performs integrated processing based on the output results of the discrimination units, wherein each discrimination unit outputs a discrimination result obtained by discriminating the target image using a machine learning model, and an individual heat map showing the degree of response to each pixel of the target image during the discrimination process; the integration unit generates an integrated discrimination result obtained by discriminating the target image based on the plurality of discrimination results, and generates an integrated heat map showing the degree of response to each pixel of the target image in the integrated discrimination result based on the plurality of individual heat maps.

[0009] The second invention of the present application is an image processing apparatus of the first invention, wherein the integrating unit has weighting values ​​for each of the discrimination units, the integrating unit generates the integrated discrimination result based on a total value obtained by summing the weighting values ​​obtained by multiplying each of the discrimination results by the respective weighting values, and for each pixel of the target image, the integrating unit generates the integrated heat map as the reaction degree of each pixel in the integrated heat map by summing the reaction degrees of each individual heat map by the weighting values.

[0010] The third invention of this application is an image processing apparatus of the second invention, further comprising a display unit capable of displaying an image, wherein the display unit displays the individual heatmap output by the discrimination unit.

[0011] The fourth invention of this application is an image processing apparatus of the third invention, wherein the display unit displays the individual heatmaps output by the discrimination unit and the integrated heatmaps output by the integration unit.

[0012] The fifth invention of this application is an image processing apparatus according to any one of the second to fourth inventions, wherein the integrated unit is capable of changing the value of the weighting value based on a command signal received from an external input device.

[0013] The sixth invention of this application is a computer program for image processing, wherein the computer is instructed to perform the following steps: A) input a target image to be processed into a plurality of machine learning models, and output for each machine learning model a discrimination result in which the target image has been identified and an individual heat map showing the degree of response to each pixel of the target image during the discrimination process; B) generate an integrated discrimination result in which the target image has been identified based on the plurality of discrimination results; and C) generate an integrated heat map showing the degree of response to each pixel of the target image in the integrated discrimination result based on the plurality of individual heat maps.

[0014] According to the first to sixth inventions of this application, an interpretable and uninterpretable integrated heatmap can be obtained in an ensemble learning model composed of multiple machine learning models.

[0015] In particular, according to the fifth invention of this application, the weighting values ​​can be adjusted while referring to individual heatmaps. This makes it possible to obtain more accurate inference results (integrated discrimination results).

[0016] This is a schematic diagram of an image processing apparatus according to one embodiment. This is a functional block diagram of the image processing apparatus according to one embodiment. This is a flowchart showing the flow of image recognition processing in the image processing apparatus according to one embodiment. This is a diagram showing an example of discrimination result, integrated discrimination result, individual heat map and integrated heat map.

[0017] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0018] <1. Configuration of the Image Processing Device> Figure 1 is a schematic diagram of an image processing device 1 according to one embodiment. Figure 2 is a functional block diagram of the image processing device 1. This image processing device 1 performs image recognition processing to identify objects contained in an image using a machine learning model.

[0019] As shown in Figure 1, this image processing device 1 comprises a computer body 11 and a display unit 12. The computer body 11 is connected to an external input device 13. The computer body 11 includes a processor 111 such as a CPU, memory 112 such as RAM, and storage unit 113 such as a hard disk drive. The display unit 12 displays images output from the computer body 11. The display unit 12 is, for example, a liquid crystal display such as a PC monitor. The input device 13 can transmit command signals to the computer body 11 according to user actions. The input device 13 is, for example, a keyboard and mouse, or a touch panel.

[0020] The memory 112 and the storage unit 113 are connected to the processor 111 via bus wiring (not shown). The storage unit 113 stores a computer program P. The processor 111 loads the computer program P stored in the storage unit 113 into the memory 112 and executes the code contained in the computer program P sequentially. As a result, the image processing device 1 performs image recognition processing on the target image Di, which is the object to be processed, and classifies each pixel of the target image Di into one of several classes.

[0021] Computer program P is application software that causes the computer main unit 11 to perform various processes related to area identification processing. Computer program P is read from a storage medium M such as a CD or DVD and installed on the computer main unit 11. However, computer program P may also be downloaded to the computer main unit 11 via a network N such as the Internet.

[0022] In a typical image recognition AI model, the target image Di is identified and the probability of each object (classification class) being identified is output. This image processing device 1 outputs the probability that an object in the target image Di belongs to each classification class, as well as a heat map showing the pixels that influenced the determination of whether an object belongs to that classification class and the magnitude of that influence at each pixel.

[0023] Figure 2 is a functional block diagram of the image processing device 1. The computer body 11 includes an image storage unit 20 and an image processing unit 30.

[0024] The image storage unit 20 is a storage unit provided within the computer main body 11. The image storage unit 20 may use the storage unit 113, or it may be a storage unit provided separately from the storage unit 113. The image storage unit 20 stores the target image Di that is to be processed.

[0025] The image processing unit 30 performs image processing on the target image Di. The image processing unit 30 includes a first discrimination unit 31, a second discrimination unit 32, a third discrimination unit 33, and an integration unit 34. The first discrimination unit 31, the second discrimination unit 32, the third discrimination unit 33, and the integration unit 34 are functions realized by the processor 111 executing a computer program P.

[0026] The first discrimination unit 31, the second discrimination unit 32, and the third discrimination unit 33 are all discrimination units that perform discrimination on the target image Di, which is the object to be processed, to determine whether or not it contains objects. In this embodiment, there are three discrimination units, but the number of discrimination units may be two or four or more. The discrimination units 31, 32, and 33 each have different machine learning models M1, M2, and M3.

[0027] The machine learning models M1, M2, and M3 of the discrimination units 31, 32, and 33 take an image as an input variable and output discrimination results Ra, Rb, and Rc, and individual heatmaps Ha, Hb, and Hc. The discrimination results Ra, Rb, and Rc include values ​​indicating the probability that an object contained in the input target image Di belongs to each of several classification classes. The individual heatmaps Ha, Hb, and Hc are image data showing the degree of response to each pixel of the target image Di during the discrimination process by the discrimination units 31, 32, and 33.

[0028] Specifically, the machine learning model M1 of the first discrimination unit 31 outputs a first discrimination result Ra and a first individual heatmap Ha when an image is input. The machine learning model M2 of the second discrimination unit 32 outputs a second discrimination result Rb and a second individual heatmap Hb when an image is input. Furthermore, the machine learning model M3 of the third discrimination unit 33 outputs a third discrimination result Rc and a third individual heatmap Hc when an image is input.

[0029] For machine learning models M1, M2, and M3, for example, an image recognition model based on a CNN (Convolutional Neural Network) with Grad-CAM (Gradient-weighted Class Activation Mapping) implemented is used.

[0030] Note that machine learning models M1, M2, and M3 produce different output results for the same input. For example, machine learning models M1, M2, and M3 may all be machine learning models of the same type and structure, but trained on different datasets. Also, machine learning models M1, M2, and M3 may include trained models with different algorithms.

[0031] Machine learning models M1, M2, and M3, as an example, take a photograph of an animal as an input image and output the probability that the object in the input image belongs to one of five classification classes: dog, cat, rabbit, raccoon, or fox. Simultaneously, machine learning models M1, M2, and M3 acquire a heat map for each pixel in the input image for each classification class, showing the degree of response that influenced the classification to that class.

[0032] The integration unit 34 performs integration processing based on the discrimination results Ra, Rb, and Rc, which are the output results of the discrimination units 31, 32, and 33, and the individual heatmaps Ha, Hb, and Hc. Specifically, the integration unit 34 generates an integrated discrimination result Rt, which identifies the target image Di based on the multiple discrimination results Ra, Rb, and Rc. The integration unit 34 also generates an integrated heatmap Ht, which shows the response rate to each pixel of the target image Di in the integrated discrimination result Rt. Details of the integration processing in the integration unit 34 will be described later.

[0033] The image processing unit 30 displays the target image Di, the individual heatmaps Ha, Hb, Hc, and the integrated heatmap Ht created by the image processing unit 30 on the display unit 12, in accordance with commands from the input device 13. The integration unit 34 can also adjust the weighting values, which will be described later, in accordance with commands from the input device 13. This makes it possible to obtain a more accurate integrated discriminator Rt and integrated heatmap Ht.

[0034] <2. Image Processing Flow> Next, the image recognition processing in the image processing device 1 will be explained with reference to Figure 3. Figure 3 is a flowchart showing the flow of image recognition processing in the image processing device 1.

[0035] As shown in Figure 3, first, in accordance with the command from the input device 13, the image processing unit 30 reads the target image Di to be subjected to region division processing from the image storage unit 20 (Step S101: Target image acquisition step).

[0036] Next, the first discrimination unit 31, the second discrimination unit 32, and the third discrimination unit 33 each input the target image Di to be processed into machine learning models M1, M2, and M3. As a result, each discrimination unit 31, 32, and 33 outputs discrimination results Ra, Rb, and Rc for each machine learning model M1, M2, and M3, which have determined the target image Di, and individual heat maps Ha, Hb, and Hc that show the degree of response to each pixel of the target image Di during the discrimination process (Step S102: Individual output result acquisition step).

[0037] Specifically, the machine learning model M1 of the first discrimination unit 31 outputs the first discrimination result Ra and the first individual heatmap Ha. The machine learning model M2 of the second discrimination unit 32 outputs the second discrimination result Rb and the second individual heatmap Hb. In addition, the machine learning model M3 of the third discrimination unit 33 outputs the third discrimination result Rc and the third individual heatmap Hc.

[0038] Here, the individual heatmaps Ha, Hb, and Hc each contain the same number of heatmap images as the number of classification classes, and each pixel at the same position as each pixel in the target image Di is assigned a response value for the corresponding classification class.

[0039] Figure 4 shows an example of discrimination results Ra, Rb, Rc, integrated discrimination result Rt, individual heatmaps Ha, Hb, Hc, and integrated heatmap Ht. In the example in Figure 4, the target image Di is an image of a dog. For this target image Di, the machine learning models M1, M2, and M3 of this embodiment output discrimination results Ra, Rb, and Rc, which indicate the probability that the object in the target image Di belongs to one of five classification classes: dog, cat, rabbit, raccoon, and fox, and individual heatmaps Ha, Hb, and Hc, which indicate the degree of response to each pixel of the target image Di in each classification class. In other words, the machine learning models M1, M2, and M3 each output five discrimination results Ra, Rb, and Rc and five individual heatmaps Ha, Hb, and Hc.

[0040] In the example in Figure 4, the dog class is the most probable result for all classification results Ra, Rb, and Rc. Figure 4 also shows the individual heatmaps Ha, Hb, and Hc for the dog class. In the heatmaps Ha, Hb, Hc, and Ht in the example in Figure 4, areas with a reactivity of 0 are shown in black, and areas with a higher reactivity are shown in white.

[0041] As shown in Figure 4, since machine learning models M1, M2, and M3 are different from each other, the discrimination results Ra, Rb, Rc and individual heatmaps Ha, Hb, Hc obtained in the individual output result acquisition step S102 are also different from each other.

[0042] Next, based on the discrimination results Ra, Rb, and Rc, the integration unit 34 generates an integrated discrimination result Rt for the target image Di (step S103: integrated discrimination result generation step). Since the integrated discrimination result Rt is also generated for each classification class, five integrated discrimination results Rt are generated in this embodiment.

[0043] In this embodiment, the integration unit 34 has weighting values Ca, Cb, and Cc which are predetermined parameters. These weighting values Ca, Cb, and Cc perform weighting on the discrimination results Ra, Rb, and Rc and the individual heatmaps Ha, Hb, and Hc. That is, the first weighting value Ca is a value indicating the weighting for the discrimination result Ra and the individual heatmap Ha, the second weighting value Cb is a value indicating the weighting for the discrimination result Rb and the individual heatmap Hb, and the third weighting value Cc is a value indicating the weighting for the discrimination result Rc and the individual heatmap Hc. The weighting values Ca, Cb, and Cc of this embodiment are set so that their sum is 1.

[0044] The integration unit 34 of this embodiment uses, as the integrated discrimination result Rt, the total value obtained by summing the products of the weighting values Ca, Cb, and Cc corresponding to the respective discrimination results Ra, Rb, and Rc. That is, for each classification class, the integrated discrimination result Rt is obtained as shown in Equation (1). Rt = Ra * Ca + Rb * Cb + Rc * Cc …(1)

[0045] In the example of FIG. 4, the initial-set weighting values Ca, Cb, and Cc in step S103 are all the same value, specifically 0.33 (exactly 1 / 3). The weighting values Ca, Cb, and Cc in step S103 may all be the same value in this way, or the adjusted weighting values Ca', Cb', and Cc' in the case of performing image identification processing previously may be used as the initial-set weighting values Ca, Cb, and Cc for the next image identification processing.

[0046] Next, the integration unit 34 generates an integrated heatmap Ht indicating the reactivity of each pixel of the target image Di in the integrated discrimination result Rt based on the individual heatmaps Ha, Hb, and Hc (step S104: integrated heatmap generation step). Since the integrated heatmap Ht is also generated for each classification class, five integrated heatmaps Ht are generated in this embodiment.

[0047] In this embodiment, for each pixel of the target image Di, the integration unit 34 multiplies the reactivities corresponding to the pixel in the respective individual heatmaps Ha, Hb, and Hc by the same weighting values Ca, Cb, and Cc as in step S103, sums them, and uses the result as the reactivity corresponding to the pixel in the integrated heatmap Ht, thereby generating the integrated heatmap Ht.

[0048] After obtaining the integrated discrimination result Rt and the integrated heatmap Ht, the image processing unit 30 displays the target image Di, the individual heatmaps Ha, Hb, and Hc, and the integrated heatmap Ht on the display unit 12 (step S105: display step). As a result, the user can visually grasp the reaction positions in the respective discrimination units 31, 32, and 33 with respect to the target image Di from the individual heatmaps Ha, Hb, and Hc. Also, regarding the integrated heatmap Ht, it is possible to visually grasp where in the target image Di the integrated discrimination result Rt was obtained as a result of the reaction.

[0049] Next, the image processing unit 30 can adjust the weighting values Ca, Cb, and Cc of the integration unit 34 based on the command signal received from the input device 13 and change them to new weighting values Ca', Cb', and Cc' (step S106: weighting adjustment step).

[0050] In this embodiment, while the user refers to the discrimination results Ra, Rb, and Rc and the individual heatmaps Ha, Hb, and Hc, the user can determine the new weighting values Ca', Cb', and Cc' by inputting the weighting values themselves into the input device 13.

[0051] For example, in the example shown in Figure 4, the first classification result Ra and the third classification result Rc have a high probability of corresponding to the correct classification class, the dog class, while the second classification result Rb has the highest probability of corresponding to the dog class, but the probability value is low. Also, in the first individual heatmap Ha and the third individual heatmap Hc, the reacted area roughly coincides with the dog's head in the target image Di, whereas in the second individual heatmap Hb, a reaction is only observed in an area of ​​about two-thirds of the dog's head. For this reason, the user increased the weighting of the classification results in the first classification unit 31 and the third classification unit 33, and decreased the weighting of the classification result in the second classification unit 32, changing the weighting values ​​to Ca'=0.4, Cb'=0.2, and Cc'=0.4.

[0052] Subsequently, the integration unit 34 generates a new integrated discrimination result Rt' and an integrated heatmap Ht' using the new weighting values ​​Ca', Cb', and Cc', similar to steps S103 and S104 (step S107: reacquisition step). Specifically, a new integrated discrimination result Rt' is obtained for each classification class as shown in equation (2). Rt' = Ra * Ca' + Rb * Cb' + Rc * Cc' ... (2)

[0053] In the example shown in Figure 4, the probability of the image being a dog in the re-acquired integrated classification result Rt' is higher than the probability of the image being a dog in the initial integrated classification result Rt. This suggests that the image recognition processing was performed with greater accuracy.

[0054] In this embodiment, the weighting adjustment step S106 involved the user directly setting new weighting values ​​Ca', Cb', and Cc', but the present invention is not limited thereto. For example, the integration unit 34 may have an automatic calculation unit that automatically calculates new weighting values ​​Ca', Cb', and Cc' based on the probability values ​​of each classification class in the discrimination results Ra, Rb, and Rc, based on the correct classification class input by the user. Alternatively, such an automatic calculation unit may calculate the new weighting values ​​Ca', Cb', and Cc' according to a pre-set calculation formula. Furthermore, the automatic calculation unit may be composed of a machine learning model that performs additional learning each time the image recognition process is repeated.

[0055] Thus, the image processing unit 30 is a so-called ensemble learning model that integrates the outputs of machine learning models M1, M2, and M3, which are possessed by multiple discrimination units 31, 32, and 33, in the integration unit 34. In such an ensemble learning model, by generating an integrated heatmap Ht using weight values ​​Ca, Cb, and Cc, explanatory interpretability can be obtained for the integrated discrimination result Rt, which is the final discrimination result.

[0056] Furthermore, in this embodiment, the weighting values ​​Ca, Cb, and Cc can be adjusted while referring to the individual heatmaps Ha, Hb, and Hc. This makes it possible to obtain more accurate inference results (integrated discrimination result Rt).

[0057] <3. Modifications> Although embodiments have been described above, the present invention is not limited to those described above, and various modifications are possible.

[0058] In the above embodiment, there are three discrimination units, but the present invention is not limited thereto. The number of discrimination units may be two, or four or more.

[0059] Furthermore, in the above embodiment, each discrimination unit classified the target image Di into five classification classes, but the present invention is not limited to this. The number of classification classes may be two to four, or six or more.

[0060] Furthermore, although the display unit 12 and input device 13 of the image processing device 1 were configured separately in the above embodiment, the present invention is not limited thereto. The display unit 12 and input device 13 may be touch panels.

[0061] Although this invention has been described in detail, the above description is illustrative in all respects and does not limit the invention. It is understood that countless variations not illustrated can be envisioned without falling outside the scope of this invention. The components described in each of the above embodiments and variations can be combined or omitted as appropriate, as long as they do not contradict each other.

[0062] 1 Image processing device 11 Computer main unit 12 Display unit 13 Input unit 30 Image processing unit 31 First discrimination unit 32 Second discrimination unit 33 Third discrimination unit 34 Integration unit Ca, Cb, Cc, Ca', Cb', Cc' First weighting value Di Target image Ha, Hb, Hc Individual heatmap Ht, Ht' Integrated heatmap M1, M2, M3 Machine learning model P Computer program Ra, Rb, Rc Discrimination result Rt, Rt' Integrated discrimination result

Claims

1. An image processing apparatus comprising: a plurality of discrimination units that perform image processing on a target image to be processed; and an integration unit that performs integrated processing based on the output results of the discrimination units, wherein each discrimination unit outputs a discrimination result in which the target image has been discriminated using a machine learning model, and an individual heat map showing the degree of response to each pixel of the target image during the discrimination process; the integration unit generates an integrated discrimination result in which the target image has been discriminated based on the plurality of discrimination results; and generates an integrated heat map showing the degree of response to each pixel of the target image in the integrated discrimination result based on the plurality of individual heat maps.

2. An image processing apparatus according to claim 1, wherein the integrating unit has weighting values ​​for each of the discrimination units, the integrating unit generates the integrated discrimination result based on a total value obtained by summing the weighting values ​​obtained by multiplying each of the discrimination results by the respective weighting values, and for each pixel of the target image, generates the integrated heat map as the reaction degree of each pixel in the integrated heat map by summing the reaction degrees of each individual heat map by the weighting values.

3. An image processing apparatus according to claim 2, further comprising a display unit capable of displaying an image, wherein the display unit displays the individual heatmaps output by the discrimination unit.

4. An image processing apparatus according to claim 3, wherein the display unit displays the individual heatmaps output by the discrimination unit and the integrated heatmaps output by the integration unit.

5. An image processing apparatus according to any one of claims 2 to 4, wherein the integrated unit is capable of changing the value of the weighting value based on a command signal received from an external input device.

6. A computer program for image processing, which causes the computer to perform the following steps: A) input a target image to be processed into multiple machine learning models, and output for each machine learning model a discrimination result in which the target image has been identified, and an individual heat map showing the degree of response to each pixel of the target image during the discrimination process; B) generate an integrated discrimination result in which the target image has been identified based on the multiple discrimination results; and C) generate an integrated heat map showing the degree of response to each pixel of the target image in the integrated discrimination result based on the multiple individual heat maps.

Images