Region detection method, program, and region detection device

Abstract

A representative value of a first color component of interest which is the degree of saturation, or of a first color component of interest which is a color corresponding to staining or a color approximating said color in a plurality of training images used for generating a trained model is acquired as a training color component representative value (steps S14 and S15). The representative value of the first color component of interest in at least one target image is acquired as a target color component representative value (steps S16 and S17). A correction function is acquired from the training color component representative value and the target color component representative value, a second color component of interest in the target image is corrected using the correction function, and the corrected target image is input to the trained model to detect a target region in the target image.

Description

Region detection method, program and region detection device

[0001] The present invention relates to a technique for detecting a target region to be detected based on staining from a biological stained image using a trained model. [Reference to Related Application] The present application claims the benefit of priority from Japanese Patent Application JP2025-30045 filed on February 27, 2025, and the entire disclosure of that application is incorporated herein.

[0002] Conventionally, single-cell analysis has been performed to obtain the expression level of biological substances including proteins on a cell-by-cell basis from digital images of stained pathological tissue specimens. In single-cell analysis, individual cell regions in an image are detected, and staining intensity or the like in each cell is acquired based on information on the cell regions. Thereby, for example, the presence or absence of protein expression in each cell is specified, and an analysis result is obtained.

[0003] Images of pathological tissue differ in tissue condition and staining status depending on differences in the site from which the tissue specimen was collected and the type of staining, even when the site and staining are the same. Therefore, so-called rule-based image processing in which a person determines processing conditions in advance makes it difficult to perform appropriate analysis. Accordingly, analysis using machine learning has also been proposed, and preprocessing training images before generating a trained model has also been proposed.

[0004] For example, in Japanese National Publication No. 2022-529259, in a technique for determining ovarian toxicity using machine learning, images used for training are preprocessed to have the same size scale, same color scale or saturation scale, and then the training process is executed. Further, in Japanese National Publication No. 2023-538309, a technique is disclosed for generating a positive segmentation mask and a negative segmentation mask using a trained model from an image including a staining pattern of a biomarker. Then, preprocessing is performed on training images and verification input images so that they have the same size scale, same color scale or saturation scale, and then the learning algorithm is executed.

[0005] Incidentally, when extracting target regions to be detected based on staining from stained images (hereinafter referred to as "target regions") using a trained model, the desired target regions are indicated (i.e., annotations are performed) on the images used for training, and the trained model is generated using these images. At this time, the staining density of the sample is usually not constant, and there are images that are heavily stained and images that are lightly stained. Therefore, if the staining density of the images used for training differs from the staining density of the images in which the target regions are to be detected using the trained model, the detection accuracy of the target regions will decrease. Furthermore, even if preprocessing is performed on the images used for training, as in conventional techniques, the detection accuracy of the target regions may not be sufficiently improved due to differences in staining that exceed expectations.

[0006] The present invention aims to improve the accuracy of detecting target regions using a pre-trained model.

[0007] One aspect of the present invention is a region detection method for detecting a target region to be detected based on staining from a biological staining image using a trained model, comprising: a) obtaining a representative value of the first color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, from a plurality of training images used to generate the trained model, as a representative value of the trained color component; b) obtaining a representative value of the first color component of interest from at least one target image, which is a biological staining image that is to be input to the trained model and from which a target region is to be detected, as a representative value of the target color component; c) obtaining a correction function from the representative value of the trained color component and the representative value of the target color component; d) correcting the second color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, from the at least one target image using the correction function; and e) detecting a target region from the at least one target image by inputting the corrected at least one target image to the trained model.

[0008] According to the present invention, the accuracy of detecting a target region using a pre-trained model can be improved.

[0009] Aspect 2 of the present invention is a region detection method according to aspect 1, wherein in step a), a representative value of the first color component of a pixel in the plurality of learning images whose third color component of interest is saturation, or whose third color component of interest is a color corresponding to a color or a color similar to said color and whose value is equal to or greater than a predetermined threshold is obtained as the learning color component representative value, and in step b), a representative value of the first color component of a pixel in the at least one target image whose third color component of interest is equal to or greater than a predetermined threshold is obtained as the target color component representative value.

[0010] A third aspect of the present invention is a region detection method according to aspect 1 (or aspect 1 or 2), wherein the at least one target image is each of the groups obtained by dividing a plurality of target images into a plurality of groups.

[0011] Aspect 4 of the present invention is a region detection method according to aspect 1 (or aspect 1 or 2), wherein the number of at least one target image is 1.

[0012] Aspect 5 of the present invention is a region detection method according to Aspect 1 (which may be any one of Aspects 1 to 4), further comprising, after step d), e) a step of displaying a selected target image selected from the at least one target image; f) a step of modifying the correction function; g) a step of correcting the selected target image using the modified correction function; and h) a step of displaying the selected target image.

[0013] Aspect 6 of the present invention is a region detection method according to any one of aspects 1 to 5, wherein the target region is a cellular-level region.

[0014] Aspect 7 of the present invention is a computer-readable program that causes a computer to detect a target region to be detected based on staining from a biological staining image using a trained model, wherein the execution of the program by the computer causes the computer to perform the following steps: a) obtain a representative value of the first color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, of a plurality of training images used to generate the trained model, as a representative value of the trained color component; b) obtain a representative value of the first color component of at least one target image, which is a biological staining image that is to be input to the trained model and from which the target region is to be detected, as a representative value of the target color component; c) obtain a correction function from the representative value of the trained color component and the representative value of the target color component; d) correct the second color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, of the at least one target image using the correction function; and e) input the corrected at least one target image to the trained model to detect the target region from the at least one target image.

[0015] Aspect 8 of the present invention is a region detection device for detecting a target region to be detected based on staining from a biological staining image, comprising: a trained model that detects a target region from a target image when a target image, which is a biological staining image, is input; a correction function acquisition unit that acquires a representative value of a first color component of interest, which is saturation, or a first color component corresponding to a color corresponding to staining or a color similar thereto, from a plurality of training images used to generate the trained model, and acquires a representative value of the first color component of interest from at least one target image that is to be input to the trained model and from which a target region is to be detected, as a target color component representative value, and acquires a correction function from the trained color component representative value and the target color component representative value; and a correction unit that corrects a second color component of interest, which is saturation, or a second color component corresponding to a color corresponding to staining or a color similar thereto, from at least one target image using the correction function.

[0016] The aforementioned objectives, as well as other objectives, features, embodiments, and advantages, will be revealed by the detailed description of the invention below, with reference to the attached drawings.

[0017] This is a diagram showing the configuration of the computer. This is a block diagram showing the functional configuration of the region detection device. This is a diagram showing the flow of the region extraction method. This is a diagram showing the flow of the region extraction method. This is a diagram illustrating a training image. This is a diagram illustrating a saturation image. This is a diagram illustrating a mask image. This is a diagram illustrating a target image before correction. This is a diagram illustrating a target image after correction. This is a diagram showing an example of the display unit. This is a diagram showing the flow of the correction function modification.

[0018] Figure 1 shows the configuration of a computer 4 that functions as a region detection device according to one embodiment of the present invention. The region detection device is a device that uses a trained model to detect target regions that should be detected based on staining from a biological staining image. The target region is a stained region, a region surrounded by a stained region, a region containing a stained region, etc., and in this embodiment, the target region is the region of stained cells. "Region detection" is segmentation, which separates the region to be detected from other regions in an image. Target region detection using a trained model is used, for example, in the field of image cytometry. In this embodiment, the target image for which the target region is detected is assumed to be an image of a specimen in multiplex immunohistochemistry analysis, but the use of the computer 4 is not limited to such specimen analysis. Also, the target region does not have to be a cell region. The computer 4 also functions as a device that generates trained models.

[0019] Computer 4 has the configuration of a general computer system, including a CPU 41, a GPU 42, a ROM 43, a RAM 44, a fixed disk 45, a display 46, an input unit 47, a reader 48, a communication unit 49, and a bus 40. The CPU 41 performs various arithmetic operations. The GPU 42 performs various image processing operations at high speed. The ROM 43 stores basic programs. The RAM 44 stores various information. The fixed disk 45 stores information. The display 46 displays various information such as images.

[0020] The input unit 47 includes a keyboard 47a and a mouse 47b that receive input from the operator. The reader 48 reads information from a computer-readable recording medium 9 such as an optical disk, magnetic disk, magneto-optical disk, or memory card. The display 46, keyboard 47a, mouse 47b, and reader 48 are connected to the bus 40 via an interface (I / F). The communication unit 49 sends and receives signals to and from external devices of the computer 4. The bus 40 is a signal circuit that connects the CPU 41, GPU 42, ROM 43, RAM 44, fixed disk 45, display 46, input unit 47, reader 48, and communication unit 49.

[0021] In computer 4, program 91 is read in advance from recording medium 9 via reader 48 and stored in fixed disk 45. Program 91 may also be stored in fixed disk 45 via a network. The CPU 41 and GPU 42 perform arithmetic processing using RAM 44 and fixed disk 45 according to the computer-readable program 91. The CPU 41 and GPU 42 function as arithmetic units. Other configurations besides the CPU 41 and GPU 42 that function as arithmetic units may also be employed.

[0022] Figure 2 is a block diagram showing the functional configuration of the region detection device 1, which is realized when the computer 4 performs calculations and other processing according to the program 91. All or part of each functional configuration may be realized by dedicated electrical circuits. Furthermore, these functional configurations may be realized by multiple computers. Program 91 is a computer-readable program that instructs the computer 4 to detect target regions from biological staining images using a trained model.

[0023] Of the functional configurations shown in Figure 2, the display unit 11 mainly displays the functions of the display 46 in Figure 1 and provides information to the operator. The arithmetic unit 12 is realized by the CPU 41, GPU 42, ROM 43, RAM 44, fixed disk 45 and their peripheral components. The operation unit 13 receives input from the operator and is mainly realized by the input unit 47 in Figure 1. The storage unit 14 is mainly realized by the RAM 44 and fixed disk 45. Various devices can be used as the storage unit 14 as long as they can store information.

[0024] The memory unit 14 is pre-stored with data for multiple target images. In Figure 2, the data for the target images is shown as "target image data 51". The processing of the target images in the following description is, more precisely, processing of the target image data 51. The same applies to other images. The target images are biological staining images that are to be input into the trained model 125, described later, to detect the target region. Detection of the target region is synonymous with "extraction of the target region".

[0025] The memory unit 14 also stores training image data. Training images are images used to generate the trained model 125. In Figure 2, the training image data is shown as "training image data 52". The processing of training images in the following description is, more precisely, processing of training image data 52. Annotation image data 53 is associated with the training image data 52. Annotation image data 53 is image data in which the region to be detected as a target region in the training image is indicated. Multiple combinations of training image data 52 and annotation image data 53 are stored in the memory unit 14 as a training dataset 520.

[0026] The calculation unit 12 comprises, functionally, a conversion unit 121, a correction function acquisition unit 122, a correction unit 123, an inverse conversion unit 124, and a learning unit 126. The calculation unit 12 also stores a trained model 125. The trained model 125 is a model generated when the learning unit 126 trains the learning model using the learning dataset 520, thereby determining the values ​​of various parameters included in the learning model. The training here is so-called deep learning, but other machine learning methods may be used. The trained model 125 has the function of outputting region detection image data 54 in which a target region has been detected when image data is input. In the region detection device 1, the learning unit 126 may be omitted. In this case, training is performed by a separately provided learning device to generate the trained model 125, and the trained model 125 and the learning dataset 520 are taken into the region detection device 1.

[0027] Figures 3 and 4 show the flow of the region extraction method executed by the computer 4. Before the processes shown in Figures 3 and 4 are executed, a trained model 125 is generated using the training dataset 520 and prepared in the calculation unit 12.

[0028] First, as shown in Figure 2, the storage unit 14 is prepared with training image data 52, which is data of multiple training images, and target image data 51, which is data of at least one target image (step S11). The number of target images may be one or two or more. Next, one training image is selected, and the conversion unit 121 converts the pixel values ​​of each pixel in the training image into an HSV color component group (step S12).

[0029] In this embodiment, the pixel values ​​of the training image before conversion are the values ​​of the red (R), green (G), and blue (B) color components. The pixel values ​​after conversion, that is, the values ​​of the three color components, are the hue (H), saturation (S), and lightness (V) color components. In this embodiment, the three color components are referred to as the "color component group." Also, in this embodiment, as will be described later, we focus on saturation among the converted color component group, so saturation will also be referred to as the "color component of interest" below. The training image converted to the values ​​of the color component group including the color component of interest will be referred to below as the "training-converted image."

[0030] Next, if there are unconverted training images (step S13), the unconverted training images are selected and converted (step S12). By performing step S12 for each of the multiple training images, multiple trained converted images are obtained (steps S12, S13).

[0031] Figure 5 illustrates a training image 61. The training image 61 includes tissue regions 611 and non-tissue regions 612. The tissue region 611 also includes a stained region 613. Figure 6 illustrates a training transformation image 62 in which the saturation value is used as a pixel value (hereinafter referred to as the "saturation image"). In the saturation image 62, the region 622 corresponding to the non-tissue region 612 in Figure 5 appears with extremely low saturation, while the region 623 corresponding to the stained region 613 appears as a region with high saturation. In addition, the region 621 corresponding to the tissue region 611 contains a region 624 with low saturation.

[0032] The correction function acquisition unit 122 acquires a representative value of the saturation of pixels in multiple trained transformation images that have a saturation value equal to or greater than a predetermined threshold, as a trained saturation representative value (indicated as "trained color component representative value" in Figure 3) (step S14). Specifically, first, a mask image is generated in the saturation image 62 of Figure 6 that masks pixels whose saturation value is less than the threshold. Figure 7 is an example of a mask image 63, where the mask region 630 is the region with lower saturation than the region 623 of Figure 6 that corresponds to the stained region (regions 621, 622, 624 of Figure 6). In Figure 7, the region corresponding to the stained region is denoted by the code 633. Then, the saturation image 62 is masked with the mask image 63, and the average value of the saturation of the unmasked pixels is obtained. Furthermore, the average of the above average values ​​of all saturation images is obtained as the trained saturation representative value. By using a mask to exclude low-saturation areas from the calculation, the influence of insufficiently stained areas or areas that do not show tissue on the calculation of the learned saturation representative value is suppressed.

[0033] In the above description, the representative value is the mean, but the representative value may be various summary statistics such as the 50th percentile (median), the 75th percentile (third quartile), and the 95th percentile. The same applies to the representative value of target saturation described later. In this embodiment, the representative value is the mean, but the mean in the following description may be replaced with other representative values.

[0034] Furthermore, in calculating the learned saturation representative value, the representative value of saturation at the target pixel in the saturation image may be determined, and the value obtained from the above representative values ​​of all saturation images (i.e., the representative value of the representative values) may be determined as the learned saturation representative value, or the representative value of saturation at the target pixel in all saturation images may be determined as the learned saturation representative value. The same applies to the target saturation representative value described later. Furthermore, the above definition of representative values, and the explanation of the method for determining the learned saturation representative value and target saturation representative value, are also the same for the learned color component representative value and target color component representative value described later.

[0035] Next, one target image is selected. This target image is transformed by the transformation unit 121 in the same way as the training image (step S15). That is, the pixel value (value of the RGB color component group) of each pixel of the target image is converted to the value of the color component group (HSV color component group) which includes saturation, which is the color component of interest. Hereinafter, the transformed target image will be referred to as the "transformed target image".

[0036] Next, if there are unconverted target images (step S16), the unconverted target images are selected and converted (step S15). By performing step S15 for each of the multiple target images, multiple converted target images are obtained (steps S15, S16). Note that the number of target images may be as small as one.

[0037] The correction function acquisition unit 122 performs the same processing as in step S14 for all target images (at least one target image) to acquire a representative value of the target saturation. That is, it acquires the average value of the saturation of pixels in at least one target transformation image that have a saturation value equal to or greater than a predetermined threshold as the representative value of the target saturation (indicated as "representative value of the target color component" in Figure 3) (step S17). Specifically, first, in a saturation image that has the saturation value of each pixel of the target transformation image as a pixel value, a mask image is generated that masks pixels whose saturation value is less than the threshold, the saturation image is masked with the mask image, and the average value of the saturation of the unmasked pixels is obtained. The average of the above average values ​​of all saturation images is obtained as the representative value of the target saturation.

[0038] Next, the correction function acquisition unit 122 acquires a correction function from the learned saturation representative value and the target saturation representative value (Figure 4: Step S21). Preferably, the correction function is a function that adjusts the saturation value of each pixel of the target image to a representative saturation value of multiple learned images. In other words, the correction function is a function that brings the saturation value of each pixel of the target image closer to a representative saturation value of multiple learned images. In this embodiment, the value obtained by dividing the learned saturation representative value by the target saturation representative value is obtained as a correction coefficient, and a linear function obtained by multiplying the saturation value of each pixel of the target transformed image by the correction coefficient is obtained as the correction function. If the corrected value exceeds the upper limit, the upper limit is used. Note that the correction function may be acquired in any way as long as it functions substantially as a function, and in this embodiment, the correction coefficient is acquired substantially as the correction function.

[0039] Furthermore, the correction function may be a function that makes the saturation value of each pixel in the target image greater than the representative saturation value of multiple training images. In other words, the correction function may enhance the saturation of the target image compared to the saturation of the training images.

[0040] The processing up to step S21 is a preparatory process for detecting a target region from the target image using the trained model 125. Once the preparatory process is complete, one target image is selected and its saturation is corrected (step S22). When correcting the saturation, the target image may be converted to an HSV color component group, but since the converted target image has already been obtained in step S15, the saturation correction is actually performed on this converted target image. That is, the saturation value of each pixel is multiplied by a correction coefficient. The correction may increase or decrease the saturation of the converted target image.

[0041] The corrected target image is inversely transformed in the inverse transformation unit 124, and the HSV color component group is converted to the RGB color component group (step S23). In other words, the color component group is inversely transformed back to the initial color component group. This results in a target image with corrected saturation. Figure 8 is an example of the target image 71 before correction. The target image 71 includes tissue regions 711 and non-tissue regions 712. The tissue region 711 also includes a stained region 713. If the saturation of the target image 71 is lower than the saturation of the multiple training images, the correction increases the saturation of the target image 71, and in the corrected target image 72, the stained region 723 becomes clearly visible as shown in Figure 9. In other words, the saturation of the target image is corrected to be equivalent to the saturation of the multiple training images.

[0042] The corrected target image is input to the trained model 125, and the target region is detected (step S24). That is, an image showing the target region is acquired. If there are unprocessed target images (step S25), the next target image is selected, and steps S22 to S25 are repeated. As a result, the saturation of at least one target image 71 is corrected using a correction function to acquire a corrected target image 71, which is then input to the trained model 125, and the target region is detected from that at least one target image 71.

[0043] As described above, in the region detection apparatus 1, the saturation of a target image is corrected using a plurality of learned images used when generating the trained model 125, and then the target image is input to the trained model 125. This makes it possible to reduce the influence of the staining state indicated by the learned images and the staining state indicated by the target image, and improve the accuracy of detecting a target region from the target image using the existing trained model 125.

[0044] In the above description, it has been described that the operations shown in FIGS. 3 and 4 are performed on all target images. However, the target images may be grouped based on factors that affect the staining state, such as staining conditions, equipment used for staining, and the person in charge of staining. In this case, the operations shown in FIGS. 3 and 4 are performed for each group obtained by dividing a plurality of target images into a plurality of groups. This makes it possible to further improve the accuracy of detecting the target region.

[0045] In the region detection apparatus 1, the operations shown in FIGS. 3 and 4 may be performed for each target image. That is, the operations shown in FIGS. 3 and 4 may be performed with the number of target images set to 1. This operation makes it possible to appropriately detect the target region even when the staining states between target images greatly differ.

[0046] Next, correction of a correction function and a threshold for masking will be described. FIG. 10 is a diagram showing a display example of a display unit 11 used for correcting a correction coefficient that is a correction function. That is, an operator determines the final correction coefficient and threshold while referring to the display in FIG. 10. In FIG. 10, a selected one of the learned images is displayed on the left side, and a selected one of the target images is displayed on the right side. For example, a plurality of learned images are displayed small so as to be arranged on the screen, and when the operator selects a learned image via an operation unit 13, the learned image is displayed on the left side of the screen. Similarly, a plurality of target images are displayed small so as to be arranged on the screen, and when the operator selects one target image, the target image is displayed on the right side of the screen. A plurality of learned images may be selected and displayed side by side. A plurality of target images may be selected and displayed side by side.

[0047] In FIG. 10, the learning image is denoted by reference numeral 61, and the target image is denoted by reference numeral 71. In addition, "all-image saturation representative value" is displayed above these images. In the present embodiment, the all-image saturation representative value above the learning image 61 is a value obtained by averaging, over all learning images, the average saturation value (average value after mask processing) of each individual learning image. In the present embodiment, the all-image saturation representative value above the target image 71 is a value obtained by averaging, over all target images, the average saturation value (average value after mask processing) of each individual target image.

[0048] "Selected group" indicates a selected one of groups preset for a plurality of target images. In the present embodiment, the "group saturation representative value" is a value obtained by averaging, over all target images in the group, the average saturation value of each individual target image belonging to the selected group. In the "selected group", "Group1", "Group2", ... can be selected. When "Group1" is selected, a value obtained by averaging, over all target images of Group1, the average saturation value of each individual target image belonging to Group1 is displayed as the group saturation representative value. The group saturation representative value for each group is obtained in advance.

[0049] In the present embodiment, the "saturation representative value" displayed below the learning image 61 in FIG. 10 is the average saturation value of the currently displayed learning image 61, and the "saturation threshold" therebelow is a mask threshold value used when obtaining the average saturation value of the learning image during the calculation process of step S14. In the present embodiment, the "saturation representative value" displayed below the target image 71 is the average saturation value of the currently displayed target image 71, and the "saturation threshold" therebelow is a mask threshold value used when obtaining the average saturation value of the target image 71 during the calculation process of step S17.

[0050] The "correction coefficient" below the target image 71 is a coefficient of a correction function that is a proportional function. The initial value of the correction coefficient is a value obtained using the above-described learning saturation representative value and target saturation representative value. The "corrected saturation representative value" is a value obtained by multiplying the saturation representative value of the currently displayed target image 71 by the correction coefficient. As the target image 71, the image after saturation correction is displayed.

[0051] Here, the operator can change the saturation threshold of the training image 61, as well as the saturation threshold, correction coefficient, and corrected saturation representative value of the target image 71, via the operation unit 13.

[0052] For example, if the saturation threshold of the training image 61 is changed, the correction function acquisition unit 122 recalculates the representative saturation value of the training image 61. By comparing the representative saturation value of all images with the representative saturation value of the selected training image, it is possible to confirm whether the initial saturation threshold was appropriate. Similarly, if the saturation threshold of the target image 71 is changed, the correction function acquisition unit 122 recalculates the representative saturation value of the target image 72. By comparing the representative saturation value of all images and the group saturation representative value with the representative saturation value of the selected target image, it is possible to confirm whether the initial saturation threshold was appropriate. The saturation threshold of the training image 61 and the saturation threshold of the target image 71 may be changed so that when one is changed, the other is always equal. In addition, a saturation image (see Figure 6) and a mask image based on the saturation threshold (see Figure 7) can be displayed for the training image and the target image.

[0053] If the correction coefficient of the target image 71 is changed, the corrected saturation representative value is updated, and if the corrected saturation representative value is changed, the correction coefficient is updated. When a change occurs in the correction coefficient, a process is performed to change the saturation of the data of the currently displayed target image 71, and the corrected target image 71 (corresponding to the target image 72 in Figure 9) is displayed. The operator looks at the displayed target image 71, the overall saturation representative value, and the group saturation representative value and determines whether the correction coefficient or the corrected saturation representative value is appropriate. Through the above work by the operator, the final saturation threshold and correction coefficient are determined.

[0054] Figure 11 shows the operation flow of the region detection device 1 in correcting the correction coefficient (correction function) described above. The steps shown in Figure 11 may be performed at any stage after step S21 (acquisition of the correction function) in Figure 4.

[0055] First, the system accepts the operator's selection of a target image, and the selected target image 71 is displayed on the display unit 11 (step S31). Hereinafter, the selected target image 71 will be referred to as the "selected target image". As illustrated in Figure 10, the display unit 11 displays the current representative saturation value, saturation threshold, correction coefficient, and corrected saturation threshold of the selected target image 71.

[0056] Next, when the operator changes the correction coefficient or the corrected saturation threshold via the control unit 13, the other value is appropriately changed to match the changed value, as described above. Note that the process of changing either value can be considered as a modification of the correction coefficient (correction function) (step S32).

[0057] The correction unit 123 applies the corrected correction coefficient to the selected image 71 to obtain a selected image with corrected saturation (step S33). Specifically, it converts the color component group of the selected image 71 from RGB to HSV, corrects the saturation using the corrected correction coefficient, and converts the color component group back to RGB. Then, the corrected selected image 71 is displayed on the display unit 11 (step S34).

[0058] If the process shown in Figure 11 is performed immediately after step S21, steps S22 to S25 are executed using the modified correction function. If the process shown in Figure 11 is performed after step S25, steps S22 to S25 are executed again using the modified correction function, and the detection result of the target region is updated.

[0059] As described above, the region detection device 1 allows the operator to modify the correction function while confirming the detection results via the display unit 11. This enables the detection of more appropriate target regions. Note that the correction function may be modified only for some of the target images (each group or individual target images). Furthermore, if the operator changes the saturation threshold for the training images, the representative saturation value for all images may be recalculated using the changed saturation threshold. Similarly, if the operator changes the saturation threshold for the target images, the representative saturation value for all images and the representative saturation value for groups may be recalculated using the changed saturation threshold.

[0060] Next, we will explain an example in which color components other than saturation are corrected as part of the correction in step S22. When an image is color-separated into a group of color components that includes color components other than saturation, saturation information is not obtained, but color information close to the dyed color is obtained.

[0061] The operation flow of the region detection device 1 when using color components other than saturation as the color component of interest is the same as in Figures 3, 4, and 11, except for some details, and the operation is the same except that saturation is replaced with a color component of interest other than saturation. In the following explanation of processing with respect to a color component of interest other than saturation, unless otherwise specified, a color component of interest other than saturation will simply be referred to as the "color component of interest".

[0062] As a color decomposition performed on the training image and target image, for example, the "Color Deconvolution" function is executed using the image analysis software "ImageJ," and "[H AEC]" is selected as the "Vectors," decomposing the pixel values ​​of the RGB color component group into "H color component," "AEC color component," and "other color component." Then, the AEC color component is used as the color component of interest. The color decomposition is not limited to the above example, but in this embodiment, since the dyed color is the AEC color component, the above color decomposition is used.

[0063] In order to perform color separation and correct the intensity of the stained color in an image, the color component of interest must be the color corresponding to the stain or a color that approximates that color. In other words, the color component group obtained by converting from the initial color component group RGB of each pixel in the training image includes the color component of interest that corresponds to the color corresponding to the stain or a color that approximates that color. The training-converted image obtained in step S12 is an image obtained by converting the values ​​(pixel values) of the initial color component group of each pixel to the values ​​of the color component group that includes the color component of interest. Then, by having the conversion unit 121 perform step S12 for each of the multiple training images, multiple training-converted images are obtained (step S13).

[0064] In step S14, the correction function acquisition unit 122 acquires representative values ​​of the color components of interest from multiple trained transformed images as the trained color component representative values. The trained color component representative values ​​correspond to the explanation of the trained saturation representative values ​​above, where saturation is replaced with the color component of interest (AEC color component). In this case, unlike the case of saturation, a mask image using a threshold (see Figure 7) is not generated. That is, the average value of the color component of interest from each trained image is simply calculated, and the value obtained by averaging these average values ​​over all trained images is the trained color component representative value. When correcting saturation, not only the dyed color but also various other colors are corrected simultaneously, and a threshold (saturation threshold) is used to exclude undyed areas. Of course, a threshold may also be used when calculating the trained color component representative values.

[0065] The conversion unit 121 and the correction function acquisition unit 122 perform the same processing on at least one target image to obtain representative values ​​of the target color components (steps S15 to S17). That is, the initial color component group values ​​(pixel values) of each pixel of the target image are converted to the values ​​of the color component group that includes the color component of interest to obtain a target converted image, and at least one target converted image is obtained by executing step S15 for each of the at least one target image (step S16). Then, the average value of the color component of interest in the at least one target converted image is obtained as the representative value of the target color component (step S17). In principle, thresholds and mask images are not used in the above processing either.

[0066] Subsequently, similar to the case of saturation, the correction function acquisition unit 122 acquires a correction function from the learned color component representative value and the target color component representative value (step S21), and the correction unit 123 corrects the target color component of the selected target transformation image using the correction function (step S22). The correction function is a function that adjusts the value of the target color component of each pixel of the target image to the representative value of the target color component of multiple learning images. In other words, the correction function is a function that brings the value of the target color component of each pixel of the target image closer to the representative value of the target color component of multiple learning images. Furthermore, the inverse transformation unit 124 inversely transforms the color component group in the corrected target transformation image back to the initial color component group, thereby acquiring the corrected target image (step S23).

[0067] Furthermore, the correction function may be a function that makes the value of the color component of interest for each pixel of the target image greater than the representative value of the color component of interest for multiple training images. In other words, the correction function may be one that emphasizes the color component of interest for the target image more than the color component of interest for interest for the training images.

[0068] The corrected target image is input to the trained model 125, and the target region is detected from the target image (step S24). Steps S22 to S24 are performed for each of at least one target image (step S25), so that the target region is detected from all target images.

[0069] As explained above, in the region detection device 1, the target image is input to the trained model 125 after the color component of the target image has been corrected using multiple training images used to generate the trained model 125. This reduces the influence of the staining state shown in the training images and the staining state shown in the target image, and improves the accuracy of detecting the target region from the target image using the already existing trained model 125. By using a color component of interest other than saturation instead of saturation, the accuracy of detecting the target region can be further improved.

[0070] Even when the color component of interest is something other than saturation, the correction function may be modified in the same manner as the display screen in Figure 10 and the operation flow in Figure 11. However, a threshold may or may not be set. The modification of the correction function is the same as described above, but with saturation replaced by the color component of interest. By modifying the correction function while checking the detection results via the display unit 11, an appropriate correction function can be obtained quickly.

[0071] In the above embodiment, we have described cases where saturation is used as the color component of interest and cases where a color component other than saturation is used as the color component of interest. However, when correcting the saturation of the training image or target image, a color component other than saturation may also be used. For example, although the correction is performed on saturation, a color component other than saturation may be used when obtaining the mask image when acquiring the training saturation representative value (step S14) or the target saturation representative value (step S17).

[0072] Specifically, in step S12, the pixel value of each pixel in the training image is converted to the value of a group of color components that includes the color of interest corresponding to the color of staining or a color that approximates that color, thereby obtaining a training-transformed image. For example, the RGB color component group is converted to "H color component," "AEC color component," and "other color component." Then, in step S13, multiple training-transformed images are obtained. Subsequently, in step S14, the average value of the saturation of pixels in the multiple training-transformed images that have a value of the color of interest (other than saturation) that is above a predetermined threshold is obtained as the training saturation representative value. Note that since the saturation of the training image is needed in step S14, in step S12, the training image is also converted to an HSV color component group (conversion to a group of color components that includes saturation), and a saturation image is obtained.

[0073] Similarly, in step S15, the target image is converted to a target converted image by converting the pixel value of each pixel in the target image to a value of the color component group that includes the aforementioned color component of interest, excluding saturation. Then, in step S16, multiple target converted images are obtained. Subsequently, in step S17, the average value of the saturation of pixels in the multiple target converted images that have a value of the color component of interest above a predetermined threshold is obtained as the target saturation representative value. Note that since the saturation of the target image is required in step S17, in step S15, the target image is also converted to the HSV color component group (conversion to a color component group that includes saturation), and a saturation image is obtained.

[0074] Once the learned saturation representative value and the target saturation representative value are obtained, a correction function is acquired through the process shown in Figure 4, and the target region is detected from the corrected target image. Through the above process, a highly accurate saturation correction function is obtained by utilizing color components other than saturation, while high-speed and highly accurate detection of the target region is achieved by utilizing high-speed conversion between RGB and HSV.

[0075] In the embodiments described above, examples were explained in which a mask image is generated using saturation as the color of interest component, a correction function is obtained using saturation as the color of interest component, and the target image is corrected using saturation as the color of interest component; an example is described in which a correction function is obtained using color of interest components other than saturation, and the color of interest components other than saturation of the target image are corrected; and an example is described in which a mask image is generated using color of interest components other than saturation, a correction function is obtained using saturation as the color of interest component, and the target image is corrected using saturation as the color of interest component. However, the color of interest components when obtaining the correction function, the color of interest components when making corrections, and the color of interest components when generating the mask image may be combined arbitrarily.

[0076] In this case, in step S12, the conversion unit 121 generates images of the required color components from the training image, including the color component of saturation and the color component of saturation corresponding to the color or a color similar to that color (hereinafter also referred to as "non-saturation color component of saturation"). Similarly, in step S15, the conversion unit 121 generates images of the required color components from the target image, including the color component of saturation and the non-saturation color component of saturation.

[0077] For example, if an image of the color component of interest, which is saturation, and an image of the color component of interest that is not saturation are to be used in a later process, in step S12, the pixel value of each pixel in the training image is converted to a value in the color component group that includes the color component of interest, which is saturation, thereby obtaining a training-transformed image. Furthermore, another training-transformed image is obtained by converting the pixel value of each pixel in the training image to a value in the color component group that includes the color component of interest that is not saturation. Also, in step S15, the target-transformed image is obtained by converting the pixel value of each pixel in the target image to a value in the color component group that includes the color component of interest, which is saturation, thereby obtaining a target-transformed image. Furthermore, another target-transformed image is obtained by converting the pixel value of each pixel in the target image to a value in the color component group that includes the color component of interest that is not saturation.

[0078] Then, in step S14, the representative value of the focus color component of multiple trained transformed images is obtained as the trained color component representative value, using either the focus color component that is saturation or the focus color component that is not saturation as the focus color component. Similarly, in step S17, the representative value of the focus color component of at least one target transformed image is obtained as the target color component representative value, using either the focus color component that is saturation or the focus color component that is not saturation as the focus color component. Furthermore, the correction function is obtained in step S21.

[0079] Subsequently, in steps S22 to S25, the color component of interest, whether it is saturation or not, is treated as the color component of interest, and the color component of interest in the target transformed image corresponding to that color component is corrected. After the inverse transform, the target region is detected by the trained model. The transformation unit 121, the correction unit 123, and the inverse transform unit 124 are used to correct the color component of interest. If these functions are considered as a correction unit in a broad sense, the correction unit corrects the color component of interest in the target image.

[0080] In the above operation, if we refer to the color component of interest when determining the correction function (the color component of interest that is saturation, or the color component of interest that corresponds to the color of the dye or a color that approximates that color) as the "first color component of interest", then in step S14, representative values ​​of the first color component of the multiple training images used to generate the trained model are obtained as the training color component representative values. In step S17, representative values ​​of the first color component of at least one target image are obtained as the target color component representative values. Then, in step S21, the correction function is obtained from the training color component representative values ​​and the target color component representative values.

[0081] On the other hand, if we refer to the color component to be corrected (the color component of interest which is saturation, or the color component of interest which corresponds to the color of the dye or a color that approximates that color) as the "second color component of interest", then in steps S22 to S25, the second color component of interest of at least one target image is corrected using the correction function, and by inputting the corrected at least one target image into the trained model, the target region is detected from that at least one target image. The first color component of interest may be the same as or different from the second color component of interest.

[0082] Furthermore, if a mask image is used in steps S14 and S17, the color component of interest used to generate the mask image may also be a color component of interest that is saturation, or a color component of interest that corresponds to a color or a color that approximates that color. If this color component of interest is called the "third color component of interest," then in step S14, the representative value of the first color component of the pixels in multiple training images whose third color component of interest is equal to or greater than a predetermined threshold is obtained as the representative value of the training color component. In step S17, the representative value of the first color component of the pixels in at least one target image whose third color component of interest is equal to or greater than a predetermined threshold is obtained as the representative value of the target color component. The third color component of interest may be the same as or different from the first or second color component of interest.

[0083] As described above, there are four possible combinations of the first and second color components of interest, and if a mask image is used, there are eight possible combinations of the first to third color components of interest. The explanations given with reference to Figures 1 to 11 apply to all of these combinations. In any combination, the influence of the staining state shown by the training image and the staining state shown by the target image can be reduced, and the accuracy of detecting the target region from the target image using the already existing trained model 125 can be improved.

[0084] The configuration and operation of the region detection device 1 may be modified in various ways.

[0085] In Figures 3 and 4, the order of each step may be changed as appropriate, provided that equivalent processing is performed. For example, step S22 may be performed on all target images, then step S23 may be performed on all target images, and then step S24 may be performed on all target images.

[0086] The mask image used to determine the learned saturation representative value and the target saturation representative value may be generated using an upper threshold in addition to a lower threshold for saturation. That is, not only pixels with low saturation but also pixels with high saturation may be excluded when determining the learned saturation representative value and the target saturation representative value. The learned saturation representative value and the target saturation representative value may be determined after masking obviously unnecessary areas in advance. In this way, the representative value of saturation when determining the correction function may be obtained as an approximate representative value from appropriately selected pixels. Even when determining the correction function based on color components of interest other than saturation, the learned color component representative value and the target color component representative value may be obtained using lower thresholds and upper thresholds for the color components.

[0087] The correction function is not limited to multiplying by a correction coefficient; it may also add, subtract, or divide a constant by the color component values. The correction function is not limited to a linear function.

[0088] When determining the learned saturation representative value or learned color component representative value, color components other than the color component of interest (including saturation) are not used. Therefore, only the value of the color component of interest is obtained, and the values ​​of other color components do not need to be obtained. Regarding training images, the concept of obtaining the values ​​of a group of color components by transforming pixel values ​​includes the case where only the value of one color component is obtained.

[0089] The vital staining images in which the target region is detected can be various images as long as they show cells. The training images and target images are not limited to images showing pathological tissue. For example, they may be images showing a collection of cells that do not bind to each other.

[0090] In the region detection device 1, corrections are made when the target image is input to the already existing trained model 125. Therefore, it is preferable that no corrections are made to the saturation or color component of the training image when generating the trained model 125. However, corrections to the training image may be made when generating the trained model 125.

[0091] The configurations in the above embodiments and each modified example may be combined as appropriate, as long as they do not contradict each other.

[0092] Although the invention has been described in detail, the above description is illustrative and not limiting. Therefore, it can be said that numerous modifications and embodiments are possible as long as they do not deviate from the scope of the present invention.

[0093] 1 Region detection device 4 Computer 61 Training image 71 Target image 91 Program 121 Transformation unit 122 Correction function acquisition unit 123 Correction unit 124 Inverse transformation unit 125 Trained model S12-S17, S21-S25, S31-S34 Steps

Claims

1. A region detection method for detecting a target region to be detected based on staining from a biologically stained image using a trained model, comprising: a) a step of obtaining a representative value of the first color component of interest, which is saturation, or the color corresponding to the stain or a color similar thereto, from a plurality of training images used to generate the trained model, as a representative value of the training color component; b) a step of obtaining a representative value of the first color component of at least one target image, which is a biologically stained image that is to be input into the trained model and from which a target region is to be detected, as a representative value of the target color component; c) a step of obtaining a correction function from the representative value of the training color component and the representative value of the target color component; d) a step of correcting the second color component of interest, which is saturation, or the color corresponding to the stain or a color similar thereto, of the at least one target image using the correction function; and e) a step of detecting a target region from the at least one target image by inputting the corrected at least one target image into the trained model.

2. A region detection method according to claim 1, wherein in step a), a representative value of the first color component of pixels in the plurality of learning images whose third color component of interest is saturation, or whose third color component of interest is a color corresponding to a color or a color similar to said color and whose value is equal to or greater than a predetermined threshold is obtained as the learning color component representative value, and in step b), a representative value of the first color component of pixels in at least one target image whose third color component of interest is equal to or greater than a predetermined threshold is obtained as the target color component representative value.

3. A region detection method according to claim 1, wherein the at least one target image is one of the groups obtained by dividing a plurality of target images into a plurality of groups.

4. A region detection method according to claim 1, wherein the number of at least one target image is 1.

5. A region detection method according to claim 1, further comprising, after step c), f) a step of displaying a selected target image selected from the at least one target image; g) a step of modifying the correction function; h) a step of correcting the selected target image using the modified correction function; and i) a step of displaying the selected target image.

6. A region detection method according to any one of claims 1 to 5, wherein the target region is a region at the cellular level.

7. A computer-readable program that causes a computer to detect a target region to be detected based on staining from a biological staining image using a trained model, wherein the execution of the program by the computer comprises the steps of: a) obtaining a representative value of the first color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, from a plurality of training images used to generate the trained model, as a representative value of the trained color component; b) obtaining a representative value of the first color component of at least one target image, which is a biological staining image that is to be input into the trained model and from which the target region is to be detected, as a representative value of the target color component; c) obtaining a correction function from the representative value of the trained color component and the representative value of the target color component; d) correcting the second color component of interest, which is saturation, or the color corresponding to the staining or a color similar thereto, from the at least one target image using the correction function; and e) detecting a target region from the at least one target image by inputting the corrected at least one target image into the trained model. A program that executes the command.

8. Region detection device for detecting a target region to be detected based on staining from a biological staining image, comprising: a trained model that detects a target region from a target image when a target image which is a biological staining image is input; a correction function acquisition unit that acquires a representative value of a first color component of interest, which is saturation, or a first color component corresponding to a color corresponding to staining or a color similar thereto, from a plurality of training images used to generate the trained model, and acquires a representative value of the first color component of interest of at least one target image which is to be input to the trained model and from which a target region is to be detected, as a target color component representative value, and acquires a correction function from the trained color component representative value and the target color component representative value; and a correction unit that corrects a second color component of interest, which is saturation, or a second color component corresponding to a color corresponding to staining or a color similar thereto, of the at least one target image using the correction function.

Images