Pathological image analysis method and device

Abstract

A method performed by a device according to one embodiment may comprise: an operation of acquiring tissue images labeled with BAG classes that specify whether an instance corresponding to a tumor tissue is included; a noise removal preprocessing operation of removing noise from the tissue images by applying a variable threshold value according to visual feature information for each region of the tissue images; and an operation of training a neural network model trained using multi-instance learning by dividing the preprocessed tissue images into tiles of a predefined size and labeling tiles divided from the same tissue image with the same BAG class.

Description

Pathological image analysis method and device

[0001] The present invention relates to a pathological image analysis technique, and more specifically, to a technique that improves the analysis accuracy of pathological images by providing noise removal, color normalization, lesion detection through a multi-instance learning technique, and model explainability using Grad-CAM.

[0002]

[0003] Pathological images are utilized as essential data for the diagnosis and research of diseases, and recently, research utilizing artificial intelligence to extract and classify features from these images is actively being conducted.

[0004] These AI-based pathological image analysis technologies learn patterns from large-scale pathological image data, distinguish between normal tissue and lesions, and predict the progression of specific diseases. Along with this, as technologies for analyzing the location and distribution of cancer cells within pathological images advance, there is a growing need for AI models capable of more precise identification for the histological classification and prognosis prediction of various tumors, such as gastric, breast, and kidney cancer.

[0005] Meanwhile, there are several issues that need to be addressed in existing pathological image analysis. Pathological images may contain various forms of noise, such as air bubbles, pen marks, and tissue folds, which can negatively impact training and analysis results. In particular, since analyzing minute differences in tissue structure is crucial for pathological examinations of cancers such as gastric, breast, and kidney cancers, more sophisticated preprocessing techniques are required because this noise has a significant impact on model accuracy.

[0006] In addition, since pathological images appear differently in color and density depending on various scanner equipment and staining methods, variations in image data may occur even for the same tissue sample. These differences degrade the generalization performance of artificial intelligence models, leading to problems where high accuracy is achieved in specific hospitals or datasets, but performance degrades in other environments.

[0007] Furthermore, pathological images often contain a low proportion of specific lesions within the overall image. Consequently, when traditional classification models are trained on a whole-image basis, they are prone to failing to detect subtle changes in the lesion or making misjudgments due to being influenced by the characteristics of surrounding tissues rather than the lesion itself. While multi-instance learning techniques are utilized to address this, previous studies have faced issues such as limited application methods or difficulties in clearly presenting the rationale behind the model's judgments.

[0008] To overcome these limitations, a preprocessing method is required that effectively removes noise from images while preserving important tissue features, as well as technology capable of obtaining consistent results across various scanners and staining methods. Furthermore, it is necessary to optimize training strategies based on multi-instance learning to analyze diverse tumors, such as gastric, breast, and renal cancers, and to provide a method to visually explain which regions the model focuses on analyzing. Through these efforts, it will be possible to enhance the reliability of pathological image analysis and implement AI-based diagnostic assistance technologies that can be utilized more effectively in clinical settings.

[0009]

[0010] The present invention aims to provide a technology for improving reliability and consistency in pathological image analysis. To this end, a preprocessing technique is applied to effectively remove unnecessary noise within pathological images, and color differences arising from various scanners and staining methods are normalized to enable an artificial intelligence model to learn consistent data. Furthermore, the invention seeks to enhance the reliability of analysis results by optimizing a multi-instance learning technique to accurately detect specific lesions even when their proportion within an image is low, and by applying a visual explanation technique that intuitively provides the basis for the model's judgment.

[0011] Furthermore, the present invention aims to implement an artificial intelligence model that analyzes pathological images of various tumors, such as gastric cancer, breast cancer, and kidney cancer. While existing pathological analysis systems are often limited to specific types of diseases, applying the technology of the present invention enables the construction of a more general-purpose model, thereby allowing for the effective identification of various cancer types. To this end, a multi-instance learning method considering the histological characteristics of tumors within pathological tissue is applied, and Grad-CAM is utilized to visually present key areas analyzed by the artificial intelligence model. This approach allows pathologists to place greater trust in the analysis results of the artificial intelligence and can be effectively utilized in clinical settings for the diagnosis of various types of tumors, such as gastric cancer, breast cancer, and kidney cancer.

[0012] Meanwhile, the technical problems of the present invention are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below.

[0013]

[0014] A method performed by a device operated by a processor according to one embodiment may include: acquiring a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included; a noise removal preprocessing operation that removes noise from the tissue image by applying a variable threshold value according to visual feature information for each region of the tissue image; and training a neural network model trained in a multi-instance learning method by dividing the preprocessed tissue image into tiles of a predefined size and labeling the same BAG class on the tiles divided from the same tissue image.

[0015] Additionally, the noise removal preprocessing operation may include: an operation of extracting visual feature information including resolution, contrast, histogram, and background-tissue contrast for each region of the tissue image; an operation of calculating histogram uniformity, local contrast variation, and texture consistency indices for each region based on the visual feature information to calculate a noise score for each region; and an operation of determining a threshold value according to a predefined ratio to have a value proportional to the calculated noise score, and applying the Otsu algorithm based on the threshold value to remove noise from the tissue image.

[0016] Additionally, the operation of extracting the visual feature information may include: an operation of determining the structural complexity of the tissue included in the tissue image; and an operation of dividing the area into a pre-set first size, which is a relatively fine unit, for an area where the tissue structure exceeds a pre-set standard based on the structural complexity, and dividing the area into a pre-set second size, which is a relatively large unit, for an area where the tissue structure is below the pre-set standard.

[0017] Additionally, the operation of calculating the noise score may include: an operation of normalizing the brightness distribution uniformity, local contrast change, and texture consistency indicators of the area so that they can be compared within the same scale range; and an operation of calculating the noise score by combining the normalized indicators in a weighted sum manner.

[0018] In addition, the above noise score can be calculated according to the following mathematical formula 1.

[0019] [Mathematical Formula 1]

[0020]

[0021] ( is the noise score, is the uniformity of brightness distribution; a higher value indicates an area with less noise. is a change in local contrast, and a larger value indicates an area with a higher probability of noise. is texture consistency, where a higher value indicates a tissue structure with less noise, and w1, w2, and w3 are weights reflecting the importance of each metric)

[0022] Additionally, the operation of removing noise from the tissue image may include: converting the tissue image to a Lab color space; extracting an A channel representing red-green contrast from the converted image; and performing an Otsu algorithm by comparing the extracted A channel value with the threshold value calculated for each region.

[0023] Additionally, the operation of performing the Otsu algorithm may include comparing the A channel value in each region with a threshold value to identify pixels exceeding the threshold value as tissue and increasing the A channel color intensity of the corresponding pixels to emphasize tissue contrast; and identifying pixels in each region where the A channel value is below the threshold value as noise and setting the A channel value of the corresponding pixels to 0 to remove noise.

[0024] A method performed by a device operated by a processor according to one embodiment may include: acquiring a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included; a color normalization preprocessing operation that converts the color of the tissue image to a normalized color of a preset standard using a generative adversarial neural network that has been trained to convert the color of the input image to a normalized color; and a operation of training a neural network model in a multi-instance learning manner by dividing the preprocessed tissue image into tiles of a preset size and labeling the same BAG class on the tiles divided from the same tissue image.

[0025] Additionally, the above method further includes an operation of training the generative adversarial network prior to the color normalization preprocessing operation, and the operation of training the generative adversarial network may include: an operation of acquiring a standard tissue image taken from a reference scanner to correct the staining density and color difference of tissue images acquired from various scanners; and an operation of training the generative adversarial network using the standard tissue image.

[0026] Additionally, the operation of training using the standard tissue image may include: an operation of configuring a generator of a generative adversarial network to generate a color-converted image reflecting a standard color distribution from an input tissue image to learn the color information of the standard tissue image; an operation of configuring a discriminator of a generative adversarial network to compare the color-converted image generated by the generator with the standard tissue image to determine whether the color-converted image and the standard tissue image are distinguishable; and an operation of competitively training the generator and the discriminator so that the discriminator can recognize the color-converted image generated by the generator as a real image.

[0027] Additionally, the competitive learning operation may include an operation to calculate the Peak Signal-to-Noise Ratio (PSNR) between the transformed image and the original tissue image to evaluate whether the tissue image transformed by the generative adversarial network has been normalized without loss of original tissue information.

[0028] Additionally, the competitive learning operation may include an operation to calculate a Structural Similarity Index Measure (SSIM) to evaluate the structural similarity between the tissue image transformed by the generative adversarial network and the original tissue image.

[0029] Additionally, the competitive learning operation may include an operation to calculate a color difference analysis value (CIEDE2000) to evaluate whether the color of the tissue image transformed by the generative adversarial network has been consistently normalized.

[0030] Additionally, the competitive training operation may include: an operation to normalize the PSNR, SSIM, and color difference analysis values ​​so that they can be compared within the same scale range and combine them in a predefined weighted sum manner to calculate a quality score of the tissue image transformed by the generative adversarial network; an operation to generate a new transformed image by adjusting the parameters of the generator if the quality score does not meet a pre-set threshold; and an operation to repeatedly evaluate the quality score for the newly generated transformed image and competitively train the generator and the discriminator until the quality score becomes greater than or equal to the threshold.

[0031] A method performed by a device operated by a processor according to one embodiment may include: acquiring a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included; a preprocessing operation to remove noise from the tissue image and normalize it to a color of a preset standard; and designing a neural network model in which a rectified linear unit and a dropout layer are connected to a fully connected layer of a ResNet-18 model structure, dividing the preprocessed tissue image into tiles of a preset size, and labeling the same BAG class on the tiles divided from the same tissue image to train the neural network model using a multi-instance learning method.

[0032] Additionally, the above method may include an operation of applying Grad-CAM (Gradient-weighted Class Activation Mapping) to the neural network model after the above training operation in order to interpret the training results of the neural network model and verify reliability, and the operation of applying Grad-CAM may include an operation of backpropagating the gradient for the class predicted by the neural network model to calculate weights for the activation map of the last convolutional layer; an operation of generating a heatmap reflecting the importance of each BAG class by applying the calculated weights; and an operation of overlaying the generated heatmap on the original tissue image to visually display the area that the neural network model focused on when predicting the BAG class.

[0033] In addition, the above BAG class may include normal tissue, benign tumor, and malignant tumor.

[0034] Additionally, the operation of applying the Grad-CAM may include: inputting a target tissue image, which is subject to determination of whether it contains a tumor, into a neural network model that has completed training, and determining the class for each tile that divides the target tissue image as one of normal tissue, benign tumor, or malignant tumor; aggregating the number of classes determined per tile, and determining the target tissue image as malignant if the ratio of tiles classified as malignant tumors within the target tissue image is greater than or equal to a preset first threshold; determining the target tissue image as benign if the ratio of tiles classified as benign tumors within the target tissue image is greater than or equal to a preset second threshold and the ratio of tiles classified as malignant tumors is less than a third threshold; and determining the target tissue image as normal if the ratio of tiles classified as normal tissues within the target tissue image is greater than or equal to a preset fourth threshold and the ratios of tiles classified as benign and malignant tumors are less than a fifth threshold and a sixth threshold, respectively.

[0035] Additionally, the operation of training the neural network model may include: an operation of determining the boundary of the tissue included in the tissue image; an operation of removing data from the outer region of the tissue boundary; and an operation of dividing the inner region of the tissue boundary into a preset size to generate tiles.

[0036] Additionally, the operation of training the neural network model may include: an operation of determining the tissue area and background included in the tile; and an operation of training the neural network model using only tiles in which the tissue area included in the tile is 25% or more.

[0037] In addition, the operation of training the neural network model may include an operation of improving the training data by copying the tissue area included in the tile and copying it to an empty area when the tissue area included in the tile is less than 25%.

[0038] In addition, the operation of training the neural network model may include an operation of improving the training data by mirroring the tissue area included in the tile left-right or up-down within the tile when the tissue area included in the tile is 25% or more and 50% or less.

[0039]

[0040] The present invention provides a technology capable of simultaneously ensuring accuracy and reliability in the analysis of pathological images, thereby enhancing the utility of multi-AI-based medical imaging diagnosis capable of analyzing various tumors such as gastric cancer, breast cancer, and kidney cancer.

[0041] In particular, the present invention effectively removes noise from pathological images to minimize learning interference caused by unnecessary data, and maintains the consistency of model learning by normalizing color differences arising from various scanners and staining methods. Furthermore, by analyzing pathological images on a tile-by-tile basis and applying a multi-instance learning method, more precise predictions are possible, and a deep neural network structure utilizing ResNet-18 improves learning efficiency while mitigating the vanishing gradient problem.

[0042] Furthermore, the present invention can secure explainability by generating a heatmap using Grad-CAM to visually provide which areas the AI ​​model focused on during the decision-making process. This is an important factor in helping medical professionals trust and utilize the analysis results of AI, and increases the practical clinical applicability of AI-based pathological image analysis.

[0043] Accordingly, the present invention improves the level of automation in pathological diagnosis while enabling artificial intelligence models to establish themselves as reliable tools capable of collaborating with medical professionals.

[0044] Meanwhile, the effects of the present invention are not limited to those mentioned above, and other unmentioned technical effects will be clearly understood by a person skilled in the art from the description below.

[0045]

[0046] FIG. 1 is a configuration diagram of a pathological image analysis device according to one embodiment.

[0047] FIG. 2 is a flowchart showing the steps of an operation performed by a pathological image analysis device according to one embodiment.

[0048] FIG. 3 is an example of an operation to remove noise from a tissue image by adjusting the threshold of the Otsu algorithm according to one embodiment.

[0049] FIG. 4 is an example diagram of an operation for training a generative adversarial neural network according to one embodiment.

[0050] FIG. 5 is an example diagram of an operation to divide a tissue image into tiles of a predefined size according to one embodiment.

[0051] FIG. 6 is an exemplary diagram showing the structure of a neural network model according to one embodiment.

[0052] FIG. 7 is an example diagram of an operation for training a neural network model according to one embodiment.

[0053] FIG. 8 is an example diagram of the inference operation of a neural network model according to one embodiment.

[0054] FIG. 9 is a table showing the performance evaluation results of a neural network model according to one embodiment.

[0055] FIG. 10 is a graph showing the ROC curve of a neural network model according to one embodiment.

[0056]

[0057] Detailed information regarding the purpose, technical configuration, and resulting effects of the present invention will be more clearly understood through the following detailed description based on the drawings attached to the specification of the present invention. An embodiment according to the present invention will be described in detail with reference to the attached drawings.

[0058] The embodiments disclosed herein should not be interpreted or used to limit the scope of the invention. It is obvious to those skilled in the art that the description including the embodiments herein has various applications. Accordingly, any embodiments described in the detailed description of the invention are illustrative for better explaining the invention and are not intended to limit the scope of the invention to the embodiments.

[0059] The functional blocks shown in the drawings and described below are merely examples of possible implementations. In other implementations, other functional blocks may be used without departing from the spirit and scope of the detailed description. Additionally, while one or more functional blocks of the present invention are shown as individual blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

[0060] Furthermore, the expression that it includes certain components is an “open-ended” expression that merely refers to the existence of such components and should not be understood as excluding additional components.

[0061] Furthermore, when it is stated that one component is “connected” or “joined” to another component, it should be understood that while it may be directly connected or joined to that other component, there may also be other components in between.

[0062] Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present invention.

[0063] The present invention proposes a pathological image analysis device (100) that implements a technology to improve the analysis accuracy of pathological images by providing noise removal of pathological images, color normalization, lesion detection through a multi-instance learning technique, and explainability of a model using Grad-CAM.

[0064] Hereinafter, we will examine the configuration of the pathological image analysis device (100) of the present invention and the operation of each configuration.

[0065] FIG. 1 is a diagram showing the configuration of a pathological image analysis device (100) (hereinafter referred to as 'device (100)') according to one embodiment.

[0066] Referring to FIG. 1, a device (100) according to one embodiment may each include a memory (110), a processor (120), an input / output interface (130), and a communication interface (140).

[0067] The memory (110) can store data obtained from an external device or data generated by itself. The memory (110) can store instructions that can perform operations of the processor (120). For example, the memory (110) can store data such as tissue images and neural network models, which will be described later.

[0068] The processor (120) is a computational device that controls the overall operation. The processor (120) can execute instructions stored in memory (110). The operation of the device (100) according to the embodiment of the present document can be understood as an operation performed by the processor (120).

[0069] The input / output interface (130) may include a hardware interface or a software interface for inputting or outputting information.

[0070] The communication interface (140) enables the transmission and reception of information through a communication network. To this end, the communication interface (140) may include a wireless communication module or a wired communication module.

[0071] The device (100) can be implemented in various forms of devices capable of performing calculations through a processor (120) and transmitting and receiving information through a network. For example, it can be implemented in the form of a server, a computer device, a portable communication device, a smartphone, a portable multimedia device, a laptop, a tablet PC, etc., but is not limited to these examples.

[0072] FIG. 2 is a flowchart of an operation performed by a device (100) according to one embodiment. The operation of the device (100) according to the embodiment of FIG. 2 can be understood as an operation performed by a processor (120).

[0073] Each step disclosed in FIG. 2 is merely a preferred embodiment for achieving the purpose of the present invention, and some steps may be added or deleted as needed, and any one step may be included in another step. The order of each operation disclosed in FIG. 2 is arranged only for ease of understanding and is not limited to a chronological order, and the order may be changed and operated differently according to the designer's choice.

[0074] Referring to FIG. 2, in step S1010, the device (100) can acquire a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included.

[0075] A tissue image refers to a Whole Slide Image (WSI) used in digital pathology, meaning pathological image data generated by scanning microscope slides at high resolution. Tissue images encompass the entire pathological tissue, and analysis can be performed to evaluate the presence of a tumor.

[0076] A BAG class refers to a label assigned to an entire tissue image. In the Multiple Instance Learning described below, labels are assigned at the BAG level (i.e., units of the same tissue image) rather than to individual tiles segmented from the tissue image. If some tiles within a BAG contain instances corresponding to tumor tissue, all tiles belonging to that BAG may be classified as having the label "tumor." Therefore, BAG classes are applied as a collective labeling method for the entire group of tiles belonging to the tissue image, rather than as labels for individual instances.

[0077] Meanwhile, tissue images may contain various types of noise during the microscopic imaging process, and such noise can degrade accuracy during the training and inference processes of neural network models. For example, air bubbles, pen marks, background noise, and tissue folds contain unnecessary information in the original image, which can hinder the model from learning the actual pathological features of the tissue.

[0078] To solve this, in step S1020, the device (100) can perform a preprocessing operation to remove noise from the tissue image by applying a variable threshold value according to the visual feature information of each region of the tissue image.

[0079] For example, the device (100) can determine a threshold value for performing noise removal for specific regions of a tissue image and perform noise removal based on the Otsu algorithm based on different threshold values ​​for each region. Here, the Otsu algorithm is a method of automatically setting an optimal threshold value by analyzing the pixel intensity distribution (histogram) of the image, but the present invention proposes a method of applying a newly proposed threshold derivation method as described below to derive noise scores for each region of the tissue image and obtain a threshold value, and then applying the Otsu algorithm based on the threshold value obtained in this way.

[0080] To this end, the device (100) can first determine the structural complexity of the tissue included in the tissue image to distinguish the regions of the tissue image to which different thresholds are to be applied.

[0081] For example, the device (100) can determine that a specific area is a region with a complex tissue structure if the brightness distribution is wide and uneven, or if there is a large change in local contrast, or if there is low texture consistency and high fractal dimension. Additionally, the device (100) can determine that a specific area is a region with low structural complexity if the brightness distribution is narrow and uniform, if there is little change in local contrast, or if there is high texture consistency. Based on the determined structural complexity, the device (100) can divide the area into relatively fine units (e.g., divide the area into pixel sizes smaller than or equal to a pre-set first size) for areas where the complexity of the tissue structure exceeds a pre-set standard, and divide the area into relatively large units (e.g., divide the area into pixel sizes larger than or equal to a pre-set second size) for areas where the complexity of the tissue structure is smaller than or equal to a pre-set standard.

[0082] Next, the device (100) can extract visual feature information including resolution, contrast, histogram, and background-tissue contrast for each region of the tissue image, and calculate the histogram uniformity, local contrast variation, and texture consistency indices for each region of the tissue image based on the extracted visual feature information to calculate a noise score for each region.

[0083] For example, the device (100) can normalize the brightness distribution uniformity, local contrast change, and texture consistency indicators of the tissue image by region so that they can be compared within the same scale range, and combine the normalized indicators in a weighted sum manner to calculate a noise score as shown in Equation 1 below.

[0084]

[0085] ( is the noise score, is the uniformity of brightness distribution; a higher value indicates an area with less noise. is a change in local contrast, and a larger value indicates an area with a higher probability of noise. is texture consistency, where a higher value indicates a tissue structure with less noise, and w1, w2, and w3 are weights reflecting the importance of each metric)

[0086] Accordingly, the device (100) can determine a threshold value according to a predefined ratio to have a value proportional to the calculated noise score, and apply the Otsu algorithm based on the determined threshold value to remove noise from the tissue image.

[0087] FIG. 3 is an example of an operation to remove noise from a tissue image by adjusting the threshold of the Otsu algorithm according to one embodiment.

[0088] Referring to FIG. 3, the device (100) can convert a tissue image into a Lab color space. A Lab color space is a color space that separates color information into lightness (L) and two color channels (A, Green-Red / B, Blue-Yellow), and has the advantage of being able to analyze color differences more precisely than an RGB color space. By utilizing this, the A channel (red-green contrast) of the tissue image can be analyzed to more effectively distinguish the tissue from the background, and color information can be reflected more clearly when applying the Otsu algorithm.

[0089] The device (100) can extract an A channel representing red-green contrast from an image converted to a Lab color space and perform an Otsu algorithm by comparing the extracted A channel value with a threshold value calculated for each region. For example, the device (100) can compare the A channel value and the threshold value in each region to identify pixels exceeding the threshold value as tissue and increase the A channel color intensity of the corresponding pixels to emphasize tissue contrast. Additionally, the device (100) can identify pixels in each region where the A channel value is below the threshold value as noise and remove the noise by setting the A channel value of the corresponding pixels to 0.

[0090] Meanwhile, tissue images may exhibit differences in color and brightness depending on the various pathological slide scanners and staining methods, and these differences hinder consistency during the training and inference processes of neural network models. Even for the same tissue, color distribution can vary due to differences in imaging equipment, staining density, and lighting conditions, which can interfere with the model learning the intrinsic characteristics of the tissue.

[0091] To solve this problem, in step S1030, the device (100) can perform a preprocessing operation to convert the color of the tissue image to be used for training into a normalized color of a preset standard using a Generative Adversarial Network (GAN) that has been trained to convert the color of the input image into a normalized color.

[0092] The operation for training the generative adversarial network in step S1030 is as follows.

[0093] For example, the device (100) acquires a standard tissue image taken from a first scanner to serve as a reference among various scanners that take tissue images, in order to correct the staining density and color difference of tissue images acquired from various scanners, and trains a generative adversarial network using this standard tissue image. Accordingly, the device (100) can perform training by configuring a generative adversarial network with a structure as shown in FIG. 4.

[0094] FIG. 4 is an example diagram of an operation for training a generative adversarial neural network according to one embodiment.

[0095] Referring to FIG. 4, the device (100) can learn a color normalization function using a generative adversarial neural network composed of a generator and a discriminator.

[0096] To this end, the device (100) may configure a generator that generates a color-converted image reflecting the standard color distribution of the first scanner from an input tissue image to learn the color information of the standard tissue image, and may configure a discriminator that determines whether the two images are distinguishable by comparing the color-converted image generated by the generator with the original standard tissue image of the first scanner.

[0097] Accordingly, the device (100) can competitively train the generator and the discriminator using an adversarial training method. For example, the device (100) can perform training by applying Binary Cross Entropy or WGAN-based Wasserstein Loss with Gradient Penalty as the loss function of the discriminator, and the generator can be trained to generate a transformed image closer to a standard color distribution by additionally considering Mean Squared Error (MSE) or Perceptual Loss.

[0098] In this way, the device (100) can optimize the color normalization model by applying an adversarial learning algorithm that competitively trains the generator and the discriminator so that the discriminator can recognize the color-converted image generated by the generator as a real image (= to the extent that the generator's generated version is indistinguishable from the original).

[0099] In addition, after the learning of the generator and discriminator described above is completed, the device (100) can further improve the performance of the generator and discriminator by additionally applying a newly proposed algorithm as follows.

[0100] For example, the device (100) can calculate the Peak Signal-to-Noise Ratio (PSNR) between the transformed tissue image and the original tissue image to further evaluate whether the tissue image transformed by the generative adversarial network has been normalized without loss of original tissue information. Additionally, the device (100) can calculate the Structural Similarity Index Measure (SSIM) to evaluate the structural similarity between the tissue image transformed by the generative adversarial network and the original tissue image. Additionally, the device (100) can calculate a color difference analysis value (CIEDE2000) to evaluate whether the color of the tissue image transformed by the generative adversarial network has been consistently normalized.

[0101] Accordingly, the device (100) can calculate a quality score of the tissue image transformed by the generative adversarial network by combining the derived PSNR, SSIM, and color difference analysis values. For example, the device (100) can calculate a final quality score by normalizing the PSNR, SSIM, and CIEDE2000 values ​​to a pre-set range so that they can be compared within the same scale range, and then weighting each score using a pre-defined weight-based weighted sum method.

[0102] Accordingly, if the calculated quality score does not meet a preset threshold, the device (100) can adjust the parameters of the generator to generate a new transformed image and repeatedly evaluate the quality score for the newly generated transformed image to competitively train the generator and the discriminator until the quality score becomes greater than or equal to the preset threshold.

[0103] In step S1040, the device (100) can train a neural network model in a multi-instance learning manner by dividing a preprocessed tissue image into tiles of a predefined size and labeling the tiles divided from the same tissue image with the same BAG class.

[0104] FIG. 5 is an example diagram of an operation to divide a tissue image into tiles of a predefined size according to one embodiment.

[0105] Referring to FIG. 5, the device (100) can identify the boundary of the tissue included in the tissue image and remove data from the outer area of ​​the tissue boundary to divide the inner area of ​​the tissue boundary into pre-set sizes and generate tiles. For example, the device (100) can extract the boundary of the foreground rather than the background in the pathology image through the GrabCut algorithm and remove data from the outer area of ​​the boundary to make it null. Accordingly, the device (100) can generate tiles such that only the area corresponding to the tissue (ROI) is included in the tile.

[0106] Next, the device (100) can divide the area corresponding to the tissue into tiles of a predefined pixel size (e.g., 224 x 224) and label the same BAG class on the divided tiles in the same tissue image.

[0107] At this time, among the divided tiles, the tile divided from the border portion of the tissue may include a portion of the tissue's outer area (e.g., null area). In this case, the device (100) uses only tiles in which the tissue area included in the tile is 25% or more to train the neural network model described later, and if the tissue area included in the tile is less than 25%, the tissue area included in the tile is copied to an empty area to improve the training data, and then it can be used to train the neural network model described later.

[0108] In addition, even if the tissue area included in the tile is 25% or more, if the tissue area included in the tile is 25% or more and 50% or less, the device (100) can improve the training data by mirroring the tissue area included in the tile left-right or up-down within the tile, and then use it for training the neural network model to be described later.

[0109] Subsequently, the device (100) can label the divided tiles with a BAG class. A BAG class refers to a label assigned to an entire tissue image, and the BAG class may include normal tissue, benign tumor, and malignant tumor. By using the BAG class, if a specific tissue image contains instances corresponding to tumor tissue, all tiles belonging to that BAG may be classified with a label corresponding to tumor. That is, all tiles divided from a specific tissue image have the same BAG class as the BAG class of step S1010 labeled on the tissue image.

[0110] FIG. 6 is an exemplary diagram showing the structure of a neural network model according to one embodiment.

[0111] Referring to FIG. 6, the device (100) can configure a neural network model by configuring the basic structure of a ResNet-18 model, and by connecting a Rectified Linear Unit (ReLU) and a Dropout layer to the Fully Connected Layer to configure a neural network model of S1040 steps.

[0112] The ResNet-18 model utilizes a block-based architecture containing residual connections to address the vanishing gradient problem that can occur in deep neural networks. Preprocessed tissue images, consisting of input tiles, pass through a 7x7 convolution layer to extract initial feature maps, followed by a process of progressively learning deeper features through multiple residual blocks (Block B). Additionally, each Block B consists of two 3x3 convolution layers, and downsampling can be performed in the first layer by applying a stride of 2. Subsequently, meaningful structural patterns can be extracted from the tissue images by progressively expanding features starting from 64 channels to 128, 256, and 512 channels. In the final step, a pooling layer is applied to generate unique feature vectors, which are then passed to a fully connected layer to ultimately classify the BAG class of the tissue image.

[0113] In particular, the neural network model of the present invention is configured to provide nonlinearity by applying a rectified linear unit activation function in the fully connected layer and to prevent overfitting by adding a dropout layer, thereby enabling the neural network model to stably learn various pathological tissue images and improve the generalization performance of the model.

[0114] FIG. 7 is an example diagram of an operation for training a neural network model according to one embodiment.

[0115] Referring to FIG. 7, the device (100) trains a neural network model based on tiles segmented from a preprocessed tissue image. At this time, a BAG class is mapped to each tile, and the BAG class is applied as a collective labeling method for the entire group of tiles belonging to the same tissue image. Subsequently, the device (100) performs a Tile Feature Training process, during which pathological features of each tile are extracted. The extracted tile features are trained in a ResNet-18-based neural network model, thereby constructing a model that classifies whether an individual tile is normal tissue or tumor tissue. Accordingly, the neural network model receives each tile as an independent instance, analyzes the features of the tile using a multi-instance learning method, and performs classification at the tile level through the trained model.

[0116] FIG. 8 is an example diagram of the inference operation of a neural network model according to one embodiment.

[0117] Referring to FIG. 8, the device (100) inputs a target tissue image, which is subject to determination of whether it contains a tumor, into a trained neural network model, and can determine the class for each tile of the target tissue image as one of normal tissue, benign tumor, or malignant tumor.

[0118] Subsequently, the device (100) can aggregate the tile-specific classes identified by the neural network model to analyze the proportion of each class within the target tissue image. In this process, if the proportion of tiles classified as malignant tumors among all tiles is greater than or equal to a preset first threshold, the target tissue image can be determined to be a malignant tumor. Additionally, if the proportion of tiles classified as benign tumors among all tiles is greater than or equal to a preset second threshold, and the proportion of tiles classified as malignant tumors is less than a preset third threshold, the target tissue image can be determined to be benign. Furthermore, if the proportion of tiles classified as normal tissues among all tiles is greater than or equal to a preset fourth threshold, and the proportions of tiles classified as benign and malignant tumors are less than the preset fifth and sixth thresholds, respectively, the target tissue image can be determined to be normal. Through this, the device (100) can synthesize the tile-unit prediction results analyzed by the neural network model to determine whether a tumor is included with greater reliability at the tissue image unit level.

[0119] In step S1050, the device (100) may apply Grad-CAM (Gradient-weighted Class Activation Mapping) to the neural network model to interpret the learning results of the neural network model and verify reliability.

[0120] To this end, the device (100) can backpropagate the gradient for the class predicted by the neural network model to calculate weights for the activation map of the last convolutional layer and generate a heatmap reflecting the importance of each BAG class based on the weights. The device (100) can overlay the generated heatmap onto the original target tissue image to visually provide which areas the neural network model focused on analyzing when predicting a specific class.

[0121] Through this, the device (100) can visually analyze the image area that the neural network model focused on during the process of predicting a specific class (e.g., normal tissue, benign tumor, malignant tumor), and based on this, can clearly verify the basis for the model's decision. In particular, regarding predictions made at the BAG level, it can analyze how much an individual tile contributed to the final decision, and can be used to determine whether a specific tile played an important role in the model's judgment or became the cause of an error.

[0122] Additionally, the device (100) can quantitatively analyze the features of the heatmap generated through Grad-CAM to evaluate the reliability of the class predicted by the model. For example, if the heatmap clearly highlights the major lesion area of ​​the tissue for a specific class (e.g., malignant tumor), the reliability of the prediction can be judged to be high. On the other hand, if the heatmap focuses on background areas unrelated to the specific structure of the tissue, the prediction can be judged to be uncertain or that the model has not learned an appropriate pattern.

[0123] FIG. 9 is a table showing the results of a performance evaluation of a neural network model according to one embodiment. Specifically, FIG. 9 shows the results of evaluating the performance indicators of the neural network model, Precision, Sensitivity, F1-score, and AUC for normal tissue, a benign tumor, and a malignant tumor.

[0124] Referring to Fig. 9, the neural network model of the present invention showed overall high values ​​for the classification performance of normal tissue and malignant tumor, and in particular, the AUC values ​​were 0.987 and 0.957, respectively, confirming that the model maintains stable classification performance.

[0125] FIG. 10 is a graph showing the ROC curve of a neural network model according to one embodiment. FIG. 10 shows the AUC values ​​for each class (normal tissue, benign tumor, malignant tumor).

[0126] Referring to Figure 10, the AUC value of normal tissue was the highest at 0.987 (95% CI: 0.986-0.987), the AUC value of malignant tumor was 0.957 (95% CI: 0.957-0.958), and the AUC value of benign tumor was 0.949 (95% CI: 0.949-0.950). These results indicate that the neural network model of the present invention maintains high classification performance overall, and in particular, the distinction between normal tissue and malignant tumor is excellent.

[0127] According to the above-described embodiment, the present invention provides a technology capable of simultaneously ensuring accuracy and reliability in the analysis of pathological images, thereby having the effect of increasing the utility of AI-based medical imaging diagnosis.

[0128] In particular, the present invention effectively removes noise from pathological images to minimize learning interference caused by unnecessary data, and maintains the consistency of model learning by normalizing color differences arising from various scanners and staining methods. Furthermore, by analyzing pathological images on a tile-by-tile basis and applying a multi-instance learning method, more precise predictions are possible, and a deep neural network structure utilizing ResNet-18 improves learning efficiency while mitigating the vanishing gradient problem.

[0129] Furthermore, the present invention can secure explainability by generating a heatmap using Grad-CAM to visually provide which areas the AI ​​model focused on during the decision-making process. This is an important factor in helping medical professionals trust and utilize the analysis results of AI, and increases the practical clinical applicability of AI-based pathological image analysis.

[0130] Accordingly, the present invention improves the level of automation in pathological diagnosis while enabling artificial intelligence models to establish themselves as reliable tools capable of collaborating with medical professionals.

[0131] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more items unless the relevant context clearly indicates otherwise.

[0132] In this document, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B or C,” “at least one of A, B and C,” and “at least one of A, B, or C” may include all possible combinations of items listed together in the corresponding phrase. Terms such as “1,” “2,” or “first” or “second” may be used simply to distinguish a component from another component and do not limit the components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as “coupled” or “connected” to another (e.g., 2nd) component, with or without the terms “functionally” or “communicationly,” it means that the component may be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0133] As used in this document, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be a component formed integrally, or a minimum unit of a component or part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0134] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) that can be read by a device (e.g., an electronic device). The storage medium may include random access memory (RAM), a memory buffer, a hard drive, a database, erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), read-only memory (ROM), and / or the like.

[0135] Additionally, the processor of the embodiments of this document may call at least one instruction among one or more instructions stored from a storage medium and execute it. This enables the device to operate to perform at least one function according to at least one called instruction. Such one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The processor may be a general-purpose processor, a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), and / or the like.

[0136] A device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily.

[0137] Methods according to the various embodiments disclosed in this document may be provided as part of a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as a manufacturer's server, an application store's server, or the server's memory.

[0138] According to various embodiments, each component (e.g., module or program) of the described components may include a singular or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the components of the multiple components in the same or similar manner as they were performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically; one or more of the operations may be executed in a different order; omitted; or one or more other operations may be added.

Claims

1. A method performed by a device operated by a processor, The operation of acquiring a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included; A noise removal preprocessing operation for removing noise from the tissue image by applying a variable threshold value according to the visual feature information of each region of the tissue image; and The operation of training a neural network model using a multi-instance learning method by dividing a preprocessed tissue image into tiles of a predefined size and labeling the tiles divided from the same tissue image with the same BAG class, method.

2. In Paragraph 1, The above noise removal preprocessing operation is An operation to extract visual feature information including resolution, contrast, histogram, and background-tissue contrast for each region of the above tissue image; An operation to calculate the Histogram Uniformity, Local Contrast Variation, and Texture Consistency indices for each region based on the above visual feature information, and to calculate the noise score for each region; and The method includes determining a threshold value according to a predefined ratio to have a value proportional to the calculated noise score, and applying an Otsu algorithm based on the threshold value to remove noise from the tissue image. method.

3. In Paragraph 2, The operation of extracting the above visual feature information is An operation to determine the structural complexity of an organization included in an organization image; and Based on the above structural complexity, the operation includes dividing the area into a pre-set first size, which is a relatively fine unit for the area where the organizational structure exceeds a pre-set standard, and dividing the area into a pre-set second size, which is a relatively large unit for the area where the organizational structure is below a pre-set standard. method.

4. In Paragraph 3, The operation of calculating the above noise score is An operation to normalize the brightness distribution uniformity, local contrast change, and texture consistency indicators of the above region so that they can be compared within the same scale range, respectively; and The operation of calculating a noise score by combining the above-mentioned normalized indicators in a weighted sum manner, method.

5. In Paragraph 4, The above noise score is calculated according to the following mathematical formula 1, [Mathematical Formula 1] ( is the noise score, is the uniformity of brightness distribution; a higher value indicates an area with less noise. is a change in local contrast, and a larger value indicates an area with a higher probability of noise. is texture consistency, where a higher value indicates a tissue structure with less noise, and w1, w2, and w3 are weights reflecting the importance of each metric) method.

6. In Paragraph 5, The operation of removing noise from the above tissue image is The operation of converting the above tissue image into the Lab color space; The operation of extracting an A channel representing red-green contrast from the above-described converted image; and The operation of performing the Otsu algorithm by comparing the extracted A channel value with the threshold value calculated for each region, method.

7. In Paragraph 6, The operation of performing the above Otsu algorithm is An action of comparing A channel values ​​in each region with a threshold, identifying pixels exceeding the threshold as tissue, and increasing the A channel color intensity of the corresponding pixels to emphasize tissue contrast; and Includes an operation to identify pixels in each region where the A channel value is below a threshold as noise, and to remove the noise by setting the A channel value of the corresponding pixel to 0. method.

8. A method performed by a device operated by a processor, The operation of acquiring a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included; A color normalization preprocessing operation that converts the color of the tissue image into a normalized color according to a preset standard using a generative adversarial network that has been trained to convert the color of the input image into a normalized color; and A method comprising the operation of training a neural network model using a multi-instance learning method by dividing a preprocessed tissue image into tiles of a predefined size and labeling the tiles divided from the same tissue image with the same BAG class. method.

9. In Paragraph 8, The above method Prior to the color normalization preprocessing operation above, the operation of training the generative adversarial network above is further included, and The operation of training the above generative adversarial neural network is, The operation of acquiring a standard tissue image taken from a reference scanner to correct for differences in staining density and color of tissue images acquired from various scanners; and The operation of training the above generative adversarial network using the above standard organization image, method.

10. In Paragraph 9, The operation of training using the above standard tissue image is An operation to configure a generator of a generative adversarial network that generates a color-converted image reflecting a standard color distribution from an input tissue image to learn the color information of the above standard tissue image; Operation of configuring a discriminator of a generative adversarial network that compares a color-converted image generated by the generator with a standard tissue image to determine whether the color-converted image and the standard tissue image are distinguishable; and A method comprising an operation of competitively training the generator and the discriminator so that the discriminator can recognize the color-converted image generated by the generator as a real image. method.

11. In Paragraph 10, The above competitive learning action is To evaluate whether the tissue image transformed by the above generative adversarial network has been normalized without loss of original tissue information, the operation of calculating the Peak Signal-to-Noise Ratio (PSNR) between the transformed image and the original tissue image is included. method.

12. In Paragraph 11, The above competitive learning action is A method including an operation to calculate a Structural Similarity Index Measure (SSIM) to evaluate the structural similarity between the tissue image transformed by the above-mentioned generative adversarial network and the original tissue image, method.

13. In Paragraph 12, The above competitive learning action is A method including an operation to calculate a color difference analysis value (CIEDE2000) to evaluate whether the color of the tissue image transformed by the above generative adversarial network has been consistently normalized, method.

14. In Paragraph 13, The above competitive learning action is The operation of normalizing the above PSNR, SSIM, and color difference analysis values ​​so that they can be compared within the same scale range, and combining them in a predefined weighted sum method to calculate a quality score of the tissue image transformed by the generative adversarial network; If the above quality score does not meet a preset threshold, the operation of adjusting the parameters of the generator to generate a new converted image; and The operation of repeatedly evaluating the quality score of a newly generated transformed image and competitively training the generator and the discriminator until the quality score exceeds a threshold value. method.

15. A method performed by a device operated by a processor, The operation of acquiring a tissue image labeled with a BAG class that specifies whether an instance corresponding to tumor tissue is included; A preprocessing operation to remove noise from the above tissue image and normalize it to a preset standard color; and Designing a neural network model in which a rectified linear unit and a dropout layer are connected to the fully connected layer of the ResNet-18 model structure, and training the neural network model using a multi-instance learning method by dividing a preprocessed tissue image into tiles of a predefined size and labeling the tiles divided from the same tissue image with the same BAG class. method.

16. In Paragraph 15, The above method After the above-mentioned learning operation, To interpret the learning results of the above neural network model and verify its reliability, the method includes applying Grad-CAM (Gradient-weighted Class Activation Mapping) to the above neural network model, and The operation of applying the above Grad-CAM is The operation of backpropagating the gradient for the class predicted by the above neural network model to calculate the weight for the activation map of the last convolutional layer; The operation of generating a heatmap reflecting importance by BAG class by applying calculated weights; and A method comprising overlaying a generated heatmap onto an original tissue image to visually display the area that the neural network model primarily referenced when predicting the BAG class. method.

17. In Paragraph 16, The above BAG class is Includes normal tissue, benign tumors, and malignant tumors, method.

18. In Paragraph 17, The operation of applying the above Grad-CAM is An operation of inputting a target tissue image, to be determined for the presence of a tumor, into a trained neural network model, and determining the class of each tile segmented from the target tissue image as one of normal tissue, benign tumor, or malignant tumor; An operation to aggregate the number of classes determined per tile, and to determine the target tissue image as malignant if the ratio of tiles classified as malignant tumors within the target tissue image is greater than or equal to a preset first threshold; An operation of determining the target tissue image as benign when the ratio of tiles classified as benign tumors within the target tissue image is greater than or equal to a preset second threshold and the ratio of tiles classified as malignant tumors is less than a third threshold; and The method includes an operation of determining the target tissue image as normal when the ratio of tiles classified as normal tissue within the target tissue image is greater than or equal to a preset fourth threshold, and the ratios of tiles classified as benign and malignant tumors are less than the fifth threshold and the sixth threshold, respectively. method.

19. In Paragraph 15, The operation of training the above neural network model is An operation to determine the boundaries of the tissue included in the above tissue image; The operation of removing data from the outer area of ​​the boundary of the above organization; and A method comprising the operation of generating tiles by dividing the inner area of ​​the boundary of the above organization into pre-set sizes. method.

20. In Paragraph 19, The operation of training the above neural network model is An operation to determine the tissue area and background included in the above tile; and The operation of performing training of the neural network model using only tiles in which the tissue area included in the tile is 25% or more, method.

21. In Paragraph 20, The operation of training the above neural network model is If the tissue area included within the above tile is less than 25%, the operation of copying the tissue area included within the above tile and copying it to an empty area to improve the training data is included. method.

22. In Paragraph 21, The operation of training the above neural network model is If the tissue area included within the tile is 25% or more and 50% or less, the operation of improving training data by mirroring the tissue area included within the tile left-right or up-down within the tile is included. method.

Images