Colorectal cancer grade prediction and segmentation using deep learning

Abstract

The invention relates to a method for examining colorectal cancer tumors using deep learning method.

Description

[0001] COLORECTAL CANCER GRADE PREDICTION AND SEGMENTATION USING DEEP LEARNING

[0002] Technical Field

[0003] The invention relates to a method for predicting pixel locations and tumor grades of colorectal cancer tumors on digital histopathology images using a deep learning model.

[0004] Prior Art

[0005] In cancer, early detection and active monitoring of the disease grade are critical for the treatment phase. Patent document no. US11170897B2 relates to a method for providing a tissue biopsy using digital images. In the method mentioned in the US11170897B2 patent document, the whole slide image is subjected to image enlargement and the enlarged images are subdivided into patches. These patches feed a deep learning model to detect tumor regions. The accuracy level of the measurement can be improved as the invention uses patches and is unable to perform an analysis for every pixel in the image.

[0006] The invention with application number US2023267606A1 relates to a deep learning method for cancer detection based on a sample image. This deep learning method is designed in two layers. Said deep learning method feeds the deep learning method by subdividing the images into patches, similar to the invention numbered US11170897B2.

[0007] Application numbered US11954593B2 relates to a cancer analysis prediction method using an end-to-end learning system. In this method, although the slide image is segmented into tiles, an abnormality score is obtained by classifying between the tiles. This abnormality score can be used to diagnose the tumor.

[0008] All the problems mentioned above require innovation in the technical field. Summary and Objects of the Invention

[0009] The main object of the invention relates to a method for determining the grade of colorectal cancer. Another object of the invention is to reduce potential information loss by using images at the pixel level in learning algorithms. The invention increases the level of accuracy compared to other alternatives as it uses visual information in images down to the pixel level, rather than leaving it at the segment level.

[0010] Colorectal cancer, due to its diverse pathophysiology, poses particular challenges in identifying and selecting the appropriate treatment. Biopsies taken with invasive methods can be analyzed to determine the possible presence and progression of cancer.

[0011] More than 90% of colorectal carcinomas are adenocarcinomas originating from epithelial cells of the colorectal mucosa, and as a result, CRC, an adenocarcinoma, is classified into 3 main groups: grade 1, 2 and 3. Conventional adenocarcinoma is characterized by glandular formation, which forms the basis of histological tumor grading.

[0012] The histopathologic grade may be interpreted differently by different observers, which prolongs the examination process and is a disadvantage in cases requiring urgent examination.

[0013] Another object of the invention is to provide a method capable of processing histopathological images.

[0014] Drawings and Descriptions Defining the Invention

[0015] The descriptions of the figures used for better understanding of the invention are provided below.

[0016] Fig- 1- Different deep learning models and tumor examination in different biopsy samples. Detailed Description of the Invention

[0017] The invention relates to pixel-level segmentation and grading of tumor regions on an image acquired with a digital scanner using a deep learning model.

[0018] Deep learning methods for rapidly examining histopathologic conditions are enabling progress in the field of digital pathology. Deep learning methods have enabled a breakthrough in the field of digital pathology.

[0019] The dataset used to develop the deep learning model used in the invention is processed by fixing the samples obtained by surgical removal using formalin and with the help of a machine. The samples from the tissue sample are prepared as paraffin blocks with a thickness of 4 microns. Prepared samples are stained with hematoxylin and eosin compounds in a manner similar to classical histology practices. The stained samples are then digitized with a digital scanner and used to train the deep learning model.

[0020] The dataset for Colorectal Cancer Tumor Grade Segmentation used contains a total of 103 whole slide images. Ground-truth annotations for these images were created by two independent pathologists. Dataset comprises pixel based segmentation masks belonging to annotations, 1st Grade (Grade- 1), 2nd Grade, (Grade-2) and 3rd Grade (Grade-3) tumor classes as well as "Normal-mucosa" for normal class and "Others" for other regions excluding these. In training, the prepared dataset is divided into three parts, 70% for training, 15% for validation, and 15% for testing. To determine the baseline results for this dataset, leading convolutional neural network (CNN) and transformer-based models were trained and evaluated.

[0021] Fig. 1 shows the results of the method on different examples and different deep learning systems.

[0022] The results show that the transformer-based model, named SwinT, achieves an average dice score of 63%, outperforming other transformer-based models and all CNN-based models, in line with the recent success of transformer-based models in the field of computer vision.

[0023] The first step of this method is to obtain a digital image of the tissue sample using a digital scanner or similar device and a hematoxylin and eosin stained sample. This image is magnified 40 times to obtain an image of approximately one hundred thousand pixels by one hundred thousand pixels. The digital image is then reduced in size and divided into segments. These segments feed a deep artificial neural network, enabling the detection of tumors and their grades at the pixel level within the image. The results obtained from each segment are then combined to obtain the segmentation and grading result for the whole image. The invention preferably shows the location of the tumors using a heat map.

[0024] In digital pathology, challenges arise in the WSI processing process at gigapixel resolutions. In the invention, a solution to this problem was found by using a sub-sampling method. This subsampling rate can be calculated as the ratio of the width and height into which the image is divided. The deep learning method is defined as a hyperparameter selected from values of 20, 40, 60 and 80. In addition, ground-truth scaling is also scaled by the same deep learning network.

[0025] For the selection of parameters, DeepLabv3+, UNet, SwinTransformer, SegFormer and ConvNext techniques are optionally used.

[0026] For the optimization of the Deeplabv3+ technique, the Stochastic Gradient Descent (SGD) technique with 0.9 momentum and 0.0001 weight reduction was used. A mathematical formula called weighted cross entropy loss was used to balance the difference between the data in the categories. Said formula can be explained as follows. w represents the distribution of data in categories, i represents the index of different classes, t represents the minimum accuracy value, p represents the probabilities, and C represents the number of categories. The formula for finding the data distribution can be defined as follows. nj represents the number of pixels in the jth image and ny represents the number of pixels in the jth image of category i. The UNet technique uses an SGD optimization with a momentum value of 0.9. Loss in process is defined for each category in a non-specific way as follows. p represents the predicted probability and g represents the accuracy class.

[0027] In the SwinTransformer model, 224x224 sized images are selected using the UperNet framework. When this method is used, ADAM optimization process is used with a reduction in momentum of 0.9 and density of 10'4.

[0028] Similar to the Swin transformer method, the SegF ormer and ConvNext parameter selection models use the ADAM optimization process

[0029] In a preferred embodiment of the invention, a scanner with a resolution of 0.25 pm / pixel is used.

[0030] Said deep learning model can preferably be a CNN or a transformer model.

[0031] Different metrics can also be used to measure the performance of a deep learning model used in the invention. These metrics evaluate images by categorizing them into 5 different classes.

[0032] The formulas used to calculate these metrics can be seen below.

[0033] TPC

[0034] Recallc=

[0035] TPC+ FNC

[0036] 2TR

[0037] DSCc= - - - c 2TPC+ FPC+ FNC TPC, FPCand FNCrepresents the values of true positive, false positive, and false negative numbers of pixel numbers respectively. The average values of the metric values can be seen below.

[0038] In a preferred embodiment of the invention, the model can be improved thanks to the self- attention mechanism inherent in transformer models.

Claims

CLAIMS1. A deep learning method for detecting the tumor grade of cells in colorectal cancer, characterized by:• obtaining a digital image of the tumor tissue sample,• applying a magnification factor to said digital image,• reducing the size of the digital image,• segmenting the digital image at the pixel level,• feeding segments into a deep neural network model,• analyzing, by the deep neural network model, tumors and grades thereof at the pixel level,• combining the results obtained for said tumors and grades thereof to obtain a segmentation and grading result, and obtaining a segmentation and grading result for the whole image.

2. A method according to claim 1, characterized in that when the images are merged, the regions of the tumors are indicated by a heat map.

3. A method according to claim 1, characterized in that the DeepLabv3+ model is used in parameter selection.

4. A method according to claim 3, characterized in that Weighted Cross Entropy Loss with the following formula is used for mitigating data imbalance between categories:wherein w represents the distribution of data in categories, i represents the index of different classes, t represents the minimum accuracy value, p represents the probabilities, and C represents the number of categories.

5. A method according to claim 1, characterized in that the UNet, SegF ormer, SwinTransformer, or ConvNext model is used in parameter selection.

6. The method according to claim 1, characterized in that the accuracy of the method is verified byTPCPrecision,, = - c TPC+ FPC2TPCDSCc= - - - c 2TPC+ FPC+ FNCformulas.

7. A method according to claim 1, characterized in that said deep artificial neural network model is trained with a dataset containing pixel-based segmentation masks belonging to the annotations 1st Grade, 2nd Grade, and 3rd Grade tumor classes, as well as “Normalmucosa” for the normal class and “Others” for other regions.

8. A method according to claim 7, characterized in that said deep learning method is aCNN method.

9. A method according to claim 1, characterized in that said deep learning method is a transformer method.

Images