Method for diagnosing ocular disease from fundus image and apparatus therefor

Abstract

The present invention relates to a method for diagnosing an ocular disease from a fundus image. The method for diagnosing an ocular disease from a fundus image, according to an embodiment of the present invention, comprises: a step A of extracting a grayscale map from a static field of view report by using template matching; a step B of generating a pair of images by mapping a patient's fundus image and the grayscale map, the patient's fundus image having the same examination date and the same eye position; and a step C of training a visual field prediction model by using the generated pair of images.

Description

A method for diagnosing eye diseases from fundus images and the apparatus thereof

[0001] The present invention relates to a method for diagnosing eye diseases from fundus images.

[0002] More specifically, this relates to a method for predicting the visual field from fundus images. It also relates to a method for classifying diabetic retinopathy from fundus images and lesion maps.

[0003]

[0004] Glaucoma can occur when the optic nerve is damaged due to elevated intraocular pressure. Glaucoma causes visual field damage, and if left untreated, it can lead to blindness. The presence of glaucoma can be diagnosed by comprehensively reviewing methods such as visual acuity testing, intraocular pressure measurement, evaluation of optic nerve damage, and visual field measurement.

[0005] However, glaucoma testing methods take more than an hour, and the results may become inaccurate over time. Therefore, there is a need for methods to reduce the testing time and increase accuracy.

[0006] Meanwhile, diabetic retinopathy is a condition in which vision declines due to impaired microcirculation in the retina caused by peripheral circulation disruptions resulting from diabetes. Diabetic retinopathy can lead to vision loss as it progresses slowly without distinct symptoms. Therefore, there is a need for a method that can automatically diagnose diabetic retinopathy and reduce both time and costs.

[0007]

[0008] The objective of the present invention is to solve the above-mentioned problems by providing a method for diagnosing eye diseases from fundus images.

[0009] More specifically, it provides a method for predicting a visual field from fundus images and a method for classifying diabetic retinopathy from fundus images and lesion maps.

[0010]

[0011] To achieve the above objectives, a method for diagnosing an ocular disease from a fundus image according to an embodiment of the present invention comprises: a step A of extracting a grayscale map from a static visual field report using template matching; a step B of generating pairs of images by mapping a fundus image of a patient with the same examination date and eye position to a grayscale map; and a step C of training a visual field prediction model using the generated pairs of images.

[0012] In one embodiment, in step A, static field of view reports are grouped into circular, elliptical, and teardrop shapes according to the shape of the grayscale map.

[0013] In one embodiment, in step A, the extracted grayscale map is saved as an image file.

[0014] In one embodiment, in step A, a template suitable for a group of shapes of a grayscale map is generated, and a matching is performed to extract a detected area.

[0015] In one embodiment, in step C, the visual field prediction model is Pix2Pix or CycleGAN.

[0016] In one embodiment, in step C, the size of the image input to the visual field prediction model is uniform.

[0017] In one embodiment, in step C, the visual field prediction model includes a generator based on UNet and a discriminator based on PatchGAN.

[0018] In one embodiment, the visual field prediction model further includes step D, in which a grayscale map is generated from a fundus image.

[0019] In one embodiment, the method further includes step E, which involves comparing the output grayscale map with the grayscale map designated as a target to update the parameters of the visual field prediction model.

[0020] An apparatus for diagnosing an eye disease from a fundus image according to one embodiment of the present invention includes a visual field prediction model, extracts a grayscale map from a static visual field report using template matching, maps the fundus image of a patient with the same examination date and eye position to the grayscale map to generate pairs of images, trains the visual field prediction model using the generated pairs of images, the visual field prediction model generates a grayscale map from the fundus image, and compares the output grayscale map with a grayscale map designated as a target to update the parameters of the visual field prediction model.

[0021]

[0022] By implementing the present invention, a grayscale map can be generated from a fundus image in a short time, thereby shortening the time required for glaucoma examination. Furthermore, by analyzing and comparing pixel-level errors, structural similarities, and quality between images based on performance indicators, it can be made useful for diagnosing the presence of glaucoma.

[0023] The grayscale map generated by implementing the present invention can be widely used in ocular diseases, including glaucoma, and in the medical field in general.

[0024] In addition, by integrating information from fundus images and lesion maps to perform cross-attention, the accuracy of diagnosing diabetic retinopathy can be improved.

[0025]

[0026] FIG. 1 shows the configuration of a device for diagnosing eye diseases from a fundus image according to one embodiment of the present invention.

[0027] FIG. 2 illustrates a process of converting a fundus image into a grayscale map in one embodiment of the present invention.

[0028] Figure 3 is a graph showing the performance of a visual field prediction model in one embodiment of the present invention.

[0029] Figure 4 shows a fundus image and a grayscale map in one embodiment of the present invention.

[0030] FIG. 5 illustrates a process of converting a fundus image into a grayscale map in one embodiment of the present invention.

[0031] Figure 6 is a graph showing the performance of a visual field prediction model in one embodiment of the present invention.

[0032] FIG. 7 shows a fundus image and a grayscale map in one embodiment of the present invention.

[0033] FIGS. 8A and FIGS. 8B are diagrams showing the configuration of a diabetic retinopathy classification model in one embodiment of the present invention.

[0034] Figure 9 is a graph showing the performance of a diabetic retinopathy prediction model in one embodiment of the present invention.

[0035]

[0036] The present invention may be implemented with various modifications without departing from the spirit, and may have one or more embodiments. Furthermore, the embodiments described in the “specific details for implementing the invention” and “drawings,” etc., in the present invention are examples for specifically explaining the present invention and do not limit or restrict the scope of the rights of the present invention.

[0037] Accordingly, anything that a person skilled in the art to which the present invention pertains can easily deduce from the “specific details for carrying out the invention” and “drawings,” etc., of the present invention may be interpreted as falling within the scope of the present invention.

[0038] In addition, the size and shape of each component shown in the drawings may be exaggerated for the purpose of explaining the embodiments and do not limit the actual size and shape of the invention.

[0039] Unless specifically defined otherwise in the specification of the present invention, terms used therein may have the same meaning as generally understood by those skilled in the art to which the present invention pertains.

[0040] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0041]

[0042] FIG. 1 shows the configuration of a device for diagnosing eye diseases from a fundus image according to one embodiment of the present invention.

[0043] A device (100, hereinafter “diagnostic device”) for diagnosing an eye disease from a fundus image according to one embodiment of the present invention may use a fundus image and a grayscale map obtained in advance. The fundus image may provide structural information regarding the retina and the optic nerve. The grayscale map may visualize functional information of the static visual field system in an intuitive pattern.

[0044] The diagnostic device (100) can input a fundus image (I) into a visual field prediction model (10) to obtain a grayscale map (M). The diagnostic device (100) can also input the fundus image (I) into a diabetic retinopathy classification model (20) to obtain probability information regarding the stages of diabetic retinopathy.

[0045] The dataset for training the visual field prediction model (10) and the diabetic retinopathy classification model (20) may be collected from a medical institution and may have had the patient's personal information masked to prevent exposure. The dataset may include multiple fundus images and multiple test results per patient. For example, it may include two fundus images and six test results per patient (three test results for each eye). Also, the time interval between the multiple test results may vary from patient to patient.

[0046] The diagnostic device (100) may be one or more computing devices. Alternatively, it may be a device that includes a processor, a memory device, and an input / output device, and stores and executes instructions that can be read by a computer. The components included in the diagnostic device (100) may be hardware devices, instructions that can be read by a computer, or processes generated when executing such instructions.

[0047] FIG. 2 illustrates a process of converting a fundus image into a grayscale map in one embodiment of the present invention.

[0048] First, the diagnostic device (100) can extract a grayscale map from static perimetry reports using template matching. (S1010)

[0049] Static viewfinder reports can be classified into multiple types, for example, 13 types. Depending on each type, the location, shape, and size of the grayscale map may differ.

[0050] The diagnostic device (100) can group static field of view reports having similar grayscale maps. At this time, the types of grayscale maps can be classified into groups such as circle, oval, and distorted circle (e.g., water drop). The water drop group may account for 10% of the total grayscale maps, but may be excluded because there may be a large difference in shape from the circle and oval.

[0051] The diagnostic device (100) can identify a grayscale map in a static field of view report using template matching and then save it as an image file. It can generate a template suitable for a group of shapes of the grayscale map and perform matching to extract the detected area.

[0052] The diagnostic device (100) can save the extracted grayscale map to individual patient folders and then complete the extraction process through a review and modification process.

[0053] Next, the diagnostic device (100) can generate pairs of images by mapping a grayscale map to a fundus image of a patient whose examination date and eye position are the same. (S1020)

[0054] A visual field prediction model (10) can be trained using the generated image pairs. (S1030)

[0055] The visual field prediction model (10) is a model that transforms images, and may be, for example, Pix2Pix or CycleGAN. Alternatively, the visual field prediction model (10) may be a Conditional Generative Adversarial Network (cGAN) optimized for transforming paired images.

[0056] The size of the image input to the visual field prediction model (10) may be uniform. For example, it may be 256 pixels wide and 256 pixels high, and the diagnostic device (100) may adjust the size of the image. The image file name may include the position of the eye and the date of examination. The position of the eye may be distinguished as the left eye (Oculus Sinister, OS) and the right eye (Oculus Dexter, OD).

[0057] The visual field prediction model (10) may include a generator based on UNet and a discriminator based on PatchGAN. The generator can generate a grayscale map by determining the features and structure of the fundus image. The discriminator can evaluate the generated grayscale map and feed the evaluation result back to the generator.

[0058] After the visual field prediction model (10) is trained, when a new fundus image is input into the visual field prediction model (10), the visual field prediction model (10) can perform calculations and output a grayscale map corresponding to the input fundus image. (S1040)

[0059] By comparing the output grayscale map with the actual grayscale map designated as the target, the visual field prediction model (10) can be evaluated and its parameters updated to retrain it. (S1050)

[0060] The inference and update steps of the visual field prediction model (10) can be repeated.

[0061] Figure 3 is a graph showing the performance of a visual field prediction model in one embodiment of the present invention.

[0062] The generated image pairs can be divided into training data and test data in a predetermined ratio. For example, the image pairs can be divided into training data and test data in a ratio of 8:2.

[0063] Hyperparameters of the visual field prediction model (10) can be pre-set. For example, the training period can be set over 200 epochs, and the learning rate can be 0.001 for the first 50 epochs and 6.6x10 for the remaining 150 epochs. -7 It can be set to decrease linearly. In this case, fine tuning can be performed near the optimal point of the loss function.

[0064] The graph shows the performance of Pix2Pix, cGAN, and L1 losses in the visual field prediction model (10) when combined with SSIM (Structural Similarity Index Measure) and TV (Total Variation) losses.

[0065] L1 loss can calculate the average of the absolute values ​​of pixel differences between the generated image and the target image. cGAN loss can measure the discriminator's ability to distinguish between real images and generated images. SSIM loss can evaluate the similarity between two images in terms of structure, contrast, and brightness. TV loss can calculate noise within the generated image.

[0066] Mean Absolute Error (MAE) and Mean Squared Error (MSE) can evaluate the accuracy of generated images by calculating pixel-level differences. Peak Signal to Noise Ratio (PSNR) can indicate pixel-level quality differences between the original image and the reconstructed image, and higher values ​​indicate higher image quality. SSIM can indicate similarity in terms of the structural, contrast, and brightness aspects of the image, and values ​​closer to 1 indicate that the two images are similar to each other. As a final performance metric, a combination of L1, cGAN, and TV loss with low MSE and high PSNR and SSIM can be selected.

[0067] Figure 4 shows a fundus image and a grayscale map in one embodiment of the present invention.

[0068] The fundus image (I) and the first grayscale map (M1) are designated as a pair of images, and the first grayscale map (M1) may be a ground truth to be designated as a target or answer.

[0069] When a fundus image (I) is input into a visual field prediction model (10), a second grayscale map (M2) can be output.

[0070] It can be seen that the generated second grayscale map (M2) appears to be almost identical to the first grayscale map (M1). For example, regarding the similarity between the second grayscale map (2) output by the visual field prediction model (10) and the first grayscale map (M1) designated as the target, the ratios judged as best, normal, and worst may be 30%, 45%, and 20%, respectively.

[0071] FIG. 5 illustrates a process of converting a fundus image into a grayscale map in one embodiment of the present invention.

[0072] The grayscale map generated by the visual field prediction model (10) may have areas that appear blurred due to the complexity of mapping relationships at the pixel level. Additionally, the size of the grayscale maps extracted from the field of view report may differ, or the size and position of the crosshairs within the grayscale map may differ, which may affect the generation of the grayscale map due to a lack of consistency in the training data. Furthermore, a significant amount of pixel-level noise may appear in the grayscale map.

[0073] In one embodiment of the present invention, pixel-level patterns can be replaced with block-level regions. An image can be divided into blocks, the grayscale ratio of each block can be calculated, and then the pattern regions of the blocks can be replaced with a single grayscale color. Furthermore, by simplifying the features of the image to reduce the influence of pixel-level noise and utilizing block-level regions, distortion occurring during image resizing can be minimized.

[0074] To this end, the diagnostic device (100) can extract a grayscale map from a static field of view report and then adjust it to a specific size. For example, the size of the grayscale map can be adjusted so that the width is 512 pixels and the height is 512 pixels. (S1060)

[0075] And the diagnostic device (100) can simplify the scaled grayscale map. For example, the pattern area of ​​the block can be replaced with a single grayscale color. (S1070) The remaining process may be the same as described in FIG. 2.

[0076] Accordingly, a grayscale map showing the overall degree of vision loss is generated instead of pixel-level features, which can resolve the problem of inconsistency and performance degradation in multimodal image transformation data when processing data in detail, and maintain data consistency.

[0077] Meanwhile, when the amount of prepared training data is insufficient, the visual field prediction model (10) may not be able to generate a grayscale map similar to the grayscale map set as a target. In one embodiment of the present invention, attention may be added to the GAN model to improve the performance of unpaired image-to-image conversion, or SPADE (Spatially Adaptive Normalization) may be used to improve the performance of image-to-image conversion.

[0078] Figure 6 is a graph showing the performance of a visual field prediction model in one embodiment of the present invention.

[0079] (a) shows the performance according to the combination of loss functions when image features are simplified. The loss functions can be a combination of L1 and GAN or L1, GAN, and TV. Here, SSIM loss may not be used as it causes image quality degradation. Performance was shown by adjusting the epochs for the same configuration.

[0080] (b) shows the performance according to the combination of loss functions and the presence or absence of attention when image features are simplified. In this case, the loss function can be a combination of L1 and GAN. Compared to (a), MSE can be improved by up to 17.8% and SSIM by up to 57%. Also, it can be seen that the image was transformed more finely when attention was added.

[0081] FIG. 7 shows a fundus image and a grayscale map in one embodiment of the present invention.

[0082] A diagnostic device (100) can extract a first grayscale map (M1) from a static field of view report, adjust it to a specific size, and simplify the adjusted grayscale map to generate a third grayscale map (M3).

[0083] When training the visual field prediction model (10), the third grayscale map (M3) can be targeted to generate a pair of images with the fundus image (I).

[0084] When a fundus image (I) is input into a visual field prediction model (10), a second grayscale map (M2) can be output. It can be seen that the generated second grayscale map (M2) appears almost identical to the third grayscale map (M3).

[0085] For example, regarding the similarity between the second grayscale map (2) output by the visual field prediction model (10) and the third grayscale map (M3) designated as a target, the ratios judged as best, normal, and worst may be 55%, 35%, and 10%, respectively.

[0086] FIGS. 8A and FIGS. 8B are diagrams showing the configuration of a diabetic retinopathy classification model in one embodiment of the present invention.

[0087] The diagnostic device (100) can crop the fundus image. For example, only the center area of ​​the fundus image can be cropped.

[0088] And you can adjust the size of the cropped image. For example, you can adjust the image size so that it is 640 pixels wide and 640 pixels high.

[0089] Multiple images can be generated by augmenting resized images. For example, images can be rotated, flipped horizontally or vertically, color jitter added, or normalization performed.

[0090] The diabetic retinopathy classification model (20) can generate a lesion map from a fundus image. The fundus image and the generated lesion map can be input into a pre-trained model. The diabetic retinopathy classification model (20) may include a segmentation model (21) and a classification model (22).

[0091] As shown in FIG. 8a, the segmentation model (21) can be updated by feeding back the result of inputting the generated lesion map and the ground truth designated as a target into the loss function. The classification model (22) can receive the lesion map generated from the fundus image and the fundus image, which are transferred and learned from the output result of the segmentation model (21), and calculate the probability for each stage of diabetic retinopathy. The calculated result can be updated by feeding back to the classification model (22).

[0092] As shown in FIG. 8b, the segmentation model (21) includes an encoder composed of multiple stages. Each stage includes a patch merging block that patches and merges images using self-attention, and a Swin Transformer block that captures dependencies and details necessary for segmentation. The patch merging block can combine patches into larger patches to reduce spatial dimensions and preserve essential features.

[0093] The segmentation model (21) includes a decoder composed of multiple stages. Each stage includes an upsampling layer and a convolutional block. The decoder reconstructs a segmented lesion map from encoded features and can use both low-level and high-level features through skip connections.

[0094] The fundus image can be output by generating a lesion map after passing through the encoder and decoder of the segmentation block (21).

[0095] The segmentation model (21) can be trained using binary cross entropy with logit loss applied. And it can distinguish between lesion pixels and non-lesion pixels according to the following mathematical formula.

[0096]

[0097] (N: number of samples, i: sample index, y i : Actual label of the i-th sample, x i : Predicted probability of the i-th sample)

[0098] The segmentation model (21) can be trained to segment lesion maps in fundus images using the fine-grained annotated diabetic retinopathy (FGADR) dataset. The segmentation model (21) can use pre-trained weights and then perform fine-tuning. The segmentation model (21) can generate lesion maps from EyePACS data. This transfer learning process can improve the accuracy of segmentation by capturing both general features and domain-specific features. Fine-tuning can adjust the model's weights to fit the FGADR dataset.

[0099] The classification model (22) can receive a fundus image and a lesion map generated by a generator as input and classify diabetic retinopathy. The fundus image and the lesion map are input into corresponding Swin Transformer blocks, and the output of each Swin Transformer block can be input into a cross attention block. The cross attention block (23) can combine the embedding sequence of the fundus image and the embedding sequence of the lesion map.

[0100] The classification model (22) includes multiple stages composed of a block for patching and merging images and a Swin Transformer block. The embedding sequence obtained through the multiple stages can be calculated by sequentially passing through an average pooling layer and a fully connected layer. When the calculated value is input into a softmax function, the probability of each stage of diabetic retinopathy can be output.

[0101] The classification model (22) can improve the performance of the model by capturing the interaction between the fundus image and the lesion map to distinguish features. By integrating information from the fundus image and the lesion map through cross-attention and finding key areas, it is possible to detect lesions of diabetic retinopathy.

[0102] The classification model (22) can be trained using binary cross-entropy with logit loss and can improve the ability to predict the probability of each stage of diabetic retinopathy by outputting the logit converted into a probability during the training of the model. Each stage of diabetic retinopathy can be expressed as 'negative', 'moderate', 'severe', and 'acute'.

[0103] In a cross-attention block, the fundus image (X=nxd) and the lesion map (Y=mxd) can be converted into a query (Q), a key (K), and a value (V). The query (Q), key (K), and value (V) can be calculated according to the following mathematical formula. In the mathematical formula, W can be a weight matrix.

[0104]

[0105] The attention weight (A) can be calculated according to the following mathematical formula.

[0106]

[0107] The cross-attention block can calculate and output a matrix representing the image (X). Additionally, it can be trained to distinguish between foreground and background by learning the interaction between the fundus image and the lesion map through the cross-attention block.

[0108] Figure 9 is a graph showing the performance of a diabetic retinopathy prediction model in one embodiment of the present invention.

[0109] For the segmentation task, the Dice coefficient and IoU (Intersection over Union) can be measured. And for the classification task, accuracy, AUC (Area Under the Curve), TPR (True Positive Rate), and TNR (True Negative Rate) can be measured.

[0110] In the segmentation task, the Dice coefficient was 0.89 and the IoU was 0.86, while in the classification task, the AUC was 96.2% and the accuracy was 94.6%. Additionally, the optimal threshold for classifying diabetic retinopathy can be determined using the ROC (Receiver Operating Characteristic) curve. For example, the point closest to the top-left corner of the ROC curve, where the TPR is highest and the FPR is lowest, can be determined as the threshold.

[0111] (a) shows the measured performance of the diabetic retinopathy prediction model (20). When the diabetic retinopathy prediction model (20) was compared with ResNet50, Swin Transformer (Swin-T, Swin-S, Swin-B, Swin-L), ConvNeXt, CrossViT, and CoATNet models, it can be seen that the accuracy, AUC, TPR, and TNR were measured to be higher.

[0112] (b) shows the performance measured according to whether cross-attention is applied between the fundus image and the generated lesion map in the diabetic retinopathy prediction model (20) and whether transfer learning using the Swin U-Net encoder is performed. It can be seen that when cross-attention and transfer learning are used, the accuracy and AUC are measured to be higher than when they are not used.

[0113] By integrating information from fundus images and lesion maps to perform cross-attention, it becomes possible to improve the accuracy of the model by focusing on lesions of diabetic retinopathy.

[0114]

[0115] Although embodiments of the present invention have been described above, the present invention is not limited to the above embodiments. Various modifications may be made within the scope of the detailed description of the invention and the attached drawings, provided that such modifications do not depart from the spirit of the invention and do not impair its effects. Furthermore, it is obvious that such embodiments fall within the scope of the present invention.

Claims

1. Step A: Extracting a grayscale map from a static viewfinder report using template matching; Step B, which generates pairs of images by mapping fundus images and grayscale maps of patients with the same examination date and eye position; and Step C, which trains a visual field prediction model using the generated image pairs; A method for diagnosing eye diseases from fundus images.

2. In Claim 1, In Step A above, static field of view reports are grouped into circular, elliptical, and teardrop shapes according to the shape of the grayscale map. A method for diagnosing eye diseases from fundus images.

3. In Claim 1, In step A above, saving the extracted grayscale map as an image file A method for diagnosing eye diseases from fundus images.

4. In Claim 1, In the above A step, a template suitable for the shape-based groups of the grayscale map is generated, and matching is performed to extract the detected area. A method for diagnosing eye diseases from fundus images.

5. In Claim 1, In the above step C, the visual field prediction model is Pix2Pix or CycleGAN A method for diagnosing eye diseases from fundus images.

6. In Claim 1, In step C above, the size of the image input to the visual field prediction model is uniform. A method for diagnosing eye diseases from fundus images.

7. In Claim 1, In the above step C, the visual field prediction model includes a generator based on UNet and a discriminator based on PatchGAN. A method for diagnosing eye diseases from fundus images.

8. In Claim 1, A visual field prediction model further includes step D, in which a grayscale map is generated from a fundus image. A method for diagnosing eye diseases from fundus images.

9. In Claim 8, Step E, which further includes comparing the output grayscale map with the target grayscale map to update the parameters of the visual field prediction model. A method for diagnosing eye diseases from fundus images.

10. Includes a visual field prediction model, A grayscale map is extracted from a static visual field report using template matching, and the grayscale map is mapped to fundus images of patients with the same examination date and eye position to generate image pairs, the visual field prediction model is trained using the generated image pairs, the visual field prediction model generates a grayscale map from the fundus images, and the parameters of the visual field prediction model are updated by comparing the output grayscale map with the grayscale map designated as a target. A device for diagnosing eye diseases from fundus images.

Images