Artificial intelligence-based medical image processing method and apparatus

Abstract

One aspect of the present invention discloses an apparatus for processing a medical image acquired from a capsule endoscope. The apparatus comprises: a communication unit that receives a medical image; and a processor that detects and analyzes a lesion from the medical image, wherein the processor includes: an object detection unit that receives the medical image and detects a lesion area using a pre-trained object detection model; an image size conversion unit that normalizes an image of the lesion area detected by the object detection unit to a preset size; and an image classification unit that performs a re-analysis of the lesion area by inputting the normalized image into an image classification model, which is a pre-trained neural network model, and executing the image classification model.

Description

Artificial intelligence-based medical image processing method and device

[0001] The present invention relates to an image processing method, and more specifically, to a method for more precisely detecting lesions in objects within a medical image provided from a capsule endoscope based on artificial intelligence.

[0002] Capsule endoscopes are important medical devices for diagnosing gastrointestinal diseases and generate numerous medical images. Recently, technology is being developed to automatically detect and classify lesions in these medical images using artificial intelligence. Conventional capsule endoscopes are disclosed in “Korean Patent Publication No. 10-262387 (System and method for processing medical images, Jan. 8, 2024).”

[0003] The above lesion detection technology used a method of detecting lesions using a single deep learning model. For example, it used a method of using only an object detection model such as YOLO or using only a CNN-based classification model.

[0004] However, this single-model approach has the following drawbacks. It is difficult to detect small lesions. Additionally, it leads to false positives due to the inability to analyze lesion characteristics in detail, requires unnecessary interpretation by clinicians due to low specificity, and fails to adequately reflect characteristics based on lesion size.

[0005] The objective according to one aspect of the present invention to solve the aforementioned problems is to provide a medical image processing method that improves the accuracy of lesion detection through an ensemble structure combining an object detection model and an image classification model for a reading image provided to a clinician.

[0006] According to one embodiment of the present invention for achieving the above-mentioned purpose, an apparatus for processing medical images acquired from a capsule endoscope comprises a communication unit for receiving said medical images and a processor for detecting and analyzing lesions from said medical images, wherein the processor may include an object detection unit that receives said medical images and detects a lesion area using a pre-trained object detection model, and an image classification unit that performs re-analysis of said lesion area by inputting at least one model among a pre-trained image classification model and a semantic segmentation model into an image of the lesion area detected by said object detection unit and executing the same.

[0007] Even though images of multiple lesion regions are extracted for a single input image as a result of the detection of the object detection unit, the image classification model of the image classification unit can perform reanalysis of the lesion regions by separating the multiple lesion regions and inputting them one by one.

[0008] The above device may further include an image size conversion unit that normalizes a lesion area detected by the object detection unit to a preset size and inputs it into the image classification model.

[0009] When the object detection unit executes the object detection model to detect multiple lesion regions within a single medical image, the image size conversion unit can identify the detected multiple lesion regions one by one, enlarge or reduce the identified lesion regions to a preset size, and input them into the image classification model.

[0010] The above object detection model can be trained using the original frame of a medical image for training and the location information of the labeled part within the original frame as a training dataset.

[0011] The above image classification model can be trained by separately extracting only the labeled portion of the training dataset of the above object detection model.

[0012] The above image classification model can be trained by normalizing the separately extracted image to a predetermined size.

[0013] The above preset size may be 90×90 to 1024×1024.

[0014] The above object detection model may be a neural network model of at least one of the YOLO series and the RCNN series.

[0015] The above image classification model may be at least one neural network model among the MobileNet family, ResNET, VGGNet, GoogleNet, DenseNet, and EfficientNet.

[0016] The lesions handled by the object detection model and the image classification model can be learned by classifying them into at least one of hemorrhagic, inflammatory, vascular, and polypoid lesions.

[0017] The lesions handled by at least one of the object detection model and the image classification model can be learned by classifying them into small and large sizes according to their size.

[0018] The processor can generate and control an artificial intelligence-based model to improve the specificity of medical image analysis through an ensemble structure of the object detection model and the image classification model.

[0019] The above processor can perform a determination on the final lesion by combining the detection result of the object detection unit and the classification result of the image classification unit.

[0020] The above image classification unit can be expressed by performing an image classification model that extracts detection boundaries in a rectangular shape.

[0021] In addition, the image classification unit can generate a lesion mask through semantic segmentation, extract the outline of the mask to select key vertices, and connect the selected vertices to generate a lesion boundary in the shape of a square or polygon.

[0022] A method for processing a medical image acquired from a capsule endoscope according to one embodiment of the present invention for achieving the above-mentioned purpose may include the steps of receiving the medical image, detecting a lesion region from the medical image using an object detection model, normalizing the detected lesion region to a predetermined size using an image classification model, classifying the lesion in the normalized lesion region using an image classification model, and determining a final lesion using the image classification result.

[0023] According to the medical image processing method of the present invention, the time required for medical image interpretation is reduced, thereby increasing the interpretation efficiency of clinicians and shortening the patient's waiting time.

[0024] In addition, it is effective in accurately detecting the location and size of lesions within medical images, enabling detailed feature analysis of the detected lesion areas, improving specificity to reduce false positives, and securing characteristics of lesions by size and shape.

[0025] The effects obtainable through the specific examples of this specification are not limited to those listed above. For example, there may be various technical effects that a person with ordinary skill in the related art can understand or derive from this specification. Accordingly, the specific effects of this specification are not limited to those explicitly described herein, but may include various effects that can be understood or derived from the technical features of this specification.

[0026] FIG. 1 is a conceptual diagram schematically illustrating a medical image processing system according to one embodiment of the present invention.

[0027] FIG. 2 is a detailed block diagram specifically showing the configuration of a processor of a medical image processing device according to one embodiment of the present invention.

[0028] FIG. 3 is a flowchart for explaining the operation of an object detection model according to an embodiment of the present invention,

[0029] FIG. 4 is a table showing the results detected by the object detection model of FIG. 3,

[0030] FIG. 5 is a table for explaining an input image of an image classification model according to an embodiment of the present invention,

[0031] Figure 6 is a table showing the normalized image of the image in Figure 5,

[0032] FIG. 7 is a conceptual diagram for explaining the ensemble model structure of a medical image processing device according to one embodiment of the present invention,

[0033] FIGS. 8 and 9 are drawings for comparing the performance of a medical image processing device according to an embodiment of the present invention.

[0034] The present invention is capable of various modifications and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail.

[0035] However, this is not intended to limit the invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0036] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0037] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0038] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0039] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0040] Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. In order to facilitate an overall understanding of the present invention, the same reference numerals are used for identical components in the drawings, and redundant descriptions of identical components are omitted.

[0041]

[0042] FIG. 1 is a conceptual diagram schematically illustrating a medical image processing system according to an embodiment of the present invention. As shown in FIG. 1, the image processing system according to an embodiment of the present invention may include a capsule endoscope device (10) and an image processing device (100).

[0043] Referring to FIG. 1, a capsule endoscope device (10) captures the inside of a patient's digestive tract to generate a medical image and transmits the generated medical image to an image processing device (100). Although one embodiment of the present invention is intended for a capsule endoscope, it does not necessarily have to be a capsule endoscope, and any image captured from a device (such as a general endoscope) that captures the inside of a human body or the inside of an animal's organ, particularly organs related to the digestive tract, may be used.

[0044] The image processing device (100) may include a communication unit (110), a memory (120), a processor (130), and a display unit (140). The image processing device (100) may be referred to as a medical image processing device, a device, a receiver, etc.

[0045] The communication unit (110) receives the captured image provided by the capsule endoscope (10). At this time, the communication unit (110) can receive the captured image through human body communication with the capsule endoscope (10). However, it is not necessary to transmit and receive image data only through human body communication, and it is acceptable to use other communication methods such as RFID, Bluetooth, or wired connection.

[0046] The memory (120) contains instructions related to the function of a processor (130) that generates a reading image based on a captured image. For example, the memory (120) stores an ensemble neural network algorithm including an object detection model and an image classification model. Additionally, the memory (120) can store a received medical image and an image processing result.

[0047] The processor (130) inputs the medical image received through the communication unit (110) into an ensemble neural network algorithm stored in memory (120) to detect and classify lesions. Specifically, the processor (130) detects lesion regions through an object detection model and classifies lesions in detail by normalizing the detected regions and inputting them into an image classification model.

[0048] The display unit (140) outputs the image processing results. The display unit (140) can visually display the location, size, and classification results of the detected lesion along with the original medical image. Through this, medical staff (clinicians) can intuitively identify the presence and characteristics of the lesion.

[0049] FIG. 2 is a detailed block diagram specifically showing the configuration of a processor of a medical image processing device according to an embodiment of the present invention. As shown in FIG. 2, according to an embodiment of the present invention, the processor (200) may include an object detection unit (210), an image size conversion unit (220), and an image classification unit (230).

[0050] Referring to FIG. 2, the object detection unit (210) detects an area suspected of being a lesion from an input medical image. The object detection unit (210) can classify the lesion using an object detection model. According to an embodiment of the present invention, the object detection model may include an RCNN series (RCNN / Fast-RCNN / Faster-RCNN / Mask-RCNN), a YOLO series (v1~v11), SSD, RetinaNet, etc. Using the object detection model, an inference area containing location and size information of the lesion is detected in a medical image input at a preset specific size (e.g., 320×320). The object detection unit (210) can detect various types of lesions, including hemorrhagic, inflammatory, vascular, and polypoid lesions. In particular, the accuracy of detection can be improved by applying learned weights according to the characteristics of each lesion.

[0051] The image size conversion unit (220) normalizes the lesion area (which may be an inferred area) detected by the object detection unit (210) to a size suitable for image classification. Specifically, it converts the previously detected area to a preset size (224×224). The size to be converted can be set by the user to a size suitable for input into the image classification unit (230). The range can be from 90×90 to 1024×1024. This reflects clinical experience that areas that are too small or too large are likely to be false positives. The image size conversion unit (220) can convert the size while minimizing image quality degradation by using interpolation, etc.

[0052] Finally, the image classification unit (230) receives an image of a normalized (i.e., transformed) lesion area and performs detailed classification related to the lesion. The image classification unit (230) can classify the lesion using an image classification model. According to an embodiment of the present invention, ResNet, VGGNet, GoogleNet, MobileNet series (v1 to v4), DenseNet, EfficientNet, etc., may be used as image classification models. Preferably, it is better to use a neural network model of the MobileNet series, specifically MobileNetV3. The image classification unit (230) can classify the lesion into small and large depending on the size of the lesion. In addition, the classification is performed by classifying the type into at least one of hemorrhagic, inflammatory, vascular, and polypoid. In particular, by enlarging or reducing the size of only the lesion area and analyzing only that part intensively, detailed feature extraction is possible, which leads to improved classification accuracy. The image classification unit (230) can calculate a confidence score for each lesion type. Only when this score exceeds a specific threshold is it finally classified as the corresponding lesion. Meanwhile, depending on the case, the image classification unit (230) may perform cross-validation with the detection result of the object detection unit (210) and determine it as the final lesion only when the results of the two models match. In particular, the object detection unit (210) detects all lesions included in the input image. That is, multiple lesion regions may be detected for a single input image. However, the image classification unit (230) infers at least one lesion region per image to allow for a more precise analysis of the lesion region. That is, it derives at least one inference result of the lesion region. In other words, the inference result of the image classification unit (230), which infers in a 1:1 manner, can add reliability to the inference result of the object detection unit (210), which infers in a 1:many manner.To this end, when the object detection unit (210) executes an object detection model to detect multiple lesion regions within a single medical image, the image size conversion unit (220) identifies the detected multiple lesion regions one by one and normalizes the identified lesion regions by enlarging or reducing them to a preset size. Then, the images of the lesion regions, each identified and normalized, are input into the image classification model of the image classification unit (230) so that reanalysis of the lesion regions can be performed.

[0053] The processor (200) can improve specificity by more than about 2% compared to a single model through this ensemble structure. Specificity is the probability of recognizing a normal area as a normal area rather than a lesion. In particular, it can exhibit high detection and classification performance even for small lesions or unclear lesions. The processor (200) visualizes the detection and classification results and transmits them to a display unit, and if necessary, can store detailed information of the detected lesions in memory (database). Through this configuration, the processor (200) can effectively combine object detection and image classification to detect and classify lesions with high accuracy, and this can be utilized as a reliable tool to assist the clinician's diagnosis.

[0054] Meanwhile, according to one embodiment of the present invention, the object detection unit (210), image size conversion unit (220), and image classification unit (230) are used to generate accurate information about lesions through inference by executing a pre-trained object detection model and an image classification model, but at the same time, the training of the above neural network models can also be performed directly. That is, the inference result is derived from the input image, and self-learning can be performed by using it directly as a training data set. Alternatively, the inference result may be used to perform neural network model training by having an expert add processing work, such as text or labeling modification. The description of the embodiments of FIGS. 4 to 6 may be a description of the inference result using a neural network model, or it may be understood as a description of labeling for generating training data in the same or similar way. It will be obvious to a person skilled in the art to which the present invention belongs that the artificial intelligence model training can be linked with the labeling method and the result inferred through the said artificial intelligence model.

[0055] FIG. 3 is a flowchart illustrating the operation of an object detection model according to an embodiment of the present invention.

[0056] Referring to FIG. 3, an object detection unit (210) according to one embodiment of the present invention executes an object detection model, and the object detection model sequentially performs an input (INPUT) step, a feature extraction (BACKBONE) step, a feature combination (NECK) step, and an object location classification (Head) step.

[0057] First, in the INPUT stage, medical images acquired from a capsule endoscope are input. The images are input as RGB images with a size of 320×320. In particular, image preprocessing is performed during the input stage, which includes brightness normalization, contrast enhancement, and noise removal.

[0058] In the backbone stage, features are extracted from the input image using the CSPDarknet53 architecture. CSPDarknet53 utilizes a Cross-Stage Partial Network (CSP) structure, which allows for reduced computational costs while maintaining accuracy compared to the existing Darknet. In the backbone stage, features are extracted hierarchically from low-level to high-level images through convolutional layers and residual connections.

[0059] In the NECK stage, the features extracted from the BACKBONE are integrated and enhanced. In this stage, a PANet (Path Aggregation Network) structure is used to effectively combine features of different scales. In particular, by fusing multi-scale features through bottom-up and top-down paths, the information necessary to detect lesions of various sizes is reinforced.

[0060] In the Head stage, dense predictions are performed based on the features enhanced in the NECK stage. In this stage, predictions are made at three different scales (13×13, 26×26, 52×52). At each scale, anchor boxes are used to predict the location and size of the lesion, and at the same time, the probability for the lesion class is calculated.

[0061] The results of density prediction are post-processed to derive the final detection results. In this step, the Non-Maximum Suppression (NMS) algorithm is applied to eliminate duplicate detections, and a confidence threshold is applied to select only detection results with a high probability. In particular, an NMS algorithm adjusted for the characteristics of medical images is used to ensure that adjacent lesions are not suppressed by each other.

[0062] The object detection unit (210) can accurately detect the location and size of a lesion in an input medical image through this four-step process. In particular, it exhibits robust detection performance for lesions of various sizes through a multi-scale structure and feature fusion, and real-time processing is possible due to the optimized network structure.

[0063] The detected results may include the bounding box coordinate information (x, y, width, height) along with confidence scores for each lesion class. This is utilized in the subsequent image size conversion unit and image classification unit. Additionally, intermediate feature maps at each stage can be visualized if necessary, allowing them to be used for analysis and debugging of the detection process.

[0064] Meanwhile, the object detection unit (210) according to the present embodiment can binary classify the captured image. In particular, in an embodiment where the object detection model is executed as a YOLO model, a method in which an expert labels text associated with the lesion and draws a rectangle (or polygon) around the lesion may be applied to the labeling of the YOLO model.

[0065] The YOLO model can be implemented as any YOLO family model, such as YOLOv4 or YOLOv8. As previously explained, the YOLO v4 model adopts the YOLO v3, CSPDarknet53, SPP, PAN, BoF, ALC, and BoS architectures. Conventional YOLO models had a weakness in detecting small objects. However, the YOLOv4 model can demonstrate high performance even in the detection of small lesions by using a high input resolution.

[0066] Basically, when a captured image is input, the object detection model to which the YOLO model is applied classifies images containing predefined lesions and performs binary classification regardless of the type of lesion to classify them into lesion images and normal images. At this time, the predefined lesions may be images containing hemorrhagic, inflammatory, vascular, and polypoid lesions. Generally, when a captured image is input, the YOLO model can classify the image into four classes to determine whether it contains hemorrhagic, inflammatory, vascular, and polypoid lesions.

[0067] Meanwhile, as the number of classes to be classified increases, the performance of the deep learning model inevitably deteriorates. According to another embodiment of the present invention, the object detection model may not necessarily perform classification into four classes when a captured image is input. The object detection model classifies images containing lesions regardless of the type of lesion and performs binary classification into two classes: lesion images and normal images. That is, the object detection model classifies captured images containing at least one of hemorrhagic, inflammatory, vascular, and polypoid lesions, and reclassifies the classified captured images into lesion images and normal images. At this time, the criterion for distinguishing between lesion images and normal images can be determined based on four thresholds after statistically calculating thresholds for each of the hemorrhagic, inflammatory, vascular, and polypoid lesions.

[0068] Figure 4 is a table showing the labeled appearance for training the object detection model of Figure 3.

[0069] Referring to FIG. 4, as previously described, the object detection model executed through the object detection unit (210) is trained using the entire image (320×320) (which can be called the original image) and the location information of the labeled rectangle as a training data set. In particular, when labeling the image, the object detection model applies a bounding box to the area where the lesion is located within the lesion image and can input information regarding the type and size of the lesion. The types of lesions can be classified as hemorrhagic, inflammatory, vascular, and polypoid. Additionally, the size of the lesion is classified as small or large. The classification into small and large can be achieved by the labeling expert directly determining whether the lesion is larger or smaller than a reference size value, taking into account the reference size value. The reference size value can be set by the user and can be changed arbitrarily. The labeling expert can label the information regarding the size without inputting it as text, but rather by converting it into a small data reference format using boxing and by boxing it to a large format.

[0070] Meanwhile, various object detection models other than YOLO can be used. In particular, depending on the characteristics of the available object detection models, various methods can be used for labeling image data.

[0071] For example, in image data labeling, an expert identifies lesions present in the image data. The expert can then label the corresponding lesions with text (information regarding the type and size of the lesion).

[0072] For example, in image data labeling, an expert identifies lesions present in the image data. The expert then labels the lesions with text and directly marks the lesions within the image data. At this stage, the expert can mark the location of the lesions by drawing a rectangle around them.

[0073] For example, in image data labeling, an expert identifies lesions present in the image data. The expert then labels the lesions with text and directly marks the lesions within the image data. In this process, the expert can mark the location of the lesions by drawing polygons based on the lesions present in the image data.

[0074] When medical images are input into an object detection model to which a training dataset generated by labeling through the above method is applied, specific lesion regions can be extracted as bounding boxes, as shown in the table in Figure 4. At this time, the lesions can be classified into four types: hemorrhagic, inflammatory, vascular, and polypoid, and information on the size of the lesion, indicating whether it is small or large, can also be extracted. In another example, the result of the object detection model may be derived as a binary result of normal regions and lesion regions.

[0075] FIG. 5 is a table for explaining the input image of an image classification model according to one embodiment of the present invention.

[0076] Referring to FIG. 5, in order to derive more effective learning and inference results between the object detection model and the image classification model, it may be necessary to normalize the input image of the image classification model. To this end, only the bounding box area of ​​the inference result (or labeling result) of the object detection model in FIG. 4 is separately extracted. As in the embodiment of FIG. 5, this can be divided into four classes of lesions and two classes of sizes. It can be divided into a total of eight categories. To ensure that learning or inference in the image classification model is performed more appropriately, the device enlarges or reduces the extracted image of FIG. 5 to a certain size. The result is described in FIG. 6.

[0077] Figure 6 is a table showing the normalized image of Figure 5.

[0078] Referring to Fig. 6, the extracted image of Fig. 5 is normalized and converted to a preset size. This can be changed and adjusted according to the preset input data size of the deep learning model. As previously explained, the image size suitable for the image classification model is preferably in the range of 90×90 to 1024×1024. In particular, it is desirable to have a size of 224×224. Basically, square image shapes allow for good learning and, consequently, good extraction of inference results. The normalized image can also be labeled by distinguishing four types of lesions and two lesion sizes, as shown in the table of Fig. 5. Furthermore, during inference, results representing eight distinct categories can be derived as described above. As described above, when labeling is performed in the image classification model, it has the advantage of extracting better features by intensively learning the abnormal lesion regions. This can improve classification performance and has the characteristic of making it robust even to small lesions.

[0079] Meanwhile, an image classification unit according to one embodiment of the present invention may include a structure that performs at least one model among an image classification model and semantic segmentation. Semantic segmentation is a technique for classifying each pixel of an image into a specific class. In one embodiment of the present invention, it can be used to detect the precise boundaries of a lesion at the pixel level. Specifically, the image classification unit (230) of FIG. 2 adopts a Fully Convolutional Network (FCN) structure to perform precise segmentation of the lesion area. The FCN has an encoder-decoder structure, where the encoder extracts features through MobileNetV3, and the decoder generates a segmentation map of the original image size based on the extracted features. Furthermore, the image classification unit (230) can adaptively detect boundaries according to the shape of the lesion. In particular, for irregularly shaped lesions, the limitations of the existing rectangular bounding box, which made it difficult to accurately represent the area, can be overcome, and the boundaries of the lesion can be represented in a polygonal shape.

[0080] To achieve this, the following processing steps are performed: (i) A pixel-level lesion mask is generated through semantic segmentation. (ii) The outlines of the generated mask are extracted. (iii) Key vertices are selected by applying an outline simplification algorithm (e.g., the Douglas-Peucker algorithm). (iv) The selected vertices are connected to create a polygonal lesion boundary. (v) At this stage, the number of vertices in the polygon is automatically adjusted according to the complexity of the lesion. Simple lesions are represented by a small number of vertices, while complex lesions use a larger number of vertices to represent accurate boundaries.

[0081] The results of semantic segmentation can be used for (i) accurate calculation of the area of ​​the lesion, (ii) analysis of the morphological characteristics of the lesion, (iii) tracking changes in the shape of the lesion over time, (iv) providing basic data for 3D reconstruction, etc.

[0082] Through this semantic segmentation-based polygonal detection method, the actual shape of the lesion can be represented more accurately. Furthermore, the inclusion of unnecessary normal tissue regions can be minimized. Additionally, adjacent multiple lesions can be accurately distinguished. Moreover, quantitative analysis of the size and shape of the lesion is possible. By applying such semantic segmentation technology, the image processing method according to one embodiment of the present invention enables not only the detection of the location of the lesion but also accurate boundary detection, thereby supporting more accurate diagnosis by medical professionals.

[0083] FIG. 7 is a conceptual diagram illustrating the ensemble model structure of a medical image processing device according to one embodiment of the present invention.

[0084] Referring to Fig. 7, an inference result for an input image, i.e., information about a lesion, can be obtained using the object detection model and image classification model learned as previously described. The inference result includes information regarding the location and size of the lesion area in the input image. This may include information regarding the type of lesion (one of hemorrhagic, inflammatory, vascular, and polypoid) and the size of the lesion (small, large). This can be output in the form of a bounding box.

[0085] To explain more specifically, first, capsule endoscopy medical images are input as data to be analyzed. The input medical images may have a size of 320×320. The input images are digital images with RGB color channels and may be images of the inside of the digestive tract.

[0086] The input medical image is input into an object detection model. At this time, YOLOv4 may be used as the object detection model. As a result of object detection, hemorrhagic, inflammatory, vascular, and polypoid lesions are detected in the input image. This may be a part of a 320×320 area. In the embodiment of FIG. 7, a vascular lesion is detected as the inference area of ​​the object detection model, and the detected area is indicated by a yellow bounding box. Subsequently, the device enlarges the detected lesion area and displays it separately. This becomes a Region of Interest (ROI) for subsequent processing.

[0087] After detecting the inference region of the object detection model, the inference region is extracted and its size is adjusted. That is, the detected lesion region is normalized to a fixed reference size. It is preferable for the reference size to range from 90×90 to 1024×1024. In particular, a size of 224×224 is desirable. This is intended to reduce false positives by excluding detection regions that are too small or too large. The lesion region of the object detection model is normalized to a size of 224×224 and input into the image classification model. Meanwhile, bilinear interpolation may be used during the size normalization process. This is used to adjust the size while minimizing image quality degradation. The normalized size is tailored to the input requirements of the subsequent image classification model and may be the standard input size for models pre-trained on large datasets such as ImageNet.

[0088] The image classification model performs a re-analysis of the inference domain of the object detection model. For this purpose, MobileNetV3 can be used as the image classification model. MobileNetV3 is a lightweight neural network optimized for mobile environments that can achieve high classification performance with minimal computation. In particular, this system enables more detailed feature extraction by focusing exclusively on the lesion region compared to analyzing the entire image. MobileNetV3 performs detailed classification on normalized lesion images, and the results can then be comprehensively analyzed in conjunction with the object detection results.

[0089] Meanwhile, in the final decision stage, the results of object detection and image classification are combined using an ensemble approach. During this process, a final decision may be made by weighting the prediction probabilities of the two models or by applying rule-based judgment logic. This ensemble approach has been confirmed to improve specificity by 2% compared to a single model. In particular, this improvement in specificity leads to a reduction in false positives, which can reduce unnecessary review time for medical staff.

[0090] The processing of this embodiment can achieve high accuracy and specificity by performing object detection and image classification sequentially while utilizing the results of the two models in a complementary manner. A hierarchical approach in which the object detection model identifies the location and size of the lesion, followed by the image classification model analyzing the corresponding area in greater detail, enables the achievement of high accuracy and specificity. In particular, the reliability of the analysis can be enhanced and unnecessary computations reduced through the process of limiting the size of the lesion area and normalization.

[0091] FIGS. 8 and 9 are drawings for comparing the performance of a medical image processing device according to an embodiment of the present invention.

[0092] Referring to FIG. 8, a performance comparison test was conducted with a conventional AI for a medical image processing system based on the method described in the present invention, in accordance with one embodiment. The test was performed on a total of 5,300,000 data points, comprising 4,900,000 normal images and 400,000 images of abnormal lesions. While the conventional AI uses a single model structure employing only YOLOv4, the image processing system according to one embodiment of the present invention adopts an ensemble structure combining YOLOv4 and MobileNetV3. As a result of the performance evaluation, the sensitivity of both systems was identical at 93%; however, regarding specificity, the image processing system according to one embodiment of the present invention showed a performance improvement of 2%, reaching 62% compared to 60% for the conventional AI. Furthermore, in terms of accuracy, the image processing system according to one embodiment of the present invention showed a result of 2% improvement, reaching 66% compared to 64% for the conventional AI.

[0093] Referring to FIG. 9, an image according to an embodiment of the present invention presents qualitative comparison results for three detected lesions (inflammatory, polypoid, and vascular) among four types of lesions: hemorrhagic, inflammatory, vascular, and polypoid. For each lesion type, the Reference Standard (correct answer key), the analysis results of a conventional AI, and the analysis results of an image processing system according to an embodiment of the present invention were compared. Particularly noteworthy is the detection accuracy of the lesion area. In the case of inflammatory lesions, the image processing system according to an embodiment of the present invention detected an area closer to the correct answer key, and it also demonstrated more accurate boundary detection for polypoid lesions. In the case of vascular lesions, it can be confirmed that the image processing system according to an embodiment of the present invention detects the lesion with a more accurate location and size. These qualitative comparison results visually demonstrate that the image processing system according to an embodiment of the present invention possesses more sophisticated lesion detection capabilities compared to conventional AI. The ensemble structure of object detection and image classification enables more accurate identification of the location and shape of the lesion, which is analyzed to have led to improved specificity. Furthermore, it can be confirmed that stable detection performance is demonstrated for lesions of various shapes and sizes. In particular, false positives or missed detections that may occur in existing AI have been significantly reduced, and it can be confirmed that the boundaries of lesions are distinguished more clearly. This performance improvement reduces the time required to interpret medical images, thereby increasing the interpretation efficiency of clinicians and shortening patient waiting times. Information regarding lesion detection or prediction of normality can be provided to clinicians using the method described above in the present invention.

[0094]

[0095] The system or device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the system, device, and component described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0096] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0097] The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiments, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiments, and vice versa.

[0098] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0099] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

1. A device for processing medical images acquired from a capsule endoscope, A communication unit receiving the above medical image; and A processor for detecting and analyzing lesions from the medical image above is included, wherein the processor, An object detection unit that receives the above medical image as input and detects a lesion region using an object detection model, which is a neural network model of at least one of the pre-trained YOLO series and RCNN series; and An image size conversion unit that normalizes the image of the lesion area detected by the object detection unit to a preset size; and A medical image processing device comprising an image classification unit that performs reanalysis of the lesion area by inputting the normalized image into an image classification model, which is a neural network for pre-trained image classification, and executing the same.

2. In Paragraph 1, A medical image processing device that performs reanalysis of said lesion regions by separating and inputting said lesion regions one by one into the image classification model of said image classification unit, even though images of multiple lesion regions are extracted for a single input image as a result of detection by said object detection unit.

3. In Paragraph 1, A medical image processing device in which, when the object detection unit executes the object detection model to detect multiple lesion regions within a single medical image, the image size conversion unit identifies the detected multiple lesion regions one by one, enlarges or reduces the identified lesion regions to a preset size, and inputs them into the image classification model.

4. In Paragraph 1, A medical image processing device in which the object detection model is trained using the original frame of a medical image for training and location information of a labeled part within the original frame as a training dataset.

5. In Paragraph 4, A medical image processing device in which the above image classification model is trained by separately extracting only the labeled portion of the training dataset of the above object detection model.

6. In Paragraph 5, A medical image processing device in which the above image classification model is trained by normalizing the separately extracted image to a predetermined size.

7. In Paragraph 6, A medical image processing device having a preset size of 90×90 to 1024×1024.

8. In Paragraph 1, A medical image processing device in which the lesions handled by the object detection model and the image classification model are learned by classifying at least two of hemorrhagic, inflammatory, vascular, and polypoid lesions.

9. In Paragraph 1, A medical image processing device in which lesions handled by at least one of the object detection model and the image classification model are learned by being classified into small and large according to their size.

10. In Paragraph 1, A medical image processing device wherein the image classification model of the image classification unit is at least one neural network model among the MobileNet series, ResNET, VGGNet, GoogleNet, DenseNet, and EfficientNet.

11. In Paragraph 1, The above processor is a medical image processing device that controls the specificity of medical image analysis to be improved through an ensemble structure of the object detection model and the image classification model.

12. In Paragraph 1, The above processor is a medical image processing device that performs a determination on the final lesion by combining the detection result of the object detection unit and the classification result of the image classification unit.

13. In Paragraph 1, A medical image processing device in which the image classification unit performs semantic segmentation to detect the boundaries of the lesion in pixel units and represents the detected boundaries in a polygonal form.

14. In Paragraph 1, The above image classification unit is a medical image processing device that generates a lesion mask through semantic segmentation, extracts the outline of the mask to select key vertices, and connects the selected vertices to generate a polygonal lesion boundary.

15. A method for processing medical images acquired from a capsule endoscope, Step of receiving the above medical image, A step of detecting a lesion region from the medical image using an object detection model; A step of normalizing the detected lesion area to a predetermined size; A step of classifying lesions in the normalized lesion area using an image classification model; and A medical image processing method comprising a step of determining a final lesion using image classification results.

Images