Method and device for diagnosing region suspected of bladder cancer through cystoscopy image

Abstract

The present invention relates to a method and device for diagnosing a region suspected of bladder cancer through a cystoscopy image. Specifically, the present invention relates to a method and device for automatically detecting a region suspected of bladder cancer in a cystoscopy image by using a deep learning-based convolution neural network (CNN) and a regional proposal network (RPN), and providing, as a confidence score and visual information, whether a bladder cancer is malignant.

Description

Method and device for diagnosing a site suspected of bladder cancer through cystoscopy images

[0001] The present invention relates to a method and apparatus for diagnosing a suspected bladder cancer area through cystoscope images. Specifically, the present invention relates to a method and apparatus for automatically detecting a suspected bladder cancer area in a cystoscope image using a deep learning-based CNN (convolutional neural network) and RPN (regional proposal network), and for providing a confidence score and visual information regarding whether the area is malignant.

[0002] Bladder cancer (BCa) is the 10th most common malignant tumor worldwide, posing significant challenges to diagnosis and treatment due to its high recurrence rate and the need for continuous monitoring. White light cystoscopy (WLC) is currently used as the standard for the diagnosis and surveillance of BCA. Patients with suspected lesions identified via WLC generally undergo transurethral resection of bladder tumors (TURBT) to obtain a definitive pathological diagnosis and staging. Non-muscle invasive BCA, which accounts for approximately 75% of cases, can often be managed with TURBT, followed by regular cystoscopic surveillance every 36 months to detect recurrence.

[0003] White light cystoscopy (WLC) is widely used but has several significant limitations. In particular, there is a high likelihood of missing flat lesions, such as carcinoma in situ (CIS) or very small tumors, resulting in a misdiagnosis rate of 20–30% and an incomplete resection rate of up to 50%. These diagnostic limitations lead to high rates of early recurrence and progression in patients with BCA. Although cystoscopy improvement technologies, such as narrow-band imaging and photodynamic diagnostics, have been developed to address these limitations, their adoption is limited due to the need for specialized equipment and additional training.

[0004] In recent years, artificial intelligence (AI) has established itself as an innovative technology in the field of medical diagnosis, providing the capability to automatically extract and analyze complex image data. AI applications have demonstrated promising results in various fields, such as radiology, dermatology, and gastroenterology, by improving diagnostic accuracy and efficiency. AI has also been studied in the field of bladder cancer diagnosis, exhibiting excellent performance in detecting bladder cancer and determining its stage or grade through CT (computed tomography), MRI (magnetic resonance imaging), hematoxylin and eosin staining, or urine cytology images. Furthermore, numerous studies have been conducted on the ability of AI to accurately diagnose BCA based on cystoscopy images, and high accuracy has been demonstrated in most of these studies.

[0005] Cystoscopic images contain information such as the location, size, shape, and texture of tumors, providing crucial data for the diagnosis of bladder cancer. However, visual inspection by medical professionals may have limitations in interpreting these complex images. Artificial intelligence (AI) can compensate for these limitations by automatically detecting suspected tumor sites in cystoscope images and providing medical staff with objective diagnostic information. This can reduce the possibility of misdiagnosis during cystoscopy and improve diagnostic efficiency.

[0006] Deep learning technology plays a particularly important role in AI-based bladder cancer diagnosis. Deep learning models learn the features of cystoscope images through Convolutional Neural Networks (CNNs) and can distinguish between tumors and normal tissues. In particular, Region Proposal Networks (RPNs) are suitable for detecting Regions of Interest (RoIs) in cystoscope images, and can efficiently detect candidate regions and evaluate their reliability by utilizing anchor boxes and bounding box regression. These technologies can significantly improve the accuracy of automated analysis in bladder cancer diagnosis.

[0007] AI-based software can be used to automatically mark the boundaries of suspected tumor sites in cystoscopy images and predict whether those areas are malignant. This software supports medical professionals in making more precise diagnoses and contributes to lowering the risk of recurrence for patients. Furthermore, the confidence scores and visualized results provided by AI facilitate communication between medical staff and patients and can serve as important reference materials in the clinical decision-making process.

[0008] Based on the discussion described above, various embodiments of the present invention provide a method and apparatus for diagnosing a site suspected of bladder cancer through cystoscope images.

[0009] Various embodiments of the present invention provide a method and apparatus for more accurately detecting suspected bladder cancer sites through data preprocessing techniques including noise removal, color adjustment, and brightness adjustment in cystoscope images.

[0010] Various embodiments of the present invention provide a method and apparatus for detecting and classifying suspected bladder cancer sites by integrating various information, such as the color, size, and texture of a tumor in a cystoscope image, through multi-channel based deep learning analysis.

[0011] Various embodiments of the present invention provide a method and apparatus for detecting candidate regions of a cystoscope image using a region proposal network (RPN) and a convolutional neural network (CNN) header and determining whether a tumor is malignant based on a confidence score.

[0012] Various embodiments of the present invention provide a method and apparatus for visualizing the boundaries of a suspected bladder cancer site as a three-dimensional image and intuitively expressing the size and malignancy of a tumor through a color-coded mask.

[0013] Various embodiments of the present invention provide a method and apparatus for generating highly reliable diagnostic results by applying an adaptive loss function and a shape normalization technique to improve the accuracy of model learning and correcting the distorted shape of a suspected bladder cancer site.

[0014] According to various embodiments of the present invention, a method performed by an electronic device for providing information on a suspected bladder cancer area from a cystoscope image comprises: a step of receiving a cystoscope image taken of a bladder by a cystoscope via the input device or the transceiver; a step of generating a preprocessed image by the processor by performing data preprocessing on the cystoscope image, including noise removal, color adjustment, and brightness adjustment; a step of extracting feature vectors in multiple channels from the preprocessed image by the processor using a CNN (convolution neural network) based backbone model; a step of detecting a suspected bladder cancer area from the preprocessed image by the processor based on the feature vectors using a regional proposal network (RPN); a step of determining a confidence score regarding whether the suspected bladder cancer area is a malignant tumor, and determining by the processor whether the suspected bladder cancer area is a malignant tumor based on a comparison between the confidence score and a set threshold value. A method is provided comprising the steps of: marking a boundary line on the outer edge of the suspected bladder cancer area in the preprocessed image, and generating a processed image by the processor in which the suspected bladder cancer area of ​​the preprocessed image is marked with a color code mask depending on whether the suspected bladder cancer area is a malignant tumor; and outputting the processed image by the output device.

[0015] According to various embodiments of the present invention, an electronic device for providing information on a suspected bladder cancer area from a cystoscope image comprises: an input device; an output device; a transceiver; a processor; and a memory, wherein the memory stores instructions for operations to be performed by the processor, and the operations include: receiving a cystoscope image taken of a bladder by a cystoscope via the input device or the transceiver; generating a preprocessed image by the processor by performing data preprocessing on the cystoscope image, including noise removal, color adjustment, and brightness adjustment; extracting feature vectors in multiple channels from the preprocessed image by the processor using a CNN (convolution neural network) based backbone model; and detecting a suspected bladder cancer area from the preprocessed image by the processor based on the feature vectors using a regional proposal network (RPN). An electronic device is provided comprising the steps of: determining a confidence score regarding whether the suspected bladder cancer area is a malignant tumor, and determining by the processor whether the suspected bladder cancer area is a malignant tumor based on a comparison of the confidence score and a set threshold value; marking a boundary line on the outer edge of the suspected bladder cancer area in the preprocessed image, and generating by the processor a processed image in which the suspected bladder cancer area in the preprocessed image is marked with a color code mask according to whether the suspected bladder cancer area is a malignant tumor; and outputting the processed image by the output device.

[0016] Various embodiments of the present invention can provide a method and apparatus for diagnosing a site suspected of bladder cancer through cystoscope images.

[0017] Various embodiments of the present invention can provide a method and apparatus for more accurately detecting suspected bladder cancer sites through data preprocessing techniques including noise removal, color adjustment, and brightness adjustment in cystoscope images.

[0018] Various embodiments of the present invention can provide a method and apparatus for detecting and classifying suspected bladder cancer sites by integrating various information, such as the color, size, and texture of a tumor in a cystoscope image, through multi-channel based deep learning analysis.

[0019] Various embodiments of the present invention may provide a method and apparatus for detecting candidate regions of cystoscope images and determining whether a tumor is malignant based on a confidence score by utilizing a region proposal network (RPN) and a convolutional neural network (CNN) header.

[0020] Various embodiments of the present invention can provide a method and apparatus for visualizing the boundaries of a suspected bladder cancer site as a three-dimensional image and intuitively expressing the size and malignancy of a tumor through a color-coded mask.

[0021] Various embodiments of the present invention can provide a method and apparatus for generating highly reliable diagnostic results by applying an adaptive loss function and a shape normalization technique to increase the accuracy of model learning and correcting the distorted shape of a suspected bladder cancer area.

[0022]

[0023] FIG. 1 illustrates an overview of the operation of an electronic device according to various embodiments of the present invention.

[0024] FIG. 2 illustrates an example of a system for diagnosing a suspected bladder cancer area through a cystoscope image according to various embodiments of the present invention.

[0025] FIG. 3 illustrates the configuration of an electronic device according to various embodiments of the present invention.

[0026] FIG. 4 illustrates a method of operation of an electronic device according to various embodiments of the present invention.

[0027] FIG. 5 illustrates the structure of a multi-layer perceptron (MLP) for implementing a deep learning algorithm according to various embodiments of the present invention.

[0028] Figure 6 illustrates an example of the structure of an AI model based on a two-stage model.

[0029]

[0030] Figure 7 illustrates an example of a ResNet block and a RexNext block.

[0031] Figure 8 illustrates an example of an RPN structure for cutting out a suspected tumor region and passing it to the final header Neural Networks.

[0032] Figure 9 illustrates an example of an AI header structure for determining whether a suspected area received via RPN is a malignant tumor and expressing the tumor size.

[0033] FIG. 10 illustrates an example of a structure for training a deep learning model.

[0034] Figure 11 illustrates an example of a structure for training a deep learning model.

[0035] FIG. 12 illustrates an example of a system for diagnosing a suspected bladder cancer area through a cystoscope image according to various embodiments of the present invention.

[0036] FIG. 13 illustrates an example of an output screen of an electronic device for diagnosing a suspected bladder cancer area through a cystoscope image according to various embodiments of the present invention.

[0037] Figure 14 illustrates an example of the definition and calculation formula of the die coefficient.

[0038]

[0039] Hereinafter, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. The present invention may be embodied in various different forms and is not limited to the embodiments described herein.

[0040] Hereinafter, in the contents of this specification, cystoscope and endoscope may be used interchangeably and have the same meaning.

[0041]

[0042] The present invention relates to a method and apparatus for diagnosing a suspected bladder cancer area through cystoscope images. Specifically, the present invention relates to a method and apparatus for automatically detecting a suspected bladder cancer area in a cystoscope image using a deep learning-based CNN (convolutional neural network) and RPN (regional proposal network), and for providing a confidence score and visual information regarding whether the area is malignant.

[0043]

[0044] FIG. 1 illustrates an overview of the operation of an electronic device according to various embodiments of the present invention.

[0045] Referring to FIG. 1, an electronic device (120) according to various embodiments of the present invention operates in a structure that receives a cystoscope image (110) and outputs analysis information (130) of a suspected bladder cancer area.

[0046]

[0047] FIG. 2 illustrates an example of a system for diagnosing a suspected bladder cancer area through a cystoscope image according to various embodiments of the present invention.

[0048] Referring to FIG. 2, a system according to various embodiments of the present invention is configured to extract image information from a cystoscope image and visualize and output the results of performing AI analysis.

[0049] A system according to various embodiments of the present invention receives an image captured by a cystoscope device as input and performs a data preprocessing process including noise removal, color adjustment, and brightness correction of the image. This process minimizes visual distortion caused by the complex internal structure of the bladder and lighting conditions, thereby preparing the system to derive more accurate results during the AI ​​analysis stage.

[0050] Preprocessed images are analyzed using a deep learning-based Convolutional Neural Network (CNN) backbone model and a Regional Proposal Network (RPN). The CNN extracts unique feature vectors from the images, and the RPN detects suspected bladder cancer sites based on these feature vectors and establishes Regions of Interest (RoIs). Subsequently, a Confidence Score is calculated to determine malignancy regarding these regions of interest, and bounding boxes and polygon masks are generated as needed.

[0051] The results of the AI ​​analysis are visualized to be intuitively conveyed to the user. Suspected bladder cancer sites are indicated in the form of bounding boxes or polygon masks, and a confidence score is additionally provided for each suspected site. This visualization supports medical professionals in diagnosing bladder cancer more clearly and efficiently, and clearly conveys information regarding the location and shape of the lesions.

[0052] The system according to various embodiments of the present invention is designed to be applicable in various medical environments. It can further enhance diagnostic accuracy by integrating with cystoscopy technologies such as narrow band imaging (NBI) or photodynamic diagnosis (PDD). Furthermore, it maximizes clinical applicability by improving generalization performance through multi-institutional learning and designing it to be compatible with existing systems. This scalability supports the work of medical staff and contributes to improving the quality of patient care.

[0053]

[0054] FIG. 3 illustrates the configuration of an electronic device according to various embodiments of the present invention.

[0055] Referring to FIG. 3, the electronic device (120) includes a memory (121), a processor (122), an input device (123), and an output device (124).

[0056] The memory (121) is connected to the memory (121), processor (122), input device (123), and output device (124), and can store information input through the input device (123). Additionally, the memory (121) is connected to the processor (122) and can store data such as a basic program, application program, setting information, and information generated by the operation of the processor (122) for the operation of the processor (122). The memory (121) may be composed of volatile memory, non-volatile memory, or a combination of volatile memory and non-volatile memory. Furthermore, the memory (121) can provide stored data upon the request of the processor (122).

[0057] The processor (122) may be configured to implement the procedures and / or methods proposed in the present invention. The processor (122) controls the overall operations of the electronic device (120). For example, the processor (122) writes and reads data to and from memory (121). Additionally, the processor (122) receives information through an input device (123). Additionally, the processor (122) outputs information through an output device (140). The processor (122) may include at least one processor.

[0058] The input device (123) is connected to the processor (122) and can input information, etc. The input device (123) may include a touch display, a keypad, a keyboard, an information input module, etc. According to one embodiment, the electronic device (120) may further include a transceiver, and the input device (123) can input information, etc. received from another device connected to a wired / wireless communication network through the transceiver.

[0059] The output device (124) is connected to the processor (122) and can output information, etc. in the form of video / audio, etc. The output device (124) may include a display, a speaker, an information output module, etc. According to one embodiment, the electronic device (120) may further include a transceiver, and the output device (124) can transmit and output information, etc. to another device connected to a wired / wireless communication network through the transceiver.

[0060] Although not illustrated in FIG. 3, according to one embodiment, the electronic device (120) may further include a transceiver. The transceiver is connected to the processor (122) and transmits and / or receives a signal. All or part of the transceiver may be referred to as a transmitter, a receiver, or a transceiver. The transceiver may support at least one of various wireless communication standards, such as the IEEE (Institute of Electrical and Electronics Engineers) 802.xx system, IEEE Wi-Fi system, 3GPP (3rd generation partnership project) system, 3GPP LTE (long term evolution) system, 3GPP 5G NR (new radio) system, 3GPP 6G system, 3GPP2 system, and Bluetooth, which are wired and wireless access systems.

[0061]

[0062] FIG. 4 illustrates a method of operation of an electronic device according to various embodiments of the present invention.

[0063] An embodiment of FIG. 4 illustrates an example of a method performed by an electronic device for providing information on a suspected bladder cancer site from a cystoscope. The electronic device includes an input device, an output device, a processor, and a memory.

[0064] In step S401, the electronic device receives a cystoscope image of the bladder taken by the cystoscope via the input device.

[0065] In step S402, the electronic device generates a preprocessed image by the processor by performing data preprocessing on the cystoscope image, including noise removal, color adjustment, and brightness adjustment.

[0066] In step S403, the electronic device uses a CNN (convolutional neural network)-based backbone model stored in the memory to extract feature vectors from multiple channels from the preprocessed image by the processor.

[0067] In step S404, the electronic device detects a suspected bladder cancer area from the preprocessed image based on the feature vectors using the regional proposal network (RPN) stored in the memory by the processor.

[0068] In step S405, the electronic device determines a confidence score regarding whether the suspected bladder cancer site is a malignant tumor, and the processor determines whether the suspected bladder cancer site is a malignant tumor based on a comparison of the confidence score and a set threshold value.

[0069] In step S406, the electronic device marks a boundary line on the outer edge of the suspected bladder cancer area in the preprocessed image, and the processor generates a processed image in which the suspected bladder cancer area of ​​the preprocessed image is marked with a color code mask depending on whether the suspected bladder cancer area is a malignant tumor.

[0070] In step S407, the electronic device outputs the processing image by the output device.

[0071] According to various embodiments of the present invention, the CNN may be an artificial neural network that repeatedly applies one or more of a convolution operation, a pooling operation, and an activation function to extract features of an image.

[0072] According to various embodiments of the present invention, the RPN may be a module that uses anchor boxes, a classifier, and a regression unit to suggest regions within an image where an object is likely to exist.

[0073] According to various embodiments of the present invention, the loss function may be a function that quantifies the difference between the model's predicted value and the actual value to provide an error signal for model learning.

[0074] According to various embodiments of the present invention, the CNN may be a deep learning model that learns spatial characteristics and patterns of image data. The backbone model may be a network responsible for extracting features of the preprocessed image based on the structure of the CNN. The multiple channels may be channels for analyzing the preprocessed image individually by separating it into feature maps for different information regarding color, brightness, and texture. The preprocessed image may be separated into the multiple channels by the backbone model. The multiple channels may be analyzed by the CNN.

[0075] According to various embodiments of the present invention, the CNN may be configured to hierarchically extract high-level features of the shape, size, and texture of the tumor from low-level features of edges, points, and lines. The last layer among the plurality of layers included in the CNN may be configured to integrate the feature maps of the multi-channels to generate a single final feature vector, which is numerical data. The final feature vector may include information related to the color, size, and texture of the malignant tumor. The feature maps and the final feature vector may be used by the RPN to detect the suspected bladder cancer site and to determine whether the suspected bladder cancer site is the malignant tumor.

[0076] According to various embodiments of the present invention, the CNN may be configured to generate the final feature vector by extracting channel-independent intermediate feature vectors for the multiple channels, removing the remaining information excluding tumor-related information for each channel from the intermediate feature vectors through filtering, and then integrating them.

[0077] According to various embodiments of the present invention, the CNN may be configured to acquire information on unique feature vector patterns for normal tissue, benign tumors, and malignant tumors through a pre-training process. Whether the suspected bladder cancer site is a malignant tumor may be determined based on a comparison of the final feature vector and the feature vector pattern.

[0078] According to various embodiments of the present invention, the RPN may be configured to detect the suspected bladder cancer site, which is a region of interest (RoIs), from the preprocessed image. The RPN may be configured to determine proposal regions for the suspected bladder cancer site based on the final feature vector generated from the CNN-based backbone model.

[0079] According to various embodiments of the present invention, the RPN may be configured to set grid-shaped anchor boxes covering the entire preprocessed image, determine the probability that each anchor box for each part of the preprocessed image corresponds to a tumor site by comparing the feature maps with the anchor boxes, continuously perform initial detection and fine detection using a coarse-to-fine detection structure, adjust the boundaries between the anchor boxes and the actual tumor through bounding box regression, remove duplicate regions in the preprocessed image using a non-maximum suppression (NMS) algorithm, and determine the candidate regions by leaving only the most reliable candidate regions in the preprocessed image.

[0080] According to various embodiments of the present invention, the determined candidate regions may be the suspected bladder cancer sites.

[0081] According to various embodiments of the present invention, the candidate regions determined by the RPN may be transmitted to the header of the CNN. The header of the CNN may be an output layer configured to detect and classify the malignant tumor based on the CNN-based backbone model and the feature maps output by the RPN. The confidence score of the probability that the candidate regions correspond to the malignant tumor, and the classification of whether the candidate regions are the malignant tumor, the benign tumor, or normal tissue, may be determined by the header of the CNN. Whether each candidate region is the malignant tumor may be determined based on whether the confidence score is higher than the threshold value.

[0082] According to various embodiments of the present invention, based on information related to the malignant tumor determined by the headers of the RPN and the CNN, the processed image may be output in the form of a three-dimensional image so that the depth, volume, and structure of the tumor are intuitively expressed for the suspected bladder cancer site. In the processed image, the size and malignancy of the malignant tumor may be visualized and output using the color code mask.

[0083] According to various embodiments of the present invention, the CNN may be configured to calculate a final loss value by using a standard loss function (Categorical Cross-Entropy, Smooth L1, Binary Cross-Entropy) when performing the pre-training process, along with (a) an adaptive loss function that improves the accuracy of discrimination of suspected regions by dynamically adjusting loss weights according to the confidence score, and (b) a shape regularization loss function to minimize the shape distortion of the tumor.

[0084] According to various embodiments of the present invention, the final loss value may be used to adjust the CNN during the pre-training process to minimize the difference between the predicted value and the actual value. The CNN may be configured to learn areas with a high suspicion of bladder cancer more accurately by adjusting the loss weight according to the confidence score for areas suspected of bladder cancer. The processed image may be output with the distorted shape of the malignant tumor corrected through shape regularization for the areas suspected of bladder cancer.

[0085] According to various embodiments of the present invention, the processing image may be configured to concisely display the diagnostic result for the suspected bladder cancer site by grouping multiple suspected tumor sites at the same location based on the confidence score for the suspected bladder cancer site.

[0086] According to various embodiments of the present invention, an electronic device comprises an input device; an output device; a transceiver; a processor; and a memory, wherein the memory stores instructions for operations performed by the processor, and the operations are configured to perform a method of operation of the electronic device according to the embodiments of FIG. 4.

[0087] According to various embodiments of the present invention, a computer program is provided that is configured to perform the operation method of an electronic device according to the embodiments of FIG. 4 and is recorded on a computer-readable storage medium.

[0088]

[0089] FIG. 5 illustrates the structure of a multi-layer perceptron (MLP) for implementing a deep learning algorithm according to various embodiments of the present invention.

[0090] The bladder cancer suspected site analysis model according to various embodiments of the present invention may be configured with an MLP structure according to the embodiment of FIG. 5. The bladder cancer suspected site analysis model may be implemented with various architectures such as a Convolutional Neural Network (CNN), DenseNet, U-net, GoogLeNet, and Generative Adversarial Network. For example, when the bladder cancer suspected site analysis model is implemented as a CNN, the bladder cancer suspected site analysis model performs a learning process to adjust the connection weights of the artificial neural network using training data.

[0091] Deep learning is one of the technologies that has recently emerged in the field of machine learning, and it is a neural network composed of multiple hidden layers and multiple hidden units contained within them. When low-level features are input into a deep learning model, these features pass through multiple hidden layers and are transformed into high-level features that can better explain the problem to be predicted. Since this process does not require expert prior knowledge or intuition, subjective factors in feature extraction can be eliminated, making it possible to develop a model with higher generalization capabilities. Furthermore, because feature extraction and model building in deep learning are structured as a single set, it has the advantage of forming a final model through a simpler process compared to existing machine learning theories.

[0092] A multi-layer perceptron (MLP) is a type of artificial neural network (ANN) based on deep learning that consists of multiple nodes. Each node is a neuron similar to the connection patterns of animals that uses non-linear activation functions. This non-linear property enables linear separation of inseparable data.

[0093] Referring to FIG. 5, the artificial neural network (400) of the MLP model according to various embodiments of the present invention is composed of an input layer (410), a plurality of hidden layers (430), and an output layer (450).

[0094] Input data, such as multiple factors related to the cystoscope image, is input to the nodes of the input layer (410). Here, the multiple factors (411) related to the cystoscope image correspond to the basic features (low-level features) of the deep learning model.

[0095] In the nodes of the hidden layer (430), calculations based on input factors are performed. The hidden layer (430) is a layer in which units defined by multiple nodes formed by combining multiple factors (411) related to the cystoscope image are stored. The hidden layer (430) may be composed of multiple hidden layers as shown in FIG. 5.

[0096] For example, when the hidden layer (430) is composed of a first hidden layer (431) and a second hidden layer (433), the first hidden layer (431) is a layer in which first units (432), defined as multiple nodes formed by combining multiple factors (411) related to the lowest feature, the cystoscope image, are stored, and the first units (432) correspond to the upper features of the multiple factors related to the cystoscope image. The second hidden layer (433) is a layer in which second units (434), defined as multiple nodes formed by combining the first units of the first hidden layer (431), are stored, and the second units (434) correspond to the upper features of the first units (432).

[0097] The nodes of the output layer (450) represent the calculated prediction results. The output layer (450) may be equipped with a plurality of prediction result units (451). Specifically, the plurality of prediction result units (451) may consist of two units: a True unit and a False unit. Specifically, the True unit is a prediction result unit that signifies that the cystoscope image is related to a specific feature, and the False unit is a prediction result unit that signifies that the cystoscope image is not related to a specific feature.

[0098] Each weight is assigned to the connection between the second units (434) included in the second hidden layer (433), which is the last layer of the hidden layer (430), and the prediction result units (451). Based on these weights, the relationship with specific features of the cystoscope image is predicted.

[0099] For example, if one of the second units (434) predicts that the cystoscope image will be related to a specific feature, it will be connected to the true unit and the false unit, respectively, and a weight with a positive value will be assigned to the connection with the true unit, and a weight with a negative value will be assigned to the connection with the false unit. Conversely, if one of the second units (434) predicts that the cystoscope image will not be related to a specific feature, it will be connected to the true unit and the false unit, respectively, and a weight with a negative value will be assigned to the connection with the true unit, and a weight with a positive value will be assigned to the connection with the false unit.

[0100] Multiple connection lines will be formed between multiple second units (434) and the true unit. If the total sum of the multiple connection lines has a positive value, the multiple factors (411) related to the cystoscope image in the input layer (410) will be predicted as factors related to the cystoscope image with a specific feature. According to one embodiment, whether such a cystoscope image is related to a specific feature may be predicted by comparing the total sum of the multiple connection lines with a preset value.

[0101] The artificial neural network (400) of the MLP model learns by adjusting the learning parameters. According to one embodiment, the learning parameters include at least one of weights and deviations. The learning parameters are iteratively adjusted through an optimization algorithm called gradient descent. Whenever a prediction result is calculated from a given data sample (forward propagation), the performance of the network is evaluated through a loss function that measures the prediction error. Each learning parameter of the artificial neural network (400) is adjusted by gradually increasing in the direction that minimizes the value of the loss function, a process called back-propagation.

[0102]

[0103] Hereinafter, the specific design and structure of an AI model according to various embodiments of the present invention will be described.

[0104] Figure 6 illustrates an example of the structure of an AI model based on a two-stage model.

[0105] The first step of the AI ​​model is to process the input cystoscope images using a ResNet-based 101-layer backbone model. Unlike a standard ResNet structure, this backbone is designed with separate convolution layers for each channel. This allows each channel to individually extract various features from the cystoscope images, which are then filtered through a refinement process to retain only the specific features of the tumor. This process helps to analyze the complex features of the cystoscope images more effectively.

[0106] The extracted feature maps are processed through the Region Proposal Network (RPN), and potential regions of interest (tumor candidate regions) are detected. During this process, score and offset information are generated, and the degree to which the candidate regions align with the model's interests is evaluated. Subsequently, the candidate regions are aligned via RoI Align and converted to a fixed size, preparing them for use in subsequent steps.

[0107] The aligned RoIs are passed to the Meta-Learner Predictor Head, where two main tasks are performed. The first is to classify candidate regions and define boundaries by predicting the class and bounding box of each RoI. The second is to perform segmentation of the corresponding regions to isolate the precise shape of the tumor. These segmentation results contribute to a more precise diagnosis of the tumor's location and shape.

[0108] The model's final output is the result of marking the tumor region on the input image in the form of bounding boxes and segment masks. The bounding boxes indicate the approximate location of the tumor, while the segment masks visualize the specific boundaries and shape of the tumor. Through this process, the AI ​​model supports the diagnosis of tumors in cystoscopy images with high accuracy.

[0109]

[0110] Figure 7 illustrates an example of a ResNet block and a RexNext block.

[0111] ResNet blocks are the basic components of a residual network and are structures designed to solve the vanishing gradient problem during the learning process through connections between inputs and outputs. The ResNet block illustrated on the left side of Fig. 7 uses a "bottleneck structure" and consists of three convolutional layers. The first layer reduces the number of channels using a 1×1 filter, the second layer extracts spatial information using a 3×3 filter, and the third layer restores the number of channels using a 1×1 filter. This configuration increases the computational efficiency of the model while minimizing performance degradation.

[0112] The ResNeXt block illustrated on the right side of Fig. 7 is an extended version of the ResNet block that increases the number of parallel paths by introducing the concept of "cardinality." In this structure, with a cardinality set to 32, each path follows the same bottleneck structure and processes the inputs separated into 4 channels independently before summing the results. This process contributes to improving expressiveness without significantly increasing the complexity of the model.

[0113] Cardinality is an important hyperparameter of ResNeXt, representing the number of parallel paths. Increasing cardinality allows for the learning of diverse features per path, thereby enhancing the network's expressiveness. This enables more effective learning of complex patterns in images, which is particularly advantageous in complex medical image processing, such as cystoscopy images. ResNeXt blocks provide better performance while maintaining nearly the same level of computational complexity.

[0114] Both ResNet and ResNeXt blocks are used to extract features from cystoscope images and are combined with the Region Proposal Network (RPN) to accurately estimate the location and size of tumors. Compared to ResNet blocks, ResNeXt blocks can learn a wider range of features through a parallel structure, providing improved results in terms of accuracy and efficiency. This design demonstrates superior performance in the analysis of complex medical data.

[0115]

[0116] Figure 8 illustrates an example of an RPN structure for cutting out a suspected tumor region and passing it to the final header Neural Networks.

[0117] Faster R-CNN's Region Proposal Network (RPN) serves as the first stage of the object detection network, responsible for proposing regions of interest (areas suspected of being tumors) from input images. The RPN is composed of a complete convolutional neural network that generates fixed-length feature vectors from the input images. These vectors are subsequently passed to the classification and regression layers. Because the RPN shares image convolution computations with Faster R-CNN's detection network, it significantly improves the speed of region proposal.

[0118] RPN generates feature vectors for each small window using a sliding window method on the input convolutional feature map. These feature vectors are passed to an intermediate layer and compressed into a 256-dimensional representation. Subsequently, k anchor boxes of various sizes and ratios are applied to each window to provide a basis for predicting the presence and location of objects.

[0119] The output of the RPN is divided into two main tasks. The first is to determine the presence of objects (2k scores) for each anchor box through a classification layer (cls layer). The second is to predict coordinate information (4k coordinates) to adjust the precise boundaries of each anchor box through a regression layer (reg layer). Through this process, the objectness score and coordinates of each region are calculated simultaneously.

[0120] RPN is efficient in the region proposal process because it shares the computation of image convolutional features with Faster R-CNN's detector network. This accelerates the region proposal speed and improves the model's overall performance. Finally, the regions of interest proposed by RPN are used to learn the precise location and size of tumors and are passed to the final header neural network for further detailed classification and detection.

[0121]

[0122] Figure 9 illustrates an example of an AI header structure for determining whether a suspected area received via RPN is a malignant tumor and expressing the tumor size.

[0123] The AI ​​header of Mask R-CNN processes the Region of Interest (RoI) received from the Region Proposal Network (RPN) to determine whether a tumor is malignant and to define its boundaries. In the first step, the RoI is converted into a 7×7×256 feature map. It is then reduced to a 1024-dimensional vector through a Fully Connected Layer, and finally, class and bounding box regression values ​​are predicted. This process focuses on determining the presence and location of the tumor.

[0124] One of the key features of the AI ​​header structure is that the mask branch operates in parallel with the class and bounding box prediction branches. The mask branch takes positive RoIs passed from the RPN as input and uses a Fully Convolutional Network (FCN) to predict binary masks. This enables the derivation of the precise shape of the malignant tumor region.

[0125] The input to the mask branch starts with a 14×14×256 feature map, which is processed through four iterations of convolutional layers. Subsequently, the feature map is upsampled to 28×28×256, and finally, a 28×28×80 mask is predicted. Here, each channel represents a binary mask corresponding to a specific class (e.g., tumor type).

[0126] Mask prediction results represent the precise size and shape of malignant tumors and complement bounding box and class prediction results. This structure goes beyond simply determining the presence of a tumor to providing information that allows for the visual confirmation of its specific shape. Such high-resolution mask prediction contributes to diagnosis and treatment planning by providing detailed tumor information crucial for medical image analysis.

[0127]

[0128] AI models according to various embodiments of the present invention use various malignant tumor data for training to detect malignant tumors (CIS, low grade, high grade, etc.). Through this, the model is designed to learn characteristic patterns of malignant tumors, such as size, shape, and color. The training data provides information that enables accurate differentiation of tumor sites in cystoscope images and contributes to improving the accuracy of malignant tumor detection.

[0129] During the training process of the AI ​​model, the location of malignant tumors is masked to learn the entire tumor area. This allows the model to understand not only specific parts of the tumor but also its overall characteristics. Areas such as normal skin tissue, benign tumors, and inflammation are all trained to be classified as normal, so the model is designed to clearly recognize only malignant tumors.

[0130] As cystoscope images pass through the Backbone (CNN), normal skin tissue is filtered out to values ​​close to zero. Conversely, malignant tumors are converted into specific numerical feature vectors and extracted. Through this learning structure, the model can effectively identify the characteristics of malignant tumors, while minimizing unnecessary information. This process significantly improves the accuracy and reliability of tumor detection.

[0131] Suspicious tumor regions extracted through the Region Proposal Network (RPN) undergo a further filtering process in the header. This step ensures that ambiguous areas are definitively removed, allowing only malignant tumors to be extracted. This structure is designed to enable the model to detect malignant tumors with high confidence and plays a crucial role in medical image analysis.

[0132]

[0133] Hereinafter, preprocessing and postprocessing techniques for cystoscope images according to various embodiments of the present invention will be described.

[0134] Cystoscopic images contain various characteristics such as degradation, light intensity, and light wavelength; therefore, it is necessary to compensate for these characteristics and increase the amount of data during the preprocessing stage. Data augmentation techniques such as brightness adjustment, the addition of Gaussian noise, angle transformation, and image resizing are used during preprocessing. In particular, since images processed with specific wavelengths, such as Narrow Band Imaging (NBI), are difficult to reproduce through simple color conversion, as many NBI images as possible are collected and augmented using the same method. This process resolves the problem of insufficient data and provides a rich amount of data necessary for model training.

[0135] To minimize image degradation, cystoscope images are extracted from the DICOM format into PNG, a lossless image format. All images are read and processed as uint8 data types (0–255), and brightness adjustment is performed by adding to or subtracting from the entire image matrix. At this time, to prevent overflow and underflow, values ​​are adjusted so that they do not fall below 0 or exceed 255. Gaussian noise is added by applying a Gaussian filter, and various data are generated by adjusting image size and angle through affine transformation.

[0136] In the post-processing stage, a Confidence Score is calculated based on the results extracted from the AI ​​model, representing the reliability of the diagnostic outcome. This value expresses the probability of similarity to previously trained malignant tumor data; it is used to measure the AUROC and set a cut-off value. Through this process, the presence of a malignant tumor is ultimately determined, and relevant information is provided. The post-processing stage serves to complement the model's prediction results to enhance their reliability.

[0137] Preprocessing and postprocessing directly impact diagnostic accuracy. Without the application of preprocessing and data augmentation techniques, diagnostic accuracy can generally decrease by 5–10%, which can undermine the reliability of patient information provided to physicians. Furthermore, image degradation caused by differences in cystoscope specifications can also be compensated for through preprocessing. In fact, testing cystoscopy data from private clinics rather than university hospitals demonstrated high accuracy, successfully detecting most malignant tumors. As such, preprocessing and postprocessing are essential steps in medical image analysis, contributing to improved diagnostic accuracy and reliability.

[0138]

[0139] Hereinafter, a method for automatically setting and displaying boundaries of suspected areas according to various embodiments of the present invention will be described.

[0140] The AI ​​model provides the location and size of detected malignant tumors in the form of xy coordinates, and based on this, sets the tumor's bounding box or polygon (mask). In particular, after determining whether a tumor is malignant using the Confidence Score, it displays the tumor's location to the user as only an outline so that it can be easily identified. However, the bounding box is not displayed separately; information is provided by emphasizing only the outline.

[0141] If there are overlapping parts of masks during the tumor display process, the overlapping masks are integrated, and only the boundary coordinates of the integrated area are extracted. This enables a clear and concise display without redundant information, allowing users to more easily understand the location of the tumor. This process prevents information duplication and enhances the efficiency of visual representation.

[0142] Masks are fundamentally composed of binary data consisting of 0s and 1s, and each mask result is merged through an OR operation. This method integrates based on the parts where the mask is 1 (areas where tumors are detected). By combining overlapping masks through the OR operation, a single integrated boundary can ultimately be derived. This process is essential for concisely organizing the AI ​​detection results and providing them to the user.

[0143] Contours are extracted based on boundary points in the mask's binary values ​​that transition from 0 to 1. This contour information accurately represents the boundaries of the tumor, allowing users to intuitively understand its location and size. This boundary display method enhances diagnostic efficiency and accuracy, helping medical professionals quickly assess the tumor's condition and take appropriate action. As such, automatic boundary setting and display technology plays a key role in the tumor diagnosis process.

[0144]

[0145] Hereinafter, a method for training an AI model and optimizing model performance according to various embodiments of the present invention will be described.

[0146] FIG. 10 illustrates an example of a structure for training a deep learning model.

[0147] Imaging data from bladder cancer patients was used to train the deep learning model. The data was collected from a diverse range of patients and various forms of bladder cancer, and the quality was enhanced by having medical experts process the data directly. This dataset composition was designed to ensure that the model maintains generalized performance even in real-world diagnostic settings.

[0148] The collected data was detailed by labeling according to lesion type into benign tumor, CNI (Chronic Non-specific Inflammation), CIS (Carcinoma In Situ), metastatic cancer, low grade, and high grade. This process helps the model clearly learn the characteristics of each lesion. After the labeling, the model was designed to effectively remember and distinguish lesion-specific features by conducting training on each label.

[0149] During the training process, weights were adjusted so that each label was separated in the embedding space. This allows the model to learn by isolating the unique features of each lesion, enabling it to distinguish between malignant and benign tumors more effectively. The optimization algorithm and loss function were used with a focus on simultaneously improving sensitivity and specificity.

[0150] The AI ​​model is designed to avoid overfitting to specific data by training on multi-institutional data, thereby enhancing its applicability in various clinical settings. It improves diagnostic efficiency through the accurate classification and detection of lesions and strengthens diagnostic reliability based on high sensitivity and specificity. This structure contributes to the model operating stably in real-world medical environments.

[0151]

[0152] Figure 11 illustrates an example of a structure for training a deep learning model.

[0153] Afterwards, the AI ​​model was trained by dividing the samples into three categories—normal tissue, benign tumor, and malignant tumor—once again so that only malignant tumor samples would be combined in the embedding space.

[0154] During the initial training phase of the AI ​​model, data was classified into normal tissue, benign tumors (CNI), and malignant tumors (CIS, metastatic, low-grade, high-grade). Subsequently, to enable the AI ​​model to distinguish malignant tumors more clearly, additional training was conducted by reclassifying the data into three groups—normal, benign, and malignant—so that only malignant tumor samples were aggregated and analyzed within the embedding space. This resulted in a further improvement in the model's ability to distinguish malignant tumors.

[0155] Three main loss functions are used in the training process of AI models. The first is the classification loss for learning labels, which is calculated using Categorical Cross-Entropy. The second is the bounding box loss for learning bounding box coordinates, which utilizes the Smooth L1 loss function. The third is the mask loss for identifying malignant tumor locations, which is calculated based on Binary Cross-Entropy. These loss functions are designed to maximize model performance by separating and integrating their respective training objectives.

[0156] The three loss functions mentioned above are calculated individually and then summed to derive a final loss function. This final loss function determines the overall learning direction of the model and enables it to accurately learn the differences between normal tissue, benign tumors, and malignant tumors. Through this, the model learns more stably and can demonstrate generalized performance even in diagnostic environments.

[0157] Reclassification learning and loss function optimization contribute to simultaneously improving the model's sensitivity and specificity. A model capable of accurately distinguishing the location and characteristics of malignant tumors significantly enhances diagnostic accuracy and supports medical professionals in efficiently detecting and analyzing tumors in clinical settings. As such, the structural design of the learning process plays an essential role in strengthening the reliability and practicality of the model.

[0158]

[0159] Hereinafter, performance evaluation criteria for an AI model in the AI ​​model-based diagnosis process for a suspected bladder cancer area according to various embodiments of the present invention will be described.

[0160] Diagnosis Result Confirmed / Confirmed Information Medical Staff Diagnosis Result + / - Read Result Positive Negative Total AI-assisted Read Test Device Positive 473(97.3%) 111(8.0%) 584 Negative 13(2.7%) 1293(92.0%) 1306 Total 486(25.7%) 1404(74.3%) 1890(100.0%) Sensitivity(%)[95% CI] 1) [95.5 to 98.4][95% CI] 2) [95.6 to 98.5][95% CI] 3) [95.5 to 98.6] Specificity (%) [95% CI] 1) [90.6to93.4][95% CI] 2) [90.6to93.4][95% CI] 3) [90.6 to 93.5] Reading Accuracy (%) [95% CI] 1) [92.2 to 94.5][95% CI] 2) [92.3 to 94.5][95% CI] 3) [92.2 to 94.5] Average of Dice Factors [95% CI] 4) 90.3% [89.1 to 91.4]

[0161] Note:

[0162] 1) Wald's Confidence Interval with Continuity Correction Method

[0163] 2) Confidence Interval by Jeffreys-Perks method

[0164] 3) Confidence Interval by Clopper-Pearson's method

[0165] 4) Confidence Interval for one sample mean

[0166]

[0167] - Refer to similar research papers for the basis of using the relevant evaluation indicators.

[0168] 1) Ikeda, Atsushi, et al. "Support system of cystoscopic diagnosis for bladder cancer based on artificial intelligence." Journal of endourology 34.3 (2020): 352-358.

[0169] 2) Shkolyar, Eugene, et al. “Augmented bladder tumor detection using deep learning.” European urology 76.6 (2019): 714-718.

[0170] 3) Misgana Negassi, et al. “Application of artificial neural networks for automated analysis of cystoscopic images: a review of the current status and future prospects” World Journal of Urology volume 38, pages2349-2358 (2020).

[0171]

[0172] The AI ​​model evaluated sensitivity and specificity in bladder tumor diagnosis based on malignant tumors (benign) and normal (negative). Sensitivity represents the ability to detect malignant tumors, while specificity represents the ability to correctly classify normal tissues. [Table 1] shows that the AI ​​model provides high diagnostic performance, with sensitivity and specificity each exceeding 95.5%. These indicators play an important role in evaluating the reliability of malignant tumor diagnosis in a medical setting.

[0173] Diagnosis accuracy represents the proportion of correct classifications in the total diagnosis, and the Dice Score was used to verify the reliability of the tumor location and size provided by the model. The Dice Score evaluates how similar the tumor region detected by the AI ​​model is to the actual tumor region and, equivalent to the F1 score, recorded 90.3%. This suggests that the AI ​​model can accurately detect the size and location of the tumor.

[0174] [Table 1] enhanced the reliability of performance indicators using various confidence interval calculation methods, such as Wald's method, Jeffreys-Perks method, and Clopper-Pearson's method. These confidence intervals present the statistical variability of model performance and assist medical professionals in interpreting the model's diagnostic results. This methodology ensures that the model's evaluation results are reproducible and reliable.

[0175] Research papers related to bladder cancer diagnosis were referenced as the basis for evaluating model performance. For example, studies by Ikeda et al. and Shkolyar et al. analyzed the sensitivity and specificity of AI-based bladder cancer diagnostic models, demonstrating that AI models can be utilized as useful tools in clinical diagnosis. These performance evaluation metrics are crucial for assessing the model's applicability in real-world medical settings and provide high reliability to both medical professionals and patients.

[0176]

[0177] Hereinafter, performance evaluation criteria for an AI model in the AI ​​model-based diagnosis process for a suspected bladder cancer area according to various embodiments of the present invention will be described.

[0178] FIG. 12 illustrates an example of a system for diagnosing a suspected bladder cancer area through a cystoscope image according to various embodiments of the present invention.

[0179] The system according to the present invention consists of three main components. First, an input device that receives a cystoscope (DICOM format) image as input selects a DICOM folder and uploads the image. Second, a processor converts the uploaded image and runs an AI model to generate lesion analysis and results. Third, an output device visualizes the analysis results and delivers them to the user, helping medical staff intuitively understand the suspected tumor area.

[0180] The software begins by selecting a DICOM folder and converts the endoscopic image data into a readable format. The DICOM images undergo an endoscopic image conversion process to be processed into analyzable images. This conversion step serves to correct image quality and prepare necessary data so that the AI ​​model can analyze them accurately. The converted images are used for AI analysis in the next step.

[0181] The transformed images are analyzed by an AI model during the deep learning interpretation stage. The AI ​​model determines the presence of lesions based on the input data and outputs the location and size of the lesions in the form of a Bounding Box, Polygon Mask, and Confidence Score. At this stage, the AI ​​accurately distinguishes lesions and improves the reliability of the diagnostic results by providing a confidence score for each lesion.

[0182] The AI ​​interpretation results are visually displayed to the user after undergoing a result processing step. The results highlight the lesion location using a polygonal mask and display the Confidence Score alongside it, enabling medical professionals to clearly identify the lesion. In addition, an interface is provided on the screen that intuitively displays the suspected tumor area, aiding user understanding and supporting rapid diagnostic decisions.

[0183] The system according to various embodiments of the present invention ensures high reliability and efficiency in bladder cancer diagnosis by systematically designing all processes from input to output. In particular, the design based on performance evaluation criteria for AI models maintains diagnostic accuracy and provides faster and more accurate results than existing diagnostic methods. By supporting medical professionals in improving diagnostic accuracy and utilizing the system efficiently in clinical settings, this system sets a new standard for bladder cancer diagnosis.

[0184]

[0185] The name of the AI ​​model according to various embodiments of the present invention is INF-M01. However, this is merely illustrative, and it is evident that the essence of the AI ​​model remains unchanged regardless of what name it is called.

[0186] Hereinafter, the differences between the INF-M01 AI model according to various embodiments of the present invention and existing technologies will be explained.

[0187] The INF-M01 AI model according to various embodiments of the present invention has overcome the limitations of existing manual diagnostic methods. In the current diagnostic environment of urology, there is a lack of methods to clearly differentiate between normal mucosa, inflammatory tissue, and borderline tumors (CIS). Although surgical biopsy is possible for definitive diagnosis, it entails surgical risks, and even experienced medical professionals face difficulties due to unclear diagnostic criteria. INF-M01 is an AI model designed to overcome these limitations; it analyzes cystoscopic images to provide a clearer diagnosis and serves to assist medical professionals in their judgment.

[0188] The INF-M01 AI model according to various embodiments of the present invention possesses training data and high accuracy. INF-M01 was trained on various types of tumors based on data from approximately 1,300 patients and enhanced reliability by using high-quality data directly processed by medical specialists. Furthermore, through multi-center testing utilizing data from other university hospitals, it maintained an accuracy of over 90%, thereby demonstrating superior performance compared to existing manual methods and other AI diagnostic studies. This indicates that INF-M01 can be utilized with high reliability in actual clinical settings.

[0189] The INF-M01 AI model according to various embodiments of the present invention is capable of real-time identification and visualization of tumor distribution. INF-M01 goes beyond merely determining the presence or absence of cancer in cystoscopy images to specifically visualize the area where the tumor is located and its distribution pattern. The AI ​​model can clearly indicate where and in what form malignant tumors are distributed in cystoscopy images, and since it can be processed in real time, it contributes immediately to the diagnostic process of physicians. This function is a diagnostic assistance feature that cannot be provided by existing methods.

[0190] The INF-M01 AI model according to various embodiments of the present invention can enhance clinical utility and efficiency. INF-M01 significantly improves efficiency compared to existing manual diagnostic methods by utilizing cystoscopic images. By increasing diagnostic speed and providing the precise location and morphology of lesions, it supports medical professionals in making quick and accurate judgments. Furthermore, through productization and software development, INF-M01 is provided in a form that can be immediately applied in clinical settings, thereby increasing diagnostic efficiency and providing practical assistance in patient management.

[0191]

[0192] Hereinafter, an evaluation of the diagnostic performance of the INF-M01 AI model according to various embodiments of the present invention is described.

[0193] Diagnostic Performance Evaluation of AI-based Software INF-M01 for Detecting Suspected Bladder Cancer Sites: Using Cystoscopic Images

[0194] Abstracts

[0195] Background / Objective: This study aimed to evaluate the accuracy of INF-M01, an AI-based software for diagnosing suspected bladder tumor sites using cystoscope images. Additionally, it sought to assess INF-M01's ability to distinguish and mark suspected bladder cancer sites within whole cystoscope images.

[0196] Methods: A randomized retrospective clinical trial was conducted using 1,890 cystoscopy images (486 from bladder cancer patients and 1,404 from healthy individuals) collected from three institutions. The images were divided into a total of five sets, A through E, each containing 1,890 photographs. INF-M01 analyzed the images in Set A to evaluate sensitivity, specificity, and accuracy. Sets B through E were analyzed by INF-M01 and four urologists to mark areas suspected of bladder tumors. The ability to distinguish bladder tumors was compared using the Dice count.

[0197] Results: The AI ​​showed a sensitivity range of 0.952 to 0.977, specificity of 0.882 to 0.962, and accuracy of 0.904 to 0.959. The average Dice coefficient of the AI ​​was 0.873 to 0.927, compared to 0.903 to 0.963 for clinical specialists. The AI ​​achieved a level similar to that of specialists in its ability to distinguish between normal bladder tissue and bladder tumors.

[0198] Conclusion: INF-M01 demonstrated satisfactory accuracy in diagnosing bladder tumors. Furthermore, it exhibited performance comparable to that of a urologist in its ability to distinguish and mark tumor sites from normal bladder tissue. These results suggest that the AI ​​possesses diagnostic capabilities and implies potential clinical utility in assisting urologists.

[0199]

[0200] Introduction

[0201] Bladder cancer (BCa) is the tenth most common malignant tumor worldwide, and its diagnosis and treatment are significantly challenging due to its high recurrence rate and the need for continuous monitoring. White light cystoscopy (WLC) is currently used as the standard for the diagnosis and surveillance of BCa. Patients with suspected lesions identified via WLC generally undergo transurethral resection of bladder tumors (TURBT) to confirm the pathological diagnosis and staging. Non-muscle invasive BCa, which accounts for about 75% of cases, can often be managed with TURBT (2, 3), followed by regular cystoscopic surveillance every 36 months to detect recurrence (4).

[0202] White light cystoscopy (WLC) is widely used but has several significant limitations. In particular, it is highly likely to miss flat lesions such as carcinoma in situ (CIS) or very small tumors, resulting in a misdiagnosis rate of 20–30% (5) and an incomplete resection rate of up to 50% (6). These diagnostic limitations lead to high rates of early recurrence and progression in BCA patients. Although cystoscopy improvement techniques such as narrowband imaging and photodynamic diagnosis have been developed to address these limitations, their adoption is limited because they require specialized equipment and additional training (5).

[0203] In recent years, artificial intelligence (AI) has established itself as an innovative technology in the field of medical diagnosis, providing the ability to automatically extract and analyze complex image data. AI applications have shown promising results in various fields, such as radiology, dermatology, and gastroenterology, by improving diagnostic accuracy and efficiency (7-9). AI has also been studied in the field of bladder cancer diagnosis and has demonstrated excellent performance in detecting bladder cancer and setting or grading stages using CT (computed tomography), MRI (magnetic resonance imaging), hematoxylin and eosin staining, or urine cytology images (10-13). Additionally, numerous studies have been conducted on the ability of AI to accurately diagnose BCA based on cystoscopy images, and high accuracy has been demonstrated in most of these studies (14-23).

[0204] Following this trend, we developed INF-M01, an AI-based software that automatically marks the boundaries of suspected bladder cancer areas in cystoscope images. In this clinical trial, we aimed to evaluate the performance of INF-M01 in automatically analyzing cystoscope images to detect and diagnose suspected bladder cancer sites.

[0205]

[0206] Materials and Methods

[0207] Sample Selection Criteria

[0208] This study was designed as a randomized retrospective confirmatory clinical trial. Cystoscopic images of patients who underwent cystoscopy to differentiate bladder tumors or other suspected bladder conditions were reviewed. The inclusion criteria for the normal group were as follows: (1) male patients who underwent cystoscopy to evaluate the treatment response for benign prostatic hyperplasia; (2) female patients who underwent cystoscopy due to urinary symptoms; and (3) patients who underwent cystoscopy to differentiate microscopic hematuria. None of the patients in the normal group were diagnosed with bladder tumors. The inclusion criteria for the cancer group were as follows: (1) patients in whom bladder cancer was confirmed via biopsy following cystoscopy due to suspected bladder tumors on imaging tests (ultrasound / CT / MRI); and (2) patients who were previously diagnosed with bladder cancer and confirmed to have bladder cancer via biopsy following cystoscopy for follow-up.

[0209] Exclusion criteria were established to ensure the integrity of the samples. The general exclusion criteria for the cancer group are as follows: (1) patients with other cancers associated with the bladder, such as rectal or cervical cancer; (2) pregnant women; (3) patients who have undergone bladder imaging using equipment other than designated endoscopic equipment; and (4) patients who have not received a histopathological diagnosis through transurethral resection of a bladder tumor. The specific exclusion criteria for the normal group are as follows: patients with benign prostatic hyperplasia showing abnormal findings on cystoscopy; patients with acute cystitis; patients in whom malignancies of the urinary tract are detected on CT or MRI; and patients suspected of bladder deformation due to pelvic radiation therapy.

[0210]

[0211] Sample size calculation

[0212] To calculate the sample size, estimates for sensitivity and specificity were established based on previously published studies and the preliminary results of this study. The weighted average sensitivity of the two studies was 93.6%, with a 95% confidence interval lower bound of 90%. The weighted average specificity was 91.6%, with a 95% confidence interval lower bound of 89.2%. Therefore, in this study, the estimated sensitivity was set at 93.6% to expect a minimum performance of 90% at a 95% confidence level, and the estimated specificity was set at 89.2% to expect a minimum performance of 89.2% at a 95% confidence level. Based on these criteria, the calculated sample size consisted of 486 images of bladder cancer and 1,404 images of normal control subjects. (See below)

[0213]

[0214] The sample size calculation was based on the formula presented in the Ministry of Food and Drug Safety's guidelines for the approval and review of in vitro diagnostic medical devices.

[0215]

[0216] Here, P1 represents the estimated sensitivity (or specificity) of the device, P0 represents the lower bound of the clinical target sensitivity (or specificity) confidence interval, and Z α / 2 is the threshold for Type 1 error, and Z β is the threshold for Type 2 error.

[0217] It was calculated that the required number of images for bladder cancer patients is 486 to achieve a power of 90% at a significance level of 5%. This result is based on an estimated sensitivity P1=0.936, a 95% confidence interval lower bound P0=0.900, a significance level α=0.05, and a power of 1-β=0.90.

[0218]

[0219]

[0220] For normal control groups without bladder cancer, the number of images required to achieve a power of 90% at a significance level of 5% was calculated to be 1,404. This result is based on an estimated specificity of P1=0.916, a 95% confidence interval lower bound of P0=0.892, a significance level of α=0.05, and a power of 1-β=0.90.

[0221]

[0222] Data Collection and Screening

[0223] After obtaining approval from the review committees of the three institutions, cystoscopy images containing the presence of bladder cancer in each image were collected from Yonsei University Severance Hospital (Approval No.: 1-2022-0052), Ewha Womans University Seoul Hospital (Approval No.: SEUMC 2022-10-017), and National Health Insurance Service Ilsan Hospital (Approval No.: NJIMC 2022-09-031). The collected dataset was transferred to Gangnam Severance Hospital (Approval No.: 3-2023-0112) for data screening. Clinicians participating in the development of the reference standard reviewed each image and excluded images with the following issues: (1) low-quality images that were too blurry or dark to be interpreted, and (2) images unrelated to the cystoscopy, e.g., images taken before the endoscope entered the bladder. After screening, each institution provided 486 images of cancer patients and 1,404 images of healthy individuals.

[0224]

[0225] Establishing Reference Standards

[0226] Reference standards were established using selected images, and three urologists participated in their development. Notably, the specialists involved in this process were distinct from those who participated in the interpretation of cystoscopy images. The process of establishing the reference standards involved extracting image frames from DICOM files containing endoscopic images, uploading them to labeling software, and having urologists review and label the images based on the presence of tumors. The labeled images were classified into folders containing images confirmed to be cancerous and normal images. This thorough process ensures the reliability and accuracy of the reference standards used in this study.

[0227] Subsequently, unique serial numbers were assigned to the images, and the interpretation results were recorded along with random numbers generated using Python's random.shuffle. This randomization process is essential for maintaining the independence and unbiased nature of the sample data, ensuring that AI-assisted interpretation and clinical experts could receive the data in a random order.

[0228]

[0229] FIG. 13 illustrates an example of an output screen of an electronic device for diagnosing a suspected bladder cancer area through a cystoscope image according to various embodiments of the present invention.

[0230] Figure 13 shows the operation screen of INF-M01. INF-M01 reads DICOM files from a specified folder to create a list of cystoscopy images. For each image, click the Read Run button to read it, and when the reading is complete, the reading results are displayed on the right. If there is a suspected bladder cancer area, the bladder tumor area is marked with a bounding box.

[0231] AI and Urologists' Interpretation of Cystoscopic Images

[0232] In this clinical trial, five sets of sample images—A, B, C, D, and E—arranged in a random order were prepared and simultaneously provided to four clinicians and AI-assisted software for diagnosing bladder tumors. The analysis was structured as follows: Set A was analyzed solely by AI-assisted diagnostic software called INF-M01, and predicted diagnostic values ​​for each image were recorded. The operation screen and reading process of INF-M01 are briefly described in Fig. 13. Set A was used to evaluate the performance of the AI ​​diagnosing bladder tumors by analyzing cystoscopy images. In Sets B through E, the AI ​​and four urologists analyzed the images to determine the presence of suspected bladder cancer sites and marked their boundaries. This task was performed independently by the four clinicians at different locations under standard clinical conditions, and appropriate environments and rest periods were provided to ensure optimal working conditions. After all analyses were completed, the results were reported to the principal investigator. The results interpreted by INF-M01 in Sets B through E were compared with the results obtained by the specialists. The research team at Gangnam Severance Hospital reviewed and verified the initial results and checked for data manipulation. If no abnormalities were found, the final results were forwarded to an external statistical analyst for further analysis.

[0233]

[0234] Figure 14 illustrates an example of the definition and calculation formula of the die coefficient.

[0235] Statistical Analysis

[0236] The sensitivity, specificity, and accuracy of the AI-assisted diagnostic device and the interpretation by a urologist were evaluated. Additionally, the Dice coefficient was calculated. The Dice coefficient is calculated by comparing the bladder cancer area identified by the AI-assisted diagnostic device or a urologist with the actual bladder cancer area. In the case of multiple lesions, the entire area of ​​all lesions is included in the calculation. As a result, one Dice coefficient is generated per image, with a value ranging from 0 to 1 (Fig. 14). A Dice coefficient of 1 indicates that the area identified by the urologist or the AI-assisted diagnostic device perfectly matches the actual result. Therefore, the Dice coefficient is used as an indicator to quantify how closely the AI ​​prediction matches the actual result.

[0237]

[0238] Results

[0239] Table 2 below shows the results of artificial intelligence interpreting cystoscopy images.

[0240] Interpretation of AICancerNormalTotalCancer479111584Normal1312931306SensitivityWald's with continuity correction method95.5 - 98.4Jeffreys-Perks method95.6 - 98.5Clopper-Pearson's method95.5 - 98.6SpecificityWald's with continuity correction method90.6 - 93.4Jeffreys-Perks method90.6 - 93.4Clopper-Pearson's method90.6 - 93.5AccuracyWald's with continuity correction method92.2 - 94.5Jeffreys-Perks method92.3 - 94.5Clopper-Pearson's method92.2 - 94.5Mean Dice coefficient90.3% (89.1 - 91.4)

[0241] [Table 2] shows the results of the AI ​​model interpreting cystoscopy images. The AI ​​model analyzed a total of 1,890 cases, of which 584 were determined to be cancerous and 1,306 were determined to be normal. Among the cases determined to be cancer, 479 were confirmed to be actual cancer, demonstrating high sensitivity, while among the 1,306 cases determined to be normal, 1,293 were confirmed to be actual normal, showing high specificity.

[0242] Sensitivity refers to the probability of correctly identifying cancer and was calculated using three methods: Wald's, Jeffreys-Perks, and Clopper-Pearson's. All three methods evaluated sensitivity between 95.5% and 98.6%, demonstrating that the AI ​​models possess very high accuracy in detecting cancerous lesions. This result indicates the potential for AI models to be utilized as effective tools in the diagnosis of bladder cancer.

[0243] Specificity refers to the probability of accurately identifying normal tissue, and all three methods were evaluated to be between 90.6% and 93.5%. This indicates that the AI ​​models demonstrated excellent performance in identifying normal tissue. Additionally, overall reading accuracy was measured at between 92.2% and 94.5%, proving that the AI ​​models provide stable and reliable diagnostic results overall.

[0244] The average Dice coefficient was calculated to be 90.3% (confidence interval 89.1–91.4%), indicating that the lesion area detected by the AI ​​model is very similar to the actual lesion area. The Dice coefficient is an important indicator for evaluating the accuracy of lesions, such as the location and size of tumors, demonstrating the precision of the AI ​​model's visual diagnostic results. These results suggest that the AI ​​model can significantly improve diagnostic efficiency in the medical field.

[0245] The interpretation results for one dataset of INF-M01 are shown in [Table 2]. Of the 486 cancer images, 473 were identified as cancer and 13 were interpreted as normal. Of the 1,404 normal images, 1,293 were correctly identified as normal. Sensitivity, specificity, and accuracy were calculated with 95% confidence intervals (CI) using the Wald method with continuity correction, and were 0.973 (0.955 - 0.984), 0.921 (0.906 - 0.934), and 0.934 (0.922 - 0.945), respectively. The mean Dice coefficient was 0.903 (0.891 - 0.914).

[0246]

[0247] The following [Table 3] shows a comparison of urologists and AI interpretations in four cystoscopy image datasets.

[0248] SensitivitySpecificityAccuracyMean Dice coefficientUrologist 10.9560.9960.9860.954AI0.9770.8790.9040.876Urologist 20.9950.9890.9910.963AI0.9830.8820.9080.879Urologist 30.9560.9930.9840.943AI0.9520.9620.9590.927Urologist 40.7770.9840.9310.903AI0.9710.8870.9080.873

[0249] [Table 3] compares the results of cystoscopy data interpretation by urologists and AI models. Sensitivity represents the probability of correctly detecting lesions, and in most cases, the AI ​​model recorded sensitivities similar to or higher than those of urologists. For example, Urologist 1 showed a sensitivity of 0.956, while the AI ​​showed a higher sensitivity of 0.977. This indicates that the AI ​​model is superior to the level of human experts in lesion detection capabilities.

[0250] Specificity is the probability of correctly classifying normal tissue, and urologists generally showed higher specificity than AI models. For example, Urologist 1's specificity was 0.996, while the AI ​​showed somewhat lower performance at 0.879. This suggests that the AI ​​may exhibit slight over-detection during the lesion detection process, indicating an area that requires future improvement.

[0251] Accuracy and the average Dice coefficient represent a comprehensive evaluation of the diagnostic results of AI and physicians. While urologists outperformed AI models in accuracy in most cases, the Dice coefficient indicates that AI models demonstrated performance similar to physicians in visually representing the location and size of tumors. For example, the average Dice coefficient for Urologist 2 is 0.963, and for AI it is 0.879, which is a similar level.

[0252] Although AI models exhibit lower performance than physicians in some metrics, they demonstrate high performance, particularly in sensitivity, showing potential as important tools for lesion detection. The real-time processing capabilities and visual result delivery functions of AI models are advantageous for shortening diagnosis times and assisting physicians' judgment. As such, combined with physicians' experience, AI models can contribute to enhancing the efficiency and reliability of clinical diagnosis.

[0253] [Table 3] shows the interpretation results of four urologists and AI for the remaining four sets. The average Dice coefficients of the urologists ranged from 0.903 to 0.963, while the average Dice coefficients of the AI ​​ranged from 0.873 to 0.927. The sensitivity of the urologists ranged from 0.777 to 0.995, specificity from 0.984 to 0.996, and accuracy from 0.931 to 0.991. The sensitivity of the AI ​​ranged from 0.952 to 0.977, specificity from 0.882 to 0.962, and accuracy from 0.904 to 0.959. It can be confirmed that the accuracy and average Dice coefficients are at similar levels between the clinicians and the AI.

[0254]

[0255] Discussion

[0256] The following [Table 4] shows previous studies on artificial intelligence used in research to detect bladder cancer using cystoscope images.

[0257] AuthorsYearAI algorithm or modelPerformanceEminata et al. (14)2018CNNAccuracy: 0.99Shkolyar et al. (15)2019CNNSensitivity: 0.909Specificity: 0.955Ikeda et al. (16)2020CNNSensitivity: 0.90Specificity: 0.94AUC: 0.98Lorencin et al. (17)2020ANNAUC: 0.99Yang et al. (18)2021CNNAccuracy: 0.969Sensitivity: 0.968Du et al. (19)2021CNNAccuracy: 0.969Sensitivity 0.968Wu et al. (20)2022CNNAccuracy: 0.977Sensitivity: 0.987Specificity: 0.975Yoo et al. (21)2022SVMAccuracy: 0.992Sensitivity: 0.993Specificity: 0.980Zhang et al. (22)2023U-NetDice: 0.83Chang et al. (23)2023CystoNetFor cystoscopySpecificity: 0.988For TURBTSpecificity: 0.954

[0258]

[0259] [Table 4] summarizes previous studies applied to detect bladder cancer using cystoscope images. Most studies evaluated accuracy, sensitivity, and specificity using Convolutional Neural Networks (CNNs). Some studies also utilized other models such as Artificial Neural Networks (ANNs), Support Vector Machines (SVMs), U-Net, and CystoNet, and each algorithm demonstrated high performance in bladder cancer diagnosis.

[0260] Studies utilizing CNNs have established themselves as major research areas and have achieved high accuracy and sensitivity. For example, Eminata et al. (2018) recorded an accuracy of 0.99, and Wu et al. (2022) demonstrated the robust performance of a CNN-based model with an accuracy of 0.977, sensitivity of 0.987, and specificity of 0.975. These results show that CNNs have high potential for analyzing cystoscope images and detecting lesions.

[0261] Lorencin et al. (2020) achieved an AUC of 0.99 using an ANN, which indicates that the model possesses very high reliability in cancer diagnosis. Additionally, Yoo et al. (2022) achieved excellent performance with an accuracy of 0.992, sensitivity of 0.993, and specificity of 0.980 through an SVM-based study. These studies demonstrate that traditional AI algorithms can still be utilized as competitive tools in medical image analysis.

[0262] Zhang et al. (2023) used U-Net to achieve a Dice coefficient of 0.83, focusing on visualizing the precise location and boundaries of lesions. Chang et al. (2023) developed CystoNet, specialized for cystoscopy, achieving a specificity of 0.988 in Cystoscopy and 0.954 in TURBT. These recent studies have demonstrated that the specialized design of models can further improve accuracy and efficiency in specific medical procedures.

[0263] AI has been actively researched in the medical field, and early studies in the field of bladder cancer focused on predicting recurrence or survival using machine learning based on clinical pathology data (10-13). Medical image analysis using AI was first attempted in radiology, dermatology, and gastroenterology (7-9), and AI-assisted diagnostic technology, particularly in the field of thoracic radiology, has been used in clinical practice for several years (24). As image analysis techniques have advanced, numerous studies on machine learning applications for detecting cancer using cystoscopy images have been published ([Table 4]).

[0264] Eminaga et al. (14) developed five deep convolutional neural network (CNN) models by reproducing 479 images using a digital atlas for cystoscopy and generating a total of 18,681 images. Validation results showed that the most accurate model recorded an accuracy of 0.99, demonstrating the potential of deep learning for the diagnostic classification of cystoscopy images. Shkolyar et al. (15) developed a deep learning algorithm called CystoNet, which showed a sensitivity of 90.9% and a specificity of 98.6% in detecting bladder cancer in cystoscopy videos. Similarly, Wu et al. (20) reported a sensitivity of 97.5% and a specificity of 98.3% in an AI system validated with a multi-institutional dataset. This study overcame some of the limitations of previous research by testing AI in various tumor stages and real clinical settings.

[0265] The integration of AI utilizing cystoscope images into the bladder tumor diagnosis process represents a significant advancement that enhances the accuracy and efficiency of tumor detection. Accordingly, we developed INF-M01, an AI-based software capable of analyzing cystoscope images to diagnose bladder tumors and marking suspected tumor sites by distinguishing them from normal bladder mucosa. This study aimed to verify whether INF-M01 possesses reliable diagnostic capabilities for bladder tumors in cystoscope images and whether it can detect areas suspected of bladder cancer. As a result, INF-M01 demonstrated higher diagnostic performance than targeted, with a bladder tumor diagnostic sensitivity of 0.973 and a specificity of 0.921. When comparing the areas marked as suspected bladder cancer by the AI ​​with those marked by urologists, the average Dice coefficient of the AI ​​ranged from 0.873 to 0.927, while that of the specialists ranged from 0.903 to 0.963, demonstrating that the AI's tumor detection capability is comparable to that of urologists. Based on these results, we believe that our AI-based software can assist urologists in analyzing cystoscopy images in clinical practice.

[0266] Compared to previous studies, the INF-M01 device exhibited a sensitivity range that closely matched the sensitivity observed in those studies. This similarity in sensitivity indicates that the AI ​​can effectively identify a high rate of true positives, making it a reliable tool for detecting bladder cancer. Furthermore, regarding the mean Dice coefficient, our results were similar to or slightly improved upon the 0.87 reported in the previous study by Ikeda et al. However, in the readings of sets B through E, the specificity of the INF-M01 was observed to be slightly lower than in the previous study, ranging from 0.882 to 0.962. While this is not the endpoint of our study, when comparing the interpretation results of the AI ​​with those of urologists, the AI's sensitivity in diagnosing bladder tumors was similar to that of urologists, but its specificity was lower. We believe these results suggest that our software needs to slightly improve its diagnostic accuracy in the future.

[0267] Our study has several limitations. One major issue is the relatively low specificity in some datasets, which raises concerns regarding the potential for overdiagnosis and unnecessary interventions. Therefore, when introducing the INF-M01 software into clinical practice, it is crucial to establish mechanisms to review and validate the software's results, particularly when the AI ​​identifies suspected lesions that are difficult for clinicians to immediately identify. Additionally, since the primary objective of this study was to evaluate the AI's detection capabilities and compare them with the performance of urologists, there is a limitation in that it is difficult to assess how much the AI ​​contributes to urologists' bladder cancer diagnostic abilities. To accurately evaluate this, future research needs to investigate how clinicians' detection performance is affected when aided by AI. Another limitation is that the model used in this study was designed to detect bladder cancer in acquired images, making it difficult to apply to real-time clinical situations. As previously mentioned, Chang et al. (23) evaluated the potential for AI integration while performing real-time cystoscopy and transurethral resection of bladder tumors (TURBT) in clinical practice. Future research will need to improve this model for real-time diagnosis of bladder cancer. Nevertheless, this study has strengths in that it compared the accuracy of bladder cancer identification in cystoscopy images between AI and urologists. According to the results, urologists showed higher specificity than AI, while sensitivity was similar. These results are expected to help urologists interpret cystoscopy images.

[0268]

[0269] Conclusion

[0270] INF-M01 demonstrated reliable accuracy in diagnosing bladder tumors using cystoscope images. Furthermore, it exhibited performance comparable to that of a urologist in its ability to mark tumor sites by distinguishing them from surrounding normal bladder tissue. These results suggest that the AI ​​possesses promising diagnostic capabilities and implies potential clinical utility in assisting urologists.

[0271]

[0272] References

[0273] 1. Sung H, Ferlay J, Siegel RL, Laversanne M, Soerjomataram I, Jemal A, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA Cancer J Clin (2021) 71(3):209-49. doi:10.3322 / caac.21660

[0274] 2. Burger M, Catto JW, Dalbagni G, Grossman HB, Herr H, Karakiewicz P, et al. Epidemiology and risk factors of urothelial bladder cancer. Eur Urol (2013) 63(2):234-41. doi:10.1016 / j.eururo.2012.07.033

[0275] 3. erlay J, Steliarova-Foucher E, Lortet-Tieulent J, Rosso S, Coebergh JW, Comber H, et al. Cancer incidence and mortality patterns in Europe: estimates for 40 countries in 2012. Eur J Cancer (2013) 49(6):1374-403. doi:10.1016 / j.ejca.2012.12.027

[0276] 4. Babjuk M, Burger M, Capoun O, Cohen D, Comperat EM, Dominguez Escrig JL, et al. European Association of Urology Guidelines on Non-muscle-invasive Bladder Cancer (Ta, T1, and Carcinoma in Situ). Eur Urol (2022) 81(1):75-94. doi:10.1016 / j.eururo.2021.08.010

[0277] 5. Russo GI, Sholklapper TN, Cocci A, Broggi G, Caltabiano R, Smith AB, et al. Performance of Narrow Band Imaging (NBI) and Photodynamic Diagnosis (PDD) Fluorescence Imaging Compared to White Light Cystoscopy (WLC) in Detecting Non-Muscle Invasive Bladder Cancer: A Systematic Review and Lesion-Level Diagnostic Meta-Analysis. Cancers (Basel) (2021) 13(17). doi:10.3390 / cancers13174378

[0278] 6. Yanagisawa T, Kawada T, von Deimling M, Bekku K, Laukhtina E, Rajwa P, et al. Repeat Transurethral Resection for Non-muscle-invasive Bladder Cancer: An Updated Systematic Review and Meta-analysis in the Contemporary Era. Eur Urol Focus (2024) 10(1):41-56. doi:10.1016 / j.euf.2023.07.002

[0279] 7. Ahn JS, Ebrahimian S, McDermott S, Lee S, Naccarato L, Di Capua JF, et al. Association of Artificial Intelligence-Aided Chest Radiograph Interpretation With Reader Performance and Efficiency. JAMA Netw Open (2022) 5(8):e2229289. doi:10.1001 / jamanetworkopen.2022.29289

[0280] 8. Wongvibulsin S, Yan MJ, Pahalyants V, Murphy W, Daneshjou R, Rotemberg V. Current State of Dermatology Mobile Applications With Artificial Intelligence Features. JAMA Dermatol (2024) 160(6):646-50. doi:10.1001 / jamadermatol.2024.0468

[0281] 9. Uche-Anya E, Anyane-Yeboa A, Berzin TM, Ghassemi M, May FP. Artificial intelligence in gastroenterology and hepatology: how to advance clinical practice while ensuring health equity. Gut (2022) 71(9):1909-15. doi:10.1136 / gutjnl-2021-326271

[0282] 10. Ma X, Hadjiiski LM, Wei J, Chan HP, Cha KH, Cohan RH, et al. U-Net based deep learning bladder segmentation in CT urography. Med Phys (2019) 46(4):1752-65. doi:10.1002 / mp.13438

[0283] 11. He C, Xu H, Yuan E, Ye L, Chen Y, Yao J, et al. The accuracy and quality of image-based artificial intelligence for muscle-invasive bladder cancer prediction. Insights Imaging (2024) 15(1):185. doi:10.1186 / s13244-024-01780-y

[0284] 12. Jansen I, Lucas M, Bosschieter J, de Boer OJ, Meijer SL, van Leeuwen TG, et al. Automated Detection and Grading of Non-Muscle-Invasive Urothelial Cell Carcinoma of the Bladder. Am J Pathol (2020) 190(7):1483-90. doi:10.1016 / j.ajpath.2020.03.013

[0285] 13. Awan R, Benes K, Azam A, Song TH, Shaban M, Verrill C, et al. Deep learning based digital cell profiles for risk stratification of urine cytology images. Cytometry A (2021) 99(7):732-42. doi:10.1002 / cyto.a.24313

[0286] 14. Eminaga O, Eminaga N, Semjonow A, Breil B. Diagnostic Classification of Cystoscopic Images Using Deep Convolutional Neural Networks. JCO Clin Cancer Inform (2018) 2:1-8. doi:10.1200 / cci.17.00126

[0287] 15. Shkolyar E, Jia X, Chang TC, Trivedi D, Mach KE, Meng MQ, et al. Augmented Bladder Tumor Detection Using Deep Learning. Eur Urol (2019) 76(6):714-8. doi:10.1016 / j.eururo.2019.08.032

[0288] 16. Ikeda A, Nosato H, Kochi Y, Kojima T, Kawai K, Sakanashi H, et al. Support System of Cystoscopic Diagnosis for Bladder Cancer Based on Artificial Intelligence. J Endourol (2020) 34(3):352-8. doi:10.1089 / end.2019.0509

[0289] 17. Lorencin I, Andelic N, Spanjol J, Car Z. Using multi-layer perceptron with Laplacian edge detector for bladder cancer diagnosis. Artif Intell Med (2020) 102:101746. doi:10.1016 / j.artmed.2019.101746

[0290] 18. Yang R, Du Y, Weng X, Chen Z, Wang S, Liu X. Automatic recognition of bladder tumours using deep learning technology and its clinical application. Int J Med Robot (2021) 17(2):e2194. doi:10.1002 / rcs.2194

[0291] 19. Du Y, Yang R, Chen Z, Wang L, Weng X, Liu X. A deep learning network-assisted bladder tumour recognition under cystoscopy based on Caffe deep learning framework and EasyDL platform. Int J Med Robot (2021) 17(1):1-8. doi:10.1002 / rcs.2169

[0292] 20. Wu S, Chen X, Pan J, Dong W, Diao X, Zhang R, et al. An Artificial Intelligence System for the Detection of Bladder Cancer via Cystoscopy: A Multicenter Diagnostic Study. J Natl Cancer Inst (2022) 114(2):220-7. doi:10.1093 / jnci / djab179

[0293] 21. Yoo JW, Koo KC, Chung BH, Baek SY, Lee SJ, Park KH, et al. Deep learning diagnostics for bladder tumor identification and grade prediction using RGB method. Sci Rep (2022) 12(1):17699. doi:10.1038 / s41598-022-22797-7

[0294] 22. Zhang Q, Liang Y, Zhang Y, Tao Z, Li R, Bi H. A comparative study of attention mechanism based deep learning methods for bladder tumor segmentation. Int J Med Inform (2023) 171:104984. doi:10.1016 / j.ijmedinf.2023.104984

[0295] 23. Chang TC, Shkolyar E, Del Giudice F, Eminaga O, Lee T, Laurie M, et al. Real-time Detection of Bladder Cancer Using Augmented Cystoscopy with Deep Learning: a Pilot Study. J Endourol (2023). doi:10.1089 / end.2023.0056

[0296] 24. Kwak SH, Kim KY, Choi JS, Kim MC, Seol CH, Kim SR, et al. Impact of AI-assisted CXR analysis in detecting incidental lung nodules and lung cancers in non-respiratory outpatient clinics. Front Med (Lausanne) (2024) 11:1449537. doi:10.3389 / fmed.2024.1449537

Claims

1. A method performed by an electronic device for providing information on a suspected bladder cancer site from a cystoscope image, wherein the electronic device comprises an input device, an output device, a processor, and a memory, and A step of receiving a cystoscope image of the bladder captured by a cystoscope via the input device; A step of generating a preprocessed image by the processor by performing data preprocessing, including noise removal, color adjustment, and brightness adjustment, on the above cystoscope image; A step of extracting feature vectors from multiple channels from the preprocessed image by the processor using a CNN (convolutional neural network)-based backbone model stored in the memory; A step of detecting a suspected bladder cancer area from the preprocessed image by the processor based on the feature vectors using the RPN (regional proposal network) stored in the memory; A step of determining a confidence score regarding whether the suspected bladder cancer site is a malignant tumor, and determining by the processor whether the suspected bladder cancer site is a malignant tumor based on a comparison of the confidence score and a set threshold value; A step of marking a boundary line on the outer edge of the suspected bladder cancer area in the preprocessed image, and generating a processed image by the processor in which the suspected bladder cancer area of ​​the preprocessed image is marked with a color code mask depending on whether the suspected bladder cancer area is a malignant tumor; A step comprising outputting the above-mentioned processed image by the above-mentioned output device, method.

2. In Paragraph 1, The above CNN is a deep learning model that learns the spatial characteristics and patterns of image data, the above backbone model is a network responsible for extracting features of the above preprocessed image based on the structure of the above CNN, the above multi-channels are channels for separating the above preprocessed image into feature maps for different information such as color, brightness, and texture and analyzing them individually, the above preprocessed image is separated into the above multi-channels by the above backbone model, and the above multi-channels are analyzed by the above CNN, and The above CNN is configured to hierarchically extract high-level features of the shape, size, and texture of the tumor from low-level features of edges, points, and lines, and The last layer among the multiple layers included in the above CNN is configured to integrate the feature maps of the multiple channels to generate a single final feature vector, which is numerical data. The above final feature vector includes information related to the color, size, and texture of the malignant tumor, and The above feature maps and the above final feature vector are used by the RPN to detect the suspected bladder cancer site and to determine whether the suspected bladder cancer site is a malignant tumor. method.

3. In Paragraph 2, The above CNN is configured to generate the above final feature vector by extracting channel-independent intermediate feature vectors for the above multiple channels, removing information other than tumor-related information from the above intermediate feature vectors through filtering for each channel, and then integrating them. method.

4. In Paragraph 3, The above CNN is configured to acquire information on unique feature vector patterns for normal tissue, benign tumors, and malignant tumors through a pre-training process, and Whether the suspected bladder cancer site is a malignant tumor is determined based on a comparison of the final feature vector and the feature vector pattern. method.

5. In Paragraph 4, The above RPN is configured to detect the suspected bladder cancer site, which is a region of interest (RoIs), from the above preprocessed image, and The above RPN is configured to determine proposal regions for the suspected bladder cancer site based on the final feature vector generated from the above CNN-based backbone model, and The above RPN is configured to set grid-shaped anchor boxes covering the entire preprocessed image, compare the feature maps with the anchor boxes to determine the probability that each anchor box for each part of the preprocessed image corresponds to a tumor site, continuously perform initial detection and fine detection using a coarse-to-fine detection structure, adjust the boundaries between the anchor boxes and the actual tumor through bounding box regression, remove duplicate regions in the preprocessed image using a non-maximum suppression (NMS) algorithm, and determine the candidate regions by leaving only the most reliable candidate regions in the preprocessed image. The above-determined candidate regions are the above-determined suspected bladder cancer sites, method.

6. In Paragraph 5, The candidate regions determined by the above RPN are transmitted as the header of the above CNN, and The header of the above CNN is an output layer configured to detect and classify the malignant tumor based on the feature maps output by the above CNN-based backbone model and the above RPN, and The confidence score of the probability that the candidate regions correspond to the malignant tumor, and the classification of whether the candidate regions are malignant tumors, benign tumors, or normal tissues are determined by the header of the above CNN, and Whether each of the candidate regions is a malignant tumor is determined based on whether the above confidence score is higher than the above threshold value, method.

7. In Paragraph 6, Based on information related to the malignant tumor determined by the headers of the RPN and the CNN, the processed image is output in the form of a 3D image so that the depth, volume, and structure of the tumor are intuitively represented for the suspected bladder cancer site, and In the above processed image, the size and malignancy of the malignant tumor are visualized and output using the color code mask, method.

8. In Paragraph 6, The above CNN is configured to calculate a final loss value by using a standard loss function (Categorical Cross-Entropy, Smooth L1, Binary Cross-Entropy) when performing the above pre-training process, along with (a) an adaptive loss function that improves the accuracy of discrimination of suspected regions by dynamically adjusting loss weights according to the confidence score, and (b) a shape regularization loss function to minimize the shape distortion of the tumor. The above final loss value is used to adjust the CNN's pre-training process to minimize the difference between the predicted value and the actual value, and The above CNN is configured to learn more accurately the areas with a high suspicion of bladder cancer by adjusting the loss weights according to the confidence score for the areas suspected of bladder cancer, and The above processed image is output with the distorted shape of the malignant tumor corrected through shape regularization with respect to the suspected bladder cancer area. method.

9. In Paragraph 1, The above processing image is configured to concisely display the diagnostic results for the suspected bladder cancer site by grouping multiple suspected tumor sites at the same location based on the confidence score for the suspected bladder cancer site. method.

10. An electronic device for providing information on a suspected bladder cancer site from a cystoscope image, Includes an input device; an output device; a processor; and memory, The above memory stores instructions for operations performed by the processor, and The above operations are, A step of receiving a cystoscope image of the bladder captured by a cystoscope via the input device; A step of generating a preprocessed image by the processor by performing data preprocessing, including noise removal, color adjustment, and brightness adjustment, on the above cystoscope image; A step of extracting feature vectors from multiple channels from the preprocessed image by the processor using a CNN (convolution neural network) based backbone model; A step of detecting a suspected bladder cancer area from the preprocessed image by the processor based on the feature vectors using a regional proposal network (RPN); A step of determining a confidence score regarding whether the suspected bladder cancer site is a malignant tumor, and determining by the processor whether the suspected bladder cancer site is a malignant tumor based on a comparison of the confidence score and a set threshold value; A step of marking a boundary line on the outer edge of the suspected bladder cancer area in the preprocessed image, and generating a processed image by the processor in which the suspected bladder cancer area of ​​the preprocessed image is marked with a color code mask depending on whether the suspected bladder cancer area is a malignant tumor; A step comprising outputting the above-mentioned processed image by the above-mentioned output device, Electronic device.

Images