Artificial intelligence solutions for medical dilemmas

Abstract

Methods and accompanying systems for deploying a predictive model for classification of at least one medical image are disclosed. An example method comprises, from a repository of medical images, selecting a subset of the repository. The subset includes an initial iteration of selecting a randomized image from each patient of the cohort and at least one subsequent iteration. The method further comprises creating an annotation associated with each medical image of the subset and training a predictive model based on the content of the medical images of the subset and the associated annotations created. The method further comprises integrating the predictive model trained with a computer-based system communicatively coupled to a medical imaging device and configured to apply the predictive model for classification of the at least one medical image. Embodiments described herein may be useful for creation of machine learning-based tools for real-time guidance of medical procedures or interventions.

Description

5439.1039001Artificial Intelligence Solutions for Medical DilemmasRELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 772,296, filed on March 14, 2025. The entire teachings of the above applications are incorporated herein by reference.

[0002] BACKGROUND

[0003] Medical imaging may be useful for a wide range of diagnostic procedures or therapeutic interventions through imaging of anatomical or physiological features or through image-guided procedures. Additionally, large amounts of medical imaging data exist, generated on a regular basis for care of clinical patients and research subjects. Recent advancements in artificial intelligence and machine learning may be capable of leveraging such medical imaging data to develop predictive models as solutions for challenges in medical diagnostic or interventional procedures.SUMMARY

[0004] Embodiments of the present invention may be helpful for developing new artificial intelligence-based tools for analyzing medical images and for deploying the tools developed to evaluate, manage, diagnose, monitor, and or treat clinical dilemmas. For example, it may be feasible to create computer vision (CV) artificial intelligence (Al) models that leverage an extensive repository of patient data. Such an Al model may be useful toward performing imaging procedures, e.g., assessing images acquired during the imaging procedures, or image-guided procedures, e.g., determining a quality of an imaging view of the image-guided procedures or identifying a stage of the image-guided procedures. Example embodiments may include without limitation an endoscopic tool for cancer diagnosis, a surgical copilot, an ultrasound tool for diagnosing and monitoring disease, or a video microscopy tool. The embodiments described herein may be helpful for enabling broader accessibility to patient care modalities or procedures typically performed in specialized facilities.- 1 - 5045914 vl5439.1039001

[0005] An example embodiment can be directed to a method for deploying a predictive model for classification of at least one medical image of a clinical or research patient. The method comprises, from a repository of medical images acquired from a cohort of patients, selecting a subset of the repository. The selecting the subset includes an initial iteration of selecting a randomized image from each patient of the cohort and at least one subsequent iteration of selecting a randomized image from each remaining patient of the cohort. The method further comprises creating an annotation associated with each medical image of the subset selected and training a predictive model based on content of the medical images of the subset selected and the associated annotations created. The method further includes integrating the predictive model trained with a computer-based system communicatively coupled to a medical imaging device. The computer-based system is configured to apply the predictive model for classification of the at least one medical image of the clinical or research patient.

[0006] Integrating the predictive model trained with the computer-based system can include acquiring the at least one medical image of the clinical patient using the medical imaging system. The integrating can further include identifying a selection of the at least one medical images acquired and generating a prediction for each image of the selection identified using the predictive model trained. The integrating can further include producing data representative of a composite image for each image of the selection and the corresponding prediction generated. The method can further comprise generating a composite prediction based on at least a portion of the plurality of images received. The method can further comprise repeating the acquiring the at least one medical image, identifying the selection, generating the prediction, and produce the data in an iterative or real-time manner until an imaging procedure of the clinical or research patient is complete.

[0007] Creating the annotation associated with each medical image can include providing, for a given medical image, one or more of a quality, a segmentation, a stage of a medical procedure, or a suggestion on a suggested medical device. According to some embodiments, the annotations created can comprise annotations associated with a given image or a frame or an image. In other embodiments, the medical images can comprise a video , which can be a sequence of images, and an annotation can comprise time stamps or time intervals within a video.- 2 - 5045914 vl5439.1039001

[0008] The method can further comprise anonymizing the at least one medical image of the clinical patient to produce at least one anonymized image of the clinical patent. The method can further comprise updating the repository of medical images from the retrospective cohort of patients with the at least one anonymized medical images. The method can further comprise retraining the predictive model based on the repository updated.

[0009] The method can further comprise preprocessing the repository of medical images prior to creating the annotation, wherein preprocessing includes at least one of normalizing, cropping, or labeling.

[0010] The repository of medical images can comprise a repository that includes videos. The method can further comprise extracting images from the videos.

[0011] The training the predictive model can comprise feeding the medical images and the associated annotations created through a convolutional neural network.

[0012] Another example embodiment can be directed to a computer-based copilot providing guidance during an image-guided procedure. The copilot comprises a processor configured to: generate a prediction using a predictive model for an image of a sequence of images. The prediction includes a quality of the image and a current stage of a sequence of stages of the image-guided procedure. The processor is further configured to determine a score for each image of the sequence based on the prediction generated. The score indicates one or more of a safety grade of the image-guided procedure or a readiness to progress from the current stage to a subsequent stage of the sequence of stages. The processor is further configured to produce data representative of a composite image based on the image, the prediction generated for the image, and the score determined for the image. The data produced can provide guidance during an image-guided procedure.

[0013] The computer-based co-pilot can further comprise a medical imaging device, communicatively coupled with the processor, configured to acquire images of the image-guided procedure. The sequence of images can be a subset of the images acquired. The medical imaging device can be an optical, ultrasound, or an x-ray imaging device.

[0014] The processor can be configured to cause a therapeutic device, communicatively coupled to the processor, to perform at least part of the image-guided procedure based on the prediction generated, the score determined, or a user input.- 3 - 5045914 vl5439.1039001

[0015] The processor can be configured, generate the prediction, determine the score, and produce the data representative of the composite image in real time.

[0016] The computer-based copilot can further comprise an electronic display communicatively coupled to the processor. The processor can be configured to render the composite image on the electronic display.

[0017] The computer-based copilot can further comprise a therapeutic device including an imaging sensor. The imaging sensor can be configured to acquire an additional sequence of images and the processor can be configured to generate the prediction using the image of the sequence of images and an other image of the additional sequence of images

[0018] The prediction generated can further include an optimal tool for performing the image-guided procedure.

[0019] The data produced can include, for an image of the subset with a low score, guidance for improving the image.

[0020] Another example embodiment can be directed to computer-based system for evaluating a microscopy sample. The computer-based system comprises a processor configured to generate a prediction using a predictive model for an image of a sequence of microscopy images. The prediction includes a quality of the image and an assessment of the image, the assessment identifying evidence of a target feature in the image. The processor is further configured to determine a score for the image based on the prediction generated. The score indicates a presence of the target feature, an extent of the target feature, or a presence of an imaging artifact confusable with the target feature. The processor is further configured to produce data representative of a composite image based on the image, the prediction generated, and the score determined. The data produced indicates a likelihood of the target feature in the image to evaluate the microscopy sample.

[0021] The computer-based system can further comprise a video microscopy system, communicatively coupled with the processor, configured to acquire images of a microscopy sample. The sequence of microscopy images can be a subset of the images acquired.

[0022] The processor can be configured to generate the prediction, determine the score, and produce the data in real time.- 4 - 5045914 vl5439.1039001

[0023] The computer-based system can further comprise an electronic display communicatively coupled to the processor, wherein the processor is configured to render the composite image on the electronic display.

[0024] The data produced can further include likelihood of a tissue processing artifact.

[0025] Another example embodiment can be directed to a computer-based system for assisting a diagnostic imaging procedure. The computer-based system comprises a processor configured to generate a prediction using a predictive model for an image of a sequence of medical images from a diagnostic imaging procedure. The prediction includes a quality of the image, a presence of an anatomical feature within the image, and evidence of a pathology in the anatomical feature within the image. The processor is further configured to determine a score for each image of the sequence based on the prediction generated, the score indicating a likelihood of the pathology. The processor is further configured to produce data representative of a composite image based on the image, the prediction generated for the image, and the score determined for the image. The data produced indicates one or more of the presence of the anatomical feature, the evidence of the pathology in the anatomical feature, or the score determined to assist the diagnostic medical imaging procedure.

[0026] The computer-based system can further comprise a medical imaging system, communicatively coupled with the processor, configured to acquire images of a human or animal, wherein the sequence of medical images is a subset of the images acquired.

[0027] The processor can be configured to generate the prediction, determine the score, and produce the data in real time.

[0028] The computer-based system can further comprise an electronic display communicatively coupled to the processor, wherein the processor is configured to render the composite image on the electronic display.BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawings will be provided by the Office upon request and payment of the necessary fee.- 5 - 5045914 vl5439.1039001

[0030] The foregoing will be apparent from the following more particular description of example embodiments, as illustrated in the drawings interspersed in accompanying documents being fded herewith. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating embodiments.

[0031] FIG. 1A is a flow diagram of an example embodiment of a method for deploying a predictive model for classification of at least one medical image of a clinical or research patient.

[0032] FIG. IB is a diagram that illustrates an example embodiment of a system for deploying a predictive model for classification of at least one medical image of a clinical or research patient.

[0033] FIGS. 2A-2C are schematic diagrams that illustrate an example retrospective component of an example embodiment of a method for deploying a predictive model.

[0034] FIGS. 3 A and 3B are flow diagrams of an example streaming component of an example embodiment of a method for deploying a predictive model.

[0035] FIG. 4A is an example composite image that may be rendered based on data generated by an example embodiment of a computer-based co-pilot.

[0036] FIGS. 4B-4D are example images that illustrate features that may used in composite images similar to the composite image of FIG. 4 A.

[0037] FIGS. 5A-5F are example composite images that may be rendered based on data generated by an example embodiment of a computer-based co-pilot during a endoscopic ultrasound gallbladder drainage (EUS-GBD) procedure.

[0038] FIGS. 6A-6C are example images that may be output by an example embodiment of a computer-based co-pilot providing guidance during an image-guided procedure.

[0039] FIGS. 7A-7E are plots of example receiver operating characteristic (ROC) curves for safety scores determined by an example embodiment of a computer-based co-pilot.

[0040] FIGS. 8A-8D are plots of example receiver operating characteristic (ROC) curves for predictions of a tool for performing a procedure, according to an example embodiment of a computer-based co-pilot.

[0041] FIG. 9 is a diagram that illustrates an example embodiment of a computer-based system 906 evaluating a microscopy sample.- 6 - 5045914 vl5439.1039001

[0042] FIGS. 10A and 10B are images of example slides of donor liver samples that may be processed by an example embodiment of a computer-based system for evaluating a microscopy sample.

[0043] FIGS. 10C and 10D are occlusion block heatmaps of the example slides of FIGS. 10A and 10B, respectively.

[0044] FIG. 11 is a plot of a receiver operating characteristic (ROC) curve of an example embodiment of a computer-based system for evaluating a microscopy sample.

[0045] FIGS. 12A and 12B are an example raw image and an example composite image, respectively, of a curvilinear ultrasound scan that may be acquired or generated by an example embodiment of a computer-based system for assisting a diagnostic imaging procedure.

[0046] FIG. 12C are an example composite image of a curvilinear ultrasound scan that may be generated by an example embodiment of a computer-based system for assisting a diagnostic imaging procedure.

[0047] FIGS. 12D and 12E are an example raw image and an example composite image, respectively, of a linear ultrasound scan that can be acquired or generated by an example embodiment of a computer-based system for assisting a diagnostic imaging procedure.

[0048] FIG. 13 is a schematic view of a computer network in which embodiments may be implemented.

[0049] FIG. 14 is a block diagram illustrating an example embodiment of a computer node (e.g, client processor(s) / device(s) or server computer(s)) in the computer network of FIG. 13.DETAILED DESCRIPTION

[0050] A description of example embodiments follows.

[0051] Medical imaging may provide non-invasive or minimally invasive guidance for the diagnosis or therapeutic intervention of a number of health-related dilemmas. However, evaluation of medical images to inform a medical diagnosis or procedure may require analyzing increasingly large numbers or sequences of medical images, extensive expertise in reading images, or specialized instruments or facilities. Artificial intelligence may be helpful in addressing some of these challenges. The prevalence of medical imaging has resulted in large repositories of images, which may be useful for generating new artificial intelligence-based- 7 - 5045914 vl5439.1039001models. Furthermore, such models may be helpful in forming a decision based upon content present in an image to provide guidance for physicians during diagnostic or therapeutic procedures.

[0052] In some embodiments, applying artificial intelligence or machine learning methodologies to high volume or time-constrained imaging settings, for example, real-time imaging or videos, may present a significant challenge. Addressing such issues may enable AI / ML techniques to be applied towards, for non-limiting example, real-time or bedside guidance during medical interventions.

[0053] As described herein, an example embodiment of the present invention can include an overarching toolchain to rapidly develop artificial intelligence (Al) based models, including machine learning (ML) or computer vision (CV) based techniques, for evaluating an image or a series of images. The toolchain may leverage previously acquired images to rapidly train new Al models, which may be used, for example, for evaluating a medical image for presence of a pathological feature or an imaging artifact or for guiding a surgeon during a medical procedure.

[0054] Al models generated using the embodiments described herein may be useful for evaluating static images (or a series of static images) or real-time images and videos. In some embodiments, evaluating one or more static medical images may be used to determine a feature, e.g., a classification, segmentation, or pathology, of the one or more static medical images for applications that may not be or may be less sensitive to time constraints, e.g., histological analysis of a biopsy sample. In other embodiments, the Al models may be deployed in real time. An example embodiment of the present invention may further include a real-time device system for deploying the Al models developed within a clinical setting, processing medical images and displaying a result or a prediction of the Al model to an operator during an imaging or image- guided procedure.

[0055] In another example embodiment, a toolchain may include a retrospective process and a streaming process. The retrospective process may be used to develop new Al models for a breadth of imaging modalities. In general, the retrospective process may include preprocessing images of a repository of images, training an Al-based model, and evaluating the Al-based model trained. The streaming process may include acquiring medical images using an imaging modality communicatively coupled with a processor configured to deploy the Al-based model- 8 - 5045914 vl5439.1039001trained, generating a prediction based on the images acquired using the Al-based model, rendering the prediction on a graphical interface, and saving the prediction and medical images for further refinement of the Al-based model.

[0056] In the example embodiment, preprocessing images may include identifying images or image sequences from a repository of images. The identified images may include or be associated with ground truth information, for example, biopsy-confirmed anatomical or physiological features. Conventional preprocessing techniques, including normalization, labeling, and cropping, may be applied to the identified images or image sequences. In some embodiments, the images include image sequences or videos, which may be converted into individual images assigned to a given patient.

[0057] According to some embodiments, one or more images may be selected or identified from a repository of medical images using a round-robin (RR) technique during pre-processing of the medical images. Such a round-robin technique can comprise, for non-limiting example, a first round of randomly selecting an image associated with each patient of the repository. In some embodiments, the round robin may further comprise at least one additional round of randomly selecting an image associated with each patient of the repository. Selected images may be used to train a machine learning model, as further described hereinbelow. According to some embodiments, implementation of such a technique to methodically select for images may be useful for ensuring that images from a subset of patients or imaging sequences are not overrepresented in a training set (e.g., the training set may be “patient-balanced” and may reduce bias of patients comprising larger numbers of images than other patients of the training set). Such considerations may be of particular importance in applications such as, for non-limiting example, videos, e.g., videos acquired during endoscopic pictures. In some embodiments, video-based medical imaging modalities may, based on a length of a procedure, comprise datasets with vastly different sizes. In some embodiments, the images selected using the RR shuffle may then be annotated by a blinded expert.

[0058] In the example embodiment, training may include combining the ground truth labels and the expert annotations for the images selected using the RR shuffle. A portion of the images may be used for training and a portion of the images may be used for testing the model. Training the model may comprise using an Al architecture for image analysis, for example, a- 9 - 5045914 vl5439.1039001convolutional neural network, a residual network, a generative adversarial network, among others. The hyperparameters of the model may be optimized using an iterative process, for example, Bayesian optimization, grid search, or random search, to generate a predictive function to be applied to testing images or newly acquired images. The choice of hyperparameter may be selected based on the type of Al architecture used for training the model. For example, for a convolutional neural network, hyperparameters of convolutional layers may include a number of kernels, a size of kernels, an activation function, a number of convolutional layers, stride, or padding; hyperparameters of fully connected layers may include dropout or connectivity pattern; and general hyperparameters may include batch size, number of epochs, or learning rate.

[0059] Analysis of the trained model may include evaluating images of the training group or continued refinement of the trained model on newly acquired images. Clinical evaluation of the trained model may include sensitivity, specificity, receiver operating characteristic (ROC) curves, and area under the ROC (AUROC). Evaluation of the elements of an image most important in classifying the image may be achieved using occlusion heatmap analysis. An aggregate prediction may further be generated based on the individual predictions generated for a sequence of images or a video of a given patient.

[0060] In the example embodiment, acquiring images in the streaming flow may include receiving images acquired by a medical imaging modality. The images may be a sequential set of images. A processor configured to receiving the acquired images acquired, for example, using a video input / output interface, may, for a given image, preprocess the given image, generate a prediction based on the preprocessed given image, and update a prediction series based upon at least a portion of the sequential set of images acquired. Based on the given image and the prediction generated or the prediction series, an output image may be rendered to inform an operator of a prediction. The output image may be a composite image, for example, an overlay or a side-by-side rendering of the given image and the prediction or prediction series. Example embodiments of rendering a prediction or prediction series may include rendering a score based on the prediction of the given image or a curve of rolling averages of scores determined for a prediction series. The predictions and the images acquired may be saved and exported to a data structure. The data structure may be used for generating an after-exam summary or for refining- 10 - 5045914 vl5439.1039001the training model, for example, by comparing scores generating using the model against a ground truth.

[0061] In another example embodiment, a method for deploying a predictive model for classification of at least one medical image of a clinical or research patient may include selecting, from a repository of medical images acquired from a retrospective cohort of patients, a subset of the repository. Selecting the subset includes an initial iteration of selecting a randomized image from each patient of the retrospective cohort and at least one subsequent iteration of selecting a randomized image from each remaining patient of the retrospective cohort. The method may further include creating an annotation associated with each medical image of the subset selected and training a predictive model based on content of the medical images and the associated annotations created. The method may further include integrating the predictive model trained with a computer-based system communicatively coupled to a medical imaging device. The computer-based system may be configured to apply the predictive model for classification of the at least one medical image of the clinical or research patient.

[0062] Creating the annotation, which may be performed by an expert, associated with each medical image may include assigning a type, score, grade, or classification to an image. In an example embodiment, creating the annotation may include at least in part assigning raw images to a data structure or folder structure based on the type, score, grade, or classification, the data structure or folder structure indicating the annotation when used as an input for training a machine learning model.

[0063] FIG. 1A is a flow diagram of an example embodiment of a method 101 for deploying a predictive model for classification of at least one medical image of a clinical or research patient. The method 101 can comprise, from a repository of medical images acquired from a cohort of patients, selecting 103 a subset of the repository. The selecting 103 the subset can include an initial iteration of selecting a randomized image from each patient of the cohort and at least one subsequent iteration of selecting a randomized image from each remaining patient of the cohort. The method can further comprise creating 105 an annotation associated with each medical image of the subset selected and training 107 a predictive model based on content of the medical images of the subset selected and the associated annotations created. The method can further comprise integrating 109 the predictive model trained with a computer-based system- 11 - 5045914 vl5439.1039001communicatively coupled to a medical imaging device. The computer-based system can be configured to apply the predictive model for classification of the at least one medical image of the clinical or research patient.

[0064] FIG. IB illustrates an example embodiment of a system 100 for deploying a predictive model for classification of at least one medical image of a clinical or research patient. The system 100 can be configured to perform the elements described hereinabove with respect to FIG. 1A. For example, the system 100 can comprise a processor 110 communicatively coupled to a repository 102 of medical images, the medical images being acquired from a cohort of patients, and can be configured to select a subset of the repository. The system 100 can be further configured to train a predictive model 104 based on content of the medical images of the subset selected and associated annotations created for each medical image of the subset selected.

[0065] The predictive model 104 trained can be integrated with a computer-based system 106 communicatively coupled to a medical imaging device 108. According to some embodiments, the system 100 and the computer-implemented system 106 can be the same system, and the system 100 can be coupled communicatively to the medical imaging device 108. In some embodiments, the computer-implemented system 106 can be a separate system, and the system 100 can be configured to export the predictive model 104 to be integrated with the computer-implemented system 106. The computer-implemented system 106 can be communicatively coupled to the medical imaging device 108 via a connection 112, which can be, for example, a wired or wireless connection.

[0066] FIGS. 2A-2C illustrate schematically an example retrospective component of an example embodiment of a method for deploying a predictive model. FIG. 2A is a flow diagram 214a that may represent a preprocessing element 215 of the retrospective workflow, and FIG. 2B is a flow diagram 214B that may represent training 217 and analysis 219 elements of the retrospective workflow. The flow diagram 214a may indicate that the preprocessing element 215 can comprise acquiring 221 a repository of medical images and information regarding ground truths, e.g., ground truths for diagnoses (which may be incorporated as labels with respect to the medical images). In some embodiments, the medical images may be videos or another form of sequences of images. The images may undergo preprocessing 223, for example, a batch image preprocessing, which may output 225 a dataset of labeled and / or cropped images based on the- 12 - 5045914 vl5439.1039001repository of medical images. The preprocessing element 215 can further comprise selecting 227 a subset 229 may from the dataset output (which can comprise the labeled and / or cropped images), for non-limiting example, using a round robin shuffle, to be annotated by a blinded expert.

[0067] The flow diagram 214b of FIG. 2B may indicate that, according to an example embodiment, the training element 217 can comprise training a predictive model based on the subset selected, the subset comprising images 231 with ground truth labels (e.g., CCA-positive) and expert-annotated labels. The training element can comprise partitioning or assigning 233 the images 231 into a training dataset and a test data set and can further comprise normalizing 235 the images for a predictive model to be trained. According to some embodiments, a machine learning model can comprise, for non-limiting example, a convolutional neural network (CNN) or a recurrent neural network (RNN). According to an embodiment, the predictive model can comprise a multi-layered CNN architecture such as ResNet50 for non-limiting example. The normalized data may also used for a model hyperparameter search 237, which may be implemented as a training loop 239 for optimizing a predictive model 241 for model training. A predict function 243 based on the predictive model trained can be output. According to an example embodiment, predictions made by the predictive model trained can be tracked in a predictions database 245 for subsequent analysis. The flow diagram 214b may further indicate that the analysis 219 element can comprise using the predictions database 245 to generate, for example, analytical metrics 247 such as receiver operating characteristic (ROC) curves, determining occlusion heatmap, and evaluating aggregate predictions (e.g., computer-aided detection (CADe) or computer-aided diagnosis (CADx)).

[0068] FIG. 2C illustrates schematically an example round robin shuffle 216 that may be for selecting a subset of images for training a predictive model. The round robin shuffle 216 may comprise a first round 218, wherein a random image, e.g., image 224, may be selected from each patient, e.g., patient 222, within a repository of medical images. According to the example embodiment illustrated, the example repository comprises 5 patients, each patient having a stack of images indicated by, for example, the image 224. In some embodiments, the order of patients (e.g., order of selection from patients) may further be randomized as well. The round robin shuffle may further comprise at least one additional round (e.g., at least a second round 220),- 13 - 5045914 vl5439.1039001wherein a random image may be selected from each remaining patient within the repository of medical images. According to the example embodiment, the round robin shuffle may create a “patient-balanced” shuffle, as illustrated after two rounds, which may be helpful for ensuring that representation of patients in the subset selected is less skewed toward patients with a lot of images (e.g.., longer videos).

[0069] FIGS. 3A and 3B are flow diagrams 314a, 314b, respectively, of an example streaming component of an example embodiment of a method for deploying a predictive model. The flow diagram 314a of FIG. 3 A may represent an input element 315 of the streaming component. The input element 315 can comprise a computer-based system 349 configured to deploy a predictive model based on a configuration stored thereon. The predictive model, e.g., the predictive model 343, can comprise, for example, a predictive model trained the process described herein with reference to FIGS. 2A-2C. In some embodiments, the computer-based system 349 can be communicatively coupled to a medical imaging device, which may be a videography device, configured for capturing medical images or videos 353. The computer- based system may be configured to apply the predictive model 343 to the medical images or videos (which may be frames of the videos) to perform a classification of the medical images. In some embodiments, the computer-based system may apply the predictive model to the medical images in a streaming loop 355 (e.g., in real-time during a videography session). According to some embodiments, the computer-based system may apply the predictive model to each frame to be processed by performing, for non-limiting example, actions 357 comprising one or more of preprocessing and normalizing a given frame, forming a prediction of the given frame using the predictive model, and updating a prediction series based on the prediction. The prediction can be saved to a prediction database 345.

[0070] The flow diagram 314b of FIG. 3B may represent an output element 317 of the streaming component. According to an example embodiment, a prediction of the given frame, for example, the prediction formed as part of the actions 357 performed in FIG. 3 A, can be used for updating 359 a user interface or heads-up display (HUD). The display can be configured to show one or more of, for non-limiting example, an instantaneous prediction 361 (which may be based on one frame), a rolling prediction 363 (which may be based on a rolling average over a period of time or frames, e.g., 30 frames), and an aggregate diagnosis 365 (which can be based on a- 14 - 5045914 vl5439.1039001subset or an entirety of frames of an acquired video). In some embodiments, the display may be configured to show a composite image comprising one or more of the predictions, the diagnosis, and the frame. The predictions and diagnosis can be rendered 367 to an electronic display (e.g., a HUD) to provide guidance (e.g., real-time or actionable guidance) to an operator. According to some embodiments, the predictions generated can further be used in a report and export stage 369. For example, the predictions database 354 and the composite image 371 of each frame can be saved 377 to a storage or repository, exported 373, or used to create 375 an after-exam summary. The after exam summary can comprise an extended analysis 379 on significant images or can be used for generating 381 a statistical analysis (which may comprise definitive graphics and metrics) of a course of the video or sequence of frames.Example Embodiment 1 - Surgical Copilot

[0071] In some embodiments, an Al model developed using the toolchain disclosed herein may be deployed as a copilot for a procedure, e.g., a surgical procedure. The surgical copilot may use medical imaging, e.g., endoscopy, ultrasound, MRI, CT, or angiography, to provide guidance for a physician during an interventional procedure. Such interventional procedures may conventionally require extensive operator training or expertise or specialized facilities, potentially limiting accessibility of the aforementioned procedures to specific healthcare settings. An Al tool as described herein, which may be trained using expert annotation of medical images used in image-guided procedures, may provide additional guidance or real-time feedback for surgeons performing such interventional procedures.

[0072] According to an example embodiment, laparoscopic cholecystectomy remains the standard of care for patients with Acute Cholecystitis (AC). Traditionally, percutaneous cholecystostomy tubes (PT-GBD) may be a second line treatment option for patients who were not deemed surgical candidates. Recently however, endoscopic ultrasound gallbladder drainage (EUS-GBD) has emerged as an alternative option to PT-GBD. EUS-GBD may improve quality of life by obviating a need for an external drain and reducing readmissions for drain occlusion. Unfortunately, EUS-GBD may be restricted to experienced endosonographers or at tertiary care centers due to its technical complexity and potential for adverse events. This restriction may- 15 - 5045914 vl5439.1039001lead to a clinical care gap, where novice endosonographers may be unable to perform the procedure.

[0073] According to the example embodiment, an artificial intelligence (Al) tool may be developed that assists endosonographers in identifying safe treatment windows for EUS-GBD. The Al tool may be evaluated through a clinical trial, which may assess performance of the Al tool to guide endosonographers in real time. During this real time evaluation, an example primary outcome may comprise assessing an Al tools ability to mimic an expert’s grade of safety window. A secondary outcome may comprise proving non-inferiority compared to amateur endosonographers.

[0074] An example embodiment of a surgical copilot system may be directed to endoscopic ultrasound (EUS)-guided drainage, e.g., of structures adjacent to the stomach and small intestine, which my comprise a minimally invasive method to manage a variety of diseases including, for non-limiting example, cholecystitis and pancreatic walled-off necrosis. During this procedure, under EUS guidance, a target structure (e.g., the gallbladder, a pancreatic fluid collection) can be identified. Subsequently, the target structure may be accessed, e.g., through a wall of a physiological lumen (for non-limiting example, a stomach or small intestine). In some embodiments, a cautery-enhanced lumen apposing metal stent may be used to puncture the wall of the physiological lumen and of the target structure. A lumen apposing metal stent (LAMS) may then be deployed to connect the stomach or small intestine to the target structure.

[0075] According to some embodiments, such interventions, which may comprise the aforementioned procedure (which may be called EUS-LAMS), may be increasingly performed around the world given their clinical efficacy and non-invasive nature. An example embodiment of a LAMS may comprise a stent called the "AXIOS®" stent, which is made by Boston Scientific. While EUS-LAMS is being performed more frequently, it may still represent a technically challenging procedure. Significant amounts of money and energy may be invested in training physicians on how to perform the procedure safely and effectively, including having physicians attend live courses and practice EUS-LAMS using a VR headset.

[0076] While EUS-LAMS may be shown to perform favorably in comparison to other treatment procedures, for example, endoscopic retrograde cholangiopancreatogram (ECRP) for choleocystis, deployment of EUS-LAMS may be limited due to a number of factors. Some- 16 - 5045914 vl5439.1039001factors may include operator variability. The EUS-LAMS procedure may require highly trained operators to determine a quality of the EUS images guiding the procedure, a correct moment to initiate a step for a procedure, for example, puncturing the wall of an organ, a duration of puncturing or cauterization, or a choice of stent size. Other factors limiting access to a given procedure may be imposed by technological or other factors at a given medical facility or environment. An Al model capable of guiding a physician through the surgical procedure, determining imaging quality of the EUS-guidance, or identifying an optimal stent size may be helpful for training junior physicians or for enabling broader accessibility to the EUS-LAMS procedure.

[0077] According to an example embodiment, a predictive model, e.g., an artificial intelligence-based model that can be integrated with a computer-based system, can be developed to serve as a "copilot" to aid physicians in real time during the deployment process. The Al can provide recommendations during or over a course of a procedure, for example, regarding whether an approach is safe or what stent physicians should use, and can also guide physicians through the deployment process.

[0078] In an example embodiment, a computer-based copilot providing guidance during an image-guided procedure can comprise a processor configured to generate a prediction using a predictive model for an image of a sequence of images. The prediction can include a quality of the image and a current stage of a sequence of stages of the image-guided procedure. The processor can be further configured to determine a score for each image of the sequence based on the prediction generated. The score can indicate readiness to progress from the current stage to a subsequent stage of the sequence of stages. The processor can be still further configured to produce data representative of a composite image for each image of the sequence based on the image, the prediction generated for the image, and the score determined for the image. The data produced can provide guidance during an image-guided procedure.

[0079] FIG. 4A is an example composite image 424a that may be rendered based on data generated by an example embodiment of a computer-based co-pilot. The composite image 424a can comprise, for non-limiting example, a frame 426 captured by a medical imaging device. The frame 427 may be used to generate a prediction based upon a predictive model and to determine a score 428 for the frame 426 based on the prediction generated. The composite image 424a can- 17 - 5045914 vl5439.1039001comprise the score 428, which may be a probabilistic representation of the score. The composite image 424a can further comprise a representation 430a of a current step of a procedure being performed, which may be determined by the co-pilot, while using the co-pilot and a recommendation 432a of a tool to be used for the procedure. In some embodiments, the composite image 424a can further comprise a view 434 from an auxiliary imaging modality, for non-limiting example, an endoscopic camera.

[0080] FIGS. 4B-4D illustrate example images features that may be used in composite images similar to the composite image of FIG. 4A. FIG. 4B illustrates example representations 430b of a prediction of a stage (e.g., a current stage based on a frame) of a procedure, which may further include or be associated with an indication or a score of a quality of a view of the frame. For non-limiting example, the indication or score can comprise, for non-limiting example, a letter score (e.g., A, B, C, D, F) or a determination (e g., Satisfactory View 436-1, Suboptimal View 436-2, Unsatisfactory View 436-3). The representations 430b of the stage in the composite image may include a recommendation or instructions for proceeding to a subsequent stage and information regarding the current stage. In some embodiments, wherein the view is unsatisfactory, the stage may be difficult to predict (e.g., Unsatisfactory View). The instructions may therefore comprise obtaining a better view.

[0081] FIG. 4C illustrates example recommendations 432b for a tool to use during a procedure. For example, with respect to a EUS-GBD procedure, the recommendation can comprise a stent size (e.g., stent smaller than 10 mm 438-3, stent smaller than 15 mm 438-2, or stent smaller than 20 mm 438-1).

[0082] FIG. 4D illustrates an example composite image 424b comprising guidance for a procedure. With respect to a EUS-GBD procedure, a step may comprise cauterization through a lumen of a gastrointestinal tract into a gallbladder. The composite image 424b, at the step of cauterization, may provide instructions 440 for performing the step of the procedure. The composite image may further comprise a frame and an auxiliary view, which may be similar to those of the composite image 424a of FIG. 4A.

[0083] FIGS. 5A-5F are example composite images that may be rendered based on data generated by an example embodiment of a computer-based co-pilot during a EUS-GBD procedure. The composite images of FIGS. 5A-5D (steps 1-4, respectively) illustrate satisfactory- 18 - 5045914 vl5439.1039001views guiding delivery of a stent to an anatomical site for deployment of said stent. The composite image of FIG. 5E illustrates guidance for performing a cautery-enhanced stent deployment following the steps of FIGS. 5A-5D. The composite image of FIG. 5F illustrates an unsatisfactory view of a procedure, as indicated by a Safety Grade and instructions to improve positioning. In the composite image of FIG. 5F, a catheter may be detected but positioning of said catheter may be suboptimal. In other examples of unsatisfactory views, the catheter may not be in view.

[0084] According to some embodiments, a computer-based co-pilot may be configured to automatically proceed between steps or stages of a procedure, for example, the stages illustrated hereinabove with respect to FIGS. 5A-5F (e.g., an AUTO mode). In some embodiments, the computer-based co-pilot may be manually toggled to transition between the steps or stages of the procedure (e.g., a MANUAL mode). An example embodiment of a co-pilot can additionally comprise a combination of the automatic and manual implementations and enable an operator to toggle between the modes. In some embodiments, the co-pilot may be configured to perform a portion of the procedure. For non-limiting example, the co-pilot may be configured to perform a cautery portion of the procedure, which may be, for non-limiting example, time-controlled or informed by a frame or image acquired from a medical imaging device.

[0085] According to an example embodiment, deployment of an example predictive model for such a copilot application can comprise two phases. A first phase can comprise identifying prior medical imaging or videography data for training the predictive model. For example, a chart review may be performed to identify who had undergone LAMs placement for different indications. According to the example embodiment, 75 patients were previously identified and corresponding whole case videos were annotated for appropriate targets for LAMs (e.g., gallbladder, walled off necrosis etc.). The patients had undergone EUS-GBD after acute cholecystitis (AC) had been proven on cross sectional imaging or based on clinical presentation. All procedures had been performed under general anesthesia using a linear-array echoendoscope and a combination of ultrasound processors.

[0086] Once appropriate patients, videos associated with the patients can be extracted. EUS- video segments (e.g., “windows”) can be annotated by an expert, for example, based on safety for EUS-GBD. Safety can be determined based on, for non-limiting examples, size of the- 19 - 5045914 vl5439.1039001gallbladder, an ability of the gallbladder to accommodate a LAMS, and a gallbladder lumen’s distance from a gastrointestinal lumen. According to the example embodiment, grade A and B annotations may indicate safe gallbladder targets, grade C annotations may indicate borderline targets, and grade D and F annotations may comprise unsafe or nonexistent targets. All extracted still images may be grouped by patient, which may be helpful for ensuring that no individual contributed data to both the training and testing sets. Annotated whole videos were split into training / validation (84.3%) and test (15.6%) cohorts.

[0087] FIGS. 6A-6C are example images that may be output by an example embodiment of a computer-based co-pilot providing guidance during an image-guided procedure. As described hereinabove, the co-pilot may be configured to output, as a non-limiting example, a safety grade for a given procedure, e.g., EUS-GBD. In some embodiments, the co-pilot may be configured to generate a prediction or determination of a safety of a window or frame of a medical image of the image-guided procedure based on an A, B, C, D, and F scale. A safe view or frame may be indicated by an A grade (FIG. 6A), a borderline view or frame may be indicated by a C grade (FIG. 6B), and an unsafe view or non-existent target (e.g., in a current view of the medical image) may be indicated by a D grade (FIG. 6C).

[0088] In the example embodiment described hereinabove with respect to a LAMS EUS- GBD procedure, a total of 473,563 extracted frames may be annotated by the expert. Of these 396,597 frames (84.3% of all stills) may be allocated to a training set, of which 128,964 frames may be allocated for a validation set, and 76,966 frames may be allocated to a test set. A machine learning model, e.g., a Resnet architecture (according to a specific embodiment, a ResNet50V2) can be used to train a predictive model and to mimic expert annotations (EUS-AI). EUS-AI performance can be assessed in accurately identifying video frames as safe (Grade AZB), or not (Grade C / D / F) on the reserved test set. According to an example embodiment, the EUS-AI can attribute different weights or “scores” to grades A, B, C, D and F. For example, a EUS-AI can classify images as “safe” on a per frame analysis if the cumulative score of Grade A or B windows surpassed 0.5. EUS-AI can classify images as “unsafe” was made if the per-frame cumulative score of Grade C, D or F windows did not surpass 0.4.

[0089] With respect to the aforementioned extracted frames, the frames may be further grouped into unique views. According to the example embedment, 836 unique views may be- 20 - 5045914 vl5439.1039001identified for usage from the frames (for example, using a selection technique such as a round robin shuffle, as described hereinabove) in EUS- Al development and testing. According to an example embodiment, of the 836 unique views, 531 views (63.5%) can be allocated to training, 170 (20.3%) can be allocated to validation., and 135 unique views (16.1%) can be allocated for testing. In some embodiments, EUS-GBD can be performed with cautery enhanced LAMS via a free-hand technique.

[0090] Furthermore, of the cases used for model training 28(87.5%) were EUS-GBD, 2 (6.2%) were EUS cystgastrostomy, and 2(6.2%) were EUS guided gastrojejunostomy (EUS-GJ).. Of the 28 EUS-GBD cases 27 (96.4%) were successful stent deployments, 1(3.6%) patient did not have AC and a LAMS was not deployed, and all patients who had a successful LAMS deployment had AC at the time of their exam. The remaining 4 cases of EUS cystgastrostomy and EUS-GJ also comprised successful stent deployments. The training cohort comprised of 32(6.0%) Grade “A” views, 81(15.3%) Grade “B” views, 103(19.4%) Grade “C” views, 94(17.7%) Grade “D” views, and 221 (41.6%) Grade “F” views. In total 267,633 stills may be extracted for training.

[0091] According to an example embodiment, testing may be performed on a per frame basis. For example, of the 76,966 frames used for testing, the EUS-AI performed with a sensitivity of 87.5%, specificity of 90.8%, and accuracy of 89.2% (AUROC 0.954) when tasked with mimicking expert frame annotations of “safe” vs “unsafe”. Table 1 presented hereinbelow provides a summary of demographics of an example dataset used to train a computer-based copilot.Table 1 - Example Demographics of Dataset for Development of Surgical Co-pilot Characteristic Training Validation Test cohort cohort cohort (n = 9) (n = 32) (n= 18)No. of windows assessed 531 170 135 No. of extracted still frames 267,633 128,964 76,966 Age, years (mean ± SD) 76.2 ± 14.8 75.3 ± 14.8 81.2 ± 10.9 Male sex, n (%) 19 (59.3%) 16 (88.8%) 5(55.5%) Race- White 30 (93.7%) 15 (83.3%) 8(88.8%) - Black 0 (0%) 2 (11.1%) 0 (0%) - Hispanic 1 (03.3%) 1 (5.5%) 1 (11.1%)Other 1 (03.3%) 0 (0%) 0 (0%)- 21 - 5045914 vl5439.1039001Case BreakdownGallbladder 28(87.5%) 14 (77.7%) 9(100%) Cystgastrostomy 2(6.2%) 1(5.5%) 0(0%) GJ 2(6.2%) 3(16.6%) 0(0%) No. of cases with Acute Cholecystitis 27 / 28 (96.4%) 9 / 14 (64.2%) 7 / 9 (77.7%) No. of cases with MBO 0 / 27(0%) 1 / 14 (7.1%) 1 / 9(11.1%) Annotated safety grades, n (% ofwindows)- Grade A 32 (6.0%) 6 (3.5%) 14 (10.4%) - Grade B 81 (15.3%) 32 (18.8%) 19 (14.1%) - Grade C 103 (19.4%) 43 (25.3%) 22 (16.3%) - Grade D 94 (17.7%) 21 (12.4%) 31 (23.0%)- Grade F 221 (41.6%) 68 (40.0%) 49 (36.3%)

[0092] According to the example embodiment, deployment of the predictive model can comprise a second phase directed to, for non-limiting example, real-time deployment of the model. For example, a clinical trial may be conducted using a computer-implemented system configured to deploy the predictive model, e.g., an Al model, in real-time. In some embodiments, patients of such a trial may be included based on a criteria of having an intact gallbladder, regardless of the presence of AC. The predictive model can be deployed during EUS cases to assess performance of an Al tool developed (e.g., the predictive model) in identifying safe windows for EUS-GBD. Endosonographers may be asked to locate the gallbladder and find an optimal position for EUS-GBD. An optimal position may comprise one where the gallbladder was visualized, for example, either from the duodenum or stomach, and where there were no intervening blood vessels nor bowel. An optical position may also comprise the echoendoscope being stable enough to allow for prolonged interrogation of the gallbladder. During an exam, expert endosonographers may be blinded to outputs from a EUS- Al and may be asked if a given window was “safe” or “unsafe”. A second observer may document the real-time EUS-AI safety recommendations as “safe” (Grade A / B), or “unsafe” (Grade C / D / F). Al safety recommendations may be compared to that of the blinded experts.

[0093] A secondary outcome of the second phase may comprise assessing such computer- implemented systems with a predictive model integrated with respect to real world applicability by comparing a novice and junior endosonographer’ s ability to identify a drainable gallbladder to EUS-AI’s interpretation. In some embodiments, a junior can be defined as someone with less- 22 - 5045914 vl5439.1039001than 10 years of experience and less than 25 EUS-GBD cases performed. In some embodiments, a novice can be defined as someone who has no sub stand ative training in EUS-GBD. Eligible participants may be recruited from the gastroenterology fellowship at a given institution.

[0094] According to an example embodiment and with respect to a second phase described hereinabove, a predictive model was tested in real time during 38 live cases. During these cases 135 windows were assessed by a panel of experts who demonstrated excellent interobserver agreement regarding EUS-GBD safety (Fleiss kappa coefficient > 0.6). Overall, experts noted 70 windows were “safe” and 65 windows were “unsafe”. EUS Al was 82% accurate in mimicking blinded experts’ safety recommendations.

[0095] For the secondary outcome of the second phase, 13 Junior endoscopists and 5 novice endoscopists, based on the endosonographic criteria as noted above. Junior endosongraphers performed with an accuracy of 74% and novice endosonographers performed with an accuracy of 71% when mimicking experts. The EUS Al was significantly more accurate than both Junior endosonographers (p < 0.01) and Novices (p < 0.01).

[0096] EUS-GBD may continue to emerge as a safe and viable option for treatment of AC. However, EUS-GBD can require expert endosonographic interpretation of the gallbladder for safe stent deployment. This hurdle may limit EUS-GBD to tertiary care centers and to expert endosonographers .

[0097] As described herein, an artificial intelligence that aids with EUS-GBD, for example, as a computer-based surgical co-pilot tool, may be useful for guiding EUS-GBD procedures. According to an example embodiment, the co-pilot may be developed based on expert annotated images, which may comprise 531 different views of various gallbladders with different safety grades based on target size, distance, and internal debris. An Al-based co-pilot, e.g., a EUS Al, may be created to mimic the expert annotations and may achieve an accuracy of 89.2% with an area under the ROC curve (AUROC) of 0.954.

[0098] According to the example embodiment, in the second phase the EUS-AI may be exposed to 38 unique cases and 135 gallbladder views in real time. The example embodiment of the real-time EUS-AI was 82% accurate in mimicking blinded experts’ safety recommendations. In addition, the system did not have any crashes and maintained latency, which may highlight its ability to be used in real time.- 23 - 5045914 vl5439.1039001

[0099] In some embodiments, the EUS-AI may be significantly superior in grading gallbladder windows as “safe” or “unsafe” when compared to Junior and Novice endosonographers .

[0100] Prior studies applying Al to EUS may have focused on lesion detection, tissue characterization, or elastography interpretation. However, example embodiments of models for an Al-based co-pilot described herein may be directed to procedural guidance in real time, which may not have previously been reported.

[0101] According to some embodiment, a first strength of the study described herein can comprise the diversity of images used for EUS-AI training. The data set may include a full spectrum of targets with borderline windows making up nearly 20% of a total number of targets. In practice, these borderline scenarios may create a clinical conundrum because safety of their drainage is not certain. Including many of these cases in training may be critical for the EUS-AI to provide appropriate guidance when a diagnosis of drainability was ambiguous. In addition, according to the example embodiment, “F” grade windows made up 41% of the training set. “F” windows may comprise images where the gallbladder was not present on exam. This could include EUS images of other structures such as the pancreas, bowel, etc. Training the EUS-AI on F windows may be critical so that a predictive model trained on the data could operate during an actual case. In some cases, EUS-AIs that are trained solely on one target may tend to crash when the EUS-Ais are exposed to new structures. Therefore, by training the EUS-AI on whole videos, and therefore all structures in an EUS exam, a trained predictive model and the computer-based co-pilot system may be more robust.

[0102] According to some embodiment, a second strength of the study described herein can comprise the EUS-AI’ s performance on unedited whole videos to mimic a real time environment. Previous studies may have shown that EUS-AI performance drops when exposed to a video test set. For example, prior studies may have described that a EUS-AI may achieve 98% sensitivity when classifying esophageal squamous cell carcinoma when provided still images. However, sensitivity dropped to 61% when provided a video test set [(7)]. Example embodiments of an EUS-AI described herein may be trained on whole unedited videos, which may therefore expose the EUS-AI co-pilot to a full spectrum of structures that could be encountered on a typical EUS exam. The benefit of whole video training may be reflected in the- 24 - 5045914 vl5439.1039001promising results from real time deployment where the EUS AT had zero crashes, minimal latency, and an accuracy of 82% when used during real time procedures

[0103] According to some embodiment, a third strength of the study described herein can comprise the EUS-AI being evaluated on a per frame basis. EUS-GBD may not only require a safe window for GB drainage but may also need a stable endoscopic position. This position can change rapidly based on the small movements by the operator. Therefore, it may be useful for any guidance program to provide a most up-to-date analysis and, in some embodiments, be able to function on a per frame basis. Embodiments of a EUS-AI described herein may perform with adequate accuracy on a per frame basis with minimal lag.

[0104] Finally, this study may highlight a clinical impact of EUS Al in successful GB drainage. Novices may perform with a sensitivity of 57% and a specificity of 87% when mimicking expert determinations of safe or unsafe EUS-GBD windows. In addition, they may perform with a false positive rate of 13%, which may be reflective of attempting drainage of a structure that would most likely lead to stent misdeployment and emergent surgery. Junior endosongraphers may perform with a sensitivity of 62% and a specificity of 87% when mimicking expert determinations of safe or unsafe EUS-GBD windows. In addition they may perform with a false positive rate of 13%, which may be reflective of attempting drainage in an unsafe situation. The EUS Al has a sensitivity of 74% and a specificity of 91% when mimicking expert determinations of safe or unsafe windows. Thus, the Al may identify more opportunities to perform EUS-GBD compared to Juniors or Novices but may also improve the false positive rate to 9%. This reduction of false positive cases may highlight the EUS Al’s ability to improve patient safety. In addition, novice may perform with a true positive rate of 57% and Juniors may perform with a true positive rate of 62%. The difference between novice and Junior endosonographers and EUS Al was 12-17%, which may reflect a lost opportunity for EUS GBD

[0105] According to some embodiments, embodiments of predictive models described herein may be tuned for a target operator. For example, a predictive model can be trained to be more liberal in its grading of “safe” windows. A different model with annotations by a less experience endosonographer may provide more a more conservative model of “safe” windows. Such a more- 25 - 5045914 vl5439.1039001conservative model may be more helpful to novice endosonographers who are at the beginning of their EUS GBD experience.

[0106] Table 2, presented hereinbelow, summarizes results from the example embodiment of the study of the EUS-AI presented hereinabove. In total, the study may comprise 23 observers (5 seniors, 13 juniors, 5 novices). Metrics of the study may comprise specificity, sensitivity, Fleiss’ K (which may be indicative of inter-rater reliability), positive predictive value (PPV), negative predictive value (NPV) and accuracy. The study may further comprise an independent panel of 3 senior observers serving as a reference standard. The panel determined 70 windows as safe and 65 windows as unsafe for EUS-GBD, and demonstrated substantial interobserver agreement (Fleiss’ K = 0.67, 95% confidence interval (CI): 0.58-0.77) in classifying EUS-GBD windows as being safe or not. For classifying EUS-GBD windows, senior observers showed moderate interobserver agreement (K = 0.51,95% CI 0.46-0.57), while junior endosonographers and novices each demonstrated only fair agreement (K =0.37, 95% CI 0.35-0.39 and K = 0.29, 95% CI 0.24-0.35, respectively). Among observers, seniors had the highest accuracy (0.81) for classifying EUS-GBD windows as safe. Senior accuracy was significantly higher than juniors (0.74; p<0.01) and novices (0.71; p<0.01).Table 2. Example Performance Metrics of Al and Operators for EUS-GBD Observer Sensitivity Specificity PPV (95% NPV (95% Accuracy Fleiss’ KGroup (95% CI) (95% CI) CI) CI) (95% CI)0.74 0.91 0.90 0.77 0.82 Al - (0.62-0.84) (0.81-0.97) (0.80-0.95) (0.69-0.83) (0.75-0.88) Seniors 0.51 0.82 0.79 0.81 0.81 0.81 (N = 5) (0.46-0.57) (0.76-0.87) (0.74-0.83) (0.77-0.84) (0.77-0.84) (0.78-0.84) Juniors 0.37 0.62 0.87 0.84 0.68 0.74 (N = 13) (0.35-0.39) (0.59-0.65) (0.85-0.89) (0.81-0.86) (0.66-0.70) (0.72-0.76) Novices 0.29 0.57 0.87 0.82 0.65 0.71 (N = 5) (0.24-0.35) (0.51-0.62) (0.82-0.90) (0.77-0.86) (0.62-0.68) (0.67-0.74)

[0107] EUS-AI had similar accuracy to seniors (0.82; p = 0.72) but significantly outperformed junior observers(p=0.036) and novices (p < 0.01)- 26 - 5045914 vl5439.1039001

[0108] FIGS. 7A-7E are plots 742a-742e of example receiver operating characteristic (ROC) curves for safety scores determined by an example embodiment of a computer-based co-pilot. The ROC curves may indicate an accuracy of the co-pilot with respect to classifying an image acquired as safety grade A (742a), B (742b), C (742c), D (742d), or F (742e ). As demonstrated in the study, EUS-AI may demonstrate effectiveness in identifying windows that are not safe for EUS-GBD

[0109] FIGS. 8A-8D are plots 842a-842d of example receiver operating characteristic (ROC) curves for predictions of a tool for performing a procedure, according to an example embodiment of a computer-based co-pilot. The example embodiment of the computer-based co-pilot may be directed to placement of LAMS, as described hereinabove, and may provide a recommendation for a stent of size no larger than 10 mm (842a), of size 15 mm or smaller (842b), or of size 20 mm or smaller (842c), or that a stent is not suitable for the procedure (842d).Example Embodiment 2 - Al-based Video Microscopy Analysis

[0110] According to some embodiments, an Al-based model developed using an embodiment of the presently disclosed invention may be helpful for assessing tissue samples in video microscopy, for example, for assessing microscopy samples of donor organs to evaluate transplant viability. Thousands of patients may die annually awaiting liver transplantation (LT) because of insufficient organ availability, which may be exacerbated by an increasing number of organs recovered but not transplanted. Donor macrosteatosis impacts allograft function and may be an essential component of donor organ viability. However, expertise for quantifying steatosis and differentiating steatosis from freezing artifacts which occurs because of the slide preparation process may be variable. Developing an artificial intelligence model for real-time point of care donor liver steatosis assessment, including differentiating steatosis versus artifact, e.g., freezing artifacts or imaging artifacts, using acquired images of a liver, for non-limiting example, video recordings of flash frozen liver biopsy (FFLB) slides, may be helpful for delivering more consistent and equitable analysis of donor liver steatosis in a wide range of clinical environments.

[0111] According to an example embodiment of an Al-based model for evaluating donor liver samples, FFLB donor specimens slides from multiple hospitals may be obtained. A high-- 27 - 5045914 vl5439.1039001definition camera attached to a standard light microscope was used to record slide examinations at lOx and 40x magnification. Still images of all video microscopy examinations were extracted and randomly sampled for annotation. An expert, blinded to the official pathologic interpretation, provided annotations of the still images. Annotations included: “microsteatosis”, “macrosteatosis”, “mixed steatosis”, “artifact”, “low quality”, and “normal.” Annotated images were randomly assigned on a per-slide basis to train and validate a convolutional neural network (CNN). Receiver operating characteristics were used to assess CNN performance for mimicking expert annotation of “macrosteatosis” and artifact.” To assess CNN performance for autonomous identification of lipid droplets versus artifact, occlusion block heatmaps (OBH) were created. OBH analysis was performed to assess what subregions the CNN valued most when classifying an image as macrosteatosis or artifact.

[0112] Thirty-five FFLB slide examinations from 15 donors were captured using video microscopy. From the videos, 2,001 still images were randomly extracted and annotated by the expert (“macrosteatosis” n=152 and “artifact” n=278). For differentiating artifact from steatosis, the CNN was 87.5% sensitive, 90.2% specific, and 89.5% accurate (AUROC 0.91, Figure 2A). Artifact was more accurately identified by occlusion block heatmap analysis when compared to macrosteatosis (96.1% vs 71.8%, p<0.01).

[0113] This study may indicate that it may be feasible to develop a video microscopy -based CNN model for the task of differentiating macrosteatosis from freezing artifact present on FFLB donor specimens. With additional training, model could be the foundation of a system that could allow for video-microscopy-based quantification of degree of steatosis. Ultimately, this AL based tool may be used for more accurate and uniform on-premises assessment of organ viability.

[0114] In an example embodiment, a computer-based system for evaluating a microscopy sample can include a processor configured to generate a prediction using a predictive model for an image of a sequence of microscopy images. The prediction can include a quality of the image and an assessment of the image, the assessment identifying evidence of a feature in the image. The processor can be further configured to determine a score for each image of the subset based on the prediction generated, the score indicating a presence of the feature, an extent of the feature, or a presence of an imaging artifact confusable with the target feature. The processor can- 28 - 5045914 vl5439.1039001be still further configured to produce data representative of a composite image for each image of the subset based on the image, the prediction generated for the image, and the score determined for the image. The data produced may indicate a likelihood of the true feature in the image to evaluate the microscopy sample. According to some embodiments, an imaging artifact confusable with a target feature can comprise, for non-limiting example, an artifact confusable with said target feature (e.g., freezing artifact with respect to steatosis) to a human eye or by conventional processing techniques but discernable by embodiments disclosed herein based on Al-based microscopy analysis.

[0115] FIG. 9 illustrates an example embodiment of a computer-based system 906 for evaluating a microscopy sample. The system 906 can comprise a computer 944, which may comprise a processor configured to generate a prediction using a predictive model. The computer 944 (and the processor) can be communicatively coupled to a microscopy device 946 configured to acquire a sequence of microscopy images. In some embodiments, the computer 944 may be communicatively coupled to a camera 948, the camera 948 in turn being configured to acquire the sequence of images from the microscopy device 944. In some embodiments, the camera 946 may be configured to acquire a video or sequence of images, which may be streamed to the computer 944 and rendered to an electronic display 950.

[0116] As described hereinabove, the prediction can include a quality of an image of the sequence of microscopy images and an assessment of the image. The assessment may identify evidence of a feature in the image. According to the example embodiment described hereinabove with respect to liver transplantation, the feature can comprise steatosis (e.g., microsteatosis, macrosteatosis, or mixed steatosis) or an artifact, which may be visually similar to steatosis under visual evaluation of the microscopy images. The processor may be further configured to determine a score for each image of the subset based on the prediction generated, the score indicating a presence of the feature, an extent of the feature, or a presence of an artifact. The processor may be further configured to produce data representative of a composite image for each image of the subset based on the image and the score determined for the image. According to some embodiments, the processor can be configured to cause a rendering of the composite images on an electronic display communicatively coupled to the processor, e.g., the electronic display 950, which may be configured for live streaming slides.- 29 - 5045914 vl5439.1039001

[0117] FIGS. 10A and 10B illustrate example slides 1042a, 1042b of donor liver samples that may be processed by an example embodiment of a computer-based system for evaluating a microscopy sample. The slide 1042a is an example slide of a liver biopsy sample comprising macrosteatosis and the slide 1042b is an example slide of a liver biopsy sample with a freezing artifact that may be visually similar to macrosteatosis.

[0118] FIGS. 10C and 10D illustrate occlusion block heatmaps 1042c, 1042d of the example slides of FIGS. 10A and 10B, respectively. Occlusion block heatmaps may indicate relative importance of regions of the slides in, by a predictive model, forming a prediction.

[0119] FIG. 11 is a plot 1142 of a receiver operating characteristic (ROC) curve of an example embodiment of a computer-based system for evaluating a microscopy sample. As illustrated, the example embodiment of the computer-based system may exhibit an area under the curve (AUC) of 0.91 for identifying steatosis (e g., with respect to absence of features or artifacts) in liver biopsy slides.Embodiment 3 - Al-model for Ultrasound Evaluation of Colon

[0120] In some embodiments, an Al-based model developed using an embodiment of the presently disclosed invention may be helpful in diagnostic imaging procedures, for example, guiding a diagnostic imaging procedure or identifying a pathology present in a diagnostic imaging procedure. In an example embodiment, ultrasonography may be useful as a tool for identifying evidence of pathophysiological alterations associated with inflammatory bowel disease (IBD). While ultrasound (US) is widely accessible, technical limitations in identifying a colon in ultrasound imaging and identifying the pathophysiological alterations previously mentioned may prevent widespread use of such a technique.

[0121] An Al model may be helpful for identifying the colon in US imaging and for determining whether evidence of IBD is present within US images. The Al model may, based on the evidence, generate a score of active IBD based on the US images acquired. Such embodiments may be helpful for training physicians to utilize US imaging for IBD diagnosis or for enabling greater accessibility of US imaging for IBD evaluation.

[0122] In an example embodiment, a computer-based system for assisting a diagnostic imaging procedure may include a processor configured to generate a prediction using a- 30 - 5045914 vl5439.1039001predictive model for an image of a sequence of medical images. The sequence of medical images may be acquired by medical imaging modality, for example, US imaging. The prediction includes a quality of the image, identification of a target anatomy, and evidence of a pathology in the image. The processor may be configured to determine a score for each image of the sequence based on the prediction generated, the score indicating a likelihood of the pathology. The processor may be further configured to produce data representative of a composite image for each image of the sequence based on the image, the prediction generated for the image, and the score determined for the image. The data produced may indicate a likelihood of the pathology in the image to assist the diagnostic medical imaging procedure.

[0123] In a specific embodiment, a medical imaging modality may be ultrasound imaging, and the images may be an image or a sequence of images acquired by an ultrasound system. The ultrasound system may acquire images of one or more types of views, for example, curvilinear scans or linear scans. In some embodiments, an Al model may use images from a given view, e.g., the curvilinear view, to identify the colon in the images. The model may provide information regarding a quality of the given image or a presence of the colon in the given image as part of a composite image. For example, data of the composite image may include scores for “Colon in View,” indicating that the colon is in view in the given image, or “Inadequate View,” indicating that the colon could not be identified in the given image. Based on the scores, the Al model may present a prediction, for example, a “Satisfactory View” or an “Unsatisfactory View.” The “Satisfactory View” may prompt an operator to progress to a next stage of the imaging procedure while the “Unsatisfactory View” may indicate further scanning is needed in the current step and imaging view, e.g., the curvilinear scans.

[0124] In the next stage, the operator may switch from curvilinear scanning to linear scanning to identify signs of a pathological condition, e.g., IBD. The Al model may continue to determine the scores of the view and the prediction of the view in the next stage. Additionally, the Al model may further determine an activity score, which may be a score of the likelihood of the pathological condition. For example, the activity score may include a low likelihood score, a medium likelihood score, and a high likelihood score. The Al model may further generate an interpretation, which predicts an overall likelihood of the pathological condition based on the activity score.- 31 - 5045914 vl5439.1039001

[0125] FIGS. 12A and 12B illustrate an example raw image and an example composite image, respectively, of a curvilinear ultrasound scan that may be acquired or generated by an example embodiment of a computer-based system for assisting a diagnostic imaging procedure. The images of FIGS. 12A and 12B may illustrate a curvilinear ultrasound view. The composite image of FIG. 12B may include one or more of the prediction generated based on the image or a score determined for the image. For example, the prediction or inference can comprise whether an organ, e.g., a colon, is in view, whether a view is satisfactory, and whether there is evidence of a pathology in the view. As illustrated, the composite image of FIG. 12B may indicate an inadequate view wherein a target organ has not been identified. In some embodiments, features of the composite image may provide a probabilistic estimation or prediction of the view of the target organ (e.g., status bars for “Colon in view” or “Inadequate view”, as illustrated) or a deterministic evaluation (e.g., the “SATISFACTORY VIEW” determination).

[0126] FIG. 12C illustrates an example composite image of a curvilinear ultrasound scan that may be generated by an example embodiment of a computer-based system for assisting a diagnostic imaging procedure. The composite image may indicate that the colon is in view and that the diagnostic imaging procedure can proceed to a subsequent step.

[0127] FIGS. 12D and 12E illustrate an example raw image and an example composite image, respectively, of a linear ultrasound scan that can be acquired or generated by an example embodiment of a computer-based system for assisting a diagnostic imaging procedure. In some embodiments, the linear scan may be utilized after a target organ, e.g., a colon, has been identified using a curvilinear scan and may be used for an assessment or diagnosis of the target organ. The composite image of FIG. 12E may be similar to the composite image of FIG. 12B and may further include one or more of a prediction or a score associated with a presence of a pathology, e.g., inflammatory bowel disease (IBD) in the colon. The image may further comprise an interpretation based on said activity or score.

[0128] It should be understood that the embodiments described hereinabove are provided for exemplification and not intended to be limiting in any way.

[0129] Computer Support- 32 - 5045914 vl5439.1039001

[0130] FIG. 13 is a schematic view of a computer network in which embodiments may be implemented. Client computer(s) / devices 50 and server computer(s) 60 provide processing, storage, and input / output (I / O) devices executing application programs and the like. Client computer(s) / device(s) 50 can also be linked through communications network 70 to other computing devices, including other client device(s) / processor(s) 50 and server computer(s) 60. The communications network 70 can be part of a remote access network, a global network (e.g., the Internet), cloud computing servers or service, a worldwide collection of computers, local area or wide area networks, and gateways that currently use respective protocols (e.g, TCP / IP, Bluetooth®, etc.) to communicate with one another. Other electronic device / computer network architectures are also suitable.

[0131] FIG. 14 is a block diagram illustrating an example embodiment of a computer node (e.g., client processor(s) / device(s) 50 or server computer(s) 60) in the computer network 70 of FIG. 13. Each computer node 50, 60 contains system bus 79, where a bus is a set of hardware lines used for data transfer among components of a computer or processing system. The system bus 79 is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, I / O ports, network ports, etc.) that enables transfer of information between the elements. Attached to the system bus 79 is an I / O devices interface 82 for connecting various input and output devices (e.g., keyboard, mouse, display(s), printer(s), speaker(s), etc.) to the computer node 50, 60. A network interface 86 allows the computer node to connect to various other devices attached to a network (e.g., the network 70 of FIG. 13). A memory 90 provides volatile storage for computer software instructions 92a and data 94a used to implement embodiments of the present disclosure (e.g, the method 101 of FIG. 1, etc.). A disk storage 95 provides non-volatile storage for the computer software instructions 92b and data 94b used to implement an embodiment of the present disclosure. A central processor unit (CPU) 84 is also attached to the system bus 79 and provides for execution of computer instructions.

[0132] In an embodiment, the processor routines 92a-92b and data 94a-94b are a computer program product (generally referenced as 92), including a non-transitory, computer readable medium (e.g, a removable storage medium such as DVD-ROM(s), CD-ROM(s), diskette(s), tape(s), etc.) that provides at least a portion of the software instructions for the disclosure methods. The computer program product 92 can be installed by any suitable software installation- 33 - 5045914 vl5439.1039001procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication, and / or wireless connection. In other embodiments, the disclosure programs are a computer program propagated signal product embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)). Such carrier medium or signals provide at least a portion of the software instructions for the present disclosure routines / program 92.

[0133] In alternative embodiments, the propagated signal is an analog carrier wave or digital signal carried on the propagated medium. For example, the propagated signal may be a digitized signal propagated over a global network (e.g., the Internet), a telecommunications network, or other networks (such as the network 70 of FIG. 13). In one embodiment, the propagated signal is a signal that is transmitted over the propagation medium over a period of time, such as the instructions for a software application sent in packets over a network over a period of milliseconds, seconds, minutes, or longer. In another embodiment, the computer readable medium of the computer program product 92 is a propagation medium that the computer system 50 may receive and read, such as by receiving the propagation medium and identifying a propagated signal embodied in the propagation medium, as described above for computer program propagated signal product.

[0134] Generally speaking, the term “carrier medium” or transient carrier encompasses the foregoing transient signals, propagated signals, propagated medium, storage medium, and the like.

[0135] In other embodiments, the program product 92 may be implemented as a so-called Software as a Service (SaaS), or other installation or communication supporting end-users.

[0136] Embodiments or aspects thereof may be implemented in the form of hardware including but not limited to hardware circuitry, firmware, or software. If implemented in software, the software may be stored on any non-transient computer readable medium that is configured to enable a processor to load the software or subsets of instructions thereof. The processor then executes the instructions and is configured to operate or cause an apparatus to operate in a manner as described herein.- 34 - 5045914 vl5439.1039001

[0137] Further, hardware, firmware, software, routines, or instructions may be described herein as performing certain actions and / or functions of the data processors. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

[0138] It should be understood that the flow diagrams, block diagrams, and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

[0139] Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and / or some combination thereof, and, thus, the data processors described herein are intended for purposes of illustration only and not as a limitation of the embodiments.

[0140] The teachings of all patents, published applications and references cited herein or in documents being filed herewith are incorporated by reference in their entirety.

[0141] While example embodiments have been particularly shown and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the embodiments encompassed by the appended claims.References

[0142] 1. Lai PB, Kwong KH, Leung KL, et al. Randomized trial of early versus delayed laparoscopic cholecystectomy for acute cholecystitis Br J Surg 6, 1998;85: 764-767. PMID: 9667702

[0143] 2. Okamoto K, Suzuki K, Takada T, et al. Tokyo Guidelines 2018: flowchart for the management of acute cholecystitis J Hepatobiliary Pancreat Sci 1, 2018;25: 55-72. PMID:29045062- 35 - 5045914 vl5439.1039001

[0144] 3. Teoh AYB, Serna C, Penas I, et al. Endoscopic ultrasound-guided gallbladder drainage versus percutaneous cholecystostomy in high-risk surgical patients with acute cholecystitis: a systematic review and meta-analysis. Endoscopy. 2020;52(2):96-106.

[0145] 4. Choi JH, Lee SS, Choi JH, et al. EUS-guided transmural gallbladder drainage in patients with acute cholecystitis and high surgical risk: a randomized clinical trial. Gastrointest Endosc. 2022;95(3):568-577.el.

[0146] 5. Jang JW, Lee SS, Song TJ, et al. Long-term outcomes of EUS-guided gallbladder drainage compared with percutaneous drainage for acute cholecystitis. Gastrointest Endosc. 2018;88(3):527-536.el.

[0147] 6. Mohan BP, Khan SR, Trakroo S, et al. EUS-guided gallbladder drainage: a systematic review and meta-analysis comparing EUS-GBD and percutaneous cholecystostomy. Endosc Int Open. 2020;8(l):E632-E642.

[0148] 7. Guo, L., Xiao, X., Wu, C., Zeng, X., Zhang, Y., Du, I, Bai, S., Xie, J., Zhang, Z., Li, Y., Wang, X., Cheung, O., Sharma, M., Liu, J., & Hu, B. (2020). Real time automated diagnosis of precancerous lesions and early esophageal squamous cell carcinoma using a deep learning model (with videos). Gastrointestinal Endoscopy, 91(1), 41-51.5045914 vl

Claims

5439.1039001CLAIMSWhat is claimed is:

1. A method for deploying a predictive model for classification of at least one medical image of a clinical or research patient, the method comprising:from a repository of medical images acquired from a cohort of patients, selecting a subset of the repository, the selecting the subset including an initial iteration of selecting a randomized image from each patient of the cohort and at least one subsequent iteration of selecting a randomized image from each remaining patient of the cohort; creating an annotation associated with each medical image of the subset selected; training a predictive model based on content of the medical images of the subset selected and the associated annotations created; andintegrating the predictive model trained with a computer-based system communicatively coupled to a medical imaging device, the computer-based system being configured to apply the predictive model for classification of the at least one medical image of the clinical or research patient.

2. The method of claim 1, wherein integrating the predictive model trained with the computer-based system includes:acquiring the at least one medical image of the clinical patient using the medical imaging system;identifying a selection of the at least one medical images acquired; generating a prediction for each image of the selection identified using the predictive model trained; andproducing data representative of a composite image for each image of the selection and the corresponding prediction generated.

3. The method of claim 2, further comprising generating a composite prediction based on at least a portion of the plurality of images received.

4. The method of claim 2, further comprising repeating the acquiring the at least one medical image, identifying the selection, generating the prediction, and produce the data- 37 - 5045914 vl5439.1039001in an iterative or real-time manner an imaging procedure of the clinical or research patient is complete.

5. The method of claim 1, wherein creating the annotation associated with each medical image includes providing, for a given medical image, one or more of a quality, a segmentation, a stage of a medical procedure, or a suggestion on a suggested medical device.

6. The method of claim 1, further comprising anonymizing the at least one medical image of the clinical patient to produce at least one anonymized image of the clinical patent.

7. The method of claim 6, further comprising updating the repository of medical images from the retrospective cohort of patients with the at least one anonymized medical images.

8. The method of claim 7, further comprising re-training the predictive model based on the repository updated.

9. The method of claim 1, further comprising preprocessing the repository of medical images prior to creating the annotation, wherein preprocessing includes at least one of normalizing, cropping, or labeling.

10. The method of claim 1, wherein the repository of medical images is a repository that includes videos, and wherein the method further comprises extracting images from the videos.

11. The method of claim 1, training the predictive model comprises feeding the medical images and the associated annotations created through a convolutional neural network.

12. A computer-based copilot providing guidance during an image-guided procedure, the copilot comprising:a processor configured to:- 38 - 5045914 vl5439.1039001generate a prediction using a predictive model for an image of a sequence of images, the prediction including a quality of the image and a current stage of a sequence of stages of the image-guided procedure;determine a score for each image of the sequence based on the prediction generated, the score indicating one or more of a safety grade of the image-guided procedure or a readiness to progress from the current stage to a subsequent stage of the sequence of stages; andproducing data representative of a composite image based on the image, the prediction generated for the image, and the score determined for the image, the data produced providing guidance during an image-guided procedure.

13. The computer-based copilot of claim 12, further comprising a medical imaging device, communicatively coupled with the processor, configured to acquire images of the image- guided procedure, wherein the sequence of images is a subset of the images acquired.

14. The computer-based copilot of claim 13, wherein the medical imaging device is an optical, ultrasound, or an x-ray imaging device.

15. The computer-based copilot of claim 12, wherein the processor is configured to cause a therapeutic device, communicatively coupled to the processor, to perform at least part of the image-guided procedure based on the prediction generated, the score determined, or a user input.

16. The computer-based copilot of claim 12, wherein the processor is configured, generate the prediction, determine the score, and produce the data representative of the composite image in real time.

17. The computer-based copilot of claim 12, further comprising an electronic display communicatively coupled to the processor, wherein the processor is configured to render the composite image on the electronic display.

18. The computer-based copilot of claim 12, further comprising a therapeutic device including an imaging sensor, the imaging sensor configured to acquire an additional- 39 - 5045914 vl5439.1039001sequence of images, and wherein the processor is configured to generate the prediction using the image of the sequence of images and an other image of the additional sequence of images.

19. The computer-based copilot of claim 12, wherein the prediction generated further includes an optimal tool for performing the image-guided procedure.

20. The computer-based copilot of claim 12, wherein the data produced includes, for an image of the subset with a low score, guidance for improving the image.

21. A computer-based system for evaluating a microscopy sample, the computer-based system comprising:a processor configured to:generate a prediction using a predictive model for an image of a sequence of microscopy images, the prediction including a quality of the image and an assessment of the image, the assessment identifying evidence of a target feature in the image;determine a score for the image based on the prediction generated, the score indicating a presence of the target feature, an extent of the target feature, or a presence of an imaging artifact confusable with the target feature; and producing data representative of a composite image based on the image, the prediction generated, and the score determined, the data produced indicating a likelihood of the target feature in the image to evaluate the microscopy sample.

22. The computer-based system of claim 21, further comprising a video microscopy system, communicatively coupled with the processor, configured to acquire images of a microscopy sample, wherein the sequence of microscopy images is a subset of the images acquired.

23. The computer-based system of claim 21, wherein the processor is configured to generate the prediction, determine the score, and produce the data in real time.- 40 - 5045914 vl5439.103900124. The computer-based system of claim 21 , further comprising an electronic display communicatively coupled to the processor, wherein the processor is configured to render the composite image on the electronic display.

25. The computer-based system of claim 21, wherein the data produced further includes likelihood of a tissue processing artifact.

26. A computer-based system for assisting a diagnostic imaging procedure, the computer- based system comprising:a processor configured to:generate a prediction using a predictive model for an image of a sequence of medical images from a diagnostic imaging procedure, the prediction including a quality of the image, a presence of an anatomical feature within the image, and evidence of a pathology in the anatomical feature within the image;determine a score for each image of the sequence based on the prediction generated, the score indicating a likelihood of the pathology; and produce data representative of a composite image based on the image, the prediction generated for the image, and the score determined for the image, the data produced indicating one or more of the presence of the anatomical feature, the evidence of the pathology in the anatomical feature, or the score determined to assist the diagnostic medical imaging procedure.

27. The computer-based system of claim 26, further comprising a medical imaging system, communicatively coupled with the processor, configured to acquire images of a human or animal, wherein the sequence of medical images is a subset of the images acquired.

28. The computer-based system of claim 26, wherein the processor is configured to generate the prediction, determine the score, and produce the data in real time.

29. The computer-based system of claim 26, further comprising an electronic display communicatively coupled to the processor, wherein the processor is configured to render the composite image on the electronic display.- 41 - 5045914 vl

Images