Medical foundation model for diagnostic image classification across multiple imaging modalities

Abstract

A method for training a medical foundation model comprises obtaining a corpus of medical journal papers, extracting images from each medical journal paper in the corpus, filtering out abstract figures from the extracted images wherein the abstract figures comprise non-medical images, generating augmented figure descriptions based on context extracted from publication text associated with each extracted image, generating image / description pairs by combining filtered medical images with corresponding augmented figure descriptions, annotating each image / description pair using a medical ontology to extract medical concepts from textual content associated with each image, and training an image encoder model and a text encoder model using contrastive language-image pre-training based on the annotated image / description pairs to yield a trained medical foundation model.

Description

MEDICAL FOUNDATION MODEL FOR DIAGNOSTIC IMAGE CLASSIFICATION ACROSS MULTIPLE IMAGING MODALITIES CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Application No. 63 / 772,394, titled "Physician-Level Classification Performance Across Multiple Imaging Domains with a Diagnostic Medical Foundation Model and a Large Dataset of Annotated Medical Images", filed March 14, 2025, which is hereby incorporated by reference in its entirety.FIELD OF INVENTION

[0002] The present disclosure relates to artificial intelligence systems for medical image analysis, and more particularly to a diagnostic medical foundation model trained on annotated medical images for physician-level classification performance across multiple imaging modalities and medical specialties.BACKGROUND

[0003] Machine learning is a subset of artificial intelligence that enables computer systems to learn from data and improve their performance on tasks without being explicitly programmed for each specific task. Machine learning algorithms build mathematical models based on training data, identifying patterns and relationships within the data that can then be applied to new, unseen data. Various types of machine learning approaches exist, including supervised learning where models are trained on labeled data with known outputs, unsupervised learning where models identify patterns in unlabeled data, and reinforcement learning where models learn through interaction with an environment. The training process for machine learning models typically involves optimizing model parameters to minimize a loss function that measures the difference between predicted outputs and actual outputs, with techniques such as gradient descent being commonly employed to iteratively adjust parameters toward improved performance.

[0004] Large language models are neural networks trained on vast amounts of text data that can understand and generate human language, enabling capabilities such as text completion, question answering, and natural language understanding acrossdiverse domains. Contrastive Language-Image Pre-training (CLIP) is a neural network architecture that learns visual concepts from natural language supervision by training on image-text pairs. CLIP jointly trains an image encoder and a text encoder to predict which images and texts in a batch are paired together, enabling the model to understand relationships between visual and textual information.SUMMARY

[0005] Systems and methods for medical image classification in accordance with embodiments of the invention are illustrated. In one embodiment, a method for training a medical foundation model includes obtaining a corpus of medical journal papers, extracting images from each medical journal paper in the corpus, filtering out abstract figures from the extracted images, wherein the abstract figures include non-medical images, generating augmented figure descriptions based on context extracted from publication text associated with each extracted image, generating image / description pairs by combining filtered medical images with corresponding augmented figure descriptions, annotating each image / description pair using a medical ontology to extract medical concepts from textual content associated with each image, and training an image encoder model and a text encoder model using contrastive language-image pre-training based on the annotated image / description pairs to yield a trained medical foundation model.

[0006] In one embodiment, filtering out abstract figures includes classifying extracted images using a convolutional neural network into a plurality of image types including individual medical images, image panels containing multiple medical images, mixed panels containing both medical images and abstract visuals, and abstract visuals only, wherein the filtering retains individual medical images, image panels, and mixed panels while excluding abstract visuals only.

[0007] In one embodiment, generating augmented figure descriptions includes using a fine-tuned large language model to extract diagnoses from a paper title and paragraphs that reference a figure associated with each extracted image.

[0008] In one embodiment, the fine-tuned large language model includes a BioMedLM model pre-trained on biomedical literature.

[0009] In one embodiment, annotating each image / description pair using a medical ontology includes extracting medical concepts using a Unified Medical Language System parser.

[0010] In one embodiment, each identified medical concept is associated with a Concept Unique Identifier from the Unified Medical Language System.

[0011] In one embodiment, the image encoder model includes a vision transformer.

[0012] In one embodiment, the vision transformer includes a ViT-Large model pretrained on a large-scale dataset of image-text pairs.

[0013] In one embodiment, the text encoder model includes a transformer-based language model pre-trained on biomedical literature.

[0014] In one embodiment, the transformer-based language model includes PubMedBert.

[0015] In one embodiment, the method further includes applying data augmentation to the image / description pairs during training.

[0016] In one embodiment, training the image encoder model and the text encoder model includes using a contrastive loss function that maximizes similarity between embeddings of corresponding image / description pairs and minimizes similarity between embeddings of non-corresponding pairs.

[0017] In one embodiment, a method for classifying a medical image includes obtaining an input medical image, generating image-specific domain labels including medical terms relevant to a classification task, providing the input medical image and the image-specific domain labels to a trained medical foundation model, wherein the trained medical foundation model includes an image encoder component and a text encoder component, obtaining cosine similarities between the input medical image and each of the image-specific domain labels from the trained medical foundation model, wherein the image encoder component generates an image embedding for the input medical image and the text encoder component generates text embeddings for each of the image-specific domain labels, refining a prediction using a modified SoftMax function, wherein the modified SoftMax function aggregates logits belonging to a same medical condition within positive and negative categories to generateaggregated scores for each medical condition before applying a SoftMax transformation to generate probability scores, and providing the prediction.

[0018] In one embodiment, the input medical image includes at least one of a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, or a clinical photograph.

[0019] In one embodiment, the image-specific domain labels include terms describing disease conditions and terms describing normalcy.

[0020] In one embodiment, generating image-specific domain labels includes automatically generating the domain labels based on a type of the input medical image.

[0021] In one embodiment, the cosine similarity between the image embedding and each text embedding is calculated as a dot product of the image embedding and the text embedding divided by a product of their respective magnitudes.

[0022] In one embodiment, the modified SoftMax function calculates an average of cosine similarity logits for medical terms within a positive category for a medical condition and an average of cosine similarity logits for medical terms within a negative category for the medical condition.

[0023] In one embodiment, the modified SoftMax function applies a SoftMax transformation to the averaged logits for the positive and negative categories to generate probability scores for each medical condition.

[0024] In one embodiment, the positive category includes medical terms indicative of a target condition and the negative category includes medical terms indicative of normalcy or alternative conditions for binary classification.

[0025] In one embodiment, the method performs zero-shot classification of the input medical image without requiring task-specific training data for the classification task.

[0026] In one embodiment, the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

[0027] In one embodiment, the modified SoftMax function calculates probability scores according to a cosine similarity between the image embedding and each textembedding calculated as Zi;= cos(7 7}), where represents the cosine similarity logit for image i and text description j, represents the image embedding vector, and Tj represents the text embedding vector, an aggregation of logits for medical terms belonging to a same medical condition calculated as Sik= where Sikrepresents the aggregated score for medical condition k, Nkrepresents the number of medical terms belonging to condition category k, and Ckrepresents the set of medical term indices belonging to condition category k, and a probability that the input medical esikimage belongs to medical condition k calculated as Pik= - — , where K representsthe set of all medical condition categories being considered for classification.

[0028] In one embodiment, a method for classifying a medical image includes obtaining an input medical image, generating image-specific domain labels including medical terms relevant to a classification task, providing the input medical image and the image-specific domain labels to a trained medical foundation model, wherein the trained medical foundation model includes an image encoder component and a text encoder component, obtaining cosine similarities between the input medical image and each of the image-specific domain labels from the trained medical foundation model, wherein the image encoder component generates an image embedding for the input medical image and the text encoder component generates text embeddings for each of the image-specific domain labels, and using a domain-specific classical machine learning model to refine a diagnostic prediction based on the cosine similarities.

[0029] In one embodiment, the input medical image includes at least one of a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, or a clinical photograph.

[0030] In one embodiment, the image-specific domain labels include terms describing conditions relevant to a diagnostic domain of the input medical image.

[0031] In one embodiment, generating image-specific domain labels includes automatically generating the domain labels based on a type of the input medical image.

[0032] In one embodiment, the cosine similarities serve as input features to the domain-specific classical machine learning model.

[0033] In one embodiment, the domain-specific classical machine learning model includes at least one of a Logistic Regression model, a Random Forest model, a Gradient Boosting model, or a Support Vector Machine.

[0034] In one embodiment, the domain-specific classical machine learning model is trained on a set of domain-specific medical images using cosine similarity features extracted from the domain-specific medical images by the trained medical foundation model.

[0035] In one embodiment, the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

[0036] In one embodiment, a method for classifying a pathology slide image includes obtaining an input pathology slide image, segmenting the input pathology slide image into a plurality of image patches, providing each image patch to a trained medical foundation model along with potential disease labels, wherein the trained medical foundation model includes an image encoder component and a text encoder component, obtaining cosine similarities between each of the potential disease labels and each image patch from the trained medical foundation model, wherein the image encoder component generates an image embedding for each image patch and the text encoder component generates text embeddings for each of the potential disease labels, and calculating a mean of each cosine similarity for each potential disease label across the plurality of image patches to generate a classification prediction for the input pathology slide image.

[0037] In one embodiment, the input pathology slide image includes a whole-slide image captured from a histopathology specimen using a digital pathology scanner.

[0038] In one embodiment, segmenting the input pathology slide image into a plurality of image patches includes extracting tiles from the input pathology slide image at specified magnification levels and spatial dimensions.

[0039] In one embodiment, the tile extraction includes automated tissue detection to ensure that extracted tiles contain sufficient tissue content for analysis.

[0040] In one embodiment, the sufficient tissue content includes a minimum of 80% tissue coverage.

[0041] In one embodiment, the tissue coverage is validated using grayscale conversion and Otsu thresholding.

[0042] In one embodiment, the potential disease labels include cancer subtypes, benign conditions, and normal tissue classifications relevant to the histopathology specimen being analyzed.

[0043] In one embodiment, the method further includes aggregating from tile-level predictions to a patient-level diagnosis by excluding normal tissue scores.

[0044] In one embodiment, the aggregating includes identifying tiles with top scores for cancer subtypes for generating the patient-level diagnosis.

[0045] In one embodiment, the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

[0046] In one embodiment, a method for retrieving relevant medical literature based on an input medical image includes obtaining an input medical image, providing the input medical image and a database of image / description pairs to a trained medical foundation model, wherein the trained medical foundation model includes an image encoder component, identifying most similar images to the input medical image based on image embeddings generated by the image encoder component of the trained medical foundation model, = ranking the most similar images based on cosine similarities between the image embedding of the input medical image and text embeddings of diagnoses extracted from descriptions associated with each similar image using the trained medical foundation model, and returning a paper from which a highest ranked image originates.

[0047] In one embodiment, the input medical image includes at least one of a clinical photograph, a radiographic image, ora histopathology slide.

[0048] In one embodiment, the database of image / description pairs includes medical images extracted from medical journal papers with associated textual descriptions including augmented figure descriptions and medical ontology annotations.

[0049] In one embodiment, image embeddings for images in the database are precomputed and stored for efficient retrieval.

[0050] In one embodiment, cosine similarity is used to measure similarity between image embeddings.

[0051] In one embodiment, the predicted diagnosis is generated based on cosine similarities between an embedding of the input medical image and text embeddings for medical terms.

[0052] In one embodiment, the ranking prioritizes images whose associated descriptions contain diagnoses that align with the predicted diagnosis for the input medical image.

[0053] In one embodiment, the method further includes returning multiple papers corresponding to top-ranked similar images.

[0054] In one embodiment, returning the paper includes returning publication details including at least one of a paper title, authors, journal name, publication date, or a link to a full publication.

[0055] In one embodiment, the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

[0056] In one embodiment, a method for detecting medical emergencies using a machine includes obtaining a video stream from at least one camera sensor on the machine, extracting image frames from the video stream, providing the extracted image frames and medical emergency description labels to a trained medical foundation model to obtain cosine similarities between each image frame and each medical emergency description label, classifying each image frame as depicting a medical emergency ora non-medical emergency based on the cosine similarities, and in response to classifying an image frame as depicting a medical emergency, initiating an emergency communication by the machine including contacting an emergency service line and transmitting a description of a nature of the medical emergency and a number of individuals involved.

[0057] In one embodiment, the machine comprises a humanoid robot equipped with at least one camera sensor for capturing the video stream.

[0058] In one embodiment, the method further includes determining whether the medical emergency was caused by an action of the machine itself and including this determination in the emergency communication.

[0059] In one embodiment, the description of the nature of the medical emergency is generated by the trained medical foundation model based on the cosine similarities between the image frame and the medical emergency description labels.

[0060] In one embodiment, a method for classifying an image / text pair as medical or non-medical using a trained medical foundation model includes obtaining an input image and an input text description, computing a direct cosine similarity between an image embedding of the input image and a text embedding of the input text description using the trained medical foundation model, computing a medical image-text similarity as an average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known medical text descriptions, computing a medical text-image similarity as an average of cosine similarities between the text embedding of the input text description and image embeddings of a plurality of known medical images, computing a non-medical image-text similarity as an average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known non-medical text descriptions, computing a non-medical text-image similarity as an average of cosine similarities between the text embedding of the input text description and image embeddings of a plurality of known non-medical images, calculating a medical domain specificity score based on the direct cosine similarity, the medical image-text similarity, the medical text-image similarity, the non-medical image-text similarity, and the non-medical text-image similarity, and classifying the image / text pair as medical or non-medical based on the medical domain specificity score.

[0061] In one embodiment, the medical domain specificity score is calculated as MDSS = (S\_img\_med + S\_txt\_med) - (S\_img\_nonmed + S\_txt\_nonmed), where S\_img\_med represents the medical image-text similarity, S\_txt\_med represents themedical text-image similarity, S\_img\_nonmed represents the non-medical image-text similarity, and S\_txt\_nonmed represents the non-medical text-image similarity.

[0062] In one embodiment, the medical domain specificity score is calculated using a weighted combination of cosine similarity ratios according to MDSS = a (S_direct / S_img_med) + b (S_direct / S_img_nonmed) + c (S_direct / S_txt_med) + d (S_direct / S_txt_nonmed), where S_direct represents the direct cosine similarity between the image embedding and the text embedding, S_img_med represents the medical image-text similarity, S_img_nonmed represents the non-medical image-text similarity, S_txt_med represents the medical text-image similarity, S_txt_nonmed represents the non-medical text-image similarity, and a, b, c, and d represent coefficients determined by a logistic regression model trained on a validation cohort of labeled medical and non-medical image / text pairs. In one embodiment, the image / text pair is classified as medical when the medical domain specificity score exceeds a predetermined threshold and as non-medical when the medical domain specificity score falls below the predetermined threshold.

[0063] In one embodiment, the predetermined threshold is optimized based on a validation set of labeled medical and non-medical image / text pairs.

[0064] In one embodiment, the plurality of known medical text descriptions includes at least 1000 text descriptions known to describe medical images.

[0065] In one embodiment, the plurality of known non-medical images includes at least 1000 images known to be non-medical images.

[0066] In one embodiment, image embeddings for the plurality of known medical images, text embeddings for the plurality of known medical text descriptions, image embeddings for the plurality of known non-medical images, and text embeddings for the plurality of known non-medical text descriptions are pre-computed and stored for efficient classification.

[0067] In one embodiment, the method is used for automated knowledge discovery on the internet by scraping image / text pairs from publicly accessible web sources, classifying the scraped image / text pairs as medical or non-medical, and collecting the medical image / text pairs to create a training dataset for improving performance of the trained medical foundation model.

[0068] In one embodiment, the method is used iteratively, wherein the trained medical foundation model is used to discover medical image / text pairs on the internet, the discovered pairs are used to retrain the trained medical foundation model, and the retrained model is used to discover additional medical content with greater accuracy.

[0069] In one embodiment, a method for creating a dermatological disease classifier includes obtaining a dermatological image dataset annotated with diagnoses, skin color classifications, and body location identifiers, providing the dermatological image dataset to a trained medical foundation model including an image encoder component and a text encoder component to generate cosine similarity features between each image and a set of dermatological disease labels, and training a domain-specific dermatological classifier based on the cosine similarity features to classify dermatological diseases across a broad range of conditions.

[0070] In one embodiment, the dermatological image dataset includes approximately 138,000 images scraped from publicly accessible internet sources.

[0071] In one embodiment, the broad range of dermatological conditions includes acne vulgaris, atopic dermatitis, contact dermatitis, seborrheic dermatitis, psoriasis, dermatophyte infections, rosacea, viral warts, scabies, and urticaria.

[0072] In one embodiment, each image in the dermatological image dataset is annotated with at least a diagnosis, a skin color classification, and a body location identifier.

[0073] In one embodiment, the skin color classifications and body location identifiers are incorporated as additional input features to the domain-specific dermatological classifier alongside the cosine similarity features.

[0074] In one embodiment, the dermatological image dataset is created using a medical domain specificity score to identify and collect dermatological image / text pairs from the internet

[0075] In one embodiment, a method for diagnosing a rare disease or an unknown disease using a trained medical foundation model includes obtaining an input medical image of a patient suspected of having the rare disease or the unknown disease, querying at least one search engine using a suspected condition name to retrieve reference images depicting the suspected condition from publicly accessible internet sources, generating image embeddings for the input medical image and for each retrieved reference image using an image encoder component of the trained medical foundation model, computing cosine similarities between the image embedding of the input medical image and the image embeddings of the retrieved reference images, repeating the querying, generating, and computing steps for at least one differential diagnosis condition, and aggregating the cosine similarities across the suspected condition and the at least one differential diagnosis condition to generate a diagnostic prediction.

[0076] In one embodiment, the reference images are retrieved in real time from the at least one search engine in response to receiving the input medical image.

[0077] In one embodiment, the at least one differential diagnosis condition is automatically generated based on the suspected condition using at least one of a large language model ora medical ontology.

[0078] In one embodiment, aggregating the cosine similarities includes computing an average cosine similarity for each condition and predicting the condition with the highest average cosine similarity as the most likely diagnosis.

[0079] In one embodiment, the diagnostic prediction includes a ranked list of conditions ordered by likelihood along with reference images most similar to the input medical image.

[0080] In one embodiment, the method further includes, for an unknown disease, obtaining a plurality of candidate images from publicly accessible internet sources depicting a potentially novel medical condition, generating image embeddings for the plurality of candidate images and the input medical image using the image encoder component, and determining whether the image embeddings of the plurality of candidate images and the input medical image form a cluster in an embedding space of the trained medical foundation model, wherein cluster formation is indicative of the input medical image depicting the unknown disease.

[0081] In one embodiment, the plurality of candidate images are obtained from social media platforms during a disease outbreak or pandemic.

[0082] In one embodiment, the clustering is performed using at least one of k-means clustering, DBSCAN, or hierarchical clustering applied to the image embeddings.

[0083] In one embodiment, the method further includes tracking formation and growth of new clusters over time to provide early detection signals for emerging medical conditions.

[0084] In one embodiment, the method further includes switching from rare disease diagnosis to unknown disease clustering mode when no candidate condition achieves a cosine similarity above a confidence threshold.

[0085] In some embodiments, all of the foregoing may be performed on a system using at least one processor and a memory.

[0086] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may berealized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.BRIEF DESCRIPTION OF FIGURES

[0087] Non-limiting and non-exhaustive examples are described with reference to the following figures.

[0088] FIG. 1 illustrates medical artificial intelligence system, in accordance with an embodiment of the invention.

[0089] FIG. 2 illustrates a block diagram of a server configured to implement a medical artificial intelligence system in accordance with an embodiment of the invention.

[0090] FIG. 3 illustrates a flowchart for a process for training a medical foundation model in accordance with an embodiment of the invention.

[0091] FIG. 4 illustrates a flowchart for a process for classifying a medical image using a trained medical foundation model and a modified SoftMax function in accordance with an embodiment of the invention.

[0092] FIG. 5 illustrates a flowchart for a process for classifying a medical image using a trained medical foundation model and a domain-specific machine learning model in accordance with an embodiment of the invention.

[0093] FIG. 6 illustrates a flowchart for a process for classifying images based on image patch analysis using a trained medical foundation model in accordance with an embodiment of the invention.

[0094] FIG. 7 illustrates a flowchart for a process for retrieving relevant medical literature based on an input medical image using a trained medical foundation model in accordance with an embodiment of the invention.

[0095] FIG. 8 illustrates a flowchart for a process for classifying image / text pairs as medical or non-medical using a trained medical foundation model in accordance with an embodiment of the invention.

[0096] FIG. 9 illustrates a flowchart for a process for generating a diagnostic prediction of a dermatological image in accordance with an embodiment of the invention.

[0097] FIG. 10 illustrates a flow chart for a process for automatically identifying medical emergencies from a video stream using a trained foundation model in accordance with an embodiment of the invention.

[0098] FIG. 11 illustrates a flow chart for a process for identifying rare and / or emerging diseases using a trained medical foundation model in accordance with an embodiment of the invention.DETAILED DESCRIPTION

[0099] Medical artificial intelligence (Al) systems for diagnostic image classification address the challenge of analyzing medical images across diverse imaging modalities and medical specialties. A diagnostic medical foundation model may be trained on a large dataset of annotated medical images to achieve classification performance comparable to that of physicians without requiring task-specific training for each diagnostic application. Such a medical foundation model may process visual information from medical images together with textual information describing medical conditions to generate diagnostic predictions, i.e. identifying the disease condition present in the image.

[0100] Designing a medical foundation model for diagnostic image classification presents substantial technical challenges. Medical diagnosis involves distinguishing between numerous conditions that may present with similar visual characteristics across different imaging modalities. For example, various pulmonary conditions may produce overlapping radiographic findings on chest X-rays, and multiple dermatological conditions may exhibit comparable morphological features in clinical photographs. A medical foundation model may need to learn discriminative features that differentiate between conditions with subtle visual differences while generalizing across different imaging technologies, anatomical regions, and patient populations. The model may also need to handle the inherent variability in medical images arising from differences in imaging equipment, acquisition protocols, patient positioning, and image quality.

[0101] To address these issues, a medical foundation model, MedFM, is described. MedFM's training process involves processing large amounts of medical journal papers to generate training data with higher signal-to-noise ratios than the papers alone. Additionally, to address the issue of similar presentation, multipledifferent output conditioning methods can be applied including a modified SoftMax function described herein, and a secondary machine learning model.

[0102] Medical Al systems may increase diagnostic accuracy through conditioning the output using a modified SoftMax function. In medical classification tasks, multiple medical terms may describe the same underlying condition or may co-occur in the same patient. A standard SoftMax function treats all classification labels as mutually exclusive, which may not reflect the clinical reality where conditions may overlap or where multiple descriptive terms may apply to a single diagnosis. The modified SoftMax function may aggregate logits belonging to the same medical condition within positive and negative categories, creating aggregated scores for each medical condition before applying a SoftMax transformation. This approach may enable MedFM to handle medical terminology that reflects the complexity of clinical diagnosis.

[0103] Medical Al systems may also increase diagnostic accuracy by combining MedFM with a secondary domain-specific machine learning model. MedFM may generate similarity scores between an input medical image and various medical term descriptions and / or various medical images. These similarity scores may then be provided to a domain-specific machine learning model that has been trained on a limited number of image / text pairs from a particular clinical domain. The domain-specific machine learning model may refine the predictions from MedFM based on patterns learned from domain-specific training data. This combination may enable medical Al systems to leverage both the broad knowledge encoded in MedFM and the specialized knowledge captured by the domain-specific model, potentially improving classification performance beyond what either model may achieve independently to achieve physician-level performance or beyond. Medical Al systems are discussed in further detail below along with a discussion with methods of training and use.Medical Artificial Intelligence Systems

[0104] Medical Al systems are computational platforms which may include a number of peripheral computing devices for interfacing with the system. In many embodiments, medical Al systems train MedFM and also provide access to the trained MedFM for use. In various embodiments, medical Al systems are loaded with a pretrained MedFM. Depending on the medical practice, medical Al systems may beintegrated directly into medical imaging systems such as (but not limited to) hospital information systems, picture archiving and communication systems (PACS), cameras, digital X-rays, magnetic resonance imaging systems, pathology microscopes, computerized tomography scanners, and / or any other medical imaging device as appropriate to the requirements of specific embodiments of the invention.

[0105] Turning now to FIG. 1, a medical Al system 100 is illustrated in accordance with an embodiment of the invention. The system 100 may include a server 110 connected to the network 140. Servers act as a computing platform capable of running MedFM. In some embodiments, the server stores medical image data, or manage user authentication and access control. The server 110 may receive requests from other components of the system 100 through the network 140 and may transmit responses containing diagnostic results, retrieved images, or other information.

[0106] The system 100 may also include a server cluster 120 connected to the network 140. The server cluster may provide distributed computing resources for handling computationally intensive tasks. In some embodiments, the server cluster can be configured to train MedFM using large datasets of annotated medical images. Server clusters can also perform inference operations on medical image data, processing input images through a trained MedFM to generate diagnostic predictions. The distributed architecture of the server cluster 120 may enable parallel processing of multiple medical images or simultaneous handling of requests from multiple client devices.

[0107] The system 100 may further include a client device 130 connected to the network 140. Client devices may be used by healthcare professionals or researchers to interact with the medical artificial intelligence system. In some cases, a client device may execute a user interface application that enables users to upload medical images for analysis, view diagnostic predictions, or retrieve similar images from a medical image database. Client devices may transmit medical images to the server or the server cluster for processing and may receive diagnostic results or other information in response.

[0108] The network 140 may serve as the communication backbone of the system 100, enabling data exchange between the server 110, the server cluster 120, and the client device 130. Networks may include various communication pathways such asthe internet, local area networks, wide area networks, other distributed network infrastructure, and / or any combination thereof. Data such as medical images, diagnostic results, medical papers, model outputs, and user requests may flow between the components of the system through the network. The topology of the system can allow the client devices to send requests and receive responses from either the server or the server cluster, facilitating flexible allocation of processing tasks based on computational requirements and system load.

[0109] Referring to FIG. 2, a server 200 configured to implement a medical artificial intelligence system is illustrated in accordance with an embodiment. The server 200 may be implemented as the server 110 or as one or more servers within the server cluster 120 of the system 100 described with reference to FIG. 1. The server 200 may include a processor 210, an input / output interface 220, and a memory 230.

[0110] Processors can be configured to execute instructions and perform computational operations for the server. In some cases, the processor may include one or more central processing units, graphics processing units, tensor processing units, field programmable gate arrays, application specific integrated circuits, other specialized processing hardware suitable for executing machine learning models, and / or any combination thereof. The processor can execute software applications stored in the memory to process medical images and generate diagnostic outputs.

[0111] The input / output interface 220 may facilitate communication between the server and external devices or networks. In some embodiments, the input / output interface enables the server to receive medical images from the client device through the network. In many embodiments, the input / output interface can also transmit diagnostic results, retrieved images, or other information to requesting devices. The input / output interface may support various communication protocols and data formats used in medical imaging systems.

[0112] The memory 230 stores data and executable instructions for the server 200. Memory can include volatile memory such as random access memory and non-volatile memory such as solid-state drives or hard disk drives, or any combination thereof. In some embodiments, the memory may store medical image data, model parameters, and software applications.

[0113] With continued reference to FIG. 2, the memory 230 includes a medical Al application 232 and a trained medical foundation model 234. Medical Al applications can provide the software framework for processing medical images and generating diagnostic outputs. In some embodiments, the medical Al application implements user interfaces, data preprocessing pipelines, and result presentation functionality. The medical Al application may coordinate the flow of data between the input / output interface, the processor, and the trained medical foundation model. In various embodiments, functionality of medical Al applications may be split out into multiple separate function specific applications. For example, a training application may be used to train a model, whereas interfacing with the trained model may be performed by a separate application. Therefore, it is to be understood that the different processes described herein need not be performed by the same piece of software.

[0114] The trained medical foundation model 234 may be a vision-language model that has been trained on a large dataset of annotated medical images to perform diagnostic classification across multiple imaging modalities and medical specialties. In various embodiments, the trained medical foundation model may include an image encoder component and a text encoder component that have been trained using contrastive language-image pre-training. In numerous embodiments, trained medical foundation models generate embeddings for input medical images and for textual descriptions of medical conditions, enabling similarity-based classification without requiring task-specific training data.

[0115] By way of example, in numerous embodiments, the trained foundation model may utilize a Contrastive Language-Image Pre-training (CLIP) architecture for visual and text comprehension. In some embodiments, the CLIP architecture may be implemented through OpenCLIP, an open-source implementation that enables training of vision-language models on custom datasets. The CLIP architecture may enable the trained medical foundation model to learn aligned representations of medical images and textual descriptions, allowing similarity-based classification across diverse imaging modalities and medical conditions. Other implementations of CLIP may be used as appropriate to the requirements of specific applications.

[0116] In numerous embodiments, the trained medical foundation model may employ a vision transformer (ViT) for image analysis combined with a transformerbased language model for text processing. Vision transformers may process inputmedical images by dividing the images into patches and applying self-attention mechanisms to learn relationships between different regions of the image. The transformer-based language model may process textual descriptions of medical conditions and generate text embeddings that can be compared with image embeddings for classification tasks.

[0117] As an example implementation, in some embodiments, the visual encoder backbone of the trained medical foundation model is a ViT-Large model pre-trained on LAION, a large-scale dataset of image-text pairs. The ViT-Large architecture may provide sufficient model capacity to capture complex visual features present in medical images across different imaging modalities. During training of the trained medical foundation model, all layers of the visual encoder backbone may be unlocked, allowing the model to adapt pre-trained representations to the medical imaging domain.

[0118] The text encoder backbone of the trained medical foundation model may be PubMedBert, a transformer-based language model pre-trained on PubMed, a database of biomedical literature. PubMedBert may provide domain-specific language understanding that captures medical terminology and relationships between medical concepts. In some cases, four layers of the text encoder backbone may be unlocked during training, enabling the model to fine-tune language representations for medical image classification tasks while retaining knowledge from pre-training on biomedical text.

[0119] During training of the trained medical foundation model, data augmentation may be applied using the TIMM library. Data augmentation parameters may include a scale range of 0.3 to 1.0 and a random erasing probability (re_prob) of 0.4. Data augmentation may increase the diversity of training examples and may improve the generalization capability of the trained medical foundation model to medical images with varying characteristics.

[0120] Training parameters for the trained medical foundation model 234 may include a learning rate of 1e-5, which may control the step size during gradient descent optimization. The training process may proceed for 40 epochs, with each epoch representing a complete pass through the training dataset. A warmup period of 2000 steps may be applied at the beginning of training, during which the learning rate may gradually increase to the target value. The batch size may be set to 256, determiningthe number of image-text pairs processed in each training iteration. Weight decay of 0.2 may be applied as a regularization technique to reduce overfitting. An epsilon value of 1e-6 may be used in the optimizer to provide numerical stability during training. These training parameters may be modified based on available computational resources and dataset characteristics. As can be readily appreciated, a specific set of parameters and training data is enumerated, but the framework is not dependent upon specific training data or parameters, and these can be modified based on available data in accordance with empirical testing based on training data and desired usage. Training processes are discussed in further detail below.Medical Foundation Model Training

[0121] Conventional methods for training medical machine learning models often involve training on widely available public data sets of medical journal papers. However, the relevant information stored within said papers are often buried in superfluous text. Methods for training MedFM differ from conventional machine learning at least in that they preprocess and package training data to increase the signal-to-noise ratio of relevant medical information stored in journal papers. Further, the packaging method of the input training data enables multiple different output use cases for MedFM several of which are described herein.

[0122] Turning now to FIG. 3, a process 300 for training a medical foundation model using medical journal papers is illustrated in accordance with an embodiment of the invention. The process 300 may be executed by the server 200 described with reference to FIG. 2, or by the server cluster 120 described with reference to FIG. 1. The process 300 may generate training data from medical publications and use the training data to train an image encoder model and a text encoder model that together form the trained medical foundation model 234.

[0123] The process 300 begins with obtaining (310) a corpus of medical journal papers. In some embodiments, medical journal papers may be obtained from open access repositories such as PubMed Central (PMC). The medical journal papers may include publications across various medical disciplines and may contain medical images along with associated textual content including figure captions, abstracts, and full article text. In various embodiments, obtaining the papers may involve downloading articles in bulk from the repository, extracting article metadata, andorganizing the articles for subsequent processing. The medical journal papers may span multiple years of publication, enabling the creation of datasets of varying sizes for studying scaling effects during model training. In various embodiments, the corpus may be used to create the PubMed Central Medical Images Dataset (PMCMID) which presently contains approximately sixteen million images extracted from approximately three million articles. While journal papers are discussed herein, other datasets can be used such as (but not limited to) medical images scraped from the Internet, or electronic health records. In various embodiments, a trained MedFM model can be used to generate classifications of scraped images which in turn can be used as training data for future training iterations.

[0124] The images from each paper are then extracted (320). In some embodiments, image extraction involves parsing the document structure of each medical journal paper to identify embedded figures. The extracted images may include a variety of content types such as medical photographs, radiographic images, histopathology slides, ultrasound images, and abstract visuals such as graphs, schematics, and flowcharts. In various embodiments, associated metadata for each image is also extracted, including figure captions and references to the figure within the article text.

[0125] With continued reference to FIG. 3, the process 300 moves to filtering (330) out abstract figures from the extracted images. Abstract figures include non-medical images such as graphs, charts, schematics, diagrams, and other visual representations that do not depict medical imaging data. In some embodiments, a convolutional neural network (CNN) may be used to classify extracted images into different types. For example, images may be classified as Type I: individual medical images, Type II: image panels containing multiple medical images, Type III: mixed panels containing both medical images and abstract visuals, or Type IV: abstract visuals only. The filtering step may retain individual medical images and image panels fortraining while excluding or separately processing abstract visuals. That is, Type I, II, and III images are retained for training and Type IV images are discarded. This filtering may increase the signal-to-noise ratio of the training data by removing images that do not contain relevant medical visual information. In various embodiments, all veterinary content or content related to non-human subjects is additionally filtered out.

[0126] Figure captions from medical journal papers may lack complete medical information needed fortraining a diagnostic model. In some cases, diagnoses may be missing from a substantial portion of figure captions, with approximately 78% of figure captions lacking explicit diagnostic information. To address this limitation, augmented figure descriptions are generated (340) based on context extracted from associated publication text. In various embodiments, a fine-tuned large language model (LLM) may be used to extract diagnoses from the paper title and paragraphs that reference the figure. The augmented figure descriptions may provide more comprehensive textual annotations that describe the medical content depicted in each image than the original caption. Automated annotation of medical images may involve extracting diagnostic information and medical concepts from textual content associated with each image. In some embodiments, a fine-tuned BioMedLM large language model may be used to extract diagnoses from associated texts. BioMedLM is a large language model that has been pre-trained on biomedical literature and may be finetuned for the task of diagnosis extraction. The fine-tuned BioMedLM may process the paper title and the referring paragraph to the image to identify diagnostic terms mentioned in the text. The referring paragraph may be the paragraph within the medical journal paper that contains a reference to the figure depicting the medical image. By extracting diagnoses from these associated texts, the fine-tuned BioMedLM may augment figure captions that lack explicit diagnostic information, providing more complete annotations for training the medical foundation model.

[0127] Each description is then annotated (350) using a medical ontology. In various embodiments, medical concepts may be extracted using a Unified Medical Language System (UMLS) parser. The UMLS is a comprehensive thesaurus and ontology of biomedical concepts that provides standardized terminology and concept identifiers across different medical vocabularies. The UMLS parser may process sentences and paragraphs referring to the image, as well as the title of the paper, to identify medical concepts mentioned in the text. In various embodiments, the UMLS parses processes the generated figure description for a given image in an image / description pair. Each identified medical concept may be associated with a Concept Unique Identifier (CUI) from the UMLS, enabling consistent labeling of medical concepts across different textual descriptions and medical publications.

[0128] The IIMLS parser may extract medical concepts belonging to 13 IIMLS semantic types. These semantic types may categorize medical concepts according to their role in medical knowledge representation. The 13 IIMLS semantic types may include TO for Anatomical Structure, which encompasses concepts describing anatomical entities at various levels of organization. T023 for Body Part may include concepts describing specific regions or components of the body. T060 for Diagnostic Procedure may encompass concepts describing procedures used to identify diseases or conditions. T047 for Disease or Syndrome may include concepts describing pathological conditions affecting the body. T033 for Finding may encompass concepts describing observations made during clinical examination or diagnostic testing. T058 for Health Care Activity may include concepts describing activities performed in the context of health care delivery. T037 for Injury or Poisoning may encompass concepts describing damage to the body resulting from external causes. T170 for Intellectual Product may include concepts describing information artifacts such as clinical guidelines or classification systems. T048 for Mental or Behavioral Dysfunction may encompass concepts describing disorders affecting mental processes or behavior. T191 for Neoplastic Process may include concepts describing abnormal tissue growth including benign and malignant tumors. T046 for Pathologic Function may encompass concepts describing disordered physiological processes. T184 for Sign or Symptom may include concepts describing manifestations of disease observable by clinicians or reported by patients. T061 for Therapeutic or Preventive Procedure may encompass concepts describing interventions intended to treat or prevent disease.

[0129] By extracting medical concepts from multiple textual sources associated with each image, including the referring sentences, referring paragraphs, and publication title, the annotation process may generate comprehensive labels that capture different aspects of the medical content depicted in each image. The combination of diagnosis extraction using the fine-tuned BioMedLM and medical concept extraction using the UMLS parser may provide both free-text diagnostic descriptions and structured ontology-based annotations for each image in the training dataset.

[0130] Image / description pairs are generated (360) by combining the filtered medical images with their corresponding augmented descriptions with the annotations. Each image / description pair may associate a medical image with textual content thatS31 -09145. POTdescribes the medical conditions, anatomical regions, imaging modalities, and other relevant information depicted in the image. In some embodiments, the image / description pairs may form the training examples used for contrastive learning, where the model learns to associate images with their corresponding textual descriptions.

[0131] An image encoder model and a text encoder model are trained (370) using contrastive language-image pre-training to yield a trained medical foundation model. Contrastive language-image pre-training involves training the image encoder and text encoder to produce aligned embeddings for matching image / description pairs while producing dissimilar embeddings for non-matching pairs. In some embodiments, the training process uses a contrastive loss function that maximizes the similarity between embeddings of corresponding image / description pairs and minimizes the similarity between embeddings of non-corresponding pairs. The trained image encoder and text encoder together form the trained medical foundation model 234, which may be capable of generating embeddings for medical images and textual descriptions that enable similarity-based classification across multiple imaging modalities and medical specialties.

[0132] In various embodiments, the process 300 may be executed periodically to update the trained medical foundation model as new medical journal papers become available. The automated nature of the process 300 may enable the training dataset to grow over time, potentially improving the performance and coverage of the trained medical foundation model across different medical conditions and imaging modalities. To ensure the performance of the model, it can be tested against a benchmark consisting of manually annotated image / text pairs evaluating performance across modalities, age groups, sexes, and skin tones. This helps ensure that the automatic pipeline does not introduce a performance regression in various patient populations including vulnerable patient populations like children or patients with darker skin tones. The process 300 may also be configured to generate datasets corresponding to different time periods, enabling analysis of how model performance scales with the amount of training data. Once trained, MedFM can be used for a wide variety of different tasks including (but not limited to) zero-shot classification of images, pathology slide processing, and relevant medical literature retrieval based on input images. These functions are described in additional detail below.Zero-Shot Classification

[0133] Zero-shot classification refers to the capability of a trained model to classify inputs into categories that were not explicitly present during training, leveraging learned representations to generalize to new classification tasks without requiring additional task-specific training data. This provides significant clinical relevancy as functional zero-shot classification enables diagnostic capability for any input image. Turning now to FIG. 4, a process 400 for classifying a medical image using a trained medical foundation model is illustrated in accordance with an embodiment of the invention. The process 400 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230. The process 400 may enable zero-shot classification of medical images, where the trained medical foundation model classifies medical conditions without requiring task-specific training data for each classification task.

[0134] The process 400 begins with obtaining (410) an input medical image. In some embodiments, the input medical image may be received through the input / output interface 220 from the client device 130 over the network 140. The input medical image may be a clinical photograph, a radiographic image such as a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, or any other arbitrary type of medical imaging data.

[0135] Image-specific domain labels are then generated (420). In various embodiments, the image-specific domain labels may include medical terms relevant to the classification task being performed. For example, when classifying chest X-ray images for pneumonia detection, the domain labels may include terms describing various types of pneumonia such as lobar pneumonia, bronchopneumonia, viral pneumonia, and aspiration pneumonia, as well as terms describing radiographic findings associated with pneumonia such as consolidation, ground-glass opacity, and air bronchogram. The domain labels may also include terms describing normalcy such as healthy lung, normal lung, and no finding. When classifying dermatological images for skin cancer detection, the domain labels may include terms describing malignant conditions such as malignant melanoma, squamous cell carcinoma, and basal cell carcinoma, as well as terms describing benign conditions and normal skin.

[0136] The domain labels may be predefined for each classification task and may be stored in the memory 230. In various embodiments, the domain labels are automatically generated based on the type of input image. For example, when presenting a chest x-ray, any number of methods including ontological methods, large language models, and others, can be used to generate a set of domain labels relevant to pulmonary and cardiac conditions. In some embodiments, text provided by the user about the image can be used to generate domain labels in a similar fashion.

[0137] The input image and labels are provided (430) to the trained medical foundation model. The image encoder component generates an image embedding representing the visual features of the input medical image. The text encoder component generates text embeddings for each domain label, representing the semantic content of each medical term.

[0138] Cosine similarities between descriptions and the input image are obtained (440) from the trained medical foundation model. In various embodiments, the trained medical foundation model calculates cosine similarities between the image embedding and each text embedding. The cosine similarity between two embeddings may be calculated as the dot product of the two embedding vectors divided by the product of their magnitudes. The cosine similarity values may range from negative one to positive one, with higher values indicating greater similarity between the image features and the medical term description. The cosine similarity values may serve as logits for classification, where each logit represents the degree of alignment between the input medical image and a corresponding medical term.

[0139] The prediction is then refined (450) using a modified SoftMax function. In medical classification tasks, multiple medical terms may describe the same underlying condition or may co-occur in the same patient. A standard SoftMax function treats all classification labels as mutually exclusive, which may not reflect clinical scenarios where conditions may overlap or where multiple descriptive terms may apply to a single diagnosis. The modified SoftMax function may address this limitation by aggregating logits belonging to the same medical condition before applying the SoftMax transformation. This mimics the diagnostic process a medical diagnostician may perform.

[0140] In various embodiments, the modified SoftMax function calculates average logits for medical terms belonging to the same medical condition within positive and negative categories. Medical terms may be grouped into positive categories representing the presence of a medical condition and negative categories representing the absence of the condition or the presence of alternative conditions. For each medical condition, the modified SoftMax function may calculate an average of the cosine similarity logits for all medical terms within the positive category for that condition. Similarly, the modified SoftMax function may calculate an average of the cosine similarity logits for all medical terms within the negative category. The averaged logits for the positive and negative categories may then be provided to a SoftMax function to generate probability scores for each medical condition.

[0141] Byway of example, when classifying an input medical image for pneumonia detection, the positive category for pneumonia may include multiple medical terms such as pneumonia, lobar pneumonia, bronchopneumonia, consolidation, and infiltrate. The negative category may include terms such as healthy lung, normal lung, and no finding. The modified SoftMax function may calculate the average of the cosine similarity logits for all terms in the positive category and the average of the cosine similarity logits for all terms in the negative category. The SoftMax function may then be applied to these averaged logits to generate a probability score indicating the likelihood that the input medical image depicts pneumonia. This aggregation approach may enable the process 400 to handle medical terminology where multiple terms may describe aspects of the same underlying condition, providing classification predictions that reflect the complexity of medical diagnosis.

[0142] The modified SoftMax function may be expressed mathematically through a series of calculations that transform raw similarity scores into probability distributions over medical conditions. The mathematical formulation may enable the trained medical foundation model to aggregate multiple related medical terms into unified condition-level predictions.

[0143] The cosine similarity between an image embedding and a text embedding may be calculated according to the formula:lij = cos(Vi, Tjwhere 1^ represents the cosine similarity logit for image i and text description j, Vtrepresents the image embedding vector generated by the image encoder for input image i, and 7} represents the text embedding vector generated by the text encoder for medical term description j. The cosine similarity function may compute the dot product of the two embedding vectors normalized by the product of their respective magnitudes, yielding values in the range from negative one to positive one.

[0144] The aggregation of logits for medical terms belonging to the same medical condition may be calculated according to the formula:where Sikrepresents the aggregated score for medical condition k for image i, Nkrepresents the number of medical terms belonging to condition category k, Ckrepresents the set of medical term indices belonging to condition category k, andrepresents the cosine similarity logit between image i and text description j. The aggregation formula may compute the arithmetic mean of cosine similarity logits across all medical terms within a given condition category, producing a single aggregated score that represents the overall alignment between the input image and the medical condition.

[0145] The modified SoftMax function transforms the aggregated scores into probability values according to the formula:ne SikPik~ ywhere Pikrepresents the probability that image i belongs to medical condition k, Sikrepresents the aggregated score for condition k for image i, K represents the set of all medical condition categories being considered for classification, and Simrepresents the aggregated score for condition m for image i. The exponential function may be applied to each aggregated score, and the resulting values may be normalized by dividing by the sum of exponentials across all condition categories. The resulting probability values may sum to one across all medical conditions, providing a probability distribution that indicates the relative likelihood of each medical condition given the input image.

[0146] In various embodiments, the condition categories Ckmay be defined to group medical terms that describe the same underlying pathology or that represent positive and negative indicators for a binary classification task. For binary classification tasks such as pneumonia detection or skin cancer detection, two condition categories may be defined: a positive category containing medical terms indicative of the target condition and a negative category containing medical terms indicative of normalcy or alternative conditions. The modified SoftMax function may then produce probability values for the positive and negative categories, enabling binary classification decisions based on the relative probabilities.

[0147] Notably, using a modified SoftMax function is not the only way to perform zero-shot classification. In numerous embodiments, a classical machine learning model trained for a domain-specific classification based on the output of the MedFM can be used to refine diagnostic predictions. Turning now to FIG. 5, a process for generating diagnostic predictions using a trained medical foundation model combined with a domain-specific machine learning model is illustrated in accordance with an embodiment of the invention. The process 500 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230. The process 500 may combine the capabilities of the trained medical foundation model with a classical machine learning model (CMLM) trained on domain-specific images to improve classification performance beyond what the trained medical foundation model may achieve through zero-shot classification alone.

[0148] The process 500 begins with obtaining (510) an input medical image. The input medical image may be a chest X-ray, a clinical photograph, a histopathology slide, or any other type of medical imaging data appropriate for the diagnostic task being performed. Image-specific domain labels are then generated (520). Domain labels are generated based on the type of input medical image and the classification task. The domain labels may include medical terms describing conditions relevant to the diagnostic domain, such as pneumonia-related terms for chest X-ray classification or skin cancer-related terms for dermatological image classification. The domain labels may be predefined for each classification domain or may be automatically generated based on characteristics of the input medical image.

[0149] With continued reference to FIG. 5, the process 500 moves to providing (530) the input image and domain-specific labels to the trained medical foundation model. In some embodiments, the input medical image is provided to the image encoder component of the trained medical foundation model, which generates an image embedding representing the visual features of the input medical image. The step 530 also provides each domain label to the text encoder component of the trained medical foundation model, which generates text embeddings for each medical term.

[0150] The process 500 continues to obtaining (540) cosine similarities between descriptions and the input image from the trained medical foundation model. In various embodiments, cosine similarities between the image embedding and each text embedding generated by the trained medical foundation model are calculated. The cosine similarity values may represent the degree of alignment between the visual features of the input medical image and the semantic content of each medical term description. The cosine similarity values may serve as feature vectors that encode the relationship between the input medical image and the set of domain-specific medical terms.

[0151] The process 500 proceeds to using (550) a domain-specific classical machine learning model to refine the diagnostic prediction based on the cosine similarities. In some embodiments, the cosine similarity values obtained from the step 540 are provided as input features to a CMLM that has been trained on a set of domain-specific medical images. The CMLM may process the cosine similarity features and generate a refined diagnostic prediction that leverages both the broad medical knowledge encoded in the trained MedFM and the specialized patterns learned from domain-specific training data.

[0152] The CMLM may be trained on a limited number of domain-specific images separate from the training data used for the trained medical foundation model. In various embodiments, the CMLM may be trained using cosine similarity features extracted from domain-specific images by the trained medical foundation model. The training process for the CMLM may involve providing domain-specific images to the trained medical foundation model, obtaining cosine similarity values between each image and the domain labels, and using the cosine similarity values along with ground truth diagnostic labels to train the CMLM.

[0153] The CMLM may be implemented using various machine learning algorithms. In some embodiments, the CMLM may be implemented as a Logistic Regression model, which may learn linear decision boundaries in the cosine similarity feature space to separate different diagnostic categories. In various embodiments, the CMLM may be implemented as a Random Forest model, which may combine multiple decision trees to capture non-linear relationships between cosine similarity features and diagnostic outcomes. In numerous embodiments, the CMLM may be implemented as a Gradient Boosting model, which may iteratively build an ensemble of weak learners to improve prediction accuracy. In some embodiments, the CMLM may be implemented as a Support Vector Machine (SVM), which may find optimal hyperplanes in the cosine similarity feature space to maximize separation between diagnostic categories. The selection of the CMLM algorithm may depend on the characteristics of the domain-specific training data and the classification task being performed.

[0154] The combination of MedFM with the CMLM may provide performance enhancement compared to zero-shot classification using MedFM alone. The trained medical foundation model provides cosine similarity features that encode broad medical knowledge learned from a large dataset of annotated medical images. The CMLM refines these features based on patterns learned from domain-specific training data, potentially improving classification accuracy for the target diagnostic task. In various embodiments, the CMLM may achieve improved performance with a relatively small number of domain-specific training images, as the cosine similarity features from the trained medical foundation model provides a rich representation of the input medical image that captures relevant diagnostic information.

[0155] As can be readily appreciated, there are numerous ways to leverage the broad medical knowledge base of MedFM to yield diagnostic classification of medical images. Certain medical images provide additional challenges. Pathology slides, for example, are very information dense and may be very large files. To perform the same type of classification on these types of images, a patch-based approach can be used. This patch based approach is described below.Patch-Based Classification of Pathology Slide Images

[0156] Histopathology whole-slide images present unique computational challenges due to their extremely large file sizes, which may exceed several gigabytes per image, and their complex tissue architecture that requires analysis at multiple magnification levels to capture both cellular-level details and broader tissue organization patterns. Turning now to FIG. 6, a process for classifying pathology slide images using a medical foundation model is illustrated in accordance with an embodiment of the invention. The process 600 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230. The process 600 may enable classification of histopathology whole-slide images (WSIs) through a patch-based analysis approach that addresses the large file sizes and high information density characteristic of pathology slide images.

[0157] The process 600 begins with obtaining (610) an input pathology slide image. In some embodiments, the input pathology slide image may be a whole-slide image captured from a histopathology specimen using a digital pathology scanner. Whole-slide images may contain gigapixels of image data representing tissue samples at high magnification levels, which can present a challenge in computational efficiency.

[0158] The slide image is segmented (620) into image patches. In various embodiments, the segmentation involves extracting tiles from the whole-slide image at specified magnification levels and spatial dimensions. In many embodiments, the tile extraction process includes automated tissue detection to ensure that extracted tiles contain sufficient tissue content for meaningful analysis. In some embodiments, tile regions are configured to contain a minimum of 80% tissue coverage. The tissue coverage can be validated through automated tissue detection methods including grayscale conversion and Otsu thresholding. Otsu thresholding is an image segmentation technique that automatically determines an optimal threshold value to separate foreground tissue regions from background regions based on the intensity histogram of the grayscale image. Tiles that do not meet the 80% tissue coverage threshold may be excluded from subsequent processing steps, reducing computational overhead and focusing analysis on regions containing relevant tissue content.

[0159] Each image patch is provided (630) to the medical foundation model along with potential disease labels. In various embodiments, each extracted tile is provided to the image encoder component of the trained medical foundation model, which generates an image embedding for each tile. Potential disease labels relevant to the pathology classification task are provided to the text encoder component of the trained medical foundation model, which generates text embeddings for each disease label. The disease labels may include terms describing cancer subtypes, benign conditions, and normal tissue classifications relevant to the histopathology specimen being analyzed.

[0160] Cosine similarities are obtained (640) between descriptions and each patch from the trained medical foundation model. In some embodiments, cosine similarities are calculated between the image embedding for each tile and the text embeddings for each disease label. The cosine similarity calculations may produce a matrix of similarity values where each row corresponds to a tile and each column corresponds to a disease label. The cosine similarity values may indicate the degree of alignment between the visual features of each tile and the semantic content of each disease label description.

[0161] The mean of each cosine similarity is calculated (650) for each label across each patch. In various embodiments, the mean cosine similarity for each disease label is calculated by averaging the cosine similarity values across all tiles extracted from the whole-slide image. This aggregation may produce a single similarity score for each disease label that represents the overall alignment between the whole-slide image and each potential diagnosis.

[0162] In some embodiments, the aggregation from tile-level predictions to patientlevel diagnosis may involve additional processing steps beyond simple averaging. The system may aggregate tile-level predictions to patient-level diagnosis by excluding normal tissue scores and identifying tiles with top 10 scores for cancer subtypes. By excluding normal tissue scores from the aggregation, the process 600 may focus on tiles that exhibit features indicative of pathological conditions. Identifying tiles with top 10 scores for cancer subtypes may enable the process 600 to base patient-level predictions on the tiles most strongly associated with malignant features, reducing the influence of tiles depicting benign or normal tissue regions on the final classification output. The aggregated scores may then be used to generate a patient-leveldiagnostic prediction indicating the most likely cancer subtype or the absence of malignancy in the histopathology specimen.

[0163] While the above focuses on oncological diagnoses, as can be readily appreciated, non-cancer related classifications can be performed using pathology slides. MedFM can further be used to retrieve relevant literature fora particular image, which is discussed in further detail below.Medical Literature Retrieval Based on Input Images

[0164] Beyond diagnostic classification, MedFM may be leveraged to retrieve relevant medical literature based on an input medical image. This capability enables clinicians to upload medical images and retrieve similar images along with publication details from a medical image dataset. This provides research assistance to critical human-in-the-loop diagnostics. T urning now to FIG. 7, a process for retrieving relevant medical literature based on an input medical image is illustrated in accordance with an embodiment of the invention. The process 700 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230. The process 700 may enable an lmage2Paper retrieval application that connects clinical images to relevant medical publications.

[0165] The process 700 begins with obtaining (710) an input medical image. In some embodiments, the input medical image may be received through the input / output interface from the client device over the network. The input medical image may be a clinical photograph, a radiographic image, a histopathology slide, or any other type of medical imaging data for which a clinician seeks relevant literature. In various embodiments, the application may provide a user interface through which clinicians upload medical images for literature retrieval.

[0166] The input image and a database of image / description pairs are provided (720) to the trained medical foundation model. In various embodiments, the database of image / description pairs may comprise the medical image dataset generated during training of the trained medical foundation model, such as the PMCMID dataset described with reference to FIG. 3. The database may contain medical images extracted from medical journal papers along with their associated textual descriptions, including augmented figure descriptions and medical ontology annotations. Thetrained medical foundation model may process both the input image and the images in the database to generate image embeddings for comparison.

[0167] The most similar images to the input image is then identified (730) based on cosine similarities between the image embedding of the input image and the image embeddings of database images, generated by the image encoder component of the trained medical foundation model.. In some embodiments, the image encoder component of the trained medical foundation model generates an image embedding for the input medical image. The image encoder component may also generate image embeddings for images in the database, or the image embeddings for database images may be pre-computed and stored for efficient retrieval. The process 700 may calculate similarity scores between the input image embedding and each database image embedding to identify images with visual features most similar to the input medical image. In various embodiments, cosine similarity may be used to measure the similarity between image embeddings, with higher cosine similarity values indicating greater visual similarity between images.

[0168] The most similar images are ranked (740) based on cosine similarities between the image embedding of the input medical image and text embeddings of diagnoses extracted via LLM from descriptions associated with each similar image, using the trained medical foundation model.. In various embodiments, the ranking step refines the initial image-to-image similarity results by incorporating textual information. The trained medical foundation model can generate a predicted diagnosis for the input medical image based on cosine similarities between the input image embedding and text embeddings for medical terms. The ranking step may then calculate cosine similarities between the image embedding of the input medical image and text embeddings of diagnoses extracted from the descriptions associated with each similar image in the database. This ranking method combines image-to-image comparison using image embeddings with cross-modal diagnosis-based ranking using cosine similarity between the input image embedding and text embeddings of database image diagnoses. In some embodiments, the ranking may prioritize images whose associated descriptions contain diagnoses that align with the predicted diagnosis for the input image, enabling retrieval of publications that address medical conditions similar to those depicted in the input image. In testing, this has led to the correctdiagnoses in top 10 results in 88.4% of cases, and at least one relevant differential diagnosis in 93.0%

[0169] The paper from which the highest ranked image originates is then returned (750). In some embodiments, the process returns publication details for the medical journal paper associated with the highest ranked similar image. The publication details may include the paper title, authors, journal name, publication date, and a link to the full publication. In various embodiments, the process may return multiple papers corresponding to the top-ranked similar images, enabling clinicians to review a set of relevant publications. The application may present the retrieved publications along with the similar images to the clinician through a user interface on the client device, facilitating access to medical literature relevant to the clinical case depicted in the input medical image.Classification of Medical versus Non-Medical Image / Text Pairs

[0170] The trained medical foundation model may be leveraged to distinguish medical image / text pairs from non-medical image / text pairs. This capability enables automated knowledge discovery on the internet, where vast quantities of image / text pairs may be scraped from publicly accessible web sources including websites, forums, social media platforms, educational resources, and online repositories. By classifying scraped image / text pairs as medical or non-medical, the system may automatically identify and collect medical image / text pairs from the internet at scale, thereby creating large training datasets that may be used to improve the performance of the trained medical foundation model. The classification of medical versus non-medical image / text pairs exploits a fundamental asymmetry in the embedding space behavior of the trained medical foundation model when applied to medical versus non-medical content.

[0171] Turning now to FIG. 8, a process 800 for classifying an image / text pair as medical or non-medical using a trained medical foundation model is illustrated in accordance with an embodiment of the invention. The process 800 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230. The process 800 may enable automated knowledge discovery on the internet by classifying image / text pairs scraped from publicly accessible web sources as medical or non-medical, therebyenabling the automated collection of medical image / text pairs at scale. In some embodiments, the internet may be scraped to obtain large quantities of image / text pairs from websites, forums, social media platforms, educational resources, online encyclopedias, and other publicly accessible sources. The process 800 may then classify each scraped image / text pair to identify medical content, which may be collected into large training datasets to improve the performance of the trained medical foundation model. The process 800 may enable automated identification of whether an input image / text pair belongs to the medical domain or the non-medical domain based on patterns of cosine similarity behavior in the embedding space of the trained medical foundation model.

[0172] The process 800 begins with obtaining (810) an input image and an input text description. In some embodiments, the input image and input text description may be obtained by scraping the internet for image / text pairs from publicly accessible web sources. Web scraping may involve systematically downloading image / text pairs from websites, forums, social media platforms, educational resources, online encyclopedias, blog posts, news articles, and other publicly accessible internet sources. In various embodiments, the input image and input text description may be received through the input / output interface 220 from the client device 130 over the network 140. The input image may be any image whose medical or non-medical nature is to be determined. The input text description may be any textual content associated with the input image, such as a figure caption, an alt-text attribute, a surrounding paragraph, a clinical description, a filename, or any other descriptive text. The input image and input text description together form an image / text pair whose domain classification is sought.

[0173] A direct cosine similarity between the input image and the input text description is computed (820). In various embodiments, the image encoder component of the trained medical foundation model generates an image embedding for the input image, and the text encoder component generates a text embedding for the input text description. The cosine similarity between the image embedding and the text embedding is calculated as the dot product of the two embedding vectors divided by the product of their respective magnitudes. This direct cosine similarity represents the degree of semantic alignment between the input image and the input text description as measured within the embedding space of the trained medicalfoundation model. If the cosine similarity is above a threshold, the image / text pair is classified as non-medical or medical using further steps.

[0174] Reference cosine similarities are computed (830) between the input image / text pair and reference sets of known medical and non-medical content. In various embodiments, four reference cosine similarities are computed. A medical image-text similarity is computed as the average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known medical image text descriptions. By way of example, the plurality of known medical image text descriptions may comprise at least 1000 text descriptions that are known to describe medical images, such as figure captions from verified medical journal papers. A medical text-image similarity is computed as the average of cosine similarities between the text embedding of the input text description and image embeddings of a plurality of known medical images. The plurality of known medical images may comprise at least 1000 images that are known to be medical images, such as clinical photographs, radiographic images, or histopathology slides from verified medical sources.

[0175] Similarly, a non-medical image-text similarity is computed as the average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known non-medical text descriptions. The plurality of known non-medical text descriptions may comprise at least 1000 text descriptions that are known to describe non-medical images, such as captions from general-purpose image datasets. A non-medical text-image similarity is computed as the average of cosine similarities between the text embedding of the input text description and image embeddings of a plurality of known non-medical images. The plurality of known non-medical images may comprise at least 1000 images that are known to be non-medical images, such as photographs of everyday objects, landscapes, or other general-purpose visual content.

[0176] The classification of medical versus non-medical image / text pairs is based on an observation regarding asymmetric behavior of cosine similarities in the embedding of the medical foundation model. When a medical image is compared against random medical text descriptions from the reference set, most of these descriptions will not be semantically relevant to the specific medical image because medical descriptions are highly specific to particular conditions, anatomical regions,and imaging modalities. Consequently, the cosine similarity between a medical image and random medical text descriptions drops substantially relative to the direct cosine similarity between the medical image and its own matching description. Similarly, when a medical text description is compared against random medical images from the reference set, most of these images will depict different conditions and modalities, causing a substantial drop in cosine similarity.

[0177] In contrast, when a non-medical image is compared against random nonmedical text descriptions, the drop in cosine similarity is smaller. Non-medical content tends to share broader semantic features in the embedding space of a medical foundation model because the model has not been trained to finely discriminate between different types of non-medical content. Similarly, when a non-medical text description is compared against random non-medical images, the drop in cosine similarity is smaller than the corresponding drop observed for medical content. This asymmetry arises because medical foundation models learn highly discriminative representations for medical content, where subtle differences between conditions are clinically significant, while non-medical content is represented more uniformly in the embedding space.

[0178] A medical domain specificity score is calculated (840) based on the computed cosine similarities. In various embodiments, the medical domain specificity score captures the asymmetric behavior described above by comparing the within-domain similarity patterns for the medical and non-medical reference sets. The medical domain specificity score may be calculated according to the formula:MDSS = (S_img_med + S_txt_med) - (S_img_nonmed + S_txt_nonmed) where MDSS represents the medical domain specificity score, S_img_med represents the medical image-text similarity computed as the average cosine similarity between the image embedding of the input image and text embeddings of the plurality of known medical text descriptions, S_txt_med represents the medical text-image similarity computed as the average cosine similarity between the text embedding of the input text description and image embeddings of the plurality of known medical images, S_img_nonmed represents the non-medical image-text similarity computed as the average cosine similarity between the image embedding of the input image and text embeddings of the plurality of known non-medical text descriptions, andS_txt_nonmed represents the non-medical text-image similarity computed as the average cosine similarity between the text embedding of the input text description and image embeddings of the plurality of known non-medical images.

[0179] For medical image / text pairs, the medical domain specificity score tends to be positive because the sum of the medical reference similarities (S_img_med + S_txt_med) exceeds the sum of the non-medical reference similarities (S_img_nonmed + S_txt_nonmed). This occurs because medical images and medical text descriptions maintain higher average cosine similarities with medical reference content than with non-medical reference content, even though the absolute cosine similarity values drop from the direct similarity. For non-medical image / text pairs, the medical domain specificity score tends to be negative because the sum of the non-medical reference similarities exceeds the sum of the medical reference similarities, reflecting the fact that non-medical content is more similar to non-medical reference content than to medical reference content.

[0180] The image / text pair is classified (850) as medical or non-medical based on the medical domain specificity score. In various embodiments, the image / text pair is classified as medical when the medical domain specificity score exceeds a predetermined threshold and as non-medical when the medical domain specificity score falls below the predetermined threshold. In some embodiments, the predetermined threshold may be optimized based on a validation set of labeled medical and non-medical image / text pairs to maximize classification accuracy, sensitivity, specificity, or other performance metrics appropriate for the intended application.

[0181] In some embodiments, the medical domain specificity score may be further refined by incorporating the direct cosine similarity. A normalized medical domain specificity score may be calculated as:NMDSS = [(S_direct - S_img_med) + (S_direct - S_txt_med)] - [(S_direct -S_img_nonmed) + (S_direct - S_txt_nonmed)]where S_direct represents the direct cosine similarity between the image embedding and the text embedding, and the remaining terms are as defined above. The normalized medical domain specificity score explicitly captures the magnitude of the cosine similarity drop from the direct similarity to the reference similarities for bothmedical and non-medical reference sets. For medical image / text pairs, the drops to medical references are smaller than the drops to non-medical references, yielding a negative normalized score. For non-medical image / text pairs, the drops to non-medical references are smaller than the drops to medical references, yielding a positive normalized score.

[0182] In various embodiments, the medical domain specificity score may alternatively be calculated using a weighted combination of cosine similarity ratios with learned coefficients. The weighted ratio-based medical domain specificity score may be calculated according to the formula:MDSS = a (S_direct / SJmg_med) + b (S_direct / S_img_nonmed) + c (S_direct / S_txt_med) + d (S_direct / S_txt_nonmed)where S_direct represents the direct cosine similarity between the image embedding of the input image and the text embedding of the input text description, S_img_med represents the medical image-text similarity computed as the average cosine similarity between the image embedding of the input image and text embeddings of the plurality of known medical text descriptions, S_img_nonmed represents the non-medical image-text similarity computed as the average cosine similarity between the image embedding of the input image and text embeddings of the plurality of known non-medical text descriptions, S_txt_med represents the medical text-image similarity computed as the average cosine similarity between the text embedding of the input text description and image embeddings of the plurality of known medical images, S_txt_nonmed represents the non-medical text-image similarity computed as the average cosine similarity between the text embedding of the input text description and image embeddings of the plurality of known non-medical images, and a, b, c, and d represent coefficients determined by a logistic regression model trained on a validation cohort of labeled medical and non-medical image / text pairs. Each term in the formula represents a ratio of the direct cosine similarity to a reference similarity, capturing the relative magnitude of the alignment between the input image / text pair compared to the alignment with reference content from each domain. The logistic regression model learns the optimal weighting of these ratios from the validation cohort, enabling the classifier to adapt to the specific characteristics of the embedding space of the trained medical foundation model. The validation cohort may comprise a small number of labeled image / text pairs, such as 100 to 1000 pairs, enabling efficient calibration ofthe coefficients without requiring a large labeled dataset. The weighted ratio formulation provides a generalized framework that subsumes the difference and quotient formulations as special cases, potentially achieving higher classification accuracy by learning data-driven coefficient values that reflect the empirical behavior of the embedding space.

[0183] In various embodiments, the reference sets of known medical and nonmedical content may be pre-computed and stored. The image embeddings for the plurality of known medical images, the text embeddings for the plurality of known medical text descriptions, the image embeddings for the plurality of known nonmedical images, and the text embeddings for the plurality of known non-medical text descriptions may be generated once using the trained medical foundation model and stored in the memory 230. During classification of a new input image / text pair, only the embeddings for the input image and input text description need to be generated, and the cosine similarities with the pre-computed reference embeddings may be calculated efficiently through matrix multiplication operations. This pre-computation approach may enable the process 800 to classify image / text pairs with minimal computational overhead beyond the initial embedding generation.

[0184] The process 800 may be applied in several practical contexts. In some embodiments, the process 800 may be used for automated knowledge discovery on the internet by scraping image / text pairs from publicly accessible web sources or from hospital information systems and classifying them as medical or non-medical. Web scraping may involve systematically downloading image / text pairs from websites, forums, social media platforms, educational resources, online encyclopedias, and other publicly accessible internet sources. The process 800 may classify each scraped image / text pair as medical or non-medical, enabling the automated collection of medical image / text pairs at internet scale. The collected medical image / text pairs may then be used to create large training datasets for the trained medical foundation model, potentially improving model performance by increasing the volume and diversity of training data beyond what is available from medical journal papers alone. In various embodiments, the process 800 may be used iteratively, where an initial trained medical foundation model is used to discover medical image / text pairs on the internet, and the discovered pairs are then used to retrain the model, resulting in an improved model that can discover additional medical content with greater accuracy. In numerousembodiments, the process 800 may also be used as a pre-processing step in a diagnostic classification pipeline to verify that input images are medical in nature before applying diagnostic classification algorithms, or to monitor data quality in production systems by flagging non-medical content that may have been incorrectly submitted for medical analysis.Dermatological Disease Classification

[0185] Turning now to FIG. 9, a process 900 for creating a dermatological disease classifier using a dermatology image dataset and a trained medical foundation model is illustrated in accordance with an embodiment of the invention. The process 900 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230.

[0186] The process 900 begins with obtaining (910) a dermatological image dataset. In various embodiments, the dermatological image dataset may comprise approximately 138,000 images collected from publicly accessible internet sources. The images may be identified as dermatological content through the medical versus non-medical classification process 800 described with reference to FIG. 8, followed by additional filtering to select images depicting skin conditions. Each image in the dermatological image dataset may be annotated with at least three types of metadata: a diagnosis identifying the dermatological condition depicted in the image, a skin color classification describing the skin tone of the individual in the image, and a body location identifier describing the anatomical region where the skin condition is present. The annotations may be generated through a combination of automated methods and manual expert review.

[0187] The dermatological image dataset may cover a broad range of dermatological diseases including the most frequently encountered conditions in clinical practice. By way of example, the dataset may include images depicting acne vulgaris, which is a chronic inflammatory skin condition affecting the pilosebaceous units. The dataset may include images depicting atopic dermatitis, also known as eczema, which is a chronic inflammatory skin condition characterized by pruritus and eczematous lesions. The dataset may include images depicting contact dermatitis, which is an inflammatory skin reaction caused by direct contact with an irritant or allergen. The dataset may include images depicting seborrheic dermatitis, which is achronic inflammatory condition affecting sebaceous gland-rich areas of the skin. The dataset may include images depicting psoriasis, which is a chronic autoimmune condition characterized by scaly erythematous plaques. The dataset may include images depicting dermatophyte infections including tinea and other fungal infections of the skin, hair, and nails. The dataset may include images depicting rosacea, which is a chronic inflammatory condition primarily affecting the central face. The dataset may include images depicting viral warts caused by human papillomavirus (HPV). The dataset may include images depicting scabies, which is a contagious skin infestation caused by the mite Sarcoptes scabiei. The dataset may include images depicting urticaria, also known as hives, which is a condition characterized by transient pruritic wheals. The dataset may further include images depicting additional dermatological conditions beyond those enumerated above.

[0188] Cosine similarity features are generated (920) by providing the dermatological image dataset to the trained medical foundation model. In various embodiments, each image in the dermatological image dataset is provided to the image encoder component of the trained medical foundation model, which generates an image embedding for each image. A set of dermatological disease labels corresponding to the conditions represented in the dataset is provided to the text encoder component of the trained medical foundation model, which generates text embeddings for each disease label. Cosine similarities between each image embedding and each disease label text embedding are calculated, producing a feature vector of cosine similarity values for each image in the dataset.

[0189] A domain-specific dermatological classifier is trained (930) based on the cosine similarity features. In various embodiments, the domain-specific classifier may be trained using the cosine similarity feature vectors as input features and the annotated diagnoses as target labels. The classifier may be implemented using classical machine learning algorithms such as logistic regression, random forest, gradient boosting, or support vector machines, or using neural network architectures as appropriate to the classification task. The combination of the broad medical knowledge encoded in the trained medical foundation model and the domain-specific dermatological training data may enable the classifier to achieve high classification performance across the broad range of dermatological conditions represented in the dataset.

[0190] A diagnostic prediction is generated (940) for an input dermatological image. In various embodiments, the trained dermatological classifier may be deployed to classify new dermatological images by generating cosine similarity features using the trained medical foundation model and applying the trained classifier to produce a diagnostic prediction. The diagnostic prediction may include a predicted diagnosis, a confidence score, and relevant differential diagnoses. The skin color classification and body location of the input image may be used to further refine the diagnostic prediction, as certain dermatological conditions present differently across skin tones and vary in prevalence and appearance across anatomical regions. In various embodiments, the skin color classifications and body location identifiers may be incorporated as additional input features to the domain-specific classifier alongside the cosine similarity features, enabling the classifier to account for these variations during training and inference.Medical Emergency Detection from Video Streams

[0191] The trained medical foundation model may also be applied to real-time video analysis for detecting medical emergencies by machines equipped with camera sensors. Such machines may include, by way of example and not limitation, robots, including humanoid robots, that operate in environments alongside humans and are equipped with camera sensors that capture continuous video streams of their surroundings; vehicles, including automobiles, equipped with a driver-facing camera, a passenger-facing camera, or both, that monitor the occupants of the vehicle during operation; and home monitoring systems equipped with one or more cameras capable of panning, tilting, and rotating in multiple directions to provide comprehensive coverage of a living space. By applying the trained medical foundation model to frames extracted from these video streams, the machine may detect medical emergencies as they occur and initiate emergency communications autonomously. This capability serves multiple purposes: it provides a safety mechanism for individuals in the vicinity of the machine who may experience a medical emergency, it enables continuous health monitoring of vehicle occupants who may experience sudden cardiac events, seizures, diabetic emergencies, or other acute conditions while driving or riding, it enables monitoring of individuals in home environments particularly elderly or at-risk individuals who may experience falls, strokes, or other medical events, and in the case of robots, it provides a safety mechanism for situations where the robot itself maycause a medical emergency through its own actions, such as through accidental collisions or other incidents involving physical contact between the machine and nearby individuals.

[0192] Turning now to FIG. 10, a process 1000 for detecting medical emergencies from a video stream captured by a machine using a trained medical foundation model is illustrated in accordance with an embodiment of the invention. The process 1000 may be executed by processors onboard the machine, by a remote server 200 described with reference to FIG. 2, or by a combination thereof.

[0193] The process 1000 begins with obtaining (1010) a video stream from at least one camera sensor on the machine. In various embodiments, the machine may be equipped with one or more camera sensors mounted at various positions on the machine to provide comprehensive visual coverage of the surrounding environment. The camera sensors may capture video at standard frame rates and resolutions sufficient for medical image analysis.

[0194] In various embodiments where the machine is a vehicle, the vehicle may be equipped with a driver-facing camera mounted on the dashboard, steering column, rearview mirror, or instrument cluster, a passenger-facing camera mounted to capture the cabin interior, or both. The driver-facing camera may monitor the driver for signs of medical distress such as loss of consciousness, seizure activity, or sudden incapacitation. The passenger-facing camera may monitor passengers for similar medical events. In various embodiments where the machine is a home monitoring system, the home monitoring system may include one or more cameras equipped with pan-tilt-zoom mechanisms that enable the cameras to rotate, pan, and tilt in multiple directions to provide comprehensive coverage of a living space. The pan-tilt-zoom mechanisms may enable the cameras to autonomously reposition to track detected individuals, scan different rooms or zones within the living space, and focus on areas of detected activity. In some embodiments, the home monitoring system cameras may be ceiling-mounted, wall-mounted, or placed on surfaces within the living space, and may be configured to cover multiple rooms or zones through scheduled or event-triggered panning sequences. The camera sensors may capture video at standard frame rates and resolutions sufficient for medical image analysis. In some embodiments, the machine may include stereo cameras, depth cameras, infraredcameras for low-light monitoring, or other imaging sensors that provide additional spatial or temporal information about the scene.

[0195] Image frames are extracted (1020) from the video stream. In various embodiments, frames may be extracted at regular intervals, such as one frame per second or one frame every few seconds, to enable real-time analysis without excessive computational overhead. In some embodiments, the frame extraction rate may be dynamically adjusted based on the activity level detected in the scene, with higher extraction rates applied during periods of rapid movement or unusual activity

[0196] The extracted image frames and medical emergency description labels are provided (1030) to the trained medical foundation model. In various embodiments, the medical emergency description labels may include textual descriptions of various types of medical emergencies such as cardiac arrest, loss of consciousness, seizure, severe bleeding, choking, anaphylactic reaction, fall with injury, stroke symptoms, severe burn, and other acute medical conditions requiring immediate emergency response. The medical emergency description labels may also include labels describing non-emergency states such as normal activity, resting, and routine movement. The image encoder component of the trained medical foundation model generates image embeddings for each extracted frame, and the text encoder component generates text embeddings for each medical emergency description label.

[0197] Each image frame is classified (1040) as depicting a medical emergency or a non-medical emergency based on the cosine similarities. In various embodiments, cosine similarities between the image embedding for each frame and the text embeddings for each medical emergency description label are calculated. The classification may be performed using a modified SoftMax function as described with reference to FIG. 4, a domain-specific classical machine learning model as described with reference to FIG. 5, or another classification method suitable for distinguishing medical emergency states from non-emergency states. In some embodiments, temporal consistency checks may be applied across multiple consecutive frames to reduce false positive detections, requiring that a medical emergency classification be sustained across a plurality of consecutive frames before triggering an emergency response.

[0198] In response to classifying an image frame as depicting a medical emergency (1050), the machine initiates an emergency communication. In various embodiments, the emergency communication includes contacting an emergency service line, such as the 911 emergency service in the United States or equivalent emergency services in other jurisdictions. The machine may transmit a description of the nature of the medical emergency based on the cosine similarities between the image frame and the medical emergency description labels. The medical emergency description label with the highest cosine similarity to the classified frame may be used to generate a natural language description of the emergency. In some embodiments, a large language model may be used to generate a comprehensive emergency description based on the cosine similarity values and the corresponding medical emergency description labels. In embodiments where the machine is a vehicle, the emergency communication may further include the vehicle’s location obtained through a global positioning system, enabling emergency responders to locate the vehicle. In embodiments where the machine is a home monitoring system, the emergency communication may include the address of the home and the specific room or zone where the emergency was detected.

[0199] The machine may also determine and communicate the number of individuals involved in the medical emergency. In various embodiments, person detection algorithms may be applied to the image frames to count the number of individuals present in the scene, and the number of individuals exhibiting signs of medical distress may be estimated based on the classification results.

[0200] In some embodiments, the process 1000 further includes determining whether the medical emergency was caused by an action of the machine itself. In embodiments where the machine is a robot, the robot may maintain a log of its recent actions including movements, physical interactions, and any contact events detected through force sensors or collision detection systems. If a medical emergency is detected temporally proximate to a recorded contact event or an unexpected force reading on the robot, the system may determine that the robot’s action may have contributed to the medical emergency. In embodiments where the machine is a vehicle, the vehicle may maintain a log of recent driving events including sudden braking, acceleration, collisions, or airbag deployments. If a medical emergency is detected in a vehicle occupant temporally proximate to a recorded collision or suddendeceleration event, the system may determine that the driving event contributed to the medical emergency. This determination may be included in the emergency communication to provide emergency responders with relevant contextual information, such as whether the emergency resulted from a vehicle collision ora robot interaction. The inclusion of machine-caused emergency detection addresses safety considerations for autonomous and semi-autonomous machines operating in proximity to humans, ensuring that the machine can report emergencies it may have caused and provide accurate information to emergency responders.

[0201] In various embodiments, the process 1000 operates continuously during the machine’s active operation, providing persistent monitoring for medical emergencies in the machine’s environment. The process 1000 may be configured to operate with minimal computational overhead to avoid interfering with the machine’s primary operational functions. The integration of the trained medical foundation model with machine systems enables autonomous emergency detection and response capabilities that may improve safety outcomes in environments where machines and humans coexist.Rare Disease and Unknown Disease Diagnosis Using Real-Time Image Retrieval

[0202] The trained medical foundation model develops a comprehensive embedding space through contrastive language-image pre-training on a large corpus of annotated medical images. A property of this embedding space is that it captures meaningful visual and semantic relationships between medical images even for conditions where only very few or no training images were present in the training dataset. This property arises because the contrastive training objective encourages the model to learn generalizable visual features that discriminate between different medical conditions based on their textual descriptions, rather than memorizing specific training examples. Consequently, the embedding space of the trained medical foundation model may encode visual similarity relationships for rare diseases and previously unknown diseases, even though the model was not explicitly trained on images of those conditions.

[0203] Turning now to FIG. 11 , a process 1100 for diagnosing rare diseases and unknown diseases using real-time image retrieval and a trained medical foundation model is illustrated in accordance with an embodiment of the invention. The process1100 may be executed by the server 200 described with reference to FIG. 2, utilizing the trained medical foundation model 234 stored in the memory 230. The process 1100 leverages the generalization capability of the embedding space to perform diagnostic classification for conditions that are underrepresented or absent from the training data.

[0204] The process 1100 begins with obtaining (1110) an input medical image of a patient suspected of having a rare disease or an unknown disease. In various embodiments, the input medical image may be received through the input / output interface 220 from the client device 130 over the network 140. The input medical image may be a clinical photograph, a dermatological image, a radiographic image, or any other type of medical imaging data. In some embodiments, a clinician may provide a suspected condition name or a set of candidate conditions along with the input medical image. In other embodiments, the system may automatically generate candidate conditions based on initial analysis of the input medical image using the trained medical foundation model.

[0205] For rare disease diagnosis, the system queries (1120) at least one search engine using the suspected condition name to retrieve reference images depicting the suspected condition from publicly accessible internet sources. In various embodiments, the search engine query may include the suspected condition name combined with medical imaging keywords to retrieve clinically relevant images. The search engine may return images from medical databases, medical journal websites, educational medical resources, clinical case repositories, dermatological atlases, and other publicly accessible internet sources. In some embodiments, multiple search engines may be queried to increase the diversity and coverage of retrieved reference images. The reference images may be retrieved in real time in response to receiving the input medical image, enabling the system to leverage the most current publicly available medical imagery for the suspected condition. In various embodiments, a predetermined number of reference images may be retrieved, such as 10, 50, or 100 reference images per queried condition.

[0206] Image embeddings are generated (1130) for the input medical image and for each retrieved reference image using the image encoder component of the trained medical foundation model. In various embodiments, the image encoder component processes each image to produce a vector representation in the embedding space ofthe trained medical foundation model. Because the embedding space captures generalizable visual features, the image embeddings for rare disease images may be positioned in regions of the embedding space that reflect the visual characteristics of the condition, even if no images of that specific condition were present during training.

[0207] Cosine similarities are computed (1140) between the image embedding of the input medical image and the image embeddings of the retrieved reference images for the suspected condition. In various embodiments, the cosine similarity between the input image embedding and each reference image embedding is calculated as the dot product of the two embedding vectors divided by the product of their respective magnitudes. The cosine similarity values indicate the degree of visual similarity between the input medical image and each reference image in the embedding space of the trained medical foundation model. A high average cosine similarity between the input medical image and the reference images for a suspected condition suggests that the input image depicts visual features consistent with that condition.

[0208] The querying, generating, and computing steps are repeated (1150) for at least one differential diagnosis condition until all queued differential conditions are queried. In various embodiments, the system queries the search engine for reference images depicting each differential diagnosis condition, generates image embeddings for the retrieved reference images, and computes cosine similarities between the input image embedding and the reference image embeddings for each differential diagnosis. The differential diagnosis conditions may be provided by the clinician, may be automatically generated based on the suspected condition using a large language model or a medical ontology, or may be derived from the initial cosine similarity analysis of the input image against domain labels. By computing cosine similarities for both the suspected condition and differential diagnoses, the system generates a comparative similarity profile that indicates the relative likelihood of each condition.

[0209] The cosine similarities are aggregated (1160) across the suspected condition and the differential diagnosis conditions to generate a diagnostic prediction. In various embodiments, the aggregation may involve computing an average cosine similarity for each condition based on the retrieved reference images and comparing the average cosine similarities across conditions. The condition with the highest average cosine similarity to the input medical image may be predicted as the most likely diagnosis. In some embodiments, a modified SoftMax function as described withreference to FIG. 4 may be applied to the aggregated cosine similarities to generate probability scores for each condition. In various embodiments, the aggregation may further incorporate text-based cosine similarities between the input image embedding and text embeddings of the condition names to provide additional discriminative information. The diagnostic prediction may include a ranked list of conditions ordered by likelihood, along with confidence scores and the reference images that were most similar to the input medical image, enabling the clinician to visually compare the input image with the highest-scoring reference images.

[0210] For unknown disease diagnosis, additional steps can be taken in a clustering mode. In various embodiments, when a potentially novel medical condition is suspected, such as during a disease outbreak or pandemic, a plurality of candidate images may be obtained from publicly accessible internet sources. The candidate images may be obtained from social media platforms, news websites, public health reporting systems, clinical case sharing platforms, or other internet sources where images of the potentially novel condition may be posted. In some embodiments, the candidate images may be identified using the medical versus non-medical classification process 800 described with reference to FIG. 8 to filter medical images from non-medical content, followed by keyword-based filtering to identify images potentially associated with the novel condition.

[0211] Image embeddings are generated for the plurality of candidate images and the input medical image using the image encoder component of the trained medical foundation model. The system then determines whether the image embeddings of the plurality of candidate images and the input medical image form a cluster in the embedding space of the trained medical foundation model. In various embodiments, clustering may be performed using algorithms such as k-means clustering, DBSCAN, hierarchical clustering, or other clustering methods applied to the image embeddings. Cluster formation is indicative that the images share common visual features that distinguish them from other medical conditions represented in the embedding space. If the input medical image falls within a cluster formed by the candidate images, the system may predict that the input medical image depicts the unknown disease associated with the cluster. In some embodiments, the system may compute an average embedding for the cluster and calculate the cosine similarity between the input image embedding and the cluster centroid to quantify the degree of association.

[0212] The clustering approach for unknown diseases leverages the property that the trained medical foundation model’s embedding space organizes images by visual similarity in a medically meaningful way. Even for a previously unknown condition, images depicting that condition would share visual features that cause their embeddings to cluster together in the embedding space, distinct from embeddings of known conditions. This enables the system to detect potential new diseases by identifying emergent clusters that do not correspond to known medical conditions. In various embodiments, the system may track the formation and growth of new clusters over time, providing early detection signals for emerging medical conditions. The system may alert public health authorities when a new cluster is detected that exceeds a predetermined size threshold, potentially contributing to early outbreak detection and surveillance.

[0213] In various embodiments, the rare disease and unknown disease diagnosis capabilities may be combined. The system may first attempt rare disease diagnosis by retrieving reference images for candidate conditions and computing cosine similarities. If no candidate condition achieves a cosine similarity above a confidence threshold, the system may switch to the unknown disease clustering mode to determine whetherthe input image may depict a previously uncharacterized condition. This two-stage approach enables the system to handle both underrepresented known conditions and genuinely novel conditions within a unified diagnostic framework.

[0214] Although specifics are discussed above, many different fabrication methods can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

CLAIMS1. A method fortraining a medical foundation model, the method comprising:obtaining a corpus of medical journal papers;extracting images from each medical journal paper in the corpus;filtering out abstract figures from the extracted images, wherein the abstract figures comprise non-medical images;generating augmented figure descriptions based on context extracted from publication text associated with each extracted image;generating image / description pairs by combining filtered medical images with corresponding augmented figure descriptions;annotating each image / description pair using a medical ontology to extract medical concepts from textual content associated with each image; andtraining an image encoder model and a text encoder model using contrastive language-image pre-training based on the annotated image / description pairs to yield a trained medical foundation model.

2. The method of claim 1, wherein filtering out abstract figures comprises classifying extracted images using a convolutional neural network into a plurality of image types comprising:individual medical images;image panels containing multiple medical images;mixed panels containing both medical images and abstract visuals; and abstract visuals only;wherein the filtering retains individual medical images, image panels, and mixed panels while excluding abstract visuals only.

3. The method of claim 1, wherein generating augmented figure descriptions comprises using a fine-tuned large language model to extract diagnoses from a paper title and paragraphs that reference a figure associated with each extracted image.

4. The method of claim 3, wherein the fine-tuned large language model comprises a BioMedLM model pre-trained on biomedical literature.

5. The method of claim 1, wherein annotating each image / description pair using a medical ontology comprises extracting medical concepts using a Unified Medical Language System parser.

6. The method of claim 5, wherein each identified medical concept is associated with a Concept Unique Identifier from the Unified Medical Language System.

7. The method of claim 1, wherein the image encoder model comprises a vision transformer.

8. The method of claim 7, wherein the vision transformer comprises a ViT-Large model pre-trained on a large-scale dataset of image-text pairs.

9. The method of claim 1, wherein the text encoder model comprises a transformerbased language model pre-trained on biomedical literature.

10. The method of claim 9, wherein the transformer-based language model comprises PubMedBert.

11. The method of claim 1, further comprising applying data augmentation to the image / description pairs during training.

12. The method of claim 1, wherein training the image encoder model and the text encoder model comprises using a contrastive loss function that maximizes similarity between embeddings of corresponding image / description pairs and minimizes similarity between embeddings of non-corresponding pairs.

13. A method for classifying a medical image, the method comprising:obtaining an input medical image;generating image-specific domain labels comprising medical terms relevant to a classification task;providing the input medical image and the image-specific domain labels to a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;obtaining cosine similarities between the input medical image and each of the image-specific domain labels from the trained medical foundation model, wherein the image encoder component generates an image embedding for the input medical image and the text encoder component generates text embeddings for each of the image-specific domain labels;refining a prediction using a modified SoftMax function, wherein the modified SoftMax function aggregates logits belonging to a same medical condition within positive and negative categories to generate aggregated scores for each medical condition before applying a SoftMax transformation to generate probability scores; and providing the prediction.

14. The method of claim 13, wherein the input medical image comprises at least one of a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, ora clinical photograph.

15. The method of claim 13, wherein the image-specific domain labels comprise terms describing disease conditions and terms describing normalcy.

16. The method of claim 13, wherein generating image-specific domain labels comprises automatically generating the domain labels based on a type of the input medical image.

17. The method of claim 13, wherein the cosine similarity between the image embedding and each text embedding is calculated as a dot product of the image embedding and the text embedding divided by a product of their respective magnitudes.

18. The method of claim 13, wherein the modified SoftMax function calculates an average of cosine similarity logits for medical terms within a positive category for a medical condition and an average of cosine similarity logits for medical terms within a negative category for the medical condition.

19. The method of claim 18, wherein the modified SoftMax function applies a SoftMax transformation to the averaged logits for the positive and negative categories to generate probability scores for each medical condition.

20. The method of claim 13, wherein the positive category comprises medical terms indicative of a target condition and the negative category comprises medical terms indicative of normalcy or alternative conditions for binary classification.

21. The method of claim 13, wherein the method performs zero-shot classification of the input medical image without requiring task-specific training data for the classification task.

22. The method of claim 13, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

23. The method of claim 13, wherein the modified SoftMax function calculates probability scores according to:a cosine similarity between the image embedding and each text embedding calculated as ltJ= cos(Vj, 7}), where Ltj represents the cosine similarity logit for image i and text description j, represents the image embedding vector, and 7) represents the text embedding vector;an aggregation of logits for medical terms belonging to a same medical condition calculated as Sik= -■■-' . jeckkj> where Sikrepresents the aggregated score for medical condition k, Nkrepresents the number of medical terms belonging to condition category k, and Ckrepresents the set of medical term indices belonging to condition category k anda probability that the input medical image belongs to medical condition k esikcalculated as Pik= - — , where K represents the set of all medical conditioncategories being considered for classification.

24. The method of claim 13, wherein the trained medical foundation model is trained according to the method of claim 1.

25. A method for classifying a medical image, the method comprising:obtaining an input medical image;generating image-specific domain labels comprising medical terms relevant to a classification task;providing the input medical image and the image-specific domain labels to a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;obtaining cosine similarities between the input medical image and each of the image-specific domain labels from the trained medical foundation model, wherein the image encoder component generates an image embedding for the input medical image and the text encoder component generates text embeddings for each of the image-specific domain labels; andusing a domain-specific classical machine learning model to refine a diagnostic prediction based on the cosine similarities.

26. The method of claim 25, wherein the input medical image comprises at least one of a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, ora clinical photograph.

27. The method of claim 25, wherein the image-specific domain labels comprise terms describing conditions relevant to a diagnostic domain of the input medical image.

28. The method of claim 25, wherein generating image-specific domain labels comprises automatically generating the domain labels based on a type of the input medical image.

29. The method of claim 25, wherein the cosine similarities serve as input features to the domain-specific classical machine learning model.

30. The method of claim 25, wherein the domain-specific classical machine learning model comprises at least one of a Logistic Regression model, a Random Forest model, a Gradient Boosting model, or a Support Vector Machine.

31. The method of claim 25, wherein the domain-specific classical machine learning model is trained on a set of domain-specific medical images using cosine similarity features extracted from the domain-specific medical images by the trained medical foundation model.

32. The method of claim 25, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

33. The method of claim 25, wherein the trained medical foundation model is trained according to the method of claim 1.

34. A method for classifying a pathology slide image, the method comprising:obtaining an input pathology slide image;segmenting the input pathology slide image into a plurality of image patches; providing each image patch to a trained medical foundation model along with potential disease labels, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;obtaining cosine similarities between each of the potential disease labels and each image patch from the trained medical foundation model, wherein the image encoder component generates an image embedding for each image patch and the text encoder component generates text embeddings for each of the potential disease labels; andcalculating a mean of each cosine similarity for each potential disease label across the plurality of image patches to generate a classification prediction for the input pathology slide image.

35. The method of claim 34, wherein the input pathology slide image comprises a whole-slide image captured from a histopathology specimen using a digital pathology scanner.

36. The method of claim 34, wherein segmenting the input pathology slide image into a plurality of image patches comprises extracting tiles from the input pathology slide image at specified magnification levels and spatial dimensions.

37. The method of claim 36, wherein the tile extraction includes automated tissue detection to ensure that extracted tiles contain sufficient tissue content for analysis.

38. The method of claim 37, wherein the sufficient tissue content comprises a minimum of 80% tissue coverage.

39. The method of claim 37, wherein the tissue coverage is validated using grayscale conversion and Otsu thresholding.

40. The method of claim 34, wherein the potential disease labels comprise cancer subtypes, benign conditions, and normal tissue classifications relevant to the histopathology specimen being analyzed.

41. The method of claim 34, further comprising aggregating from tile-level predictions to a patient-level diagnosis by excluding normal tissue scores.

42. The method of claim 41, wherein the aggregating comprises identifying tiles with top scores for cancer subtypes for generating the patient-level diagnosis.

43. The method of claim 34, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

44. The method of claim 34, wherein the trained medical foundation model is trained according to the method of claim 1.

45. A method for retrieving relevant medical literature based on an input medical image, the method comprising:obtaining an input medical image;providing the input medical image and a database of image / description pairs to a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component;identifying most similar images to the input medical image based on image embeddings generated by the image encoder component of the trained medical foundation model;ranking the most similar images based on cosine similarities between the image embedding of the input medical image and text embeddings of diagnoses extracted from descriptions associated with each similar image using the trained medical foundation model; andreturning a paper from which a highest ranked image originates.

46. The method of claim 45, wherein the input medical image comprises at least one of a clinical photograph, a radiographic image, or a histopathology slide.

47. The method of claim 45, wherein the database of image / description pairs comprises medical images extracted from medical journal papers with associated textual descriptions including augmented figure descriptions and medical ontology annotations.

48. The method of claim 45, wherein image embeddings for images in the database are pre-computed and stored for efficient retrieval.

49. The method of claim 45, wherein cosine similarity is used to measure similarity between image embeddings.

50. The method of claim 45, wherein the predicted diagnosis is generated based on cosine similarities between an embedding of the input medical image and text embeddings for medical terms.

51. The method of claim 45, wherein the ranking prioritizes images whose associated descriptions contain diagnoses that align with the predicted diagnosis for the input medical image.

52. The method of claim 45, further comprising returning multiple papers corresponding to top-ranked similar images.

53. The method of claim 45, wherein returning the paper comprises returning publication details including at least one of a paper title, authors, journal name, publication date, or a link to a full publication.

54. The method of claim 45, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

55. The method of claim 45, wherein the trained medical foundation model is trained according to the method of claim 1.

56. A system fortraining a medical foundation model, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining a corpus of medical journal papers;extracting images from each medical journal paper in the corpus; filtering out abstract figures from the extracted images, wherein the abstract figures comprise non-medical images;generating augmented figure descriptions based on context extracted from publication text associated with each extracted image;generating image / description pairs by combining filtered medical images with corresponding augmented figure descriptions;annotating each image / description pair using a medical ontology to extract medical concepts from textual content associated with each image; and training an image encoder model and a text encoder model using contrastive language-image pre-training based on the annotated image / description pairs to yield a trained medical foundation model.

57. The system of claim 56, wherein filtering out abstract figures comprises classifying extracted images using a convolutional neural network into a plurality of image types comprising:individual medical images;image panels containing multiple medical images;mixed panels containing both medical images and abstract visuals; and abstract visuals only;wherein the filtering retains individual medical images, image panels, and mixed panels while excluding abstract visuals only.

58. The system of claim 56, wherein generating augmented figure descriptions comprises using a fine-tuned large language model to extract diagnoses from a paper title and paragraphs that reference a figure associated with each extracted image.

59. The system of claim 58, wherein the fine-tuned large language model comprises a BioMedLM model pre-trained on biomedical literature.

60. The system of claim 56, wherein annotating each image / description pair using a medical ontology comprises extracting medical concepts using a Unified Medical Language System parser.

61. The system of claim 60, wherein each identified medical concept is associated with a Concept Unique Identifier from the Unified Medical Language System.

62. The system of claim 56, wherein the image encoder model comprises a vision transformer.

63. The system of claim 62, wherein the vision transformer comprises a ViT-Large model pre-trained on a large-scale dataset of image-text pairs.

64. The system of claim 56, wherein the text encoder model comprises a transformerbased language model pre-trained on biomedical literature.

65. The system of claim 64, wherein the transformer-based language model comprises PubMedBert.

66. The system of claim 56, wherein the operations further comprise applying data augmentation to the image / description pairs during training.

67. The system of claim 56, wherein training the image encoder model and the text encoder model comprises using a contrastive loss function that maximizes similarity between embeddings of corresponding image / description pairs and minimizes similarity between embeddings of non-corresponding pairs.

68. A system for classifying a medical image, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining an input medical image;generating image-specific domain labels comprising medical terms relevant to a classification task;providing the input medical image and the image-specific domain labels to a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;obtaining cosine similarities between the input medical image and each of the image-specific domain labels from the trained medical foundation model, wherein the image encoder component generates an image embedding for the input medical image and the text encoder component generates text embeddings for each of the image-specific domain labels;refining a prediction using a modified SoftMax function, wherein the modified SoftMax function aggregates logits belonging to a same medical condition within positive and negative categories to generate aggregated scores for each medical condition before applying a SoftMax transformation to generate probability scores; andproviding the prediction.

69. The system of claim 68, wherein the input medical image comprises at least one of a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, ora clinical photograph.

70. The system of claim 68, wherein the image-specific domain labels comprise terms describing disease conditions and terms describing normalcy.

71. The system of claim 68, wherein generating image-specific domain labels comprises automatically generating the domain labels based on a type of the input medical image.

72. The system of claim 68, wherein the cosine similarity between the image embedding and each text embedding is calculated as a dot product of the image embedding and the text embedding divided by a product of their respective magnitudes.

73. The system of claim 68, wherein the modified SoftMax function calculates an average of cosine similarity logits for medical terms within a positive category for a medical condition and an average of cosine similarity logits for medical terms within a negative category for the medical condition.

74. The system of claim 73, wherein the modified SoftMax function applies a SoftMax transformation to the averaged logits for the positive and negative categories to generate probability scores for each medical condition.

75. The system of claim 68, wherein the positive category comprises medical terms indicative of a target condition and the negative category comprises medical terms indicative of normalcy or alternative conditions for binary classification.

76. The system of claim 68, wherein the system performs zero-shot classification of the input medical image without requiring task-specific training data for the classification task.

77. The system of claim 68, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

78. The system of claim 68, wherein the modified SoftMax function calculates probability scores according to:a cosine similarity between the image embedding and each text embedding calculated as Itj = cos(y 7}), where Ltj represents the cosine similarity logit for image i and text description j, represents the image embedding vector, and 7) represents the text embedding vector;an aggregation of logits for medical terms belonging to a same medical condition calculated as Sik= where Sikrepresents the aggregated scorefor medical condition k, Nkrepresents the number of medical terms belonging to condition category k, and Ckrepresents the set of medical term indices belonging to condition category k anda probability that the input medical image belongs to medical condition k esikcalculated as Pik= - — , where K represents the set of all medical conditioncategories being considered for classification.

79. The system of claim 68, wherein the trained medical foundation model is trained according to the operations of claim 56.

80. A system for classifying a medical image, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining an input medical image;generating image-specific domain labels comprising medical terms relevant to a classification task;providing the input medical image and the image-specific domain labels to a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;obtaining cosine similarities between the input medical image and each of the image-specific domain labels from the trained medical foundation model, wherein the image encoder component generates an image embedding for the input medical image and the text encoder component generates text embeddings for each of the image-specific domain labels; andusing a domain-specific classical machine learning model to refine a diagnostic prediction based on the cosine similarities.

81. The system of claim 80, wherein the input medical image comprises at least one of a chest X-ray, a computed tomography scan, a magnetic resonance image, a histopathology slide, an ultrasound image, ora clinical photograph.

82. The system of claim 80, wherein the image-specific domain labels comprise terms describing conditions relevant to a diagnostic domain of the input medical image.

83. The system of claim 80, wherein generating image-specific domain labels comprises automatically generating the domain labels based on a type of the input medical image.

84. The system of claim 80, wherein the cosine similarities serve as input features to the domain-specific classical machine learning model.

85. The system of claim 80, wherein the domain-specific classical machine learning model comprises at least one of a Logistic Regression model, a Random Forest model, a Gradient Boosting model, or a Support Vector Machine.

86. The system of claim 80, wherein the domain-specific classical machine learning model is trained on a set of domain-specific medical images using cosine similarity features extracted from the domain-specific medical images by the trained medical foundation model.

87. The system of claim 80, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

88. The system of claim 80, wherein the trained medical foundation model is trained according to the operations of claim 56.

89. A system for classifying a pathology slide image, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining an input pathology slide image;segmenting the input pathology slide image into a plurality of image patches; providing each image patch to a trained medical foundation model along with potential disease labels, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;obtaining cosine similarities between each of the potential disease labels and each image patch from the trained medical foundation model, wherein the image encoder component generates an image embedding for each image patch and the text encoder component generates text embeddings for each of the potential disease labels; andcalculating a mean of each cosine similarity for each potential disease label across the plurality of image patches to generate a classification prediction for the input pathology slide image.

90. The system of claim 89, wherein the input pathology slide image comprises a whole-slide image captured from a histopathology specimen using a digital pathology scanner.

91. The system of claim 89, wherein segmenting the input pathology slide image into a plurality of image patches comprises extracting tiles from the input pathology slide image at specified magnification levels and spatial dimensions.

92. The system of claim 91, wherein the tile extraction includes automated tissue detection to ensure that extracted tiles contain sufficient tissue content for analysis.

93. The system of claim 92, wherein the sufficient tissue content comprises a minimum of 80% tissue coverage.

94. The system of claim 92, wherein the tissue coverage is validated using grayscale conversion and Otsu thresholding.

95. The system of claim 89, wherein the potential disease labels comprise cancer subtypes, benign conditions, and normal tissue classifications relevant to the histopathology specimen being analyzed.

96. The system of claim 89, wherein the operations further comprise aggregating from tile-level predictions to a patient-level diagnosis by excluding normal tissue scores.

97. The system of claim 96, wherein the aggregating comprises identifying tiles with top scores for cancer subtypes for generating the patient-level diagnosis.

98. The system of claim 89, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

99. The system of claim 89, wherein the trained medical foundation model is trained according to the operations of claim 56.

100. A system for retrieving relevant medical literature based on an input medical image, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining an input medical image;providing the input medical image and a database of image / description pairs to a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component;identifying most similar images to the input medical image based on image embeddings generated by the image encoder component of the trained medical foundation model;ranking the most similar images based on cosine similarities between the image embedding of the input medical image and text embeddings of diagnoses extracted from descriptions associated with each similar image using the trained medical foundation model; and returning a paper from which a highest ranked image originates.

101. The system of claim 100, wherein the input medical image comprises at least one of a clinical photograph, a radiographic image, or a histopathology slide.

102. The system of claim 100, wherein the database of image / description pairs comprises medical images extracted from medical journal papers with associated textual descriptions including augmented figure descriptions and medical ontology annotations.

103. The system of claim 100, wherein image embeddings for images in the database are pre-computed and stored for efficient retrieval.

104. The system of claim 100, wherein cosine similarity is used to measure similarity between image embeddings.

105. The system of claim 100, wherein the predicted diagnosis is generated based on cosine similarities between an embedding of the input medical image and text embeddings for medical terms.

106. The system of claim 100, wherein the ranking prioritizes images whose associated descriptions contain diagnoses that align with the predicted diagnosis for the input medical image.

107. The system of claim 100, wherein the operations further comprise returning multiple papers corresponding to top-ranked similar images.

108. The system of claim 100, wherein returning the paper comprises returning publication details including at least one of a paper title, authors, journal name, publication date, or a link to a full publication.

109. The system of claim 100, wherein the trained medical foundation model is trained using contrastive language-image pre-training on a dataset of annotated medical images extracted from medical journal papers.

110. The system of claim 100, wherein the trained medical foundation model is trained according to the operations of claim 56.

111. A method for classifying an image / text pair as medical or non-medical using a trained medical foundation model, the method comprising:obtaining an input image and an input text description;computing a direct cosine similarity between an image embedding of the input image and a text embedding of the input text description using the trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;computing a medical image-text similarity as an average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known medical text descriptions;computing a medical text-image similarity as an average of cosine similarities between the text embedding of the input text description and image embeddings of a plurality of known medical images;computing a non-medical image-text similarity as an average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known non-medical text descriptions;computing a non-medical text-image similarity as an average of cosine similarities between the text embedding of the input text description and image embeddings of a plurality of known non-medical images;calculating a medical domain specificity score based on the direct cosine similarity, the medical image-text similarity, the medical text-image similarity, the nonmedical image-text similarity, and the non-medical text-image similarity; and classifying the image / text pair as medical or non-medical based on the medical domain specificity score.

112. The method of claim 111, wherein the medical domain specificity score is calculated as MDSS = (S_img_med + S_txt_med) - (S_img_nonmed + S_txt_nonmed), where S_img_med represents the medical image-text similarity, S_txt_med represents the medical text-image similarity, S_img_nonmed represents the non-medical image-text similarity, and S_txt_nonmed represents the non-medical text-image similarity.

113. The method of claim 111, wherein the image / text pair is classified as medical when the medical domain specificity score exceeds a predetermined threshold and as non-medical when the medical domain specificity score falls below the predetermined threshold.

114. The method of claim 113, wherein the predetermined threshold is optimized based on a validation set of labeled medical and non-medical image / text pairs.

115. The method of claim 111, wherein the plurality of known medical text descriptions comprises at least 1000 text descriptions known to describe medical images.

116. The method of claim 111, wherein the plurality of known non-medical images comprises at least 1000 images known to be non-medical images.

117. The method of claim 111, wherein image embeddings for the plurality of known medical images, text embeddings for the plurality of known medical text descriptions, image embeddings for the plurality of known non-medical images, and textembeddings for the plurality of known non-medical text descriptions are pre-computed and stored for efficient classification.

118. The method of claim 111, wherein the method is used for automated knowledge discovery on the internet by scraping image / text pairs from publicly accessible web sources, classifying the scraped image / text pairs as medical or non-medical, and collecting the medical image / text pairs to create a training dataset for improving performance of the trained medical foundation model.

119. The method of claim 118, wherein the method is used iteratively, wherein the trained medical foundation model is used to discover medical image / text pairs on the internet, the discovered pairs are used to retrain the trained medical foundation model, and the retrained model is used to discover additional medical content with greater accuracy.

120. The method of claim 111, wherein the trained medical foundation model is trained according to the method of claim 1.

121. A method for creating a dermatological disease classifier, the method comprising:obtaining a dermatological image dataset annotated with diagnoses, skin color classifications, and body location identifiers;providing the dermatological image dataset to a trained medical foundation model comprising an image encoder component and a text encoder component to generate cosine similarity features between each image and a set of dermatological disease labels; andtraining a domain-specific dermatological classifier based on the cosine similarity features to classify dermatological diseases across a broad range of conditions.

122. The method of claim 121, wherein the dermatological image dataset comprises approximately 138,000 images scraped from publicly accessible internet sources.

123. The method of claim 121, wherein the broad range of dermatological conditions comprises acne vulgaris, atopic dermatitis, contact dermatitis, seborrheic dermatitis, psoriasis, dermatophyte infections, rosacea, viral warts, scabies, and urticaria.

124. The method of claim 121, wherein each image in the dermatological image dataset is annotated with at least a diagnosis, a skin color classification, and a body location identifier.

125. The method of claim 121, wherein the skin color classifications and body location identifiers are incorporated as additional input features to the domain-specific dermatological classifier alongside the cosine similarity features.

126. The method of claim 121, wherein the dermatological image dataset is created using a medical domain specificity score to identify and collect dermatological image / text pairs from the internet.

127. The method of claim 121, wherein the trained medical foundation model is trained according to the method of claim 1.

128. A method for detecting medical emergencies using a machine equipped with at least one camera sensor, the method comprising:obtaining a video stream from the at least one camera sensor on the machine, wherein the machine comprises at least one of a robot, a vehicle equipped with at least one of a driver-facing camera or a passenger-facing camera, or a home monitoring system equipped with at least one camera capable of panning and tilting in multiple directions;extracting image frames from the video stream;providing the extracted image frames and medical emergency description labels to a trained medical foundation model comprising an image encoder component and a text encoder component to obtain cosine similarities between each image frame and each medical emergency description label;classifying each image frame as depicting a medical emergency or a nonmedical emergency based on the cosine similarities; andin response to classifying an image frame as depicting a medical emergency, initiating an emergency communication by the machine comprising contacting an emergency service line and transmitting a description of a nature of the medical emergency and a number of individuals involved.

129. The method of claim 128, wherein the machine comprises at least one of a humanoid robot equipped with at least one camera sensor, a vehicle equipped with at least one of a driver-facing camera or a passenger-facing camera for monitoring vehicle occupants, or a home monitoring system equipped with at least one camera having pan-tilt-zoom capabilities for providing coverage of a living space.

130. The method of claim 128, wherein the medical emergency description labels comprise textual descriptions of cardiac arrest, loss of consciousness, seizure, severe bleeding, choking, anaphylactic reaction, fall with injury, stroke symptoms, and severe burn.

131. The method of claim 128, further comprising applying temporal consistency checks across a plurality of consecutive frames before initiating the emergency communication.

132. The method of claim 128, further comprising determining whether the medical emergency was caused by an action of the machine itself and including this determination in the emergency communication.

133. The method of claim 132, wherein determining whether the medical emergency was caused by an action of the machine comprises detecting a temporal proximity between the medical emergency classification and a recorded contact event, collision event, or unexpected force reading on the machine.

134. The method of claim 128, wherein the description of the nature of the medical emergency is generated based on the cosine similarities between the image frame and the medical emergency description labels.

135. The method of claim 128, wherein the number of individuals involved is determined using person detection algorithms applied to the image frames.

136. The method of claim 128, wherein the trained medical foundation model is trained according to the method of claim 1.

137. A system for classifying an image / text pair as medical or non-medical, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining an input image and an input text description;computing a direct cosine similarity between an image embedding of the input image and a text embedding of the input text description using a trained medical foundation model, wherein the trained medical foundation model comprises an image encoder component and a text encoder component;computing a medical image-text similarity as an average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known medical text descriptions;computing a non-medical image-text similarity as an average of cosine similarities between the image embedding of the input image and text embeddings of a plurality of known non-medical text descriptions;calculating a medical domain specificity score based on the computed cosine similarities; andclassifying the image / text pair as medical or non-medical based on the medical domain specificity score.

138. The system of claim 137, wherein the trained medical foundation model is trained according to the operations of claim 56.

139. A system for detecting medical emergencies, the system comprising:a machine comprising at least one camera sensor, wherein the machine comprises at least one of a robot, a vehicle equipped with at least one of a driverfacing camera ora passenger-facing camera, ora home monitoring system equipped with at least one camera capable of panning and tilting in multiple directions;one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining a video stream from the at least one camera sensor on the machine; extracting image frames from the video stream;providing the extracted image frames and medical emergency description labels to a trained medical foundation model comprising an image encoder component and a text encoder component to obtain cosine similarities between each image frame and each medical emergency description label;classifying each image frame as depicting a medical emergency or a non-medical emergency based on the cosine similarities; andin response to classifying an image frame as depicting a medical emergency, initiating an emergency communication by the machine comprising contacting anemergency service line and transmitting a description of a nature of the medical emergency and a number of individuals involved.

140. The system of claim 139, wherein the operations further comprise determining whether the medical emergency was caused by an action of the machine itself and including this determination in the emergency communication.

141. The system of claim 139, wherein the trained medical foundation model is trained according to the operations of claim 56.

142. A method for diagnosing a rare disease or an unknown disease using a trained medical foundation model, the method comprising:obtaining an input medical image of a patient suspected of having the rare disease or the unknown disease;querying at least one search engine using a suspected condition name to retrieve reference images depicting the suspected condition from publicly accessible internet sources;generating image embeddings for the input medical image and for each retrieved reference image using an image encoder component of the trained medical foundation model;computing cosine similarities between the image embedding of the input medical image and the image embeddings of the retrieved reference images;repeating the querying, generating, and computing steps for at least one differential diagnosis condition; andaggregating the cosine similarities across the suspected condition and the at least one differential diagnosis condition to generate a diagnostic prediction.

143. The method of claim 142, wherein the reference images are retrieved in real time from the at least one search engine in response to receiving the input medical image.

144. The method of claim 142, wherein the at least one differential diagnosis condition is automatically generated based on the suspected condition using at least one of a large language model or a medical ontology.

145. The method of claim 142, wherein aggregating the cosine similarities comprises computing an average cosine similarity for each condition and predicting the condition with the highest average cosine similarity as the most likely diagnosis.

146. The method of claim 142, wherein the diagnostic prediction comprises a ranked list of conditions ordered by likelihood along with reference images most similar to the input medical image.

147. The method of claim 142, further comprising, for an unknown disease:obtaining a plurality of candidate images from publicly accessible internet sources depicting a potentially novel medical condition;generating image embeddings for the plurality of candidate images and the input medical image using the image encoder component; anddetermining whether the image embeddings of the plurality of candidate images and the input medical image form a cluster in an embedding space of the trained medical foundation model, wherein cluster formation is indicative of the input medical image depicting the unknown disease.

148. The method of claim 147, wherein the plurality of candidate images are obtained from social media platforms during a disease outbreak or pandemic.

149. The method of claim 147, wherein the clustering is performed using at least one of k-means clustering, DBSCAN, or hierarchical clustering applied to the image embeddings.

150. The method of claim 147, further comprising tracking formation and growth of new clusters over time to provide early detection signals for emerging medical conditions.

151. The method of claim 142, further comprising switching from rare disease diagnosis to unknown disease clustering mode when no candidate condition achieves a cosine similarity above a confidence threshold.

152. The method of claim 142, wherein the trained medical foundation model is trained according to the method of claim 1.

153. A system for diagnosing a rare disease or an unknown disease, the system comprising:one or more processors; andmemory storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:obtaining an input medical image of a patient suspected of having the rare disease or the unknown disease;querying at least one search engine using a suspected condition name to retrieve reference images depicting the suspected condition from publicly accessible internet sources;generating image embeddings for the input medical image and for each retrieved reference image using an image encoder component of a trained medical foundation model;computing cosine similarities between the image embedding of the input medical image and the image embeddings of the retrieved reference images;repeating the querying, generating, and computing steps for at least one differential diagnosis condition; andaggregating the cosine similarities across the suspected condition and the at least one differential diagnosis condition to generate a diagnostic prediction.

154. The system of claim 153, wherein the operations further comprise, for an unknown disease:obtaining a plurality of candidate images from publicly accessible internet sources depicting a potentially novel medical condition;generating image embeddings for the plurality of candidate images and the input medical image; anddetermining whether the image embeddings form a cluster in an embedding space of the trained medical foundation model, wherein cluster formation is indicative of the input medical image depicting the unknown disease.

155. The system of claim 153, wherein the trained medical foundation model is trained according to the operations of claim 56.

156. The method of claim 111, wherein the medical domain specificity score is calculated using a weighted combination of cosine similarity ratios according to MDSS = a (S_direct / S_img_med) + b (S_direct / S_img_nonmed) + c (S_direct / S_txt_med) + d (S_direct / S_txt_nonmed), where S_direct represents the direct cosine similarity between the image embedding and the text embedding, S_img_med represents the medical image-text similarity, S_img_nonmed represents the non-medical image-text similarity, S_txt_med represents the medical text-image similarity, S_txt_nonmed represents the non-medical text-image similarity, and a, b, c, and d represent coefficients determined by a logistic regression model trained on a validation cohort of labeled medical and non-medical image / text pairs.

157. The system of claim 137, wherein the medical domain specificity score is calculated using a weighted combination of cosine similarity ratios according to MDSS = a (S_direct / S_img_med) + b (S_direct / S_img_nonmed) + c (S_direct / S_txt_med) +d (S_direct / S_txt_nonmed), where S_direct represents the direct cosine similarity between the image embedding and the text embedding, S_img_med represents the medical image-text similarity, S_img_nonmed represents the non-medical image-text similarity, S_txt_med represents the medical text-image similarity, S_txt_nonmed represents the non-medical text-image similarity, and a, b, c, and d represent coefficients determined by a logistic regression model trained on a validation cohort of labeled medical and non-medical image / text pairs.

Images