Multi-task machine learning models for digital pathology

Abstract

Techniques for using multi-task machine learning models for digital pathology are described herein. In an example, a system accesses an image of a biological sample of a patient having a medical condition. The system can process one or more embeddings using two or more models to obtain a primary continuous variable associated with an outcome for the patient and one or more auxiliary properties of the biological sample.

Description

PATENT Attorney Docket No. 110588-0894W01-1541062Client Ref. No. CMI 894.604 Multi-Task Machine Learning Models For Digital PathologyCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation claiming priority to and the benefit of U.S.Application No. 63 / 751,224, filed on January 29, 2025, and titled “Multi-Task Machine Learning Models For Digital Pathology,” U.S. Application No. 63 / 799,220, filed on May 2, 2025, and titled “Multi-Task Machine Learning Models For Digital Pathology,” and U.S. Application No. 63 / 899,929, filed on October 15, 2025, and titled “Multi-Task Machine Learning Models For Digital Pathology,” the content of which are herein incorporated in their entirety for all purposes.BACKGROUND

[0002] Histopathology refers to the examination of a biopsy or surgical specimen by a pathologist after the specimen has been processed and histological sections have been placed onto glass slides. Conventionally, pathologists review the slides and make diagnostic determinations, such as identifying or characterizing disease, based on visual inspection. This manual process can be time-consuming and susceptible to observer variability. Machine learning models can serve as assistive tools to support pathologists and researchers by enabling tasks such as image segmentation and virtual staining, as well as predicting clinical end points, including cancer subtypes, gene mutations, and patient survival from whole-slide images (WSIs). For example, a machine-learning model can be trained to predict a biomarker status that can be used as a proxy of a patient outcome (e.g., survival time, disease recurrence, etc.). However, directly predicting patient outcomes from whole-slide images — without relying on biomarker proxies may yield more accurate predictions in various settings. This is because image-based models can capture both visible and hidden features in the tissue that may be linked to biological characteristics that are not fully represented by individual biomarkers.SUMMARY

[0003] Embodiments provided herein involve accessing an image of a biological sample of a patient having a medical condition. One or more embeddings representing the image can be processed using a machine learning framework that includes two or more models to obtain a primary continuous variable associated with an outcome for the patient and one or more180147550V 1auxiliary properties of the biological sample. The one or more auxiliary properties may be auxiliary classifications of the biological sample or auxiliary continuous variable predictions of the biological sample.

[0004] In some embodiments, a computer-implemented method is provided. The method may comprise accessing an image of a biological sample of a patient having a medical condition. The biological sample may be stained using a pathology stain. The method may further comprise generating one or more feature vectors from pixels comprising at least a portion of the image. The method may include providing the one or more feature vectors to an input layer of a neural network of a machine learning framework. The method may also comprise processing, by the neural network, the one or more feature vectors to generate a shared embedding or a set of task specific embeddings representing the image. The set of task specific embeddings may include a primary regression embedding and one or more auxiliary embeddings. The one or more auxiliary embeddings may include at least one embedding selected from a first group consisting of (1) one or more auxiliary classification embeddings for obtaining one or more classifications of the biological sample and (2) one or more auxiliary regression embeddings for obtaining one or more auxiliary continuous variables of the biological sample. The method may also comprise processing the shared embedding or the primary regression embedding by a primary machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient. The method can further comprise processing the shared embedding or the one or more auxiliary embeddings by an auxiliary machine learning model of the machine learning framework to obtain one or more auxiliary properties of the biological sample. The one or more auxiliary properties may be selected from a second group consisting of (a) the one or more classifications of the biological sample and (b) the one or more auxiliary continuous variables of the biological sample. The machine learning framework can be trained using a regression loss function for the primary continuous variable, and one or more auxiliary loss functions for the one or more auxiliary properties.

[0005] In some embodiments, a computer-implemented method is also provided. The method may comprise accessing an image of a biological sample of a patient having a medical condition, wherein the biological sample may be stained using a pathology stain. The method may comprise generating one or more feature vectors from pixels comprising at least a portion of the image. The method may comprise providing the one or more feature vectors280147550V 1to an input layer of a neural network of a machine learning framework. The method may comprise processing, by the neural network, the one or more feature vectors to generate an embedding. The method may comprise processing the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient. The outcome may comprise a recurrence of the medical condition. The outcome may comprise a determination for an early distant recurrence. The machine learning regression model may include an early distant recurrence model. The early distant recurrence model may include a first EDR model and a second EDR model. Processing the embedding may include generating, using the one or more feature vectors, a first risk score using the first EDR model. The method may comprise comparing the first risk score to a threshold. The method may comprise determining, by the first EDR model, the recurrence responsive to the first risk score from the first EDR model being below a threshold. The method may comprise determining, by the second EDR model, the recurrence responsive to the first risk score from the first EDR model being greater than the threshold.

[0006] Each of the following features can be separately incorporated as part of the methods or can be incorporated together with one or more other following features. The outcome may comprise a determination for the early distant recurrence and a late distant recurrence, and the machine learning regression model may include the early distant recurrence model and a late distant recurrence model. The late distant recurrence model may be invoked when the early distant recurrence indicates low risk. The second EDR model may process sequencing data measured from the biological sample or another biological sample of the patient.

[0007] In some embodiments, a computer-implemented method is provided that may comprise accessing an image of a biological sample of a patient having a medical condition, wherein the biological sample may be stained using a pathology stain. The method may comprise generating one or more feature vectors from pixels comprising at least a portion of the image. The method may comprise providing the one or more feature vectors to an input layer of a neural network of a machine learning framework. The method may comprise processing, by the neural network, the one or more feature vectors to generate an embedding. The method may comprise processing the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient. The outcome may comprise a determination for late distant380147550V 1recurrence of the medical condition. The machine learning regression model may include a late distant recurrence model used to determine the recurrence based on a risk score from an early distant recurrence model being below a threshold.

[0008] Each of the following features can be separately incorporated as part of the methods or can be incorporated together with one or more other following features. The early distant recurrence model may use sequencing data from the biological sample or other biological sample of the patient. The method may further comprise obtaining the sequencing data and processing the sequencing data by the early distant recurrence model to determine the risk score. The early distant recurrence model may use image data of the biological sample or another biological data of the patient. The image data may include the one or more feature vectors. The method may further comprise processing the embedding by an auxiliary machine learning model of the machine learning framework to obtain one or more auxiliary properties of the biological sample, the one or more auxiliary properties selected from a second group consisting of (a) the one or more classifications of the biological sample and (b) the one or more auxiliary continuous variables of the biological sample.

[0009] In some embodiments, a computer-implemented method is also provided that may comprise accessing sequencing data measured from a biological sample of a patient having a medical condition. The method may comprise generating one or more feature vectors from the sequencing data. The method may comprise providing the one or more feature vectors to an input layer of a neural network of a machine learning framework. The method may comprise processing, by the neural network, the one or more feature vectors to generate an embedding. The method may comprise processing the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient. The outcome may comprise a recurrence of the medical condition. The outcome may comprise a determination for an early distant recurrence. The machine learning regression model may include an early distant recurrence model. Processing the embedding may include generating, by the early distant recurrence model using the one or more feature vectors, a first risk score. The method may comprise comparing the first risk score to a threshold. The method may comprise determining a first treatment is to be administered responsive to the first risk score being above the threshold. The method may further comprise, responsive to the first risk score being below the480147550V 1threshold, generating, by a late distant recurrence model, another primary continuous variable associated with a late distant recurrence for the patient.

[0010] Each of the following features can be separately incorporated as part of the methods or can be incorporated together with one or more other following features. The first treatment may include chemotherapy. The late distant recurrence model may be selected from any of the models recited in any one of the above methods.

[0011] In some embodiments, a system is provided that may comprise a sequencing device configured to generate sequencing data by sequencing nucleic acids from a biological sample of a patient having a medical condition; first early recurrence logic configured to generate one or more feature vectors from the sequencing data; provide the one or more feature vectors to an input layer of a neural network of a machine learning framework; process, by the neural network, the one or more feature vectors to generate an embedding; and process the embedding by an early distant recurrence model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient, wherein the outcome may comprise an early distant recurrence of the medical condition, and wherein processing the embedding may include generating, by the early distant recurrence model using the one or more feature vectors, a first risk score; comparing the first risk score to a threshold; and determining a first treatment is to be administered responsive to the first risk score being above the threshold; and an imaging device configured to generate an image of a biological sample of the patient, wherein the biological sample may be stained using a pathology stain; late recurrence logic configured to, responsive to the first risk score being below the threshold, generate one or more other feature vectors from pixels comprising at least a portion of the image; provide the one or more other feature vectors to an input layer of another neural network of the machine learning framework; process, by the other neural network, the one or more other feature vectors to generate another embedding; and process the other embedding by a late distant recurrence model of the machine learning framework to obtain another primary continuous variable associated with another outcome for the patient, wherein the other outcome may comprise a late distant recurrence of the medical condition.

[0012] Each of the following features can be separately incorporated as part of any embodiment or can be incorporated together with one or more other following features. The primary continuous variable may be time-dependent. The one or more auxiliary properties may include the one or more classifications of the biological sample, and the one or more580147550V 1classifications may comprise patient characteristics. At least one of the patient characteristics may be selected from a group consisting of a patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone mineral density T-score, a prior treatment, or treatment type. The one or more feature vectors may further include one or more patient characteristics. At least one of the one or more patient characteristics may be selected from a group consisting of a patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone mineral density T-score, a prior treatment, or treatment type. The outcome may comprise a survival of the patient, a time on treatment for the patient, or a recurrence of the medical condition. The recurrence may be an early distant recurrence (EDR) or a late distant recurrence (LDR), or the outcome may comprise a determination for the early distant recurrence and the late distant recurrence. The primary machine learning regression model may include an early distant recurrence model and a late distant recurrence model. The early distant recurrence model may include a first EDR model and a second EDR model, and the first EDR model may be used to determine the recurrence if a first risk score from the first EDR model is below a threshold and the second EDR model may be used to determine the recurrence if the first risk score from the first EDR model is greater than the threshold. The first EDR model may process the one or more feature vectors from the pixels, and the second EDR model may process sequencing data measured from the biological sample or another biological sample of the patient. The sequencing data may be obtained by next-generation sequencing (NGS), and the NGS may comprise whole genome sequencing (WGS), whole exome sequencing (WES), whole transcriptome sequencing (WTS), or a combination thereof. The medical condition may comprise cancer. The cancer may comprise an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumor (GIST), glioblastoma, head and neck squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), non-small cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non-epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal680147550V 1carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma. The one or more feature vectors may comprise a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of a surrounding area of the target portion, wherein the surrounding area may be within a pixel distance of the target portion. The one or more feature vectors may comprise a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of the image, wherein the additional portions may comprise an entirety of the image. The one or more feature vectors may comprise a first feature vector from first pixels comprising a first portion of the image, and the method may further comprise generating a second feature vector from second pixels comprising a second portion of the image, and processing, by the neural network, the first feature vector and the second feature vector to generate the primary regression embedding, wherein the primary regression embedding may represent an aggregation of the first feature vector and the second feature vector. The neural network may comprise a multi-instance learning model. The regression loss function for the primary continuous variable may comprise a time-dependent loss function. The time-dependent loss function may comprise a Cox partial likelihood function. The one or more auxiliary properties may include the one or more classifications of the biological sample. The one or more classifications may include a status of at least one biomarker, and the biomarker may be associated with one or more treatment. The status may comprise an expression level or a mutation. The one or more classifications may include at least one selected from a bone mineral density T-score and prior tamoxifen treatment. The method may further comprise performing one or more auxiliary regression tasks by processing the shared embedding or the one or more auxiliary regression embeddings by the auxiliary machine learning model to obtain the one or more auxiliary continuous variables of the biological sample, wherein the machine learning framework may be further trained using the regression loss function for the one or more auxiliary continuous variables. The one or more auxiliary regression tasks may comprise predicting a percentage of tumor cells in the biological sample. The biological sample may be from a tumor. The tumor may be a primary tumor or a metastatic tumor. The tumor may be a tumor of the myeloid, breast, bile ducts, colon, rectum, female genital tract, stomach, esophagus, gastrointestinal stromal cells, small intestine, brain, mouth, sinuses, nose, throat, blood, liver, nervous system, lung, lymph, male genital tract, pleura, skin, plasma cells, neuroendocrine cells, B-cells, T-cells, ovary,780147550V 1pancreas, pituitary gland, spinal cord, prostate, peritoneum, large intestine, soft tissue, connective tissue, fat tissue, thymus, thyroid, or eye. The primary tumor may be a tumor of the bladder, breast, colon, rectum, endometrium, uterus, ovary, female genital tract, kidney, blood, liver, lung, skin, lymph, pancreas, prostate, or thyroid. The biological sample may be a formalin-fixed paraffin-embedded (FFPE) tissue. The pathology stain may comprise hematoxylin and eosin (H&E). The method may further comprise providing a diagnosis, prognosis, and / or theranosis based on the primary continuous variable. The primary continuous variable may comprise a risk score. The method may further comprise outputting a report that includes the primary continuous variable and / or a patient category derived from the primary continuous variable. The report may include the patient category, and the method may further comprise determining the patient category of whether the patient is in a low risk group or a high risk group based on the primary continuous variable. The patient category may be determined by comparing the primary continuous variable to a threshold. The threshold may be determined based on a distribution of the primary continuous variable for a set of patients having the medical condition. The report may recommend providing a particular treatment responsive to the patient being within the high risk group. The method may further comprise providing the particular treatment to the patient. The report may recommend not providing either a particular treatment or any treatment responsive to the patient being within the low risk group. The particular treatment may include extended endocrine therapy (EET), and the medical condition may be a hormone-positive breast cancer, and the EET may be extended letrozole treatment (ELT). The one or more auxiliary properties may include the one or more classifications of the biological sample, and the one or more classifications may include at least one selected from a bone mineral density T-score and a prior tamoxifen use. The particular treatment may include an immune checkpoint inhibitor, and the medical condition may be lung cancer, and the immune checkpoint inhibitor may be selected from the group consisting of pembrolizumab, nivolumab, cemiplimab, dostarlimab, retifanlimab, toripalimab, atezolizumab, durvalumab, avelumab, ipilimumab, tremelimumab, relatlimab, and any useful combination thereof. The one or more auxiliary properties may include the one or more classifications of the biological sample, and the one or more classifications may include one or more biomarker statuses, and the biomarker statuses may comprise a presence, absence or level of one or more biomarker. The one or more biomarker statuses may include an expression status of at least one of PD-L1 (Programmed Death Ligand 1), PD-1 (Programmed Death- 1), CTLA-4 (Cytotoxic T-880147550V 1Lymphocyte Associated Protein 4), LAG-3 (Lymphocyte Activation Gene-3), TIM-3 (T-cell Immunoglobulin and Mucin Domain-3), TIGIT (T-cell Immunoreceptor with Ig and ITIM domains), Tumor Mutational Burden (TMB), Microsatellite Instability -High (MSLH) / Mismatch Repair Deficiency (dMMR), Tumor-Infiltrating Lymphocytes (TILs), VISTA (V-domain Ig Suppressor of T-cell Activation), B7-H3 (CD276), B7-H4, IDO1 (Indoleamine 2,3 -dioxygenase), BTLA (B and T Lymphocyte Attenuator), and a combination thereof. The immune checkpoint inhibitor may comprise pembrolizumab, the one or more biomarker statuses may include expression status of PD-L1, and the medical condition may comprise a lung cancer. The medical condition may comprise a hormone-positive breast cancer; the biological sample may comprise a tumor section from the cancer; the one or more feature vectors may further include one or more patient characteristics comprising age, node status, and surgery type; the primary continuous variable may comprise time to distant recurrence; the outcome may comprise distant recurrence of the medical condition; and the one or more auxiliary properties may include bone mineral density T-score. The distant recurrence of the medical condition may comprise late distant recurrence. The one or more auxiliary properties may further include prior treatment with tamoxifen.

[0013] These and other embodiments of the disclosure are described in detail below. For example, other embodiments are directed to systems, devices, and computer readable media associated with methods described herein.

[0014] A better understanding of the nature and advantages of embodiments of the present disclosure may be gained with reference to the following detailed description and the accompanying drawings.BRIEF DESCRIPTION OF DRAWINGS

[0015] FIG. 1 illustrates an overview of risk score prediction for a prognosis, according to embodiments of the present disclosure.

[0016] FIG. 2 illustrates an example of using a machine learning framework for a main regression task and auxiliary classification or regression tasks, according to embodiments of the present disclosure.

[0017] FIGS. 3A-3B illustrate an example of using a multi-task model, according to embodiments of the present disclosure.980147550V 1

[0018] FIG. 4 illustrates various aggregation techniques for feature vectors, according to embodiments of the present disclosure.

[0019] FIGS. 5A-5C illustrate exemplary results for predicting survival using a biomarker status (FIG. 5A), a single-task model (FIG. 5B), and a multi-task model (FIG. 5C).

[0020] FIGS. 6A-6D illustrate exemplary results of model performance for predicting survival and auxiliary classifications using a multi-task model.

[0021] FIG. 7A and FIG. 7B illustrate exemplary architectures for predicting disease recurrence in hormone receptor-positive breast cancer patients using single task and multitask models, respectively. Each figure illustrates models built using only image features or image features combined with clinical features (age, pathological node status, and surgery type (lumpectomy or mastectomy)). In FIG. 7B, two auxiliary prediction tasks are indicated: prediction of bone mineral density T-score (T-score) and prior tamoxifen use (PriorTam).

[0022] FIGS. 8A-8F illustrate cumulative incidence of distant recurrence (DR) in hormone receptor-positive breast cancer patients assigned to high or low risk of DR cohorts using single and multi-task machine learning models.

[0023] FIGS. 9A-9F illustrate hazard ratios between extended letrozole therapy (ELT) and placebo in clinical subgroups (FIG. 9A), incidence of distant recurrence in hormone receptorpositive breast cancer patients given ELT or placebo (FIG. 9B), and risk groups predicted by an image only model and a multimodal-multitask model (FIGS. 9B-9F).

[0024] FIGS. 10A-10C are Kaplan-Meier plots for all subjects (FIG. 10A), those with early DR (FIG. 10B), and those with late DR (FIG. 10C). FIGS. 10D-10F illustrate calibration plots showing observed versus predicted 10-year distant recurrence risk.

[0025] FIG. 11 illustrates various predictor models using images, clinical features, and molecular features.

[0026] FIGS. 12A-B show an example prediction pipeline including EDRP and LDRP, as well as illustrating an advanced EDRP model. FIG. 12A shows options for digital pathology and sequencing for EDRP. FIG. 12B shows using sequencing for EDRP.

[0027] FIG. 13 depicts different distributions of subjects from the training set for different redirect threshold for redirecting (reflexing) subjects from EDRP to NGSPred.1080147550V 1

[0028] FIGS. 14A-14C show comparisons among the different models.

[0029] FIG. 15 illustrates an example flow of a process for using a multi-task machine learning model in digital pathology, according to embodiments of the present disclosure.

[0030] FIG. 16 illustrates an example flow of a process for using a machine learning model for early distant recurrence in digital pathology, according to embodiments of the present disclosure.

[0031] FIG. 17 illustrates an example flow of a process for using a machine learning model for late distant recurrence in digital pathology, according to embodiments of the present disclosure.

[0032] FIG. 18 illustrates an example flow of a process for using a machine learning model for early distant recurrence using sequencing data in digital pathology, according to embodiments of the present disclosure.

[0033] FIG. 19 illustrates a measurement system according to an embodiment of the present disclosure.

[0034] FIG. 20 shows a block diagram of an example computer system usable with systems and methods according to embodiments of the present disclosure.TERMS

[0035] The term “phenotype” as used herein can mean any trait or characteristic that can be identified in part or in whole by using the systems and / or methods provided herein. In some embodiments, the systems can include one or more computer programs on one or more computers in one or more locations, e.g., configured for use in a method described herein. Phenotypes may be determined by analyzing a biological sample obtained from a subject. Phenotypes to be characterized can be any phenotype of interest, including without limitation a tissue, anatomical origin, medical condition, ailment, disease, disorder, or useful combinations thereof. A phenotype can be any observable characteristic or trait of, such as a disease or disorder, a stage of a disease or disorder, susceptibility to a disease or disorder, prognosis of a disease stage or disorder, a physiological state, or response / potential response (or lack thereof) to interventions such as therapeutics. A phenotype can result from a subject’s genetic makeup as well as the influence of environmental factors and the interactions between the two, as well as from epigenetic modifications to nucleic acid sequences. In1180147550V 1various embodiments, a phenotype in a subject is characterized by obtaining a biological sample from a subject and analyzing the sample using the systems and / or methods provided herein. For example, characterizing a phenotype for a subject or individual can include detecting a disease or disorder (including pre-symptomatic early-stage detection), determining a prognosis, diagnosis, or theranosis of a disease or disorder, or determining the stage or progression of a disease or disorder. Characterizing a phenotype can include identifying appropriate treatments or treatment efficacy for specific diseases, conditions, disease stages and condition stages, predictions and likelihood analysis of disease progression, particularly disease recurrence, metastatic spread or disease relapse. A phenotype can also be a clinically distinct type or subtype of a condition or disease, such as a cancer or tumor. Phenotype determination can also be a determination of a physiological condition, or an assessment of organ distress or organ rejection, such as post-transplantation. The compositions and methods described herein allow assessment of a subject on an individual basis, which can provide benefits of more efficient and economical decisions in treatment.

[0036] A medical condition as used herein can refer to a disease or disorder of a subject, as well as a stage or severity of the medical condition.

[0037] A subject (individual, patient, or the like) can be any animal which may benefit from the methods described herein. A subject or individual can be any animal which may benefit from the methods described herein, including, e.g., humans and non-human mammals, such as primates, rodents, horses, dogs and cats. Subjects include without limitation a eukaryotic organisms, most preferably a mammal such as a primate, e.g., chimpanzee or human, cow; dog; cat; a rodent, e.g., guinea pig, rat, mouse; rabbit; or a bird; reptile; or fish. Subjects specifically intended for treatment using the methods described herein include humans. A subject may also be referred to herein as an individual or a patient. The subject can have a pre-existing disease or disorder, including without limitation cancer. Alternatively, the subject may not have any known pre-existing condition. The subject may also be non-responsive to an existing or past treatment, such as a treatment for cancer.

[0038] Theranostics as used herein includes therapy-related diagnostic testing that provides the ability to affect therapy or treatment of a medical condition such as a disease or disease state. Theranostics testing provides a theranosis in a similar manner that diagnostics or prognostic testing provides a diagnosis or prognosis, respectively. As used herein,1280147550V 1theranostics encompasses any desired form of therapy related testing, including predictive medicine, personalized medicine, precision medicine, integrated medicine, pharmacodiagnostics and Dx / Rx partnering. Therapy related tests can be used to predict and assess drug response in individual subjects, thereby providing personalized medical recommendations. Predicting a likelihood of response can be determining whether a subject is a likely responder or a likely non-responder to a candidate therapeutic agent, e.g., before the subject has been exposed or otherwise treated with the treatment. Assessing a therapeutic response can be monitoring a response to a treatment, e.g., monitoring the subject’s improvement or lack thereof over a time course after initiating the treatment. Therapy related tests are useful to select a subject for treatment who is particularly likely to benefit or lack benefit from the treatment or to provide an early and objective indication of treatment efficacy in an individual subject. Characterization using the systems and methods provided herein may indicate that treatment should be altered to select a more promising treatment, thereby avoiding the expense of delaying beneficial treatment and avoiding the financial and morbidity costs of less efficacious or ineffective treatment(s).

[0039] Theranosis can comprise predicting a treatment efficacy or lack thereof, classifying a patient as a responder or non-responder to treatment. A predicted “responder” can refer to a patient likely to receive a benefit from a treatment whereas a predicted “non-responder” can be a patient unlikely to receive a benefit from the treatment. Unless specified otherwise, a benefit can be any clinical benefit of interest, including without limitation cure in whole or in part, remission, or any improvement, reduction or decline in progression of the condition or symptoms. The theranosis can be directed to any appropriate treatment, e.g., the treatment may comprise at least one of chemotherapy, immunotherapy, targeted cancer therapy, a monoclonal antibody, small molecule, surgery, radiation, or any useful combinations thereof.

[0040] A classification of a medical condition can include a diagnosis, prognosis, or theranosis of the subject. The term ''classification" as used herein refers to any number(s) or other characters(s) that are associated with a particular property of a sample, e.g., a medicinal condition of a subject from whom the sample was obtained. For example, a “+” symbol (or the word “positive”) could signify that a sample is classified as having deletions or amplifications. The classification can be binary (e.g., positive or negative) or have more levels of classification (e.g., a scale from 1 to 10 or 0 to 1), including probabilities. A classification of a biological sample can include a property of the patient from which the1380147550V 1biological sample was obtained, e.g., whether the patient had been treated with a particular treatment.

[0041] A “ machine learning moder (ML model) can refer to a software module configured to be run on one or more processors to provide a classification or numerical value of a property of one or more samples. An ML model can include various parameters (e.g., for coefficients, weights, thresholds, functional properties of function, such as activation functions). As examples, an ML model can include at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, one million, ten million, 100 million, or one billion parameters. An ML model can be generated using sample data (e.g., training samples) to make predictions on test data. Various number of training samples can be used, e.g., at least 10, 100, 1,000, 5,000, 10,000, 50,000, 100,000, or 200,000 training samples. One example is reinforcement learning such as Q-Learning, Deep Q-Networks (DQN), Double DQN, Dueling DQN, Policy Gradient Methods, Actor-Critic, Advantage Actor-Critic (A2C), Proximal Policy Optimization (PPO), Trust Region Policy Optimization (TRPO), and Soft Actor-Critic (SAC). Another example is an unsupervised learning model such as hidden Markov model (HMM), clustering (e.g., hierarchical clustering, k-means, mixture models, model-based clustering, density-based spatial clustering of applications with noise (DBSCAN), and OPTICS algorithm), approaches for learning latent variable models such as Expectation-maximization algorithm (EM), method of moments, and blind signal separation techniques (e.g., principal component analysis, independent component analysis, non-negative matrix factorization, singular value decomposition), and anomaly detection (e.g., local outlier factor and isolation forest).Another example type of model is supervised learning that can be used with embodiments of the present disclosure. Example supervised learning models may include different approaches and algorithms including analytical learning, statistical models, artificial neural network (e.g. including convolutional and / or transformer layers) that may have 1-10 layers as examples, recurrent neural network (e.g., long short term memory, LSTM), boosting (meta-algorithm), bootstrap aggregating (bagging) such as random forests, support vector machine (SVM), multi-class SVM, support vector regression (SVR), Bayesian statistics, case-based reasoning, decision tree learning (e.g., CART (classification and regression trees), gradient boosted trees, or random forest), inductive logic programming, linear regression, logistic regression, Gaussian process regression, genetic programming, group method of data handling, kernel estimators, learning automata, learning classifier systems, minimum message length (decision trees, decision graphs, etc.), multilinear subspace learning, naive Bayes classifier, maximum1480147550V 1entropy classifier, conditional random field, nearest neighbor algorithm, probably approximately correct (PAC) learning, ripple down rules, a knowledge acquisition methodology, symbolic machine learning algorithms, subsymbolic machine learning algorithms, minimum complexity machines (MCM), ordinal classification, data preprocessing, handling imbalanced datasets, statistical relational learning, or Proaftn (a multicriteria classification algorithm), or an ensemble of any of these types. Supervised learning models can be trained in various ways using various cost / loss functions that define the error from the known label (e.g., least squares and absolute difference from known classification) and various optimization techniques, e.g., using backpropagation, steepest descent, conjugate gradient, and Newton and quasi-Newton techniques. Some workflows may also include steps for data pre-processing or handling imbalanced datasets.DETAILED DESCRIPTION

[0042] With advances in computational algorithms and tools, significant investment has been made in the development of artificial intelligence (Al) tools to aid in the clinical assessment of biological specimens, including without limitation the prediction of clinical outcomes. For clinical outcome prediction, a machine learning framework having a multitask architecture can be used to perform a main (primary) task and one or more auxiliary tasks. The main task can involve predicting a risk score for a patient outcome from a wholeslide image. The risk score may be associated with survival, time-on-treatment, cancer recurrence, or any other suitable time-dependent prediction. The auxiliary tasks can determine auxiliary outputs, such as age, histology, gender, biomarker status, tumor percentage, etc. As such, the auxiliary tasks can include classification tasks and / or regression tasks, where regression tasks can be time-dependent predictions. Incorporating auxiliary tasks can improve the performance of the main task, particularly in weakly-supervised settings.

[0043] To generate the predictions and classifications, an image of a biological sample of a patient having a medical condition (e.g., disease, disorder, cancer, etc.) can be received. The biological sample can be imaged with a pathology stain to provide visual contrast to the sample and / or highlight certain features, thereby facilitating histologic image analysis. In some implementations, the biological sample can be imaged, e.g., using radiological instruments. The biological sample may be a tissue sample, including without limitation a tumor sample. In some implementations, the tissue sample is fixed to preserve structure. Fixation can be performed using chemical fixatives such as crosslinking agents.1580147550V 1Formaldehyde is the most common fixative used in histology applications. The tissue can also be embedded into a medium to allow sectioning. Examples of such mediums include epoxy, acrylic, agar, gelatin, celloidin, and waxes. Paraffin wax is the most common embedding material in light microscopy. In preferred embodiments, the biological sample is a formalin-fixed paraffin-embedded (FFPE) tissue. The FFPE tissue can be sectioned into slides that can be stained prior to image analysis. A classification of a biological sample can include a property of the patient from which the biological sample was obtained, e.g., whether the patient had been treated with a particular treatment.

[0044] The pathology stain can be any desired stain, dye or other imaging agent, or any useful combination thereof. In some implementations, the stain comprises a nuclear stain, cytoplasmic stain, and / or stain for the extracellular matrix. The most commonly used stain for histopathology is a combination of hematoxylin and eosin (“H&E”). Generally, hematoxylin stains the cell nuclei whereas eosin stains the cytoplasm and extracellular matrix. Such staining shows the general layout and distribution of cells and general overview of the tissue sample’s structure, which can be used to provide histological information. For review of tissue imaging including alternative imaging agents, see, e.g., Alturkistani, H et al, Histological Stains: A Literature Review and Case Study, Glob J Health Sci. 2015 Jun 25;8(3):72-79; Javaeed, A et al., Histological Stains in the Past, Present, and Future, Cureus.2021 Oct 4;13(10):el8486; Veuthey, T, et al, Dyes and stains: from molecular structure to histological application, Front Biosci (Landmark Ed) 2014 Jan 1; 19(1):91-112. See also Biological Stain Commission, available at biologicalstaincommission.org.

[0045] The image can be segmented into patches. The dimensions of the patches can vary depending on the architecture of the machine learning system. The system can consider all patches or patches from certain regions of interest, such as from tumor tissue.

[0046] A feature vector can be generated from pixels of the image, and the feature vector can be provided to an input layer of a neural network of the machine learning framework. Multiple feature vectors can be used. For example, different feature vectors can correspond to the different patches extracted or segmented from the image. The neural network can generate one or more classification embedding and / or one or more regression embedding that correspond to the image.

[0047] For the main (primary) task, the regression embedding can be processed by a machine learning regression model of the machine learning framework to obtain a risk score 1680147550V 1associated with an outcome of the patient. The machine learning regression model can be trained using a time-dependent loss function. For auxiliary tasks, the classification embedding can be input into a classification model of the machine learning framework to predict categorical variables and the regression embedding is input into the machine learning regression model to predict continuous variables. By combining risk score prediction and auxiliary output determination within a unified architecture, the risk score prediction may be more accurate than single-task (e.g., only risk-score prediction) models, as it leverages additional information.

[0048] The inclusion of one or more auxiliary regression tasks within a multi-task framework for time-dependent risk prediction would be considered counterintuitive to a typical practitioner in this field. In conventional modeling practice, a time-dependent regression target is often treated as requiring training that focuses exclusively on the timedependent outcome, because introducing additional auxiliary objectives that are not directly tied to the time-dependent endpoint would be expected to dilute the training signal and reduce performance on the primary task. This is particularly true when the auxiliary task is not timedependent (e.g., predicting a static variable such as Bone Mineral Density T-score, mutations, histology, biomarker expression status, etc.), while the primary task requires predicting a time-dependent risk function. If presented with such a technique, an expectation would be that optimizing for a static auxiliary regression target could encourage the model to learn time-invariant patterns and compromise its ability to learn the temporal structure needed for accurate time-dependent risk prediction.

[0049] However, empirical results show that training within the described multi-task framework using one or more auxiliary tasks can improve performance of the time-dependent risk prediction model. See, e.g., Section IV herein. This benefit can persist at the inference stage even when the auxiliary -task variables are not available, as long as the model was trained with the auxiliary objectives. This indicates that the auxiliary task(s) act as an effective training signal to shape shared representations that generalize better for timedependent risk prediction, even when auxiliary information is absent at deployment.Accordingly, an advantage of employing such auxiliary tasks is that an auxiliary task may assist training of the primary task but is not required during inference. Thus, auxiliary tasks expand the realm of data available for model training without adding complexity to inference. The auxiliary tasks can be related to the primary task in order to provide benefit to the1780147550V 1predictions. As a non-limiting example, if the primary task is predicting survival benefit of a particular therapeutic, it may be beneficial to employ an auxiliary task comprising status of a biomarker relevant to such therapeutic. In the example, the biomarker status may be determined using a laboratory method that will not be required at inference. In some cases, certain auxiliary tasks may provide little to no benefit.

[0050] Thus, provided herein include techniques that can employ a multi-task machine learning framework to analyze histopathological images of biological samples. Such techniques can predict patient outcomes through a combination of primary and auxiliary tasks. The method begins by generating feature vectors from the image pixels, which are processed by one or more neural networks to produce whole slide image embeddings.Depending on the framework’s architecture, these embeddings can either be shared across tasks or specifically tailored for individual tasks, such as classification and regression. The primary task can focus on predicting a time-dependent risk score, such as survival or recurrence risk. This can be achieved by processing a regression embedding with a regression model trained using a time-dependent loss function, such as the Cox partial likelihood.Simultaneously, auxiliary tasks can enrich the framework by addressing additional predictive objectives. For auxiliary classification tasks, classification embeddings can be processed by a classification model to predict categorical outcomes, such as biomarker status, histology, or disease subtypes. At the same time, auxiliary regression tasks can use regression embeddings to predict continuous variables, such as tumor percentage or other clinically significant measures. These auxiliary tasks complement the primary task, enhancing the framework’s ability to extract meaningful information from the biological sample. By enabling simultaneous execution of auxiliary classification and regression tasks, the framework maximizes the utility of the data while improving the accuracy and robustness of the primary risk score prediction. The flexibility of using either shared or task-specific embeddings allows the model to deliver comprehensive and clinically actionable predictions, advancing its diagnostic, prognostic and theranostic capabilities.

[0051] In contrast to analyses based on genomic assays that require next-generation sequencing, the framework can use an image-based model that offers advantages in cost, speed, and accessibility since risk predictions and treatment recommendations can be generated from hematoxylin and eosin (H&E) stained slides within hours. Accordingly, more rapid and scalable clinical decision-making is possible. Additionally, a smaller sample can be1880147550V 1used, e.g., a single slice of a sample can be used, as opposed to sequencing, which generally uses a larger sample, e.g., multiple slices.I. SEGMENTING AND FEATURE EXTRACTION

[0052] For a whole-slide image, patches can be generated that correspond to portions of the whole-slide image. As such, each patch can include a portion of the pixels of the whole-slide image. The patches can then be processed to extract features and make predictions based on the features.

[0053] FIG. 1 illustrates an overview of risk score prediction for a prognosis, according to embodiments of the present disclosure. A whole-slide image of a sample from a patient can be segmented into patches 102. Each patch comprises a collection of pixels corresponding to a portion of a whole-slide image. In some embodiments, a patch is a region of a whole-slide image or an area of interest having (x,y) pixel dimensions (e.g., 256 pixels by 256 pixels). For example, a whole-slide image of 1000 pixels by 1000 pixels divided into 100 pixel by 100 pixel patches would be segmented into 100 patches (each patch containing 10,000 pixels). In other embodiments, the patches 102 may overlap with each patch having (x,y) pixel dimensions and sharing one or more pixels with another patch.

[0054] Once segmented into the patches 102, a feature extractor 104 can receive the patches 102 and generate feature vectors for each of the patches 102. The feature extractor 104 may be a neural network trained to identify characteristics in the patches 102 and to generate the feature vectors indicating the characteristics. Rather than using a predicted mutation as a proxy for a patient outcome, end-to-end prognostication 106 can be performed to process the feature vectors to generate a prognosis 108. The end-to-end prognostication 106 can involve a machine learning framework that includes an aggregator for aggregating the feature vectors into one or more embeddings. Such embeddings include shared embeddings or task-specific embeddings (e.g., regression embeddings and classification embeddings).

[0055] A shared embedding is a single embedding that can be used by both a machine learning regression model and a classification model to generate outputs related to predictions and classifications for the patient. In contrast, a regression embedding is an embedding that is intended to be used by the machine learning regression model and a classification embedding is an embedding that is intended to be used by the classification model. A machine learning1980147550V 1regression model can receive a shared embedding or regression embedding and generate the prognosis 108 based on the regression embedding. As an example, the prognosis 108 can be a risk score associated with a patient outcome for the patient. For example, the prognosis 108 can be a likelihood of survival over time, where a low risk score can be associated with a higher likelihood of survival over time and a high risk score can be associated with a lower likelihood of survival over time. Although not shown in FIG. 1, the end-to-end prognostication 106 can also involve auxiliary tasks of processing the shared embedding or the classification embedding using the classification model to generate one or more classifications associated with the patient and / or processing the shared embedding or the regression embedding to generate one or more additional predictions associated with the patient.

[0056] The shared embedding and the regression / classification embeddings can differ in purpose and task alignment within the machine learning framework. The shared embedding can be a general-purpose representation of features that may be used for both regression and classification tasks. The shared embedding can balance features relevant to all tasks, providing flexibility and supporting multi-task learning. The regression embedding can be tailored specifically for predicting continuous outcomes (e.g., time-to-event or risk scores). The regression embedding can emphasize features predictive of temporal or quantitative data. The classification embedding can be designed for categorical predictions, focusing on features relevant to classifications (e.g., disease subtypes or binary labels). While all embeddings can be derived from the same feature vectors, a difference can lie in optimization, where the shared embedding supports multiple tasks, while task-specific embeddings can be specialized to maximize performance for their respective objectives (regression or classification).II. MULTI-TASK MODEL ARCHITECTURE

[0057] As mentioned, a machine learning framework can include a neural network for image embedding generation, a machine learning regression model for primary timedependent predictions, a classification model for auxiliary classification tasks, and a regression model for auxiliary regression tasks. The machine learning framework can be a multi-task model meaning that multiple tasks are performed by the model. For example, the machine learning framework can involve performing a main regression task as well as one or2080147550V 1more auxiliary tasks, which may include regression tasks for predicting continuous variables or classification tasks for categorical outcomes.A. Primary task regression

[0058] FIG. 2 illustrates an example of using a machine learning framework 210 for a main regression task 214 and auxiliary classification tasks 212, according to embodiments of the present disclosure. An input 202 to the machine learning framework 210 can be one or more feature vectors generated from pixels of an image of a biological sample of a patient having a medical condition. For instance, the image can be a whole-slide image of the biological sample. The image can be segmented into patches of pixels, where each patch includes a portion of the pixels of the whole-slide image. A feature extractor (e.g., CtransPath, RetCCL, GigaPath, etc.) can receive the patches and generate a feature vector for each patch. The feature vectors are numerical representations of characteristics of the patches. The feature vectors can then make up the input 202 to the machine learning framework 210.

[0059] In some examples, a neural network of the machine learning framework 210 can receive the input 202 of the feature vectors and generate either task-specific embeddings, such as a classification embedding and a regression embedding, or a shared embedding used across all tasks. The choice of embedding type depends on the framework’s architecture. These embeddings represent aggregated information from the whole-slide image, generated using aggregation techniques described below.

[0060] Once the embeddings are generated, the embeddings can be used to perform the main regression task 214 and one or more auxiliary tasks. The main regression task 214 involves predicting a metric for a primary continuous variable. For example, the metric can be a risk score and the primary continuous variable can be a time-dependent outcome for the patient, e.g., a risk that the medical condition recurs within a specified amount of time. In that case, the regression embedding is processed by a machine learning regression model of the machine learning framework 210 to obtain the risk score for the main regression task 214. Alternate metrics for continuous variables that could be used include without limitation survival time. Alternative continuous variables that could be measured include without limitation overall survival, disease free interval, time to next treatment, time on treatment.

[0061] In FIG. 2, the auxiliary tasks correspond to auxiliary classification tasks 212 for attributes including histology, age, gender, and biomarker status. So, the classification2180147550V 1embedding can be processed by a classification model of the machine learning framework 210 to obtain the classifications. Additionally or alternatively to classification tasks, the auxiliary tasks may include regression tasks for predicting continuous attributes such as tumor percentage. So, for the regression tasks, the regression embedding can be processed by the machine learning regression model to obtain the attribute predictions.B. Multi-task using neural network

[0062] FIGS. 3 A-3B illustrate an example of using a multi-task model, according to embodiments of the present disclosure. Patches 302 are generated from a whole-slide image 301, and each patch is processed by a feature extractor 304 to generate a feature vector 305 for the patch. The feature vectors 305 can be patch-level embeddings. The feature extractor 304 can be a neural network, e.g., a convolutional neural network such as a MoCo-V3 model, which is a deep convolutional neural network pretrained for feature extraction. Such a model can generate tile-level embeddings of K dimensions (e.g., 384 dimensions) for each extracted tile. Other foundation models may also be used as feature extractors. So, N x K represents the tile-level embeddings output by the feature extractor 304, where N is the number of tiles extracted from the whole-slide image and K is the dimensionality of the feature vector for each tile. For example, N may be 10, 100, 500, 1000, etc., and K may be 200, 250, 500, etc. The values for N and K could be at least or less than each of these numbers.

[0063] Additionally, non-image features 310 can be included in feature vector 305. Nonimage features 310 can include one or more patient, clinical and / or sample characteristics. Non-limiting examples of such characteristics patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone mineral density T-score, a prior treatment, a nodal status (e.g., whether in lymph nodes), surgery status (e.g., whether surgery has been performed or not, and possibly type of surgery such as mastectomy or lumpectomy), and treatment type. The prior treatment can comprise any treatment administered to treat the target patient population, including small molecule chemotherapies, hormone treatments, biologies, or any useful combination thereof. Any of such non-image features can instead or also be used for one or more classifications for an auxiliary task.

[0064] Continuing to FIG. 3B, a machine learning framework 316 receives the feature vectors 305 and generates at least one embedding (e.g., a shared embedding or a set of task2280147550V 1specific embeddings) for the whole-slide image. The machine learning framework 316 can include an aggregator (e.g., a transformer model) that aggregates the feature vectors 305 into the at least one embedding. The transformer model can include an encoder and a decoder. Prior to the transformer model, the machine learning framework 316 can include a linear projector that reduces the dimensionality of the feature vectors 305 from K (e.g., 384) dimensions to a lower latent space (e.g., 256) dimensions. The linear projector can act as a learned transformation layer that maps the high-dimensional feature embeddings (NxK) into a lower-dimensional latent space (e.g., N><256). This dimensionality reduction can optimize the input for the transformer encoder while preserving important information. The transformer encoder can aggregate the embeddings for N tasks, transforming them into the N><256 embeddings for task-specific layers that output predictions such as risk scores, categorical (auxiliary classification) or continuous (auxiliary regression) variable predictions. After the decoder, the embeddings can be passed through task-specific layers (multilayer perceptrons (MLPs)) for generating final outputs, such as risk scores or biomarker classifications. Non-image features 320 (e.g., same as used in non-image features 310) can be included at this point as well, e.g., as an input to the linear layer.

[0065] In a task-specific layer of the machine learning framework 316, a machine learning regression model trained using a time-dependent loss function for a risk of survival can receive the at least one embedding (e.g., a regression embedding) and predict a risk score 318 for an outcome of the patient as a primary task. For instance, the risk score 318 may be 0.8 for a survival time of five years. The risk score 318 can be compared to a threshold score (e.g., 0.6) to determine whether the risk score 318 is high risk or low risk. The threshold can be the median risk score of the training set. So, for the risk score 318 of 0.8, the risk score 318 can be determined to be high risk, meaning that the patient is high risk for not surviving five years.

[0066] The machine learning regression model can be trained using a time-dependent loss function (e.g., a Cox partial likelihood function) for a risk of survival. The main task can be trained using higher weights compared to auxiliary tasks. The Cox partial likelihood function can be expressed as:2380147550V 1wherext denotes the i-th sample, ho refers to the risk score of the model output, and ^(^i) is the list of patients with shorts survival times than patient i.

[0067] Various analyses may then be performed based on the risk score 318, such as Kaplan-Meier analysis, Cox regression analysis, and time-dependent area under the curve (AUC) analysis. These analyses collectively allow a nuanced understanding of a patient’s prognosis. The system can predict a patient-specific risk score for a clinical outcome (e.g., survival time). The subsequent analyses enable evaluation of survival probabilities, the impact of risk factors, and the predictive model’s performance over time. This information can support clinical decision-making, offering insights into survival time, potential recurrence, or other time-dependent outcomes. The risk score and Kaplan-Meier analysis can be used together to infer survival probabilities over time, offering an estimate of how long a patient might survive under specific conditions (e.g., high-risk patients might have significantly lower median survival times). The model and analyses may also be trained and calibrated to predict recurrence of disease as the clinical endpoint. The framework is generalizable to other time-to-event outcomes, provided the model is trained with the appropriate labels (e.g., time to disease progression, time to treatment failure). The specific endpoint depends on how the system is trained and the data that is input into the system.

[0068] In addition, auxiliary tasks 322 may be performed to classify or predict clinical variables for the patient. Using the regression embedding for the whole-slide image, the regression embedding may again be processed by other machine learning regression models of the machine learning framework to generate predictions for continuous clinical variables. These machine learning regression models can be trained using regression loss functions for predicting the continuous clinical variables. These auxiliary regression tasks can be leveraged to improve the performance of the primary task by providing additional context and information, enabling the framework to extract richer feature representations and enhance the accuracy of the primary risk score prediction.

[0069] A classification embedding can also be generated and processed by a classification model to predict categorical clinical variables, such as biomarker status or disease subtypes or any non-image features (e.g., patient characteristics), examples of which are provided herein. The classification embedding can be generated using, directly or indirectly, the feature vectors 305 and used to determine classifications for the patient. For categorical clinical variables, the classification embedding can be processed by a classification model of the task-2480147550V 1specific layer trained using a classification loss function for the classifications to obtain the classifications of the biological sample. During training, whole-slide images can be associated with classification labels for each of the auxiliary classification tasks, a regression label for the main regression task, and optionally a regression label for an auxiliary regression task.

[0070] Auxiliary tasks can play a dual role in enhancing the primary regression task’s accuracy and extending the framework’s capabilities to derive additional insights. By leveraging either shared or task-specific embeddings, the framework optimizes predictions across multiple objectives. In addition, auxiliary tasks can be used to derive additional insights from the data. For example, the auxiliary tasks can serve to enhance the utility of the shared embedding by leveraging it to predict continuous variables related to the biological sample, thus extending the machine learning framework’s capabilities beyond standard risk score prediction.

[0071] The auxiliary tasks 322 may include any permutation of the number of auxiliary classification tasks and auxiliary regression tasks. For instance, the auxiliary tasks 322 may include zero auxiliary classification tasks and one auxiliary regression task, one auxiliary classification task and zero auxiliary regression tasks, one auxiliary classification task and one auxiliary regression task, zero auxiliary classification tasks and two auxiliary regression tasks, one auxiliary classification task and two auxiliary regression tasks, two auxiliary classification tasks and two auxiliary regression tasks, two auxiliary classification tasks and zero auxiliary regression tasks, two auxiliary classification tasks and one auxiliary regression task, zero auxiliary classification tasks and three auxiliary regression tasks, one auxiliary classification task and three auxiliary regression tasks, two auxiliary classification tasks and three auxiliary regression tasks, three auxiliary classification tasks and three auxiliary regression tasks, three auxiliary classification tasks and zero auxiliary regression tasks, three auxiliary classification tasks and one auxiliary regression task, three auxiliary classification tasks and two auxiliary regression tasks, and so on.

[0072] During training, the model learns from both the main regression labels and the auxiliary labels. The loss function can use errors or losses for both the main regression task and the auxiliary tasks to update both the regression model and the auxiliary models. For example, the errors from the main regression task and the auxiliary regression tasks can be combined (e.g., averaged, weighted averaged, summed, etc.) and used to determined how to2580147550V 1update the models (e.g., which parameters of the model(s) of the framework should be updated). Auxiliary tasks can enrich feature learning by allowing the main regression model to generate more accurate risk scores while simultaneously enabling robust auxiliary predictions. For training the machine learning regression model, to calculate a risk score for a particular outcome for a patient (e.g., survival time), the machine learning regression model takes into account the outcome for the patient (e.g., the patient survived for five years) and all patients associated with a particular outcome (e.g., all patients that do not survive until five years).C. Example auxiliary tasks

[0073] The systems and methods provided herein were implemented for predicting distant recurrence of hormone-positive breast cancer. This section describes deployment of auxiliary tasks in this context. See also Sections IV.B-D below for further details. Other implementations can use other auxiliary tasks related to other medical conditions, such as prior treatment relevant to that medical condition.

[0074] In various embodiments, the machine learning framework 316 can be configured to simultaneously perform the primary task of risk score prediction (e.g., risk of distant recurrence in breast cancer) and one or more auxiliary tasks using patient-associated clinical features, as further shown in FIG. 7B. In the exemplary implementation for breast cancer, the auxiliary tasks 322 can leverage clinically relevant variables, including the patient’s bone mineral density (BMD) T-score (T-score) and prior tamoxifen therapy status (PriorTAM), as output predictions of the model. The following paragraphs illustrate use of auxiliary tasks using T-score and PriorTam in the context of the exemplary breast cancer setting. While T-score and PriorTAM are provided as illustrative examples of auxiliary tasks in this setting, the framework can be adaptable to other clinical variables, disease / cancer types, and treatments.

[0075] In various embodiments, the auxiliary tasks 322 can involve a single binary auxiliary task of T-score classification or T-score classification can be one of multiple auxiliary tasks. The T-score is a quantitative measure of bone mineral density and is clinically used to assess osteoporosis and fracture risk. Using T-score classification as the auxiliary task, the T-score is binarized based on a clinically meaningful threshold (e.g., < -2.0), classifying patients as having low or normal bone mineral density. This enables the2680147550V 1auxiliary task to be formulated as a binary classification, where the model predicts, based on the image and / or clinical features, whether a patient’s T-score is below or above the specified threshold. This classification can inform clinical decisions regarding therapy safety and suitability, as low bone density may influence extended endocrine therapy recommendations.

[0076] In some embodiments, the auxiliary tasks 322 can involve a single binary auxiliary task of PriorTAM classification or PriorTAM classification can be one of multiple auxiliary tasks. PriorTAM refers to whether the patient has previously been treated with tamoxifen, a common endocrine therapy agent in breast cancer. Using PriorTam classification as the auxiliary task, PriorTAM is treated as a binary variable, with the model predicting whether the patient has a history of tamoxifen use (e.g., yes / no). Prior tamoxifen exposure is a clinical factor that can influence prognosis and the likely benefit from extended letrozole therapy (ELT).

[0077] In various embodiments, the auxiliary tasks 322 can include multiple binary auxiliary tasks, where the model may be trained with two auxiliary classification branches. One classification branch can be trained for each of T-score and PriorTAM, enabling simultaneous prediction of both variables. Alternatively, the model may combine T-score and PriorTAM into a single multi-class auxiliary task, where each class represents a unique combination of T-score and PriorTAM status (e.g., T-score low / PriorTAM yes, T-score low / PriorTAM no, T-score high / PriorTAM yes, T-score high / PriorTAM no).

[0078] The machine learning framework 316 may not only generate the risk score 318 associated with a clinical outcome (e.g., the risk of distant recurrence), but also may determine and output a corresponding treatment recommendation for the patient. This treatment recommendation can be derived from the risk score 318 and, optionally, the auxiliary tasks 322 (e.g., T-score and PriorTAM in the example described above), enabling the system to provide actionable clinical guidance at the point of care.

[0079] In an example, the model processes image-derived features and, where available, clinical features to generate the risk score 318 for the patient. The risk score 318 can then be compared to a predetermined threshold (e.g., the median or a clinically validated cutoff value) to categorize the patient as “high risk” or “low risk” for adverse outcomes such as distant recurrence. Based on this categorical assignment, the system can determine whether a course of action, e.g., a specific treatment or extended treatment, should be recommended for the patient. As a non-limiting example, consider the case of hormone receptor-positive breast 2780147550V 1cancer. In this example, for patients categorized as “high risk,” the system may output a treatment recommendation advising initiation or continuation of extended letrozole therapy (ELT), as these patients are likely to derive clinical benefit from the therapy. Alternatively, for patients categorized as “low risk,” the system may output a recommendation against ELT, since the treatment may not be considered to be beneficial.

[0080] The treatment recommendation can also take into account the auxiliary tasks 322, such as BMD T-score and PriorTAM, which may impact the appropriateness or safety of therapy. For example, the system may provide an alert or tailored recommendation if the patient is predicted to have low BMD or a history of prior tamoxifen use, thereby supporting individualized treatment planning.

[0081] The system may generate a report that includes the risk score 318, risk category, the auxiliary tasks 322, and a treatment recommendation (e.g., “Recommend extended endocrine therapy” or “No extended endocrine therapy recommended”). The report may also include supporting rationale, such as the patient’s risk classification and relevant clinical factors. III. AGGREGATION TECHNIQUES

[0082] For a whole-slide image, patches are generated corresponding to portions of the whole-slide image, and a feature vector is generated for each patch. To determine a risk score and auxiliary predictions (e.g., regressions and / or classifications), aggregated embeddings representing aggregations of the feature vectors can be used. The aggregation can result in an aggregated embedding for the whole-slide image for each main and auxiliary task, or a shared embedding for all tasks. The feature vectors can be aggregated using various techniques to generate the embeddings.

[0083] FIG. 4 illustrates various aggregation techniques for feature vectors, according to embodiments of the present disclosure. Prior to each technique, patches 402 are generated from a whole-slide image 401, and the patches 402 are processed by a feature extractor 404 to generate feature vectors for each patch. The feature extractor 404 is an example of the feature extractor 304 in FIG. 3. The aggregation techniques may be performed by an aggregator, such as the transformer model (e.g., the encoder and the decoder) in FIG. 3. In a first aggregation technique 430, a model may be trained to predict a risk score corresponding to a patient outcome for each patch based on the feature vectors. These risk scores correspond to patch predictions 432. Different patches may be associated with different patch2880147550V 1predictions. As a non-limiting example, a first subset of the patches may be associated with a high-risk risk score, while a second subset of patches may be associated with a low-risk risk score.

[0084] An aggregator 434 can then determine a whole-slide image prediction 436 based on the patch predictions 432. For instance, the whole-slide image predictions 436 may be for the main task or auxiliary tasks. For the main task, the whole-slide image predictions can correspond to a maximum risk score of the patch predictions 432, a majority risk score of the patch predictions 432, an average risk score of the patch predictions 432, etc. to be the wholeslide image prediction 436 for the whole-slide image. The whole-slide image prediction 436 corresponds to an overall risk score for the whole-slide image. For instance, if the model determines that ten patches are predicted to have a high-risk risk score and seven patches are predicted to have a low-risk risk score, the aggregator 434 can predict the whole-slide image to have a high-risk risk score based on the majority of the patches being associated with the high-risk risk score prediction in the patch predictions 432.

[0085] In some examples, other aggregation techniques may involve determining weights for each patch, where the weights represent an importance of the association between the patch and the whole-slide image label. In these techniques, the aggregator can be multiinstance learning models that are used to output embeddings that represent a weight of a measure of the association or contribution from each patch to the final whole-slide image prediction.

[0086] As an example of a second aggregation technique 440, the aggregator can be a plain multi-instance learning model 442 that is trained to learn weights for each patch. The plain multi-instance learning model 442 can be considered attention-based, as it learns the relationship between the patch and the prediction, but it does not learn any interaction between the patches 402. So, for the plain multi-instance learning model 442 to generate a whole-slide image embedding 444, the features from each patch are multiplied by the weight for the patch and added to the features of the other patches times the weight of the other patches to get the whole-slide image embedding 444. The whole-slide image embedding 444 can be a shared embedding for all tasks.

[0087] The whole-slide image embedding 444 is then input into a predictor 446 (e.g., a machine learning regression model) which generates the whole-slide image prediction 436 corresponding to the risk score for the whole-slide image 401. The predictor 446 is an2980147550V 1example of the task-specific layer of the machine learning framework 316 in FIG. 3. The whole-slide image embedding 444 can be a set of task specific embeddings or a shared embedding for all tasks, depending on the model architecture. For example, if there is one primary task and four auxiliary tasks (e.g., two auxiliary regression tasks and two auxiliary classification tasks), for the task specific architecture, there can be five whole-slide image embeddings.

[0088] As an example of a third aggregation technique 450, the aggregator can be a local context multi-instance learning model 452 that is trained to learn weights for each patch based its local context (e.g., the target patch and surrounding patches). The local context multi-instance learning model 452 can be implemented as a graph neural network (GNN) that models the relationships between patches by treating patches as nodes and defining edges based on spatial proximity or feature similarity. The GNN learns how the features of each patch and its surrounding patches contribute to the overall prediction. The local context multi-instance learning model 452 aggregates a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of a surrounding area of the target portion. The surrounding area can be within a pixel distance (e.g., 100 pixels) of the target portion. Each portion (e.g., the target portion and the additional portions) corresponds to a patch.

[0089] To generate the whole-slide image embedding 454, the local context multi-instance learning model 452 can aggregate the features of all patches by applying learned weights to each patch’s features. Specifically, the feature vector for each patch is multiplied by its learned weight, and the weighted feature vectors from all patches are combined (e.g., summed or averaged) to form the whole-slide image embedding 454. These embedding captures both the local and global context of the whole-slide image. The resulting whole-slide image embedding 454 is then input into the predictor 446, which generates the whole-slide image prediction 436 corresponding to the risk score for the whole-slide image 401.

[0090] As an example of a fourth aggregation technique 460, the aggregator can be a global context multi-instance learning model 462 that is trained to learn weights for a global context (e.g., all of the patches 402). The global context multi-instance learning model 462 can be considered a transformer neural network or a similar architecture that incorporates a selfattention mechanism. This mechanism enables the global context multi-instance learning model 462 to learn both the relationship between each patch and the whole-slide image label,3080147550V 1as well as the relationship between each patch and all other patches. The self-attention mechanism assigns weights based on these relationships, reflecting the importance of each patch in the context of all patches and the label.

[0091] The global context multi-instance learning model 462 aggregates a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of the image. The additional portions make up an entirety of the image. Each portion (e.g., the target portion and the additional portions) corresponds to a patch. To generate a whole-slide image embedding 464, the global context multi-instance learning model 462 computes a weighted combination of patch features, where the weight for each patch is influenced by its interaction with all other patches and its contribution to the overall prediction. Specifically, the features from each patch are adjusted based on the learned weights and aggregated to form a holistic representation of the entire slide. The resulting whole-slide image embedding 464 is then input into a predictor 446 (e.g., a machine learning regression model or classifier), which generates the whole-slide image prediction 436 corresponding to the risk score or other target outcome for the whole-slide image 401.IV. EXAMPLE RESULTS

[0092] Embodiments for the digital pathology analysis described herein (potentially also including a molecular profiling approach) can provide techniques for selecting a candidate treatment for an individual that could favorably change the clinical course for the individual with a condition or disease, such as cancer. The techniques can provide clinical benefit for individuals, such as identifying therapeutic regimens that provide a longer progression free survival (PFS), longer disease free survival (DFS), longer overall survival (OS) or extended lifespan. Methods and systems as described herein are directed to digital pathology analysis of cancer on an individual basis that can identify optimal therapeutic regimens. Digital pathology analysis provides a personalized approach to selecting candidate treatments that are likely to benefit a cancer. The digital pathology described herein can be used to guide treatment in any desired setting, including without limitation the front-line / standard of care setting, or for patients with poor prognosis, such as those with metastatic disease or those whose cancer has progressed on standard front line therapies, or whose cancer has progressed on previous chemotherapeutic or hormonal regimens.3180147550V 1

[0093] The systems and methods provided herein may be used to classify patients as more or less likely to benefit or respond to various treatments. Unless otherwise noted, the terms “response” or “non-response,” as used herein, refer to any appropriate indication that a treatment provides a benefit to a patient (a “responder” or “benefiter”) or has a lack of benefit to the patient (a “non-responder” or “non-benefiter”). Such an indication may be determined using accepted clinical response criteria such as the standard Response Evaluation Criteria in Solid Tumors (RECIST) criteria, or other useful patient response criteria such as progression free survival (PFS), time to progression (TTP), disease free survival (DFS), time-to-next treatment (TNT, TTNT), tumor shrinkage or disappearance, or the like. RECIST is a set of rules published by an international consortium that define when tumors improve (“respond”), stay the same (“stabilize”), or worsen (“progress”) during treatment of a cancer patient.

[0094] As used herein and unless otherwise noted, a patient “benefit” from a treatment may refer to any appropriate measure of improvement, including without limitation a RECIST response or longer PFS / TTP / DFS / TNT / TTNT, whereas “lack of benefit” from a treatment may refer to any appropriate measure of worsening disease during treatment. Generally disease stabilization is considered a benefit, although in certain circumstances, if so noted herein, stabilization may be considered a lack of benefit. A predicted or indicated benefit may be described as “indeterminate” if there is not an acceptable level of prediction of benefit or lack of benefit. In some cases, benefit is considered indeterminate if it cannot be calculated, e.g., due to lack of necessary data.A. Risk score prediction using image and molecular features for immune checkpoint therapy

[0095] The immune system relies on checkpoint proteins to maintain self-tolerance and prevent autoimmunity. Cancer cells exploit these regulatory pathways by expressing ligands that bind to inhibitory receptors on T cells, effectively suppressing the anti-tumor immune response. Immune checkpoint therapy works by blocking these inhibitory signals, restoring T-cell activation and enabling the immune system to recognize and destroy tumorcells. Currently approved checkpoint inhibitors comprise monoclonal antibodies that bind to checkpoint proteins (e.g., PD-1, PD-L1, CTLA-4, LAG-3) to block their immunosuppressive effects.3280147550V 1

[0096] Immune checkpoint therapy relies on a growing roster of biomarkers to predict patient response and guide treatment decisions. The most established include PD-L1 and its receptor PD-1, along with CTLA-4, which was the first checkpoint target to reach clinical use. Newer targets like LAG-3, TIM-3, and TIGIT are expanding the therapeutic landscape, while predictive genomic signatures and biomarkers such as tumor mutational burden (TMB), microsatellite instability-high (MSI-H) / mismatch repair deficiency (dMMR), and tumorinfiltrating lymphocytes (TILs) help identify patients most likely to benefit. Additional markers for assessing immune checkpoint include VISTA, B7-H3, B7-H4, IDO1, and BTLA.

[0097] In this example, systems and methods for digital pathology provided herein were used to build models to predict survival of patients administered checkpoint therapy.

[0098] FIGS. 5A-C illustrate exemplary results of model performance for predicting survival using biomarker status, a single-task model, and a multi-task model, respectively. Plots show survival after treatment with pembrolizumab (anti-PD-Ll antibody; tradename Keytruda®; also referred to as “pembro” in the figures). FIG. 5 A illustrates a survival curve in non-small cell lung cancer patients treated with pembrolizumab stratified by the protein expression status of PD-L1 as determined by H4C (positive or negative). FIG. 5B shows a survival curve for the same patient cohort using a single model tasked with predicting survival via analysis of H&E slide images. FIG. 5C shows a joint multi-task model that combines these features, wherein the primary task was predicting survival using H&E slide images and the auxiliary task was predicting biomarker expression (PD-L1 positive or negative). The hazard ratios for the PD-L1 expression status (FIG. 5 A), the single-task model (FIG. 5B), and the multi-task model (FIG. 5C) were 0.86 (p=0.03), 0.77 (p<0.0001), and 0.71 (p<0.0001), respectively. Thus, the joint multi-task model outperformed IHC biomarker status (i.e., expression status of PD-L1) and the single-task model in predicting survival.

[0099] FIGS. 6A-6D illustrate exemplary results of model performance for predicting survival and auxiliary classifications using a multi-task model and the same patient data as shown in FIGS. 5A-C. The plots represent survival prediction with auxiliary classifications of gender (FIG. 6A), age (FIG. 6B), histology (FIG. 6C), and both histology and PD-L1 (FIG.6D). The hazard ratios associated with the auxiliary classifications are 0.75 (p<0.0001), 0.76 (pO.OOOl), 0.79 (p=0.0004), and 0.73 (p<0.0001), respectively. Based on the hazard ratios, the multi-task model performed best for predicting survival when performing auxiliary classifications of both histology and PD-L1 (FIG. 6D). With these data, the performance for3380147550V 1predicting survival with an auxiliary classification of histology (FIG. 6C) was the only setting where the hazard ratio was higher than predicting survival from either the biomarker status or the single-task model in FIGS. 5A-5B, respectively. Nevertheless, the combined auxiliary tasks of histology and PD-L1 expression led to better performance for predicting survival than using the auxiliary task of histology alone (histology x PD-L1 HR = 0.73, FIG. 6D; histology alone HR = 0.79, FIG. 6C). Among the exemplary results, the multi-task model performed best with auxiliary classification of PD-L1 alone, as shown in FIG. 5C (PD-L1 alone HR = 0.71).B. Predicting risk of late distant recurrence and extended endocrine therapy benefit in hormone-receptor-positive breast cancer

[0100] Breast cancer is the most commonly diagnosed cancer and the second leading cause of cancer-related deaths among women in the United States. See Reference 1. Approximately 70-80% of newly diagnosed breast cancers are hormone receptor-positive (HR+), which are characterized by a prolonged risk of recurrence and breast cancer mortality extending beyond five years of adjuvant endocrine therapy (ET). See References 2,3. Extended endocrine therapy (EET) has been proposed as a strategy to mitigate this late recurrence risk, and several clinical trials have investigated its potential benefits. See References 3-12. However, the results of these studies have been mixed, often influenced by variations in clinical endpoints and patient selection criteria.

[0101] The NSABP-B42 trial is one of the pivotal studies evaluating the efficacy of EET. See Reference 5. This randomized, double-blind, placebo-controlled phase III trial assessed whether an additional five years of extended letrozole therapy (ELT) could improve disease-free survival (DFS) in postmenopausal women with early-stage HR+ breast cancer. Letrozole is a type of endocrine therapy (also known as hormonal therapy) primarily used for hormone receptor-positive (HR+) breast cancer. The agent blocks the enzyme aromatase, which plays a role in the production of estrogen, and is thus classified as an aromatase inhibitor. The NSABP-B42 trial enrolled patients who remained disease-free following five years of initial ET with aromatase inhibitors or tamoxifen followed by aromatase inhibitors. While initial findings demonstrated a 3.4% improvement in DFS after a median follow-up of 6.9 years, the results did not meet the pre-specified threshold for statistical significance. A 10-year update from the B-42 trial reaffirmed ELT’s beneficial effects on DFS and its impact on breast3480147550V 1cancer-free interval (BCFI) and distant recurrence (DR) but no improvement in overall survival was observed. See Reference 13.

[0102] Despite these modest gains, there remains a pressing need for tools that can precisely identify patients most likely to benefit from EET, thereby sparing others from unnecessary treatment and its associated toxicities. Prognostic factors, including clinicopathologic features, circulating tumor cells, and molecular assays such as the Breast Cancer Index (BCI) and MammaPrint, have been used to guide EET decision-making. See References 14-19. BCI, which is adopted in both NCCN and ASCO guidelines, consists of two components: the HOXB13 / IL17BR (H / I) ratio and the molecular grade index (MGI). Its predictive segment, BCI(H / I), is based on the H / I ratio and has demonstrated the ability to predict who might benefit from EET in multiple studies. See References 17,20-25.MammaPrint, a 70-gene assay, is relatively new in this context but has shown positive results in two recent publications supporting its potential role in identifying patients who may benefit from EET. See References 26-29,30.

[0103] Both BCI and MammaPrint have been independently evaluated using subsets of the NSABP-B42 cohort, further underscoring their potential to guide EET decisions. In the BCI translational cohort analysis, while no statistically significant difference was observed between BCI (H / I)-High and BCI (H / I)-Low groups regarding ELT benefit for the primary endpoint of recurrence-free survival (RFS), a time-dependent analysis revealed that BCI (H / I)-High patients derived significant benefit from ELT after four years, whereas BCI (H / I)-Low patients did not. See Reference 24. In the MammaPrint translational cohort analysis, patients classified as low-Risk experienced significant 10-year benefits from ELT for DR, DFS, and BCFI, whereas high-Risk patients did not. See Reference 29. These studies highlight the complexities and challenges in developing reliable tools to match patients with appropriate treatments.

[0104] While molecular assays have shown promise, concurrent advancements in artificial intelligence (Al) have revolutionized medical image analysis, offering new possibilities for treatment guidance. Deep learning, particularly transformer-based models, has emerged as a powerful approach for analyzing medical images. Al-based tools leveraging histopathology images have shown remarkable potential in predicting outcomes for various cancers, including colorectal, prostate, and breast cancers. See References 31-35. Compared to molecular biomarkers, Al-based systems are more cost-effective, scalable, and capable of3580147550V 1continuous improvement through iterative learning, making them an attractive alternative or complement to traditional approaches.

[0105] As presented in this exemplary implementation, end-to-end ML models capable of accurately predicting patients’ risk of late DR and identifying subgroups likely to benefit from ELT were developed and validated. Using H&E slides alongside relevant clinical information, several ML models leveraging histopathological features extracted from WSIs were developed, with or without clinical features, and using either single task or multi-task model architecture as provided herein. All models exhibited excellent predictive performance, with the multi-task models consistently outperforming the conventional single task models in both prognostication and treatment benefit prediction.1. Methods

[0106] Below are example techniques used to produce results in section 2. Other techniques, parameters, criteria, etc. can be used, and the techniques provided below are only provided as some options.a) Patient population

[0107] NSABP B-42 was a prospective phase III trial that enrolled 3,966 postmenopausal women with hormone receptor-positive (HR+) early breast cancer who were disease-free after five years of an aromatase inhibitor or tamoxifen followed by an aromatase inhibitor. Patients were randomized to receive an additional five years of either letrozole or placebo. Eligible B-42 patients with clinical follow-up, appropriate consent, and available formalin-fixed paraffin-embedded (FFPE) primary tumor tissue were included in the study.Hematoxylin and eosin (H&E) slides from these patients were scanned using the Pramana Spectral HT scanner and / or Leica Aperio GT450 scanner.

[0108] For purposes of this implementation, 2,271 patients were included from the NSABP B-42 trial who had both clinical outcomes available and H&E slides with sufficient tissue, representing 58% of the parent cohort. In this cohort, 1,307 (60.1%) were node negative (NO) patients, 310 (13.7%) were HER2 positive (HER2+), 1,400 (61.6%) received only aromatase inhibitor as prior ET, and 871 (38.4%) received tamoxifen followed by aromatase inhibitor sequential therapy prior to randomization.3680147550V 1b) Risk score prediction pipeline

[0109] The DR risk prediction pipeline involved three main steps: 1) image preprocessing 2) feature extraction, 3) transformer-based risk score prediction. The primary analyses were conducted on whole slide images (WSIs) scanned using the Pramana scanner. To evaluate model robustness, a subset of H&E slides was scanned using the Leica scanner.c) Image preprocessing

[0110] WSIs from the Pramana scanner were converted from OME-TIFF format to TIFF format. Tissue masks were generated using a QuPath-based (see reference 36) pixel classifier, and HistoQC (see reference 37) was used to detect and remove coverslip edges and pen markings from the masks. Morphological transformations were applied to fill small holes and remove small objects, enhancing tissue mask quality. Tiles measuring 224x224 pixels were extracted at lOx magnification with a pixel size of ~1 micron.d) Feature extraction

[0111] A pretrained CTransPath model (see reference 38) was used to generate 768-dimensional (768-D) embeddings from each tile. CTransPath uses semantically related contrastive learning to improve feature representation by leveraging diverse positive samples. The backbone architecture includes a convolutional neural network (CNN) module and a Swin transformer module. The CNN stabilizes training, while the Swin transformer extracts features through local window attention and shift-window-based self-attention mechanisms. Other feature extraction techniques can be used, and these techniques are only provided as some options. See feature extractor 104 in FIG. 1 and feature extractor 304 in FIG. 3 for additional information related to feature extraction.e) Model training design

[0112] A 5-fold cross-validation was performed for the training. In each of the 5 folds, 3 splits were used for training, 1 split for validation and 1 split for testing. This strategy enabled the evaluation of the entire cohort as opposed to having a fixed small set for testing. Due to the small number of DR events in the translational cohort (n=122), a stratified splitting was adopted based on the number of DR events to divide the translational cohort into five splits. Within each test fold, the patients were divided into high / low risk group using the median3780147550V 1risk score of corresponding training cases as cutoff. Other model training techniques can be used, and these techniques are only provided as some options.f) Model architecture

[0113] An encoder-decoder transformer architecture (see reference 39) was used to predict WSI-level risk scores from tile-level embeddings generated by CTransPath. In WSI-only models, the embeddings served as input. For models which also incorporated clinical features, the clinical features were concatenated with each tile embedding using an early fusion strategy, which outperformed late fusion methods (e.g., concatenation or TensorFusion (see reference 40)). The architecture included a projection layer to reduce embedding dimensionality from 768 to 256, enabling memory-efficient processing. Encoded embeddings were passed through the transformer decoder, which generated a 256-D regression token. A fully connected layer then predicted the WSI-level risk score from this token. Other architectures can be used, and these architectures are only provided as some options. See machine learning framework 210 in FIG. 2 and machine learning framework 316 in FIG. 3 for additional information related to model architectures.

[0114] FIGS. 7A-7B illustrate models that were constructed using WSI features with or without clinical features to predict risk score only (FIG. 7A) or using the multi-task model architecture provided herein (FIG. 7B). In the multi-task model settings, the auxiliary tasks are bone mineral density (BMD) T-score and prior tamoxifen treatment (PriorTam). The models are trained to predict the risk score along with one or more auxiliary outcomes.Importantly, in the multi-task models, T-score and PriorTam are used as auxiliary task labels during training only and are not required as input variables during testing or application. This allows the model to leverage information from these variables during training without creating dependency on them during deployment.g) Model training

[0115] Models were trained using the AdamW optimizer with a batch size of 64 and 50 epochs. The Cox partial likelihood loss function (see reference 41) was used for training. To reduce overfitting, early stopping, L1 / L2 regularization (weight = P IO3), and adaptive learning rates were applied. The learning rate was set to 2 / | 0 for the image + clinical feature models and 1 x 106for the image only models. During training, 64 randomly selected tiles were used for each WSI, while all tiles were utilized for validation and testing. Optimal3880147550V 1epochs were selected based on the maximum C-index value on the validation split. Other model training techniques can be used, and these techniques are only provided as some options.h) Statistical analysis

[0116] The translational cohort’s representativeness of the parent B-42 trial population was assessed using chi-square tests for patient and tumor characteristics. The chi-square test is a non-limiting example and other tests can be used.

[0117] The primary endpoint of the model prediction was DR, defined as the time from randomization to DR. Predefined secondary endpoints included recurrence-free interval (RFI), defined as the time from randomization to local, regional, or distant breast cancer recurrence; DFS, defined as the time from randomization to breast cancer recurrence, second primary cancer or death; and breast cancer-free interval (BCFI), defined as the time from randomization to breast cancer recurrence or contralateral breast cancer as a first event.Patients who were otherwise event free were censored at the date of last clinical follow-up.

[0118] Differences in all endpoints between high-risk and low-risk groups in the entire translational cohort and clinical subgroups were assessed by log-rank test. Hazard ratio (HR) and 95% confidence intervals were calculated based on a univariate Cox model. Kaplan-Meier estimates were used for illustration purposes, with absolute difference defined as the difference in 10-year Kaplan-Meier estimate of event risk between high-risk and low-risk groups. A multi-variate analysis was performed in the entire cohort for DR endpoint to determine the effect of prognostic variables including clinical factors and predicted risk label. In this analysis, clinical variables (pathologic node status, BMD T-score, prior use of tamoxifen, age, surgery type (lumpectomy or mastectomy)), treatment label and predicted risk label were used as main effect terms. Additionally, interactions between treatment and remaining variables were also considered in the analysis. The main effect terms and interaction terms with significant effect were included in the final multi -variate Cox model. Violation of proportional hazard (PH) assumption was checked for the main-effect-only model and clinical variables violating PH assumption were used as stratification factors in the full model with interaction. Treatment and risk label satisfied PH assumption for both image and image + clinical feature models.3980147550V 1

[0119] Differences in all endpoints between ELT and placebo groups were assessed by stratified log-rank tests controlling for the stratification variables of the parent trial (pathologic node status (PNode), prior use of tamoxifen as a component of initial adjuvant therapy (PriorTam), and lowest bone mineral density T score in the lumbosacral spine, total hip, or femoral neck (T-score)). HRs and corresponding 95% confidence intervals were calculated based on the stratified Cox model. This analysis strategy for treatment benefit was used for the entire translational cohort and within clinical subgroups, which were further subdivided based on low- and high-risk groups. For each analysis, proportional hazard assumption was checked. For cases where a violation of assumption was observed, a change point was detected, and a time-dependent Cox modeling was performed. See Reference 42. The change point was determined by calculating the relative risk in the entire cohort for DR endpoint, which was found to be ~4 years.

[0120] A likelihood ratio test in the entire translational cohort was performed to test for treatment-by-clinical subgroup interaction, and treatment-by-risk interaction. For the likelihood ratio test, a reduced stratified Cox model with only main effect terms (treatment and clinical variable / risk variable) and another full stratified Cox model with main effect and interaction term (treatment-by-risk and treatment-by-clinical covariate) were used. For cases where the clinical covariate of interest for the interaction term overlapped with stratification variables, the covariate was removed from stratification factors. To account for the violation of the proportional hazards (PH) assumption by the treatment variable, a time-by-treatment interaction term was included in the Cox model. This allows the effect of treatment to vary over time, while still enabling us to estimate the interaction between treatment and risk group. A log(time) transformation was used to model the time-varying effect. Kaplan-Meier estimates were used for illustration purposes, with absolute benefit defined as the difference in 10-year Kaplan-Meier estimate of event risk between ELT and placebo treatments.

[0121] Calibration of predicted risk was evaluated at prespecified time horizons. The models were trained using Cox partial likelihood and produced continuous risk scores representing relative hazard. Absolute risk estimates were derived by fitting a Cox proportional hazards model with the model-derived risk score as the sole covariate to estimate the baseline survival function, from which predicted 10-year survival probabilities and cumulative risks were obtained. Observed event probabilities were estimated using the Kaplan-Meier method within strata defined by quantiles of predicted risk. Calibration was4080147550V 1summarized using calibration slope, calibration-in-the-large, expected calibration error, and maximum bin-level error.

[0122] To assess robustness, calibration analyses were repeated over 30 random splits of the dataset. For each split, 50% of samples were used to estimate the baseline survival function and the remaining 50% were used for calibration evaluation. Five risk strata were defined based on quantiles of predicted risk, ensuring a minimum of 10 events per stratum for stable estimation. Calibration metrics were summarized across repeats. Other statistical analysis techniques can be used, and these techniques are only provided as some options.2. Results

[0123] Below are example results illustrating aspects of various embodiments of the present disclosure.a) Predicting risk of late distant recurrence

[0124] The single-task model based solely on histopathological features from WSIs (FIG.7 A), demonstrated strong predictive performance for late DR, achieving an average C-index of 0.727 on the test dataset. Using a median risk score threshold derived from the training dataset, high-risk patients (risk scorethreshold) had significantly worse outcomes (HR = 0.292; 95% CI: 0.192 - 0.446; p < 0.001) and a 10-year absolute difference in DR of 5.79%. Similar trends were observed among patients receiving ELT (HR = 0.312; 95% CI: 0.162-0.601; p < 0.001) or placebo (HR = 0.280; 95% CI: 0.161-0.487; p < 0.001). This model also stratified patients for other endpoints in the entire translational cohort: RFI (HR = 0.369; 95% CI: 0.263-0.519; p < 0.001), BCFI (HR = 0.531; 95% CI: 0.409-0.690; p < 0.001), and DFS (HR = 0.718; 95% CI: 0.606-0.850; p < 0.001). In a multivariate Cox analysis, the risk score generated by the single-task model based solely on histopathological image features remained an independent prognostic factor for DR (HR = 0.362; 95% CI: 0.236-0.556; p < 0.0001).

[0125] The single-task model which combined readily available clinical variables (age, pathological node status, and surgery type) into a multimodal model (FIG. 7A) outperformed the above image only model (C-index 0.769 vs. 0.727), showing a stronger prognostic effect (HR = 0.222; 95% CI: 0.141-0.349; p < 0.001) and increasing the 10-year absolute difference in DR to 7.13%. Among patients receiving either ELT or placebo, the HRs were 0.218 (95% CI: 0.106-0.452; p < 0.001) and 0.221 (95% CI: 0.123-0.395; p <0.001), respectively. As4180147550V 1with the image only model, the multimodal model also identified worse outcomes for RFI, BCFI, and DFS in the high-risk group.

[0126] The performance of multi-task models was then assessed using BMD T-score and prior tamoxifen treatment (PriorTAM) as auxiliary classification tasks. See FIG. 7B. T-score was treated as a binary variable using a cutoff of BMD T-score ^-2.0. Using these auxiliary predictions, four multimodal multitask models were trained and evaluated based on different auxiliary tasks: 1) single auxiliary task / binary classification: T-score; 2) single auxiliary task / binary classification: PriorTAM; 3) single auxiliary task / multi-classification: T-score x PriorTAM (i.e., T-score + / PriorTAM +; T-score + / PriorTAM -; T-score - / PriorTAM +; and T-score - / PriorTAM -); and 4) multi auxiliary tasks (two auxiliary classification tasks): T-score and PriorTAM.

[0127] Table 1 shows a comparison of the performance of the combined WSI + clinical feature single task model (which was superior to the WSI only model) compared to the multi task models trained with the differing auxiliary tasks. The high and low risk groups were defined using a threshold set to the 50% quantile where high-risk patients have a risk score threshold. The absolute benefit (column Abs. Ben.) and hazard ratio (column HR) data reveal that both single auxiliary task models (Multi Task 1 and 2) performed better than the single task model, which performed better than either of the multiple auxiliary tasks model (Multi Task 3) or multi-class auxiliary tasks model (Multi Task 4).Table 1: Model Performance Summary (quantile = 50)4280147550V 1

[0128] Table 2 is similar to Table 1 except that the cutoff to define risk groups was set to 56%. The trends in the data are similar to those in Table 1. However, in this setting all multi task models outperformed the single task model in both absolute benefit and HR. It was found that increasing the cutoff can increase absolute benefit in the high risk group, at the cost of increased event rates in low risk group.Table 2: Model Performance Summary (quantile = 56)

[0129] FIGS. 8A-8F show the risk of distant recurrence considering prior treatment (ELT or placebo) using the single task model based on combined image and clinical features (FIGS. 8A, 8C, 8E) and multi-task model 1, which employed prediction of T-score as a single auxiliary task (FIGS. 8B, 8D, 8F). FIGS. 8A (single) and 8B (multi) show incidence of recurrence over time when considering all patients. A 50% quantile was used to separate high4380147550V 1and low risk patients. The absolute difference and HR are superior with the multi-task model (Abs Diff = 7.95% multi v 7.13% single; HR = 0.175 multi v 0.222 single).

[0130] FIGS. 8C (single) and 8D (multi) show incidence of recurrence over time when considering only the patients treated with ELT. The absolute difference and HR are superior with the multi-task model (Abs Diff = 6.15% multi v 5.73% single; HR = 0.189 multi v 0.218 single). Finally, FIGS. 8E (single) and 8F (multi) show incidence of recurrence over time when considering only the patients receiving placebo. The absolute difference and HR are superior with the multi-task model (Abs Diff = 9.75% multi v 8.54% single; HR = 0.164 multi v 0.221 single). Taken together, these data indicate that the multi-task model more accurately predicted distant recurrence in all patient populations regardless of treatment history. The largest improvement was observed in the placebo setting, demonstrating that the multi-task model has better prognostic performance.b) Identifying patients likely to benefit from extended letrozole therapy (ELT)

[0131] FIGS. 9A-9F illustrate incidence of distant recurrence in hormone receptor-positive breast cancer patients assigned to high or low risk of DR cohorts using single and multi task machine learning models provided herein. Beyond predicting survival outcomes, the deep learning-based models also provide insights for guiding ELT. Although no significant interactions were found between clinical features or risk scores and treatment, higher absolute ELT benefit was observed in patients who were node positive, underwent mastectomy, were aged ^60, had HER2 -negative disease, and a BMD T-score ^-2.0. See, e.g., FIG. 9A, which displays hazard ratios (HR) in various settings. Minimal absolute benefit of ELT (-0.06%) was observed among node negative (NO) patients; however, in node positive (N+) patients, ELT significantly reduced the risk of DR (HR = 0.511; 95% CI: 0.325-0.803) and provided a 5.94% absolute benefit. Although surgery type is not considered a direct risk factor, patients who underwent mastectomy (typically performed for larger tumors (see reference 43)) experienced a greater benefit (5.5%) compared to those who received lumpectomy (0.44%). Patients aged ^60 years experienced a 4.49% absolute benefit, compared with 0.85% for those >60. HER2 positive patients had a 3.22% absolute benefit, versus 2.13% in HER2 negative patients. Prior tamoxifen use had minimal impact on ELT4480147550V 1benefit (2.27% vs. 2.16%), and patients with a BMD T-score ^-2.0 benefited more than those with a T-score >-2.0 (5.76% vs. 1.11%).

[0132] In the entire translational cohort, patients receiving ELT had an HR of 0.621 (95% CI: 0.432-0.894; p = 0.01) and an absolute benefit of 2.2%. See FIG. 9B.

[0133] FIGS. 9C-9F show survival curves for patients stratified into low or high risk of DR cohorts by either the single task model based on combined image and clinical features (FIGS.9C, 9E) or the multi-task model 1, which employed prediction of T-score as a single auxiliary task (FIGS. 9D, 9F). A 50% quantile was used to separate high and low risk patients. FIGS.9C (single) and 9D (multi) show incidence of recurrence over time in the low risk patients. Using either model architecture, the benefit of ELT in the low risk groups was not statistically significant (p = 0.201 single and 0.381 multi). These data indicate that patients predicted to have low risk of DR did not significantly benefit from ELT. Therefore, the models not only have prognostic value but can also guide treatment decisions, as patients with better predicted prognosis (i.e., low risk) have little to no benefit from ELT. This effect appeared to be more pronounced when assigning risk via the multi-task model as the absolute benefit was smaller and HR was larger. Cf. FIGS. 9C and 9D.

[0134] FIGS. 9E (single) and 9F (multi) show incidence of recurrence over time in the patients determined by the models to be at high risk of DR. In both cases, the placebo groups had higher incidence of recurrence, indicating that patients determined to be high risk also benefitted from ELT. The absolute benefit and HR were superior with the multi-task model (Abs Ben (diff) = 4.09% multi v 3.68% single; HR = 0.614 multi v 0.644 single). Taken together, these data indicate that patients determined to be at low risk have little benefit from ELT, whereas patients determined to be at high risk have significant benefit from ELT. Thus, the single task and multi-task models provided herein have both prognostic and theranostic benefit in the care of hormone receptor-positive breast cancer patients.

[0135] The single task model combining image and clinical features further refined stratification: in the same cohort, high-risk patients treated with ELT (HR = 0.644; 95% CL 0.430-0.965; p = 0.032) had a 3.68% absolute benefit (FIG. 9E), while low-risk patients (HR = 0.575; 95% CI: 0.247-1.342; p = 0.196) had a 0.87% benefit (FIG. 9C). However, p-interaction for ELT benefit in high- vs. low-risk groups was not significant.4580147550V 1

[0136] Stratification with the single task image only model provided similar results, indicating that high-risk patients (HR = 0.67; 95% CI: 0.442-1.017; p = 0.058) experienced a 3.3% 10-year absolute benefit, whereas low-risk patients (HR = 0.581; 95% CI: 0.271-1.247; p = 0.159) had a 0.9% benefit. Although not statistically significant, the larger benefit in high-risk patients highlights the potential for identifying those who may gain most from ELT.

[0137] Overall, both the single task and multi-task models effectively stratified patients by their likelihood of deriving meaningful ELT benefit. These results were more pronounced among node positive, mastectomy, younger, and HER2 negative subgroups.3. Discussion

[0138] Building on recent advancements in pathology foundation models and advanced algorithms (see references 38,44-49), Al systems leveraging unannotated H&E slides and patient-level clinical data were developed to prognosticate long-term outcomes and stratify patients likely to benefit from EET. The single and multi-task models demonstrated robust prognostic performance for DR risk, with high-risk patients showing a significantly increased likelihood of recurrence, with particular effect in the multi-task architecture. Multivariate Cox analysis confirmed that the model-derived risk score was an independent prognostic factor, even after adjusting for clinical covariates. Furthermore, although the risk score model was trained specifically on DR, high-risk patients also exhibited worse outcomes for other endpoints, such as RFI, BCFI, and DFS.

[0139] Stratification analyses revealed variations in EET benefit across clinical subgroups, highlighting the potential for personalized treatment strategies. For example, among N+ patients, traditionally considered high-risk for DR, stratification by the models revealed low-risk patients that derived minimal benefit from EET. Sparing such patients (>150 patients in the cohort) from treatment could reduce overtreatment while maintaining efficacy in high-risk groups.

[0140] The excellent performance of the models provided herein supports their adaptability in clinical settings. The models can function as a standalone tool in settings where genomic tests are unavailable or impractical, or it can complement existing genomic assays to provide an additional layer of validation and confidence in treatment recommendations. The Al models based on image analysis may be more economical and have faster turn around time than genomic tests, e.g., NGS or other high throughput expression techniques. By integrating4680147550V 1into clinical workflows — either independently or alongside genomic tests — the models provided herein have the ability to improve treatment stratification, enhance clinicians’ decision-making processes, and ultimately advance patient outcomes.4. References (with respect to numbering in this example)1 Siegel, R. L., et al. Cancer statistics, 2024. CA: A Cancer Journal for Clinicians 74, 12-49 (2024).2 Tamoxifen for early breast cancer: an overview of the randomised trials. The Lancet 351, 1451-1467 (1998).3 Pan, H. et al. 20- Year Risks of Breast-Cancer Recurrence after Stopping Endocrine Therapy at 5 Years. N Engl J Med 377, 1836-1846 (2017).4 Gnant, M. et al. Duration of Adjuvant Aromatase-Inhibitor Therapy in Postmenopausal Breast Cancer. N Engl J Med 385, 395-405 (2021).5 Mamounas, E. P. et al. Use of letrozole after aromatase inhibitor-based therapy in postmenopausal breast cancer (NRG Oncology / NS ABP B-42): a randomised, double-blind, placebo-controlled, phase 3 trial. Lancet Oncol 20, 88-99 (2019).6 Tjan-Heijnen, V. C. G. et al. Extended adjuvant aromatase inhibition after sequential endocrine therapy (DATA): a randomised, phase 3 trial. Lancet Oncol 18, 1502-1511 (2017).7 Blok, E. J. et al. Optimal duration of extended adjuvant endocrine therapy for early breast cancer; results of the IDEAL trial (BOOG 2006-05). JNCL Journal of the National Cancer Institute 110, 40-48 (2018).8 Davies, C. et al. Long-term effects of continuing adjuvant tamoxifen to 10 years versus stopping at 5 years after diagnosis of oestrogen receptor-positive breast cancer:ATLAS, a randomised trial. Lancet 381, 805-816 (2013).9 Goss, P. E. et al. A randomized trial of letrozole in postmenopausal women after five years of tamoxifen therapy for early-stage breast cancer. N Engl J Med 349, 1793-1802 (2003).10 Goss, P. E. et al. Extending Aromatase-Inhibitor Adjuvant Therapy to 10 Years. N Engl J Med 375, 209-219 (2016).4780147550V 111 Gray, R. G. et al. (American Society of Clinical Oncology, 2013).12 Mamounas, E. P. et al. Benefit from exemestane as extended adjuvant therapy after 5 years of adjuvant tamoxifen: intention-to-treat analysis of the National Surgical Adjuvant Breast And Bowel Project B-33 trial. J Clin Oncol 26, 1965-1971 (2008).13 Mamounas, E. P. et al. Ten-year update: NRG Oncology / National Surgical Adjuvant Breast and Bowel Project B-42 randomized trial: extended letrozole therapy in early-stage breast cancer. J Natl Cancer Inst 115, 1302-1309 (2023).14 Dowsett, M. et al. Integration of Clinical Variables for the Prediction of Late Distant Recurrence in Patients With Estrogen Receptor-Positive Breast Cancer Treated With 5 Years of Endocrine Therapy: CTS5. J Clin Oncol 36, 1941-1948 (2018).15 Sparano, J. et al. Abstract GS6-03: circulating tumor cells (CTCs) five years after diagnosis are prognostic for late recurrence in operable stage II-III breast cancer. Cancer Research 78, GS6-03-GS06-03 (2018).16 Filipits, M. et al. The PAM50 risk-of-recurrence score predicts risk for late distant recurrence after endocrine therapy in postmenopausal women with endocrine-responsive early breast cancer. Clin Cancer Res 20, 1298-1305 (2014).17 Sgroi, D. C. et al. Prediction of late distant recurrence in patients with oestrogen-receptor-positive breast cancer: a prospective comparison of the breast-cancer index (BCI) assay, 21 -gene recurrence score, and H4C4 in the TransATAC study population. Lancet Oncol 14, 1067-1076 (2013).18 Wolmark, N. et al. Prognostic Impact of the Combination of Recurrence Score and Quantitative Estrogen Receptor Expression (ESRI) on Predicting Late Distant Recurrence Risk in Estrogen Receptor-Positive Breast Cancer After 5 Years of Tamoxifen: Results From NRG Oncology / National Surgical Adjuvant Breast and Bowel Project B-28 and B-14. J Clin Oncol 34, 2350-2358 (2016).19 Dubsky, P. et al. The EndoPredict score provides prognostic information on late distant metastases in ER+ / HER2- breast cancer patients. Br J Cancer 109, 2959-2964 (2013).20 Bartlett, J. M. S. et al. Breast Cancer Index and prediction of benefit from extended endocrine therapy in breast cancer patients treated in the Adjuvant Tamoxifen-To Offer More? (aTTom) trial. Ann Oncol 30, 1776-1783 (2019).4880147550V 121 Bartlett, J. M. S. et al. Breast Cancer Index Is a Predictive Biomarker of Treatment Benefit and Outcome from Extended Tamoxifen Therapy: Final Analysis of the Trans-aTTom Study. Clin Cancer Res 28, 1871-1880 (2022).22 Noordhoek, I. et al. Breast Cancer Index Predicts Extended Endocrine Benefit to Individualize Selection of Patients with HR(+) Early-stage Breast Cancer for 10 Years of Endocrine Therapy. Clin Cancer Res 27, 311-319 (2021).23 Sgroi, D. C. et al. Prediction of late disease recurrence and extended adjuvant letrozole benefit by the HOXB13 / IL17BR biomarker. J Natl Cancer Inst 105, 1036-1042 (2013).24 Mamounas, E. P. et al. Breast Cancer Index and Prediction of Extended Aromatase Inhibitor Therapy Benefit in Hormone Receptor-Positive Breast Cancer from the NRG Oncology / NS ABP B-42 Trial. Clin Cancer Res 30, 1984-1991 (2024).25 Andre, F. et al. Biomarkers for Adjuvant Endocrine and Chemotherapy in Early-Stage Breast Cancer: ASCO Guideline Update. J Clin Oncol 40, 1816-1837 (2022).26 Cardoso, F. et al. 70-Gene Signature as an Aid to Treatment Decisions in Early-Stage Breast Cancer. N Engl J Med 375, 717-729 (2016).27 van ’t Veer, L. J. et al. Gene expression profiling predicts clinical outcome of breast cancer. Nature 415, 530-536 (2002).28 Piccart, M. et al. 70-gene signature as an aid for treatment decisions in early breast cancer: updated results of the phase 3 randomised MIND ACT trial with an exploratory analysis by age. Lancet Oncol 22, 476-488 (2021).29 Rastogi, P. et al. Utility of the 70-Gene MammaPrint Assay for Prediction of Benefit From Extended Letrozole Therapy in the NRG Oncology / NS ABP B-42 Trial. J Clin Oncol 42, 3561-3569 (2024).30 van ’t Veer, L. J. et al. Selection of Patients With Early-Stage Breast Cancer for Extended Endocrine Therapy: A Secondary Analysis of the IDEAL Randomized Clinical Trial. JAMA Netw Open 7, e2447530 (2024).31 Esteva, A. et al. Prostate cancer therapy personalization via multi-modal deep learning on randomized phase III clinical trials, npj Digital Medicine 5, 71 (2022).4980147550V 132 Jiang, X. et al. End-to-end prognostication in colorectal cancer by deep learning: a retrospective, multicentre study. The Lancet Digital Health 6, e33-e43 (2024).33 Garberis, I. et al. Deep Learning Allows Assessment of Risk of Metastatic Relapse from Invasive Breast Cancer Histological Slides. bioRxiv, 2022.2011.2028.518158 (2022).34 Boehm, K. M. et al. Multimodal histopathologic models stratify hormone receptorpositive early breast cancer. bioRxiv, 2024.2002.2023.581806 (2024).35 Mondol, R. K., et al. BioFusionNet: Deep Learning-Based Survival Risk Stratification in ER+ Breast Cancer Through Multifeature and Multimodal Data Fusion. arXiv:2402.10717 (2024).36 Bankhead, P. et al. QuPath: Open source software for digital pathology image analysis. Sci Rep 7, 16878 (2017).37 Janowczyk, A., et al. HistoQC: An Open-Source Quality Control Tool for Digital Pathology Slides. JCO Clin Cancer Inform 3, 1-7 (2019).38 Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med Image Anal 81, 102559 (2022).39 Vaswani, A. et al. Attention Is All You Need. arXiv: 1706.03762 (2017).40 Zadeh, A., Chen, M., Poria, S., Cambria, E. & Morency, L.-P. Tensor Fusion Network for Multimodal Sentiment Analysis. arXiv: 1707.07250 (2017).41 Katzman, J. L. et al. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Med Res Methodol 18, 24 (2018). 42 Klein, J. & Moeschberger, M. in Statistics for Biology and Health (2003).43 Gottlieb, S. Lumpectomy as good as mastectomy for tumors up to 5 cm across. West J Med 173, 227-228 (2000).44 Vorontsov, E. et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature Medicine 30, 2924-2935 (2024).45 Lu, M. Y. et al. A visual-language foundation model for computational pathology. Nature Medicine 30, 863-874 (2024).5080147550V 146 Wang, X. et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature 634, 970-978 (2024).47 Chen, X., Xie, S. & He, K. An Empirical Study of Training Self- Supervised Vision Transformers. arXiv:2104.02057 (2021).48 Ochi, M., Komura, D. & Ishikawa, S. Pathology Foundation Models. arXiv:2407.21317 (2024).49 Vorontsov, E. et al. Virchow: A Million-Slide Digital Pathology Foundation Model. arXiv:2309.07778 (2023).50 Syed, Y. Y. Oncotype DX Breast Recurrence Score(®): A Review of its Use in Early-Stage Breast Cancer. Mol Diagn Ther 24, 621-632 (2020).51 NCCN. NCCN Guidelines Version 6.2024 Breast Cancer.52 Goyal, M. et al. A multi -model approach integrating whole-slide imaging and clinicopathologic features to predict breast cancer recurrence risk, npj Breast Cancer 10, 93 (2024).53 Wulczyn, E. et al. Deep learning-based survival prediction for multiple cancer types using histopathology images. PLOS ONE 15, e0233678 (2020).C. External Validation for late distant recurrence (LDR) model

[0141] The multi-task model for late distant recurrence (LDR) was validated against a selection of data from the TAILORx randomized clinical trial, which aimed to examine whether genes that are frequently associated with risk of recurrence for women with early-stage breast cancer could be used to assign patients to the most appropriate and effective treatment. About 6,500 samples from the trial were used to validate the model’s prognostic performance. The performance was determined for those with LDR (greater than 5 years) but also early distant recurrence, EDR (less than or equal to 5 years), as well as all subjects.

[0142] FIGS. 10A-10C show Kaplan-Meier plots for all subjects, those with early DR, and those with late DR, respectively. The separation between high and low risk groups was notable in the LDR setting (FIG. 10C), indicating that the model trained to predict LDR was accurate on the validation data set. In addition, the model provided some separation for EDR (FIG. 10B), even though the model was not trained on subjects with EDR.5180147550V 1

[0143] To test the viability of an EDR model, a training set with EDR labels was used to train various EDR models as described in sections below. Accordingly, some embodiments can predict EDR versus LDR. Such discrimination is clinically relevant as evidence supports distinct treatment approaches for early versus late distant recurrence in hormone receptorpositive breast cancer patients. For example, patients at high risk for early distant recurrence (within the first 5 years) may benefit from adjuvant chemotherapy, as demonstrated by the TAILORx trial, which showed that women with high 21 -gene recurrence scores who received chemotherapy plus hormone therapy were significantly less likely to experience distant recurrence compared to those treated with hormone therapy alone. See, e.g., Sparano JA, et al. Adjuvant chemotherapy guided by a 21-gene expression assay in breast cancer. N Engl J Med. 2018;379(2): 111-121. In contrast, patients at risk for late distant recurrence (beyond 5 years) may benefit from extended endocrine therapy rather than chemotherapy. For example, the MA.17 trial demonstrated that extended letrozole after 5 years of tamoxifen significantly improved disease-free survival (HR 0.58, p < 0.001), with node-positive patients also showing improved overall survival. See, e.g., Goss PE, et al. Randomized trial of letrozole following tamoxifen as extended adjuvant therapy in receptor-positive breast cancer: updated findings fromNCIC CTGMA.17. J Natl Cancer Inst. 2005;97(17): 1262-1271. Similarly, the ATLAS trial found that continuing tamoxifen to 10 years resulted in a 25% relative risk reduction for recurrence and a 29% relative reduction in breast cancer-related death at 15 years of follow-up. See, e.g., Davies C, et al. Long-term effects of continuing adjuvant tamoxifen to 10 years versus stopping at 5 years after diagnosis of oestrogen receptorpositive breast cancer: ATLAS, a randomised trial. Lancet. 2013 ;381 (9869): 805-816.Without being bound by theory, these differing treatment benefits are thought to reflect different biological mechanisms: early recurrences are characterized by aggressive tumor biology with sharp hazard rate peaks in the first few years, while late recurrences arise from disseminated tumor cells that escape dormancy after extended periods, a process more amenable to prolonged hormonal suppression than cytotoxic chemotherapy. See, e.g., Thomas A, et al. Late recurrence following early breast cancer. J Clin Oncol.2022;40(13): 1400-1406; Knauer M, et al. Late recurrences in early breast cancer: for whom and how long is endocrine therapy beneficial? Breast Care (Basel). 2014;9(2):97-100.

[0144] To further assess the clinical reliability of DR risk predictions, model calibration was evaluated at clinically relevant time horizons. FIGS. 10D-10F illustrate calibration plots showing observed versus predicted 10-year distant recurrence risk using three different model 5280147550V 1architectures: an image-only model (FIG. 10D, “image-only”), a multimodal model integrating histopathology and clinical variables (FIG. 10E, “Multimodal”), and a multimodal-multitask model incorporating an auxiliary task to enhance feature learning (FIG. 10F). At the 10-year horizon, all three models demonstrated acceptable overall calibration, with mean predicted risks aligned with the observed event rate (5.5%) and calibration-in-the-large values near zero, indicating minimal global bias. Among the three models, the multimodal multi-task model (FIG. 10F) showed the most favorable calibration profile, with the lowest expected calibration error and the smallest median maximum bin error (0.021), indicating improved local calibration. Calibration slopes for all models were close to 1, reflecting appropriate risk scaling across the risk range, with comparable variability across repeated splits. Overall, calibration performance was acceptable for all models, with the multimodal multi-task model providing the most stable bin-level agreement between predicted and observed risks. The observed improvement in local calibration with the multimodal multi-task model is consistent with the use of auxiliary supervision based on lowest BMD T-score (see, e.g., Table 1-2 and related discussion above), a clinically relevant characteristic related to baseline recurrence risk, which may contribute to more stable risk estimation.D. Predicting risk of early recurrence

[0145] Since the TAILORx data includes samples from subject that developed EDR, such training data was used to train an early distant recurrence predictor. Architectures and techniques described above can be used for the EDR model, including using multi-task techniques and image and clinical features. In addition, molecular data / features (e.g., sequence analysis of DNA and / or RNA of various biomarkers) can be used in prediction models. Such molecular data can also be used for LDR. Example biomarkers are provided in tables below.1. Predictor model

[0146] FIG. 11 illustrates various predictor models using images, clinical features, and molecular features. The early distant recurrence predictor (EDRP) can provide a risk score (e.g., a numerical value between 0-1 or 0-100). The risk score can be compared to a threshold that discriminates between subjects with high risk (e.g., greater than the threshold) and subjects with low risk (e.g., equal to or less than the threshold).5380147550V 1

[0147] Example image and clinical features are described above. In this example, clinical features of tumor size, menopausal status, age, and grade are shown. The example molecular features show various assays that use expression levels of various genes. The HVG assay is for highly variable genes. Example genes in such panels are provided below.

[0148] As examples, such molecular features may include point mutations, polymorphisms, deletions, insertions, substitutions, translocations, fusions, breaks, duplications, amplification, repeats, copy numbers (including determining copy number alterations (CNA); also referred to as copy number variation or CNV), transcript levels (expression levels), or any combination thereof. Information in sequence reads can also be used to determine genomic signature, including without limitation tumor mutational burden (TMB), microsatellite instability (MSI), human leukocyte antigen (HLA) genotype, mismatch repair deficiency, homologous recombination deficiency (HRD), homologous recombination repair (HRR) deficiency, loss of heterozygosity (LOH), or any combination thereof. Such features can be obtained by analysis of nucleic acids (DNA and / or RNA) extracted from patient samples, such as tumors and / or blood. In embodiments, next-generation sequencing (NGS) is used to assess the nucleic acids. Other biomarkers described herein can be used in model training as desired.2. Multi-task

[0149] As with the LDR predictor, the EDR predictor can include various tasks.Accordingly, the EDRP can be a multi-task model. Such a multi-task model can output a risk score (e.g., as shown in FIG. 11) as one or more regression outputs of regression model / layer(s) and also output one or more classifications from classification model / layer(s). The classification(s) can be viewed as the auxiliary task(s). A non-limiting example classification task can be the Oncotype DX (ODX) classification of high risk or low risk, which can be used as classification labels for the training set.3. EDRP and LDRP pipeline, including advanced EDRP

[0150] In some embodiments, an early distant recurrence model and a late distant recurrence model can be combined. Additionally, various early distant recurrence models can be used, which may also be combined, e.g., to operate on different segments of subjects. In this section, the “EDRP model” refers to a model employing image and clinical features.5480147550V 1

[0151] Another model architecture combines image and clinical features with molecular features determined using next generation sequencing (NGS). Such a model is referred to as “NGSPred” in this section. In embodiments, other types of assays such as PCR or microarrays can be used to provide molecular data, in addition to or in place of NGS.

[0152] “Advanced EDRP” refers to a model architecture employing a combination of EDRP and NGSPred. If the results for the EDRP are in a realm where similar accuracy is achieved as NGSPred (e.g., a risk below a threshold), then EDRP can be used. A goal of the advanced EDRP model is to provide a more cost-effective and efficient model while preserving accuracy. Image analysis is quicker and less expensive than NGS. Therefore advanced EDRP architecture can rely upon the EDRP model where the NGS component is not required for the prediction.a) EDRP & NGS pipeline

[0153] FIG. 12A shows an example prediction pipeline including EDRP and LDRP, as well as illustrating an advanced EDRP model. As shown, a sample is scanned and image(s) are created. This example pipeline can use two EDR models, with a first EDR model (e.g., image-based) used for an initial screen, and only if a sufficient risk of recurrence is present is a second EDR model used (e.g., sequencing based). In this example, image features are first used in the EDRP model to determine a first risk score. If the first risk score is equal to or below a redirect threshold, then the first risk score is used to determine a recommended course of action, which may include using an LDRP model. If the first risk score is above the redirect threshold, then a sequencing-based EDR model is used to determine a recommended course of action. The recommended course of action may include, e.g., treatment, no treatment, or additional testing (e.g., low risk indicates no treatment for early recurrence but determine risk of late recurrence, whereas high risk of early recurrence indicates benefit from chemotherapy).

[0154] At stage 1210, the EDRP model (image and clinical features) generates a risk score. The EDRP model can be used to determine whether a subject is high risk or low risk at stage 1220. The EDRP risk score can also be used to determine which samples to direct to the NGSPred model. For example, subjects with an EDRP risk score above a redirect threshold, or subjects without prediction from the EDRP model (e.g., image of insufficient quality, lack of sample, or model is indeterminate or has insufficient confidence), can be redirected to5580147550V 1NGSPred. For subjects with an EDRP risk score equal to or below the redirect threshold, the pipeline can proceed to stage 1220, which uses the EDRP risk score to determine the course of action. The EDRP model can correspond to one type of early recurrence logic.

[0155] At stage 1220, if the EDRP model determines a high risk, the model can recommend chemotherapy for the subject and no further analysis is required. However, if the EDRP component determines low risk at 1220, the subject’s measured data may be passed to a late recurrence model to determine whether a different treatment should be performed. . Thus, if early recurrence is predicted to be unlikely by EDRP the pipeline may continue to assess late recurrence at stage 1240.

[0156] The redirect threshold can be different than the threshold used to differentiate low risk and high risk. For example, depending on the value of the EDRP threshold relative to the redirect threshold, all of the samples sent to stage 1220 can be predicted to be low risk (e.g., if the redirect threshold is less than the EDRP threshold).

[0157] In some embodiments, the redirect threshold is determined based on a specified percentage of subjects in the training set that are above the redirect threshold. For example, the redirect threshold can be set at the risk score such that 40% of the subjects in the training set are above that risk score. Other percentages can be used such as 50%, 45%, 35%, 30%, 35%, 20%, 15%, 10%, and 5%. Additionally, the samples directed to NGSPred can have an upper limit (e.g., a maximum risk score as may also be determined using percentages of the training set) as well, such that an intermediate range is directed to NGSPred.

[0158] At stage 1230, for the subjects (samples) directed to NGSPred, a new risk score can be determined by NGSPred and compared to a NGSPred threshold to determine whether those subjects are predicted as low or high risk. The EDRP threshold and the NGSPred threshold for the respective segments of subjects can be the same or be different values. If a subject is determined to be high risk via either EDRP or NGSPred, then they can be treated accordingly, e.g., with chemotherapy, and no further analysis is required. The NGSPredmodel can correspond to another type of early recurrence logic.

[0159] At stage 1240, for subjects that are determined to be low risk via both EDRP and NGSPred, their samples can be further evaluated using an LDRP to determine if there is a low or a high risk. If a subject is determined to be high risk via LDRP, the system may recommend treatment accordingly, e.g., using extended endocrine therapy.5680147550V 1b) NGS-only pipeline

[0160] In some embodiments, the early distant recurrence may be determined solely using an sequencing-based model (e.g., an NGS-based model) that determines the risk score based on molecular features.

[0161] FIG. 12B shows an example pipeline using a sequencing-based model. In the exemplary workflow 1250 shown in the figure, a tumor specimen 1259 is provided. Nucleic acids can be extracted from the tumor for NGS analysis, and the NGS data is provided to a machine learning model to predict early recurrence 1251. Subjects found to be at high risk of early recurrence 1252 can be recommended for chemotherapy 1253 and no further analysis is required. For subjects that are determined to be low risk of early recurrence 1254, their samples can be further evaluated using a digital pathology (DP) LDRP model 1256 to determine if there is a low or a high risk of late recurrence. If a subject is determined to be high risk 1256 via the LDRP model 1255, then they can be treated accordingly, e.g., using extended endocrine therapy 1257. The LDPR model 1256 can be a multi-task model that determines the risk based on an image and clinical features as described herein.c) Example results for different redirect thresholds

[0162] FIG. 13 depicts different distributions of subjects from the training set for different redirect thresholds for redirecting (reflexing) subjects from EDRP to NGSPred. In each plot, the horizontal axis is a normalized risk score between 0-100, and the vertical axis is a count of subjects having a particular risk score. The heading for each plot provides the minimum and maximum EDRP risk score used for redirecting to the NGSPred model. The EDRP reflexed segment are ones with the risk score within the minimum and maximum thresholds. The samples within the EDRP reflexed segment are then reevaluated using NGSPred; these NGSPred risk scores have a larger spread than the previous EDRP scores for this segment.

[0163] In the scenario where 60%-100% of EDRP predictions are reflexed to NGSPred, a first step can be to calculate the normalized risk scores for EDRP or potentially for both models. Then, the top 40% of the normalized EDRP risk scores are replaced with the corresponding normalized risk scores from NGSPred after this model is run for the redirected segment. This creates a new set of combined risk scores for the population — 60% from EDRP and 40% from NGSPred. A threshold can then be applied to these combined scores to classify patients into low- and high-risk groups, e.g., a same threshold used for the entire5780147550V 1group. The 40% reflexed (redirected) cases may include both low- and high-risk patients, as they can span the full distribution.4. Different models for different feature sets

[0164] The accuracy for different models using different feature sets was compared, including (1) ODX (also referred to as RS), (2) EDRP that uses image and clinical features, (3) NGSPred that uses image and clinical features, as well as molecular features, and (4) advanced EDRP as a combination of EDRP and NGSPred.

[0165] FIGS. 14A-14C show comparisons among the different models. To have a fair comparison, the plots in FIGS. 14A-14B have similar negative and positive splits along the horizontal axis, which corresponds to the percentage positive, where positive indicates high risk. A respective threshold that provides the specified percentage positive is used for each respective model. For each percentage, a different threshold would be used. Such thresholds are not necessarily the chosen thresholds for clinical applications but are used for comparing the models. The data in FIGS. 14A-14C corresponds to using a redirect threshold of 60%, e.g., all greater than a minimum threshold of 60%.

[0166] FIG. 14A shows the sensitivity at different positive (high risk) percentages. Such sensitivity values are for the entire training set. As can be seen, the RS (ODX) 1403 and EDRP 1401 provide similar accuracy. The advanced EDRP technique 1402 provides increased accuracy relative to RS 1403 and EDRP 1401, as a result of using NGSPred for some of the samples. And NGSPred 1404 provided the highest accuracy but is the more time consuming and expensive assay as it always uses NGS. Interestingly, the advanced EDRP technique 1402 provides a comparable accuracy as NGSPred 1404 when a lower threshold is used for discriminating low and high risk, resulting in a higher percentage positive. The EDRP 1401 and NGSPred 1404 models have similar accuracy at 14 percent positive for high risk.

[0167] FIG. 14C provides further accuracy values for RS (ODX). These results show that embodiments of the present disclosure provide increased accuracy. The Precision column provides the percentage of patients that are positive for a given threshold. At 12% positive the advanced EDRP technique provides a higher sensitivity. Additionally, image-based techniques as provided herein may be faster and cheaper than molecular analysis.5880147550V 15. Feature sets

[0168] As described above, various molecular features can be used as feature sets for the molecular models. Such molecular features can be for various gene panels, e.g., for expression levels of genes in a gene panel. Various embodiments can use one or more of the following gene panels in their entirety or may include one or more genes from one or more of the following gene panels. For example, various embodiments can include one or more molecular features from at least 1, 2, 3, 4, 5, 6, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, or more of the genes from respective gene panel(s) below or collectively from all the gene panels.Table 3

[0169] Table 3 corresponds to the OncotypeDx panel. See Cobleigh MA, et al; Tumor Gene Expression and Prognosis in Breast Cancer Patients with 10 or More Positive Lymph Nodes. Clin Cancer Res 15 December 2005; 11 (24): 8623-8631Table 4

[0170] Table 4 corresponds to the BCI panel. See Jerevall PL, et al. Prognostic utility of HOXB13:IL17BR and molecular grade index in early-stage breast cancer patients from the Stockholm trial. Br J Cancer. 2011 May 24; 104(11): 1762-9.Table 55980147550V 1

[0171] Table 5 corresponds to the MammaPrint panel. See Gias AM, Floet al. Converting a breast cancer microarray signature into a high-throughput diagnostic test. BMC Genomics.2006 Oct 30;7:278.Table 6

[0172] Table 6 corresponds to the EndoPredict panel. See Filipits M, et al. A new molecular predictor of distant recurrence in ER-positive, HER2-negative breast cancer adds independent information to conventional clinical risk factors. Clin Cancer Res. 2011 Sep 15;17(18):6012-20.Table 7

[0173] Table 7 corresponds to the PAM50 panel, SeeParker JS, et al. Supervised risk predictor of breast cancer based on intrinsic subtypes. J Clin Oncol. 2009 Mar 10;27(8):l 160-7.Table 86080147550V 1

[0174] Table 8 corresponding to a high variable gene (HVG) panel based on expression value variation.V. METHODS

[0175] As described, a machine learning framework can be used to perform a main regression task and one or more auxiliary tasks. The machine learning framework can be executed by a computer system. The computer system can receive an image of a biological sample and use the machine learning framework to perform the main regression task as well as the one or more auxiliary tasks, which may include regression tasks for predicting continuous variables or classification tasks for categorical outcomes.A. Multi-task machine learning model

[0176] FIG. 15 illustrates an example flow of a process for using a multi-task machine learning model in digital pathology, according to embodiments of the present disclosure. The steps may be performed by a computer system, such as computer system 10 in FIG. 20.

[0177] At block 1502, the computer system accesses an image of a biological sample. The biological sample is from a patient having a medical condition. In addition, the biological sample is stained using a pathology stain such hematoxylin and eosin (H&E). The medical condition can correspond to a disease or disorder such as cancer.

[0178] In various examples, the cancer may be an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumor (GIST), glioblastoma, head and neck squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), non-small cell lung cancer (NSCLC), lung small cell6180147550V 1cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non-epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma. In some instances, the biological sample may be from a tumor, such as the tumor is a primary tumor or a metastatic tumor. The tumor may be a tumor of the myeloid, breast, bile ducts, colon, rectum, female genital tract, stomach, esophagus, gastrointestinal stromal cells, small intestine, brain, mouth, sinuses, nose, throat, blood, liver, nervous system, lung, lymph, male genital tract, pleura, skin, plasma cells, neuroendocrine cells, B-cells, T-cells, ovary, pancreas, pituitary gland, spinal cord, prostate, peritoneum, large intestine, soft tissue, connective tissue, fat tissue, thymus, thyroid, or eye. The primary tumor can be a tumor of the bladder, breast, colon, rectum, endometrium, uterus, ovary, female genital tract, kidney, blood, liver, lung, skin, lymph, pancreas, prostate, or thyroid.

[0179] The biological sample can comprise tumor cells. For example, the biological sample can be formalin-fixed paraffin-embedded (FFPE) tissue from a tumor biopsy. In embodiments, a threshold for tumor content in the biological sample can be set in order to perform image the analysis. For example, the threshold can be at least 10%, 15%, 20%, 25%, 30%, 40%, 50%, 60%, 70%, 75%, 80% or 90% tumor. In some cases, molecular analysis is also performed (see, e.g., discussion of NGSPred and advanced EDRP above). The threshold for image analysis and molecular analysis (e.g., NGS) may be the same or different. In some embodiments, tumor tissue is micro-dissected prior to molecular analysis.

[0180] At block 1504, the computer system generates one or more feature vectors from pixels of the image. A feature extractor (e.g., a neural network such as the feature extractor 104 in FIG. 1 and the feature extractor 304 in FIG. 3) can generate the one or more feature vectors. The pixels may correspond to at least a portion of the image. For example, the feature vectors can correspond to different patches. That is, the pixels or patches may be for an entirety of the image or only a portion of the image. The one or more feature vectors can further include one or more patient characteristics. At least one of the one or more patient characteristics can be selected from a group consisting of a patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone6280147550V 1mineral density T-score, a prior treatment, or treatment type. This step effectively captures the underlying characteristics of the image to prepare the image for downstream tasks.

[0181] At block 1506, the computer system provides the one or more feature vectors to an input layer of a neural network within the machine learning framework (e.g., machine learning framework 316 in FIG. 3). This framework can be used for multi-task prediction, leveraging the extracted features to simultaneously address multiple objectives.

[0182] At block 1508, the computer system processes the one or more feature vectors to generate either a shared embedding or a set of task specific embeddings representing the image. The set of task specific embeddings can include a primary regression embedding and one or more auxiliary embeddings. The one or more auxiliary embeddings can include one or more auxiliary classification embeddings for classification tasks and / or one or more auxiliary regression embeddings for regression tasks. Depending on the auxiliary tasks, the system may generate both classification and regression embeddings or only one type. For example, if the auxiliary tasks only involve auxiliary regression tasks, the system may create only the primary regression embedding and auxiliary regression embeddings and not auxiliary classification embeddings. The neural network processes the one or more feature vectors.

[0183] The neural network may be a multi-instance learning model that receives the one or more feature vectors, and any additional feature vectors for other portions of the image, and generates the embeddings representing the image. For instance, the neural network may include a local context multi-instance learning model, such as the local context multi-instance learning model in FIG. 4. So, the one or more feature vectors can include a first feature vector from first pixels of a target portion (e.g., a first patch) of the image and additional feature vectors from additional pixels of additional portions (e.g., additional patches) of a surrounding area of the target portion. The surrounding area can be within a pixel distance of the target portion.

[0184] Accordingly, the one or more feature vectors can comprise a first feature vector from first pixels can comprise a first portion of the image. A second feature vector can be generated from second pixels comprising a second portion of the image. The neural network can process the first feature vector and the second feature vector to generate the primary regression embedding. The primary regression embedding represents an aggregation of the first feature vector and the second feature vector.6380147550V 1

[0185] The machine learning framework may additionally or alternatively include a global context multi-instance learning model, such as the local context multi-instance learning model 452 in FIG. 4. So, the one or more feature vectors can include a first feature vector from first pixels of a target portion (e.g., a first patch) of the image and additional feature vectors from additional pixels of additional portions (e.g., additional patches) of the image. The additional portions can include an entirety of the image. The neural network can process the first feature vectors and the additional feature vectors to generate the primary regression embedding that represents an aggregation of the feature vectors.

[0186] At block 1510, the computer system performs the main (primary) task by processing the shared embedding or the primary regression embedding by a primary machine learning regression model to obtain a primary continuous variable. The primary continuous variable may be a risk score associated with a time-based outcome for the patient. For example, the risk score may predict outcomes such as a survival of the patient, time on treatment for the patient, or recurrence of the medical condition. See section IV.A and section IV.B.2.a. The primary continuous variable can be time-dependent. The machine learning framework can be trained using a regression loss function (e.g., the time-dependent loss function) to optimize the primary continuous variable prediction for survival or other relevant clinical outcomes. The computer system may output a report including the primary continuous variable. In addition, the computer system may provide a diagnosis, prognosis, and / or theranosis based on the primary continuous variable. For example, the computer system may provide a prediction of a survival time or a time on treatment for the patient.

[0187] At block 1512, the computer system processes the embeddings to perform one or more auxiliary tasks. As examples, the one or more auxiliary tasks can include a classification task and / or an auxiliary regression task, e.g., as described for auxiliary tasks 322 in FIG. 3.

[0188] For classification tasks 1512a, the shared embedding or the one or more auxiliary classification embeddings are processed by a classification model to obtain one or more auxiliary properties of the biological sample. The one or more auxiliary properties can be selected from a second group consisting of (a) the one or more classifications of the biological sample and (b) the one or more auxiliary continuous variables of the biological sample. The machine learning framework can be trained using a regression loss function for the primary continuous variable, and one or more auxiliary loss functions for the one or more6480147550V 1auxiliary properties. The one or more classifications can comprise patient characteristics, e.g., as described for FIG. 3 and in the Example Results section. As examples, the least one of the patient characteristics can selected from a group consisting of a patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone mineral density T-score, a prior treatment, or treatment type. As an example, the prior treatment can comprise tamoxifen. The one or more classifications can include a status of at least one biomarker. The biomarker can be associated with one or more treatment. The status can include an expression level or a mutation. The one or more classifications can include more than one biomarker status of multiple biomarkers, e.g., multiple genes. The biomarker statuses can include a presence, absence or level of one or more biomarker.

[0189] For regression tasks 1512b, the shared embedding or the one or more auxiliary regression embeddings are processed by a machine learning regression model to predict one or more auxiliary continuous variables of the biological sample. Exemplary auxiliary continuous variables include the percentage of tumor cells in the biological sample. The auxiliary tasks can include one or more auxiliary classification tasks 1512a, one or more auxiliary regression tasks 1512b, or any useful combination thereof. The report generated and output by the computer system may additionally include the one or more auxiliary continuous variables and / or the one or more auxiliary classifications.

[0190] The machine learning framework can employ different loss functions for its tasks. For example, a time-dependent loss function (e.g., a Cox partial likelihood function) can be used to train the main task focused on survival or other time-to-event predictions, while auxiliary classification tasks can be trained using classification loss functions (e.g., crossentropy loss), and auxiliary regression tasks can be trained using regression loss functions (e.g., mean squared error). During training and optimization, different weights can be assigned to the loss functions corresponding to each task. For instance, a higher weight can be given to the primary task to prioritize it compared to the auxiliary tasks.

[0191] In some embodiments, a report can be provided (e.g., as described in more detail elsewhere). The report can include the primary continuous variable and / or a patient category derived from the primary continuous variable. The patient category can be whether the patient is in a low risk group or a high risk group based on the primary continuous variable. The patient category can be determined by comparing the primary continuous variable to a6580147550V 1threshold. The threshold can be determined based on a distribution of the primary continuous variable for a set of patients having the medical condition.

[0192] The report can recommend providing a particular treatment responsive to the patient being within the high risk group, e.g., as described for FIG. 3 and in the Example Results section. The particular treatment can be provided / administered to the patient. The report can recommend not providing either a particular treatment or any treatment responsive to the patient being within the low risk group. In some embodiments, different treatments are provided for different predictions. As a non-limiting example, in the case of hormonepositive breast cancer, the report may recommend chemotherapy for high risk of early recurrence or extended endocrine therapy for low risk of early recurrence but high risk of late recurrence.

[0193] In some implementations, the particular treatment includes extended endocrine therapy (EET), e.g., extended letrozole therapy (ELT), when the medical condition is a hormone positive breast cancer. In such an instance, the one or more auxiliary properties can include the one or more classifications of the biological sample, where the one or more classifications include at least one selected from a bone mineral density T-score and a prior tamoxifen use. See, e.g., the description in the Example Results section.

[0194] In other implementations, where the medical condition is lung cancer, the particular treatment includes an immunotherapy such as an immune checkpoint inhibitor, which includes several classes of agents targeting different immune checkpoints. Exemplary immune checkpoint inhibitors include pembrolizumab, nivolumab, cemiplimab, dostarlimab, retifanlimab, and toripalimab, atezolizumab, durvalumab, and avelumab, ipilimumab, tremelimumab, relatlimab, and any useful combination thereof. More recently, relatlimab, a LAG-3 inhibitor, has been approved in combination with nivolumab. Checkpoint inhibitors are approved for treatment of various cancers given certain molecular marker status. As a non-limiting example, pembrolizumab may be prescribed for PD-L1 positive lung cancer. In such an instance, the one or more auxiliary properties can include the one or more classifications of the biological sample, where the one or more classifications include one or more biomarker statuses. The one or more biomarker statuses can include an expression status of at least one of PD-L1 (Programmed Death Ligand 1), PD-1 (Programmed Death- 1), CTLA-4 (Cytotoxic T-Lymphocyte Associated Protein 4), LAG-3 (Lymphocyte Activation Gene-3), TIM-3 (T-cell Immunoglobulin and Mucin Domain-3), TIGIT (T-cell6680147550V 1Immunoreceptor with Ig and ITIM domains), Tumor Mutational Burden (TMB), Microsatellite Instability-High (MSI-H) / Mismatch Repair Deficiency (dMMR), Tumor-Infiltrating Lymphocytes (TILs), VISTA (V-domain Ig Suppressor of T-cell Activation), B7-H3 (CD276), B7-H4, IDO1 (Indoleamine 2,3-dioxygenase), BTLA (B and T Lymphocyte Attenuator), and a combination thereof.

[0195] For embodiments where an outcome includes a recurrence (e.g., as described in section IV.B-D), the recurrence can be an early distant recurrence or a late distant recurrence, or the outcome can comprise a determination for the early distant recurrence and the late distant recurrence. The primary machine learning regression model includes an early distant recurrence model and a late distant recurrence model, e.g., as described for FIG. 12A. The primary continuous variable can be a time to distant recurrence (e.g., early or late distant recurrence) and the outcome can be distant recurrence of the medical condition. The one or more auxiliary properties can include bone mineral density T-score. The one or more auxiliary properties can further include prior treatment with tamoxifen.

[0196] The early distant recurrence model can include a first EDR model and a second EDR model, e.g., as described for FIG. 12A. The first EDR model (e.g., EDRP described above) can be used to determine the recurrence if a first risk score from the first EDR model is below a threshold (e.g., a redirect threshold), and the second EDR (e.g., NGSPred) model can be used to determine the recurrence if the first risk score from the first EDR model is greater than the threshold. The first EDR model can process the one or more feature vectors from the pixels, and the second EDR model can process sequencing data measured from the biological sample or another biological sample of the patient.

[0197] In some embodiments, early distant recurrence predictions are made using molecular analysis only, e.g., as described for FIG. 12B. The sequencing data can be obtained by next-generation sequencing (NGS). Optionally, the NGS can include whole genome sequencing (WGS), whole exome sequencing (WES), whole transcriptome sequencing (WTS), or a combination thereof. For example, WGS, WES, or targeted gene panel analysis can be performed on genomic DNA, and WTS or targeted transcript panel analysis can be performed on mRNA. As desired, the molecular analysis can be performed for genomic DNA, mRNA transcripts, or a combination thereof. The genomic DNA and mRNA data can be processed by machine learning models to predict risk of early recurrence. If risk of early6780147550V 1recurrence is low, the risk of late recurrence may be determined using image analysis as provided herein.

[0198] The computer system can further train one or more models of the machine learning framework using training samples of patients having known outcomes. Thus, method 1500 can be performed as part of a a training process where the predicted outcome can be compered to a known outcome to determine a portion of an error / loss term. The known outcomes can be for (a) a primary continuous variable and (b) one or more classifications or one or more auxiliary continuous variables. A loss function can be determined using differences between predicted outcomes and the known outcomes.B. Multiple models for early distant recurrence

[0199] In some embodiments, a subject can be screened for early distant recurrence using a first EDR model (e.g., EDRP) and only if a sufficient risk is measured is a second EDR model (e.g., sequencing-based) used. The first EDR model can be cheaper and faster and thus the initial screening, while the second EDR model can be more accurate. As a non-limiting example, the first EDR model may be an image only model whereas the second EDR model comprises molecular analysis such as NGS. Aspects for how the method of FIG. 15 are performed can be applied to methods described in this section, e.g., a multi-task machine learning model may be used. The late distant recurrence model and / or any of the early distant recurrence models can be selected from any of the models described herein.

[0200] FIG. 16 illustrates an example flow of a process for using a machine learning model for early distant recurrence in digital pathology, according to embodiments of the present disclosure. The steps may be performed by a computer system, such as computer system 10 in FIG. 20. All of the variations for FIG. 15 can be performed for the method of FIG. 16.

[0201] At block 1602, the computer system accesses an image of a biological sample, as described in block 1502 in FIG. 15.

[0202] At block 1604, the computer system generates one or more feature vectors from pixels of the image, as described in block 1504 in FIG. 15.

[0203] At block 1606, the computer system provides the one or more feature vectors to an input layer of a neural network, as described in block 1506 in FIG. 15.6880147550V 1

[0204] At block 1608, the computer system processes the one or more feature vectors to generate an embedding. The embedding may be a shared embedding or a set of task specific embeddings. The computer system processes the one or more feature vectors to generate the embedding as described in block 1508 in FIG. 15.

[0205] At block 1610, the computer system processes the embedding by a machine learning regression model to obtain a primary continuous variable associated with an outcome for the patient. The outcome can include a recurrence of the medical condition. The outcome can be a determination for an early distant recurrence. The machine learning regression model can include an early distant recurrence model that includes a first EDR model and a second EDR model.

[0206] Processing the embedding can include generating, using the one or more feature vectors, a first risk score using the first EDR model, comparing the first risk score to a threshold, determining, by the first EDR model, the recurrence responsive to the first risk score from the first EDR model being below a threshold, and determining, by the second EDR model, the recurrence responsive to the first risk score from the first EDR model being greater than the threshold.

[0207] The outcome can include a determination for the early distant recurrence and a late distant recurrence. The machine learning regression model can include the early distant recurrence model and a late distant recurrence model. The late distant recurrence model can be invoked when the early distant recurrence indicates low risk. The second EDR model can process sequencing data measured from the biological sample or another biological sample of the patient.C. Late distant recurrence responsive to early recurrence determination

[0208] A late distant recurrence model can be used after a determination by an early distant recurrence model, which may take various forms. Aspects for how the method of FIG. 15 are performed can be applied to methods described in this section, e.g., a multi-task machine learning model may be used. The late distant recurrence model and / or the early distant recurrence model can be selected from any of the models described herein.

[0209] FIG. 17 illustrates an example flow of a process for using a machine learning model for late distant recurrence in digital pathology, according to embodiments of the present6980147550V 1disclosure. The steps may be performed by a computer system, such as computer system 10 in FIG. 20. All of the variations for FIG. 15 can be performed for the method of FIG. 17.

[0210] At block 1702, the computer system accesses an image of a biological sample, as described in block 1502 in FIG. 15.

[0211] At block 1704, the computer system generates one or more feature vectors from pixels of the image, as described in block 1504 in FIG. 15.

[0212] At block 1706, the computer system provides the one or more feature vectors to an input layer of a neural network, as described in block 1506 in FIG. 15.

[0213] At block 1708, the computer system processes the one or more feature vectors to generate an embedding. The embedding may be a shared embedding or a set of task specific embeddings. The computer system processes the one or more feature vectors to generate the embedding as described in block 1508 in FIG. 15.

[0214] At block 1710, the computer system processes the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient. The outcome can be a determination for late distant recurrence of the medical condition. The machine learning regression model can include a late distant recurrence model used to determine the recurrence based on a risk score from an early distant recurrence model being below a threshold. The early distant recurrence model can use sequencing data from the biological sample or other biological sample of the patient. The early distant recurrence model can use image data of the biological sample or another biological data of the patient. The image data can include the one or more feature vectors.

[0215] In some implementations, the computer system can obtain the sequencing data and process the sequencing data by the early distant recurrence model to determine the risk score. The sequencing data can be obtained using next generation sequencing.

[0216] In some embodiments, the computer system can process the embedding by an auxiliary machine learning model of the machine learning framework to obtain one or more auxiliary properties of the biological sample. The one or more auxiliary properties can be from a group consisting of (a) the one or more classifications of the biological sample and (b) the one or more auxiliary continuous variables of the biological sample.7080147550V 1D. Early distant recurrence using sequencing data

[0217] A late distant recurrence model can be used after a determination by an early distant recurrence model, which may take various forms. Aspects for how the method of FIG. 15 are performed can be applied to methods described in this section, e.g., a multi-task machine learning model may be used.

[0218] FIG. 18 illustrates an example flow of a process for using a machine learning model for early distant recurrence using sequencing data in digital pathology, according to embodiments of the present disclosure. The steps may be performed by a computer system, such as computer system 10 in FIG. 20. All of the variations for FIG. 15 can be performed for the method of FIG. 18.

[0219] At block 1802, the computer system accesses sequencing data measured from a biological sample of a patient having a medical condition. The sequencing data can be accessed as described in FIG. 15.

[0220] At block 1804, the computer system generates one or more feature vectors from the sequencing data. A feature extractor (e.g., a neural network such as the feature extractor 104 in FIG. 1 and the feature extractor 304 in FIG. 3) can generate the one or more feature vectors.

[0221] At block 1806, the computer system provides the one or more feature vectors to an input layer of a neural network of a machine learning framework, as described in block 1506 in FIG. 15.

[0222] At block 1808, the computer system processes the one or more feature vectors to generate an embedding. The embedding may be a shared embedding or a set of task specific embeddings. The computer system processes the one or more feature vectors to generate the embedding as described in block 1508 in FIG. 15.

[0223] At block 1810, the computer system processes the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient. The outcome can include a recurrence of the medical condition. The outcome can be a determination for an early distant recurrence. The machine learning regression model can include an early distant recurrence model.7180147550V 1Processing the embedding can include generating, by the early distant recurrence model using the one or more feature vectors, a first risk score.

[0224] At block 1812, the computer system compares the first risk score to a threshold. The primary continuous variable can correspond to the first risk score.

[0225] At block 1814, the computer system determines a first treatment is to be administered responsive to the first risk score being above the threshold. The first treatment can include chemotherapy.

[0226] At block 1816, responsive to the first risk score being below the threshold, the computer system generates, by a late distant recurrence model, another primary continuous variable associated with a late distant recurrence for the patient. The late distant recurrence model can be selected from any of the models described herein.VI. REPORT GENERATED USING DIGITAL PATHOLOGY AND / OR MOLECULAR PROFILING

[0227] A digital pathology analysis and / or molecular profiling approach (e.g., sequence analysis) can provide a method for selecting treatments for an individual that could favorably change the clinical course of a medical condition, including without limitation cancer. These analyses provide a personalized approach to selecting treatments that are more likely to benefit a medical condition such as cancer. The methods described herein can be used to guide treatment in any desired setting, including without limitation the front-line / standard of care setting, or for patients with poor prognosis, such as those with metastatic disease or those whose cancer has progressed on standard front line therapies, or whose cancer has progressed on previous chemotherapeutic or hormonal regimens. The cancer can be a metastatic cancer or other recurrent cancer. The treatments can be on-compendium or off-compendium treatments. The treatments may be standard of care for the type of cancer in the individual, or the treatments may be typically used for other types of cancer. Thus, the image and molecular analysis provided herein may expand the choice of treatments for the individual while avoiding treatments of little to no expected benefit.

[0228] The systems and methods provided herein may be used to classify patients as more or less likely to benefit or respond to various treatments. Unless otherwise noted, the terms “response” or “non-response,” as used herein, refer to any appropriate indication that a treatment provides a benefit to a patient (a “responder” or “benefiter”) or has a lack of benefit 7280147550V 1to the patient (a “non-responder” or “non-benefiter”). Such an indication may be determined using accepted clinical response criteria such as the standard Response Evaluation Criteria in Solid Tumors (RECIST) criteria, or other useful patient response criteria such as progression free survival (PFS), time to progression (TTP), disease free survival (DFS), time-to-next treatment (TNT, TTNT), tumor shrinkage or disappearance, or the like. RECIST is a set of rules published by an international consortium that define when tumors improve (“respond”), stay the same (“stabilize”), or worsen (“progress”) during treatment of a cancer patient. As used herein and unless otherwise noted, a patient “benefit” from a treatment may refer to any appropriate measure of improvement, including without limitation a RECIST response or longer PFS / TTP / DFS / TNT / TTNT. Beneficial or desired clinical results include, but are not limited to, alleviation or amelioration of one or more symptoms, diminishment of extent of disease, stabilized (i.e., not worsening) state of disease, preventing spread of disease, delay or slowing of disease progression, amelioration or palliation of the disease state, and remission (whether partial or total), whether detectable or undetectable. Benefit also includes prolonging survival as compared to expected survival if not receiving a treatment or if receiving a different treatment. Likewise, “lack of benefit” from a treatment may refer to any appropriate measure of worsening disease during treatment. Generally disease stabilization is considered a benefit, although in certain circumstances, if so noted herein, stabilization may be considered a lack of benefit. A predicted or indicated benefit may be described as “indeterminate” if there is not an acceptable level of prediction of benefit or lack of benefit. In some cases, benefit is considered indeterminate if it cannot be calculated, e.g., due to lack of necessary data.

[0229] Molecular profiling can be performed by any known means for detecting molecules in a biological sample. Useful biological samples include tumor samples and bodily fluid samples, e.g., blood or urine. In embodiments, a tumor can be sectioned into slides such that image analysis can be performed on one or more section of the tumor and molecular analysis can be performed on other sections of the same tumor. Molecular profiling assays can include without limitation, protein and nucleic acid analysis techniques. Protein analysis techniques include, by way of non-limiting examples, immunoassays, immunohistochemistry (IHC), and mass spectrometry. Nucleic acid analysis techniques include, by way of non-limiting examples, amplification such as polymerase chain amplification (PCR) amplification (e.g., qPCR or RT-PCR), hybridization, microarrays, in situ hybridization, and DNA sequencing or RNA sequencing (e.g., dye termination sequencing, Sanger, high throughput or next7380147550V 1generation sequencing (NGS), pyrosequencing, and restriction fragment analysis). Other examples of useful assays include in situ hybridization (ISH); fluorescent in situ hybridization (FISH); chromogenic in situ hybridization (CISH); various types of microarray (mRNA expression arrays, low density arrays, protein arrays, etc); comparative genomic hybridization (CGH); Northern blot; Southern blot; and any other appropriate technique to assay the presence or quantity of a biological molecule of interest. In various embodiments, any one or more of these methods can be used concurrently or subsequent to each other for assessing target genes disclosed herein.

[0230] As part of determining a molecular profile, sequencing nucleic acid molecules can provide sequence reads that include molecular information indicating point mutations, polymorphisms, deletions, insertions, substitutions, translocations, fusions, breaks, duplications, amplification, repeats, copy numbers (including determining copy number alterations (CNA); also referred to as copy number variation or CNV), transcript levels (expression levels), or any combination thereof. Information in sequence reads can also be used to determine genomic signatures, including without limitation tumor mutational burden (TMB), microsatellite instability (MSI), human leukocyte antigen (HLA) genotype, mismatch repair deficiency, homologous recombination deficiency (HRD), homologous recombination repair (HRR) deficiency, loss of heterozygosity (LOH), or any combination thereof. Such information can be determined by analyzing the sequence reads. In preferred embodiments, NGS is used to sequence nucleic acid molecules. NGS includes whole genome sequencing (WGS), whole exome sequencing (WES) or targeted gene panel analysis, which can be performed on genomic DNA. Whole transcriptome sequencing (WTS) or targeted transcript panel analysis can be performed on mRNA. As desired, the molecular profiling analysis is performed for genomic DNA, mRNA transcripts, or a combination thereof. The high-throughput nature of NGS allows for analysis of the whole exome or whole genome and / or whole transcriptome of >22,000 genes / gene products in a single assay, thereby providing a comprehensive overview of such molecular information.

[0231] Image analysis and / or molecular profiling are not limited to identifying candidate treatments for patients in need thereof. Indeed, the data derived from these analyses can be used to characterize various phenotypes of interest. For example, molecular profiling may be used to screen for disease and monitor disease before and / or after treatment, using liquid biopsy techniques. Image analysis and / or molecular profiling can be used to characterize a7480147550V 1disease aggressiveness, such as predicting the risk that a primary tumor will recur or metastasize. Thus, image analysis and molecular profiling of a primary tumor may provide both personalized treatment options for the patient and in addition provide a metastatic potential for the tumor. The treating physician may consider the predicted metastatic potential when deciding a course of treatment for the patient.

[0232] As noted above, a patient report can be delivered to the treating physician or other caregiver of the subject whose cancer has been analyzed using the systems and methods provided herein. The report can comprise multiple sections of relevant information, including but not limited to: 1) description of the patient and sample; 2) a complete or partial listing of the biomarkers (nucleic acids, proteins, or other biological matter of interest) in the molecular profile; 3) a description of the state of one or more of the biomarkers in the molecular profile as determined for the subject; 4) a description of one or more biological signatures as determined for the molecular profile, such as microsatellite stability, tumor mutational load / burden, tissue-of-origin, recurrence predictors, treatment response predictors; and / or metastasis predictors; 5) image analysis performed and predictions thereof, 6) an indication whether one or more treatment is likely to benefit the patient, not benefit the patient, or has indeterminate benefit based on the digital pathology and / or molecular profiling results; 7) one or more clinical trials for which the patient may be eligible; 8) an indication whether the cancer is predicted to recur and / or metastasize; and / or 9) evidence relevant to the foregoing, such as literature reports and / or clinical trial results.

[0233] The description of the molecular profile within the report can include such information as the laboratory technique used to assess each biomarker, optionally including the result and any criteria used to score each technique. By way of non-limiting example, the criteria for scoring a copy number alteration (or variation, CNA or CNV) may be a presence (i.e., a copy number that is greater or lower than the “normal” copy number present in a subject who does not have cancer, or statistically identified as present in the general population, typically diploid) or absence (i.e., a copy number that is considered the same as the “normal” copy number present in a subject who does not have cancer, or statistically identified as present in the general population, typically diploid). Treatments associated with one or more of the biomarkers or biosignatures may be determined using treatment association such as in any of International Patent Publications WO / 2007 / 137187 (Inf 1 Appl. No. PCT / US2007 / 069286), published November 29, 2007; WO / 2010 / 045318 (Inf 1 Appl. No.7580147550V 1PCT / US2009 / 060630), published April 22, 2010; WO / 2010 / 093465 (Int’l Appl. No.PCT / US2010 / 000407), published August 19, 2010; WO / 2012 / 170715 (Int’l Appl. No.PCT / US2012 / 041393), published December 13, 2012; WO / 2014 / 089241 (Int’l Appl. No. PCT / US2013 / 073184), published June 12, 2014; WO / 2011 / 056688 (Int’l Appl. No.PCT / US2010 / 054366), published May 12, 2011; WO / 2012 / 092336 (Int’l Appl. No.PCT / US2011 / 067527), published July 5, 2012; WO / 2015 / 116868 (Int’l Appl. No.PCT / US2015 / 013618), published August 6, 2015; WO / 2017 / 053915 (Int’l Appl. No.PCT / US2016 / 053614), published March 30, 2017; WO / 2016 / 141169 (Int’l Appl. No.PCT / US2016 / 020657), published September 9, 2016; and W02018175501 (Int’l Appl. No. PCT / US2018 / 023438), published September 27, 2018; WO / 2020 / 113237 (based on Int’l Patent Appl. No. PCT / US2019 / 064078, filed December 2, 2019); WO / 2020 / 146554 (based on Int’l Patent Appl. No. PCT / US2020 / 012815, filed January 8, 2020); WO / 2021 / 112918 (based on Int’l Patent Appl. No. PCT / US2020 / 035990, filed June 3, 2020); WO / 2021 / 163706 (based on Int’l Patent Appl. No. PCT / US2021 / 018263, filed February 16, 2021);WO / 2021 / 222867 (based on Int’l Patent Appl. No. PCT / US2021 / 030351, filed April 30, 2021); WO / 2022 / 056328 (based on Int’l Patent Appl. No. PCT / US2021 / 049966, filed September 10, 2021); WO / 2022 / 103809 (based on Int’l Patent Appl. No.PCT / US2021 / 058741, filed November 10, 2021); and WO / 2022 / 132964 (based on Int’l Patent Appl. No. PCT / US2021 / 063603, file December 15, 2021); each of which publications is incorporated by reference herein in its entirety.

[0234] In some embodiments, the findings are associated with an ongoing clinical trial. As a non-limiting example, digital pathology or sequencing may reveal a mutation that is a requirement for enrollment in a certain trial. The therapy associations may include on-label options, off-label options, clinical trial options, or any desired combination thereof. One of skill will appreciate that on-label refers to use of a therapy in a regulatory approved setting, whereas off-label refers to use of a regulatory approved therapy in an unapproved setting, including without limitation use of a therapy to treat a certain cancer lineage when the therapy is only approved to treat other types of cancer, or use of unapproved dosage forms or regimens.

[0235] In some embodiments, therapy selection is based upon multiple marker signatures, including those that use machine learning and artificial intelligence. See, e.g., International Patent publications WO / 2020 / 113237 (based on Int’l Patent Appl. No. PCT / US2019 / 064078,7680147550V 1filed December 2, 2019); WO / 2020 / 146554 (based on Int’l Patent Appl. No. PCT / US2020 / 012815, filed January 8, 2020); WO / 2021 / 112918 (based on Int’l Patent Appl. No. PCT / US2020 / 035990, filed June 3, 2020); WO / 2021 / 163706 (based on Int’l Patent Appl. No. PCT / US2021 / 018263, filed February 16, 2021); WO / 2021 / 222867 (based on Int’l Patent Appl. No. PCT / US2021 / 030351, filed April 30, 2021); WO / 2022 / 056328 (based on Int’l Patent Appl. No. PCT / US2021 / 049966, filed September 10, 2021); WO / 2022 / 103809 (based on Int’l Patent Appl. No. PCT / US2021 / 058741, filed November 10, 2021); and WO / 2022 / 132964 (based on Int’l Patent Appl. No. PCT / US2021 / 063603, file December 15, 2021); each of which publications is incorporated by reference herein in its entirety.

[0236] Treatment associations can be updated as new information becomes available regarding various biomarkers, biosignatures, treatments, and the relationships thereof. The indication whether each treatment is likely to benefit the patient, not benefit the patient, or has indeterminate benefit may be weighted. For example, a likely or potential benefit may be a strong potential benefit or a lesser potential benefit. Such weighting can be based on any appropriate criteria, e.g., the strength of the evidence of the biomarker-treatment association, or the results of the profiling, e.g., a degree or level of over- or underexpression, mutation, or any other relevant state (e.g., wild type or altered). As the treating physician is ultimately responsible for treating their patient, such physician may use the report to assist in guiding their treatment recommendations.

[0237] The patient report can be delivered to the caregiver for the subject, e.g., the oncologist or other treating physician. The caregiver can use the results of the report to guide a treatment regimen for the subject. For example, the caregiver may administer one or more treatments indicated as likely benefit in the report. Similarly, the caregiver may avoid treating the patient with one or more treatments indicated as likely lack of benefit in the report. In some embodiments, such as when the report includes analyses such as provided herein indicating a likely recurrence or metastasis, the treating physician may choose, for example, a more aggressive treatment regimen, more frequent monitoring, or both. Such decisions are made by the caregiver with guidance from the report.

[0238] The report can be computer generated, and can be a printed report, a computer file or both. The report can be made accessible via a secure web portal. The report may be displayed using any desired medium. In some embodiments, the display is a printout, a computer file, including without limitation a pdf file, or may be displayed via an application7780147550V 1on a computer display such as a computer monitor, laptop display, tablet, smartphone, or other mobile device.

[0239] In an aspect, the disclosure provides a system for generating a patient profiling report such as described above, comprising: (a) at least one host server; (b) at least one user interface for accessing the at least one host server to access and input data; (c) at least one processor for processing the inputted data; (d) at least one memory coupled to the processor for storing the processed data and instructions for: i) accessing a digital pathology finding or molecular result determined by the systems and methodology as described herein; and ii) identifying treatments, clinical trials, phenotypes, predictions, etc., as described herein; and (e) at least one display for displaying results and outcomes of the image analysis and / or molecular profiling. In some embodiments, the system further comprises at least one memory coupled to the processor for storing the processed data and instructions for identifying, based on the digital pathology and / or molecular profile according to the methods above, at least one therapy with potential benefit for treatment of the cancer; and at least one display for display thereof. The system may further comprise at least one database comprising references for various biomarker states, data for drug / biomarker associations, or both. The at least one display can be a report provided by the present disclosure.

[0240] The methods provided herein are useful for assessing any type of cancer. As examples, the cancer can comprise an acute lymphoblastic leukemia; acute myeloid leukemia; adrenocortical carcinoma; AIDS-related cancer; AIDS-related lymphoma; anal cancer; appendix cancer; astrocytomas; atypical teratoid / rhabdoid tumor; basal cell carcinoma; bladder cancer; brain stem glioma; brain tumor, brain stem glioma, central nervous method atypical teratoid / rhabdoid tumor, central nervous method embryonal tumors, astrocytomas, craniopharyngioma, ependymoblastoma, ependymoma, medulloblastoma, medulloepithelioma, pineal parenchymal tumors of intermediate differentiation, supratentorial primitive neuroectodermal tumors and pineoblastoma; breast cancer; bronchial tumors; Burkitt lymphoma; cancer of unknown primary site (CUP); carcinoid tumor; carcinoma of unknown primary site; central nervous method atypical teratoid / rhabdoid tumor; central nervous method embryonal tumors; cervical cancer; childhood cancers; chordoma; chronic lymphocytic leukemia; chronic myelogenous leukemia; chronic myeloproliferative disorders; colon cancer; colorectal cancer; craniopharyngioma; cutaneous T-cell lymphoma; endocrine pancreas islet cell tumors; endometrial cancer;7880147550V 1ependymoblastoma; ependymoma; esophageal cancer; esthesioneuroblastoma; Ewing sarcoma; extracranial germ cell tumor; extragonadal germ cell tumor; extrahepatic bile duct cancer; gallbladder cancer; gastric (stomach) cancer; gastrointestinal carcinoid tumor; gastrointestinal stromal cell tumor; gastrointestinal stromal tumor (GIST); gestational trophoblastic tumor; glioma; hairy cell leukemia; head and neck cancer; heart cancer;Hodgkin lymphoma; hypopharyngeal cancer; intraocular melanoma; islet cell tumors; Kaposi sarcoma; kidney cancer; Langerhans cell histiocytosis; laryngeal cancer; lip cancer; liver cancer; malignant fibrous histiocytoma bone cancer; medulloblastoma; medulloepithelioma; melanoma; Merkel cell carcinoma; Merkel cell skin carcinoma; mesothelioma; metastatic squamous neck cancer with occult primary; mouth cancer; multiple endocrine neoplasia syndromes; multiple myeloma; multiple myeloma / plasma cell neoplasm; mycosis fungoides; myelodysplastic syndromes; myeloproliferative neoplasms; nasal cavity cancer; nasopharyngeal cancer; neuroblastoma; Non-Hodgkin lymphoma; nonmelanoma skin cancer; non-small cell lung cancer; oral cancer; oral cavity cancer; oropharyngeal cancer; osteosarcoma; other brain and spinal cord tumors; ovarian cancer; ovarian epithelial cancer; ovarian germ cell tumor; ovarian low malignant potential tumor; pancreatic cancer; papillomatosis; paranasal sinus cancer; parathyroid cancer; pelvic cancer; penile cancer; pharyngeal cancer; pineal parenchymal tumors of intermediate differentiation; pineoblastoma; pituitary tumor; plasma cell neoplasm / multiple myeloma; pleuropulmonary blastoma; primary central nervous method (CNS) lymphoma; primary hepatocellular liver cancer; prostate cancer; rectal cancer; renal cancer; renal cell (kidney) cancer; renal cell cancer; respiratory tract cancer; retinoblastoma; rhabdomyosarcoma; salivary gland cancer; Sezary syndrome; small cell lung cancer; small intestine cancer; soft tissue sarcoma; squamous cell carcinoma; squamous neck cancer; stomach (gastric) cancer; supratentorial primitive neuroectodermal tumors; T-cell lymphoma; testicular cancer; throat cancer; thymic carcinoma; thymoma; thyroid cancer; transitional cell cancer; transitional cell cancer of the renal pelvis and ureter; trophoblastic tumor; ureter cancer; urethral cancer; uterine cancer; uterine sarcoma; vaginal cancer; vulvar cancer; Waldenstrom macroglobulinemia; or Wilm’s tumor.

[0241] As further examples, the cancer can comprise an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumor (GIST), glioblastoma, head and neck7980147550V 1squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), non-small cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma.VII. DETERMINING SUBJECT PHENOTYPE

[0242] Embodiments can determine a phenotype of the subject, such as classifying a medical condition. As examples, such a classification can include a diagnosis of the presence / absence of a disease or disorder, a stage of a disease or disorder, susceptibility to a disease or disorder, prognosis of a disease stage or disorder, theranosis of a disease or disorder, a physiological state, or response / potential response (or lack thereof) to interventions such as therapeutics. A phenotype can result from a subject’s genetic makeup as well as the influence of environmental factors and the interactions between the two, as well as from epigenetic modifications to nucleic acid sequences.

[0243] The determination of the phenotype can use information from a patient’s biological specimens, such as derived from image analysis or by molecular profiling, as well as any patient specific and / or clinical information. The phenotype can be delivered to a treating physician in a patient report. The patient report can include various digital pathology, genetic and epigenetic information (e.g., as determined from laboratory measurements of samples, such as those described herein) as well other characteristics of a subject, such as height, weight, age, blood pressure, medical history, etc.

[0244] Embodiments can determine a phenotype (e.g., a classification of a biological sample or medical condition) multiple times over a time period (time course), e.g., intervals of 1, 2, 3, 4 weeks; 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, or 12 months; or 1, 2, 3, 4, 5 years. For instance, a liquid biopsy may be performed on a time course following treatment. Such monitoring over a time course can track a medical condition, such as remission, recurrence or disease progression.8080147550V 1A. Diagnostic applications

[0245] A diagnosis, prognosis and / or theranosis may be performed based on the image analysis and / or molecular analysis described herein. Such analysis may include digital pathology and / or molecular profiling, e.g., using NGS. In embodiments, such analysis may include the determination of a quantitative parameter (e.g., an expression level of RNA or a copy number of DNA) that is compared to one or more cutoff values that are selected to differentiate between different phenotype classifications, e.g., disease is present or not, disease is more or less aggressive, disease is more or less likely to recur, disease is more or less likely to metastasize, disease is more of less likely to respond to a treatment, or any useful combination thereof.

[0246] The phenotype can comprise detecting the presence of or likelihood of developing a tumor, neoplasm, or cancer, or characterizing the tumor, neoplasm, or cancer (e.g., primary tumor origin, stage, grade, aggressiveness, likelihood of metastasis or recurrence, etc). In some embodiments, the cancer comprises an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumors (GIST), glioblastoma, head and neck squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), lung non-small cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma. The systems and methods herein can be used to characterize these and other cancers, such as described herein. Characterizing a phenotype can be providing a diagnosis, prognosis or theranosis of the cancer.B. Treatment selection

[0247] The phenotype classification can include predicting the likely benefit of one or more treatments using the systems and methods described herein. When the phenotype8180147550V 1classification is whether the subject is a likely to benefit from a particular treatment, that particular treatment may be selected for the patient.

[0248] As a non-limiting example, a patient with lung cancer might receive immunotherapy (IO therapy), such as a checkpoint inhibitor therapy. As another non-limiting example, a hormone positive breast cancer patient may receive chemotherapy if at risk of early recurrence or extended endocrine therapy (e.g., extended letrozole therapy) if at risk of late recurrence.

[0249] The treatment selection can include predicting a likely beneficial individualized medical intervention for a particular disease state using image analysis and / or molecular profiling of a patient’s biological specimen. Various findings within the digital pathology analysis or molecular profile along with indicated treatments of likely benefit, likely lack of benefit, or indeterminate benefit may be compiled into a patient-specific report. Such a patient report may include digital pathology and / or molecular profiling test results and proposed drug therapies based on the test results. Such reports may be compiled using information in one or more databases, which may include, but are not limited to, patient biological sample / specimen information and tracking, clinical data, patient data, patient tracking, file management, study protocols, patient test results from digital pathology and / or molecular profiling, and billing information and tracking. Other useful databases may include, but are not limited to, drug libraries, gene libraries, disease libraries, and public and private databases such as UniGene, OMIM, GO, TIGR, GenBank, KEGG and Biocarta. The report can further comprise a list describing the expected benefit of treatment options based on analysis described herein (e.g., image and / or molecular analysis), thereby identifying candidate treatment options for the subject.

[0250] As described herein, machine learning can be applied to image and / or molecular data to predict a treatment course for a patient. In addition, biomarker association rules can be used to provide suggested treatments based on molecular profiling results. Simple rules can be constructed in the format of “if biomarker positive then treatment option one, else treatment option two.” Other rules may suggest no treatment with a specific drug indicated to have likely lack of benefit, or treatment with a specific regimen (e.g., immunotherapy and / or chemotherapy) of likely benefit to the patient. In some embodiments, more complex rules are constructed that involve the interaction of multiple biomarkers. The patient report may describe the association of the predicted benefit of a treatment and the biomarker and8280147550V 1optionally a summary statement of the best evidence supporting the treatment guidance. Ultimately, the treating physician will decide on the best course of treatment for a patient.

[0251] When multiple treatment options are revealed by applying the systems and methods herein, decision rules can be put in place to prioritize the selection of a treatment regimen. For example, treatments may be prioritized based on direct results of image analysis and / or molecular profiling, anticipated efficacy of therapeutic agent, prior history with the same or other treatments, expected side effects, availability of therapeutic agent, cost of therapeutic agent, drug-drug interactions, and other factors. Based on the recommended and prioritized therapeutic agent targets, a treating physician can decide on the course of treatment for a particular individual.

[0252] The systems and methods described herein can be used to provide personalized treatment options for cancer patients. In some embodiments, the subject has been previously treated with one or more therapeutic agents to treat the cancer. The cancer may be refractory to one of these agents, e.g., by acquiring drug resistance mutations. Such acquired mutations may be identified over a time course using liquid biopsy. In some embodiments, the cancer is metastatic. In some embodiments, the subject has not previously been treated with one or more therapeutic agents identified by the method. The methods and systems as described herein can identify treatments based on individual characteristics of diseased cells and tissues, e.g., tumor cells and tumor blocks, and other personalized factors in a subject in need of treatment, as opposed to relying on a traditional one-size fits all approach that is conventionally used to treat individuals suffering from a disease, especially cancer. In some cases, the recommended treatments are those not typically used to treat the disease or disorder inflicting the subject. In some cases, the recommended treatments are used after standard-of-care therapies are no longer providing adequate efficacy.VIII. EXAMPLE SYSTEMS

[0253] FIG. 19 illustrates a measurement system 1900 according to an embodiment of the present disclosure. The system as shown includes a sample 1905, such as a tissue sample, within an imaging device 1910, where an image 1908 can be generated from the sample 1905. For example, sample 1905 can be sliced, stained, and prepared on a slide for imaging. The image 1908 is sent from imaging device 1910 to logic system 1930. As an example, the8380147550V 1image 1908 can be used to predict patient outcomes. The image 1908 may be stored in a local memory 1935, an external memory 1940, or a storage device 1945.

[0254] In embodiments, measurement system 1900 may further include laboratory devices to perform molecular analysis on sample 1905 or another sample from the patient, such as next-generation sequencing of DNA and / or RNA. Such additional laboratory devices may use the same sample 1905 or additional samples from the same patient. For example, if sample 1905 is a tumor sample, sections of the tumor can be used for image analysis and other sections can be used to extract nucleic acids for use in sequencing (e.g., NGS). Thus, imaging device 1910 can be more generally considered one or more laboratory devices that can perform various measurements.

[0255] Logic system 1930 may be, or may include, a computer system, ASIC, microprocessor, graphics processing unit (GPU), etc. It may also include or be coupled with a display (e.g., monitor, LED display, etc.) and a user input device (e.g., mouse, keyboard, buttons, etc.). Logic system 1930 and the other components may be part of a stand-alone or network connected computer system, or they may be directly attached to or incorporated in a device (e.g., a sequencing device) that includes imaging device 1910. Logic system 1930 may also include software that executes in a processor 1950. Logic system 1930 may include a computer readable medium storing instructions for controlling measurement system 1900 to perform any of the methods described herein. For example, logic system 1930 can provide commands to a system that includes imaging device 1910 such that imaging or other physical operations are performed. Such physical operations can be performed in a particular order, e.g., with images being captured in a particular order. Such physical operations may be performed by a robotics system, e.g., including a robotic arm, as may be used to obtain a sample and perform imaging.

[0256] Measurement system 1900 may also include a treatment device 1960, which can provide a treatment to the subject. Treatment device 1960 can determine a treatment and / or be used to perform a treatment. Examples of such treatment can include surgery, radiation therapy, chemotherapy, immunotherapy, targeted therapy, hormone therapy, and stem cell transplant. Logic system 1930 may be connected to treatment device 1960, e.g., to provide results of a method described herein. The treatment device may receive inputs from other devices, such as an imaging device and user inputs (e.g., to control the treatment, such as controls over a robotic system).8480147550V 1

[0257] Measurement system 1900 may also include a reporting device 1955, which can present results of any of the methods described herein, e.g., as determined using the measurement system. Reporting device 1955 can be in communication with a reporting module within logic system 1930 that can aggregate, format, and send a report to reporting device 1955. The reporting module can present information determined using any of the method described herein. The information can be presented by reporting device 1955 in any format that can be recognized and interpreted by a user of the measurement system 1900. For example, the information can be presented by reporting device 1955 in a displayed, printed, or transmitted format, or any combination thereof.

[0258] In various embodiments, logic system 1930 can include first early recurrence logic (e.g., for NGSPred), late recurrence logic, and / or second early recurrence logic (e.g., EDRP).

[0259] The first early recurrence logic can be configured to: generate one or more feature vectors from the sequencing data; provide the one or more feature vectors to an input layer of a neural network of a machine learning framework; process, by the neural network, the one or more feature vectors to generate an embedding; and process the embedding by an early distant recurrence model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient, wherein the outcome comprises an early distant recurrence of the medical condition. Processing the embedding can include: generating, by the early distant recurrence model using the one or more feature vectors, a first risk score; comparing the first risk score to a threshold; determining a first treatment is to be administered responsive to the first risk score being above the threshold.

[0260] The late recurrence logic can be configured to: responsive to the first risk score being below the threshold, generate one or more other feature vectors from pixels comprising at least a portion of the image; provide the one or more other feature vectors to an input layer of another neural network of the machine learning framework; process, by the other neural network, the one or more other feature vectors to generate another embedding; and process the other embedding by a late distant recurrence model of the machine learning framework to obtain another primary continuous variable associated with another outcome for the patient, wherein the other outcome comprises a late distant recurrence of the medical condition.

[0261] Second early recurrence logic can be configured to: process, before the processing of the first early recurrence logic, the other embedding or an additional embedding generated using image data (e.g., different image feature vector than for the late recurrence model) by 8580147550V 1another early distant recurrence model of the machine learning framework to obtain an additional primary continuous variable associated with the early distant recurrence of the medical condition. Processing the other embedding or the additional embedding can include: generating, by the other early distant recurrence model using the one or more feature vectors, a second risk score; compare the second risk score to a second threshold; and determine the first treatment is to be administered responsive to the second risk score being above the second threshold.

[0262] The systems can be configured to perform any of the methods described herein.

[0263] Any of the computer systems mentioned herein may utilize any suitable number of subsystems. Examples of such subsystems are shown in FIG. 20 in computer system 10. In some embodiments, a computer system includes a single computer apparatus, where the subsystems can be the components of the computer apparatus. In other embodiments, a computer system can include multiple computer apparatuses, each being a subsystem, with internal components. A computer system can include desktop and laptop computers, tablets, mobile phones and other mobile devices.

[0264] The subsystems shown in FIG. 20 are interconnected via a system bus 75.Additional subsystems such as a printer 74, keyboard 78, storage device(s) 79, monitor 76 (e.g., a display screen, such as an LED), which is coupled to display adapter 82, and others are shown. Peripherals and input / output (I / O) devices, which couple to I / O controller 71, can be connected to the computer system by any number of means known in the art such as input / output (I / O) port 77 (e.g., USB, FireWire®). For example, I / O port 77 or external interface 81 (e.g., Ethernet, Wi-Fi, etc.) can be used to connect computer system 10 to a wide area network such as the Internet, a mouse input device, or a scanner. The interconnection via system bus 75 allows the central processor 73 to communicate with each subsystem and to control the execution of a plurality of instructions from system memory 72 or the storage device(s) 79 (e.g., a fixed disk, such as a hard drive, or optical disk), as well as the exchange of information between subsystems. The system memory 72 and / or the storage device(s) 79 may embody a computer readable medium. Another subsystem is a data collection device 85, such as a camera, microphone, accelerometer, and the like. Any of the data mentioned herein can be output from one component to another component and can be output to the user.

[0265] A computer system can include a plurality of the same components or subsystems, e.g., connected together by external interface 81, by an internal interface, or via removable 8680147550V 1storage devices that can be connected and removed from one component to another component. In some embodiments, computer systems, subsystem, or apparatuses can communicate over a network. In such instances, one computer can be considered a client and another computer a server, where each can be part of a same computer system. A client and a server can each include multiple systems, subsystems, or components. In various embodiments, methods may involve various numbers of clients and / or servers, including at least 10, 20, 50, 100, 200, 500, 1,000, or 10,000 devices. Methods can include various numbers of communication messages between devices, including at least 100, 200, 500, 1,000, 10,000, 50,000, 100,000, 500,00, or one million communication messages. Such communications can involve at least 1 MB, 10 MB, 100 MB, 1 GB, 10 GB, or 100 GB of data.

[0266] Aspects of embodiments can be implemented in the form of control logic using hardware circuitry (e.g., an application specific integrated circuit or field programmable gate array) and / or using computer software stored in a memory with a generally programmable processor in a modular or integrated manner, and thus a processor can include memory storing software instructions that configure hardware circuitry, as well as an FPGA with configuration instructions or an ASIC. As used herein, a processor can include a single-core processor, multi-core processor on a same integrated chip, or multiple processing units on a single circuit board or networked, as well as dedicated hardware. Based on the disclosure and teachings provided herein, a person of ordinary skill in the art will know and appreciate other ways and / or methods to implement embodiments of the present disclosure using hardware and a combination of hardware and software.

[0267] Any of the software components or functions described in this application may be implemented as software code to be executed by a processor using any suitable computer language such as, for example, Java, C, C++, C#, Objective-C, Swift, or scripting language such as R, Perl or Python using, for example, conventional or object-oriented techniques. The software code may be stored as a series of instructions or commands on a computer readable medium for storage and / or transmission. A suitable non-transitory computer readable medium can include random access memory (RAM), a read only memory (ROM), a magnetic medium such as a hard-drive or a floppy disk, or an optical medium such as a compact disk (CD) or DVD (digital versatile disk) or Blu-ray disk, flash memory, and the like. The computer readable medium may be any combination of such devices. In addition, the order of8780147550V 1operations may be re-arranged. A process can be terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0268] Such programs may also be encoded and transmitted using carrier signals adapted for transmission via wired, optical, and / or wireless networks conforming to a variety of protocols, including the Internet. As such, a computer readable medium may be created using a data signal encoded with such programs. Computer readable media encoded with the program code may be packaged with a compatible device (e.g., as firmware) or provided separately from other devices (e.g., via Internet download). Any such computer readable medium may reside on or within a single computer product (e.g., a hard drive, a CD, or an entire computer system), and may be present on or within different computer products within a system or network. A computer system may include a monitor, printer, or other suitable display for providing any of the results mentioned herein to a user.

[0269] Any of the methods described herein may be totally or partially performed with a computer system including one or more processors, which can be configured to perform the steps. Any operations performed with a processor may be performed in real-time. The term "real-time" may refer to computing operations or processes that are completed within a certain time constraint. The time constraint may be 1 minute, 1 hour, 1 day, or 7 days. Thus, embodiments can be directed to computer systems configured to perform the steps of any of the methods described herein, potentially with different components performing a respective step or a respective group of steps. Although presented as numbered steps, steps of methods herein can be performed at a same time or at different times or in a different order.Additionally, portions of these steps may be used with portions of other steps from other methods. Also, all or portions of a step may be optional. Additionally, any of the steps of any of the methods can be performed with modules, units, circuits, or other means of a system for performing these steps.

[0270] The specific details of particular embodiments may be combined in any suitable manner without departing from the spirit and scope of embodiments of the disclosure.However, other embodiments of the disclosure may be directed to specific embodiments relating to each individual aspect, or specific combinations of these individual aspects.8880147550V 1

[0271] The above description of example embodiments of the present disclosure has been presented for the purposes of illustration and description. It is not intended to be exhaustive or to limit the disclosure to the precise form described, and many modifications and variations are possible in light of the teaching above.

[0272] A recitation of “a”, “an” or “the” is intended to mean “one or more” unless specifically indicated to the contrary. The use of “or” is intended to mean an “inclusive or,” and not an “exclusive or” unless specifically indicated to the contrary. Reference to a “first” component does not necessarily require that a second component be provided. Moreover, reference to a “first” or a “second” component does not limit the referenced component to a particular location unless expressly stated. The term “based on” is intended to mean “based at least in part on.”

[0273] The claims may be drafted to exclude any element which may be optional. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely”, “only”, and the like in connection with the recitation of claim elements, or the use of a “negative” limitation.

[0274] All patents, patent applications, publications, and descriptions mentioned herein are incorporated by reference in their entirety for all purposes. None is admitted to be prior art. Where a conflict exists between the instant application and a reference provided herein, the instant application shall dominate.8980147550V 1

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method comprising:accessing an image of a biological sample of a patient having a medical condition, wherein the biological sample is stained using a pathology stain;generating one or more feature vectors from pixels comprising at least a portion of the image;providing the one or more feature vectors to an input layer of a neural network of a machine learning framework;processing, by the neural network, the one or more feature vectors to generate a shared embedding or a set of task specific embeddings representing the image, wherein the set of task specific embeddings include a primary regression embedding and one or more auxiliary embeddings, the one or more auxiliary embeddings include at least one embedding selected from a first group consisting of (1) one or more auxiliary classification embeddings for obtaining one or more classifications of the biological sample and (2) one or more auxiliary regression embeddings for obtaining one or more auxiliary continuous variables of the biological sample;processing the shared embedding or the primary regression embedding by a primary machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient; andprocessing the shared embedding or the one or more auxiliary embeddings by an auxiliary machine learning model of the machine learning framework to obtain one or more auxiliary properties of the biological sample, the one or more auxiliary properties selected from a second group consisting of (a) the one or more classifications of the biological sample and (b) the one or more auxiliary continuous variables of the biological sample, wherein the machine learning framework is trained using:a regression loss function for the primary continuous variable, and one or more auxiliary loss functions for the one or more auxiliary properties.

2. The computer-implemented method of claim 1, wherein the primary continuous variable is time-dependent.

3. The computer-implemented method of claim 1 or claim 2, wherein the one or more auxiliary properties include the one or more classifications of the biological sample, and wherein the one or more classifications comprise patient characteristics.

4. The computer-implemented method of claim 3, wherein at least one of the patient characteristics are selected from a group consisting of a patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone mineral density T-score, a prior treatment, or treatment type.

5. The method of any preceding claim, wherein the one or more feature vectors further include one or more patient characteristics.

6. The method of claim 5, wherein at least one of the one or more patient characteristics is selected from a group consisting of a patient age, a patient gender, a biomarker status, cancer type, histology type, tumor grade, node status, surgery type, a bone mineral density T-score, a prior treatment, or treatment type.

7. The computer-implemented method of any one of claims 1-6, wherein the outcome comprises a survival of the patient, a time on treatment for the patient, or a recurrence of the medical condition.

8. The computer-implemented method of claim 7, wherein the recurrence is an early distant recurrence (EDR) or a late distant recurrence (LDR), or wherein the outcome comprises a determination for the early distant recurrence and the late distant recurrence.

9. The computer-implemented method of claim 8, wherein the primary machine learning regression model includes an early distant recurrence model and a late distant recurrence model.

10. The computer-implemented method of claim 8 or claim 9, wherein the early distant recurrence model includes a first EDR model and a second EDR model, and wherein the first EDR model is used to determine the recurrence if a first risk score from the first EDR model is below a threshold and the second EDR model is used to determine the recurrence if the first risk score from the first EDR model is greater than the threshold.

11. The computer-implemented method of claim 10, wherein the first EDR model processes the one or more feature vectors from the pixels, and wherein the second EDR model processes sequencing data measured from the biological sample or another biological sample of the patient.

12. The computer-implemented method of claim 11, wherein the sequencing data is obtained by next-generation sequencing (NGS), optionally wherein the NGS comprises whole genome sequencing (WGS), whole exome sequencing (WES), whole transcriptome sequencing (WTS), or a combination thereof.

13. The computer-implemented method of any one of claims 1-12, wherein the medical condition comprises cancer.

14. The computer-implemented method of claim 13, wherein the cancer comprises an acute myeloid leukemia (AML), breast carcinoma, cholangiocarcinoma, colorectal adenocarcinoma, extrahepatic bile duct adenocarcinoma, female genital tract malignancy, gastric adenocarcinoma, gastroesophageal adenocarcinoma, gastrointestinal stromal tumor (GIST), glioblastoma, head and neck squamous carcinoma, leukemia, liver hepatocellular carcinoma, low grade glioma, lung bronchioloalveolar carcinoma (BAC), nonsmall cell lung cancer (NSCLC), lung small cell cancer (SCLC), lymphoma, male genital tract malignancy, malignant solitary fibrous tumor of the pleura (MSFT), melanoma, multiple myeloma, neuroendocrine tumor, nodal diffuse large B-cell lymphoma, non-epithelial ovarian cancer (non-EOC), ovarian surface epithelial carcinoma, pancreatic adenocarcinoma, pituitary carcinomas, oligodendroglioma, prostatic adenocarcinoma, retroperitoneal or peritoneal carcinoma, retroperitoneal or peritoneal sarcoma, small intestinal malignancy, soft tissue tumor, thymic carcinoma, thyroid carcinoma, or uveal melanoma.

15. The computer-implemented method of any one of claims 1-14, wherein the one or more feature vectors comprise a first feature vector from first pixels of a target portion of the image and additional feature vectors from additional pixels of additional portions of a surrounding area of the target portion, wherein the surrounding area is within a pixel distance of the target portion.

16. The computer-implemented method of any one of claims 1-14, wherein the one or more feature vectors comprise a first feature vector from first pixels of a targetportion of the image and additional feature vectors from additional pixels of additional portions of the image, wherein the additional portions comprise an entirety of the image.

17. The computer-implemented method of any one of claims 1-14, wherein the one or more feature vectors comprises a first feature vector from first pixels comprising a first portion of the image, and wherein the computer-implemented method further comprises:generating a second feature vector from second pixels comprising a second portion of the image; andprocessing, by the neural network, the first feature vector and the second feature vector to generate the primary regression embedding, wherein the primary regression embedding represents an aggregation of the first feature vector and the second feature vector.

18. The computer-implemented method of claim 17, wherein the neural network comprises a multi-instance learning model.

19. The computer-implemented method of any one of claims 1-18, wherein the regression loss function for the primary continuous variable comprises a time-dependent loss function.

20. The computer-implemented method of claim 19, wherein the timedependent loss function comprises a Cox partial likelihood function.

21. The computer-implemented method of any one of claims 1-20, wherein the one or more auxiliary properties include the one or more classifications of the biological sample.

22. The computer-implemented method of claim 21, wherein the one or more classifications include a status of at least one biomarker, optionally wherein the biomarker is associated with one or more treatment.

23. The computer-implemented method of claim 22, wherein the status comprises an expression level or a mutation.

24. The computer-implemented method of claim 21, wherein the one or more classifications include at least one selected from a bone mineral density T-score and prior tamoxifen treatment.

25. The computer-implemented method of any preceding claim, further comprising:performing one or more auxiliary regression tasks by processing the shared embedding or the one or more auxiliary regression embeddings by the auxiliary machine learning model to obtain the one or more auxiliary continuous variables of the biological sample, wherein the machine learning framework is further trained using the regression loss function for the one or more auxiliary continuous variables.

26. The computer-implemented method of claim 25, wherein the one or more auxiliary regression tasks comprise predicting a percentage of tumor cells in the biological sample.

27. The computer-implemented method of any one of claims 1-26, wherein the biological sample is from a tumor.

28. The computer-implemented method of claim 27, wherein the tumor is a primary tumor or a metastatic tumor.

29. The computer-implemented method of claim 27, wherein the tumor is a tumor of the myeloid, breast, bile ducts, colon, rectum, female genital tract, stomach, esophagus, gastrointestinal stromal cells, small intestine, brain, mouth, sinuses, nose, throat, blood, liver, nervous system, lung, lymph, male genital tract, pleura, skin, plasma cells, neuroendocrine cells, B-cells, T-cells, ovary, pancreas, pituitary gland, spinal cord, prostate, peritoneum, large intestine, soft tissue, connective tissue, fat tissue, thymus, thyroid, or eye.

30. The computer-implemented method of claim 28, wherein the primary tumor is a tumor of the bladder, breast, colon, rectum, endometrium, uterus, ovary, female genital tract, kidney, blood, liver, lung, skin, lymph, pancreas, prostate, or thyroid.

31. The computer-implemented method of any one of claims 1-30, wherein the biological sample is a formalin-fixed paraffin-embedded (FFPE) tissue.

32. The computer-implemented method of any one of claims 1-31, wherein the pathology stain comprises hematoxylin and eosin (H&E).

33. The computer-implemented method of any one of claims 1-32, further comprising providing a diagnosis, prognosis, and / or theranosis based on the primary continuous variable.

34. The computer-implemented method of any one of claims 1-33, wherein the primary continuous variable comprises a risk score.

35. The computer-implemented method of any one of claims 1-16, further comprising:outputting a report that includes the primary continuous variable and / or a patient category derived from the primary continuous variable.

36. The computer-implemented method of claim 35, wherein the report includes the patient category, the method further comprising:determining the patient category of whether the patient is in a low risk group or a high risk group based on the primary continuous variable.

37. The computer-implemented method of claim 36, wherein the patient category is determined by comparing the primary continuous variable to a threshold.

38. The computer-implemented method of claim 37, wherein the threshold is determined based on a distribution of the primary continuous variable for a set of patients having the medical condition.

39. The computer-implemented method of claim 36, wherein the report recommends providing a particular treatment responsive to the patient being within the high risk group.

40. The computer-implemented method of claim 39, further comprising: providing the particular treatment to the patient.

41. The computer-implemented method of claim 36, wherein the report recommends not providing either a particular treatment or any treatment responsive to the patient being within the low risk group.

42. The computer-implemented method of any one of claims 39-41, wherein the particular treatment includes extended endocrine therapy (EET), and wherein themedical condition is a hormone-positive breast cancer, optionally wherein the EET is extended letrozole treatment (ELT).

43. The computer-implemented method of claim 42, wherein the one or more auxiliary properties include the one or more classifications of the biological sample, and wherein the one or more classifications include at least one selected from a bone mineral density T-score and a prior tamoxifen use.

44. The computer-implemented method of any preceding claim, wherein: a) the medical condition comprises a hormone-positive breast cancer; b) the biological sample comprises a tumor section from the cancer; c) the one or more feature vectors further include one or more patient characteristics comprising age, node status, and surgery type;d) the primary continuous variable comprises time to distant recurrence; e) the outcome comprises distant recurrence of the medical condition; and f) the one or more auxiliary properties include bone mineral density T-score.

45. The computer-implemented method of claim 44, wherein the distant recurrence of the medical condition comprises late distant recurrence.

46. The computer-implemented method of claim 44 or 45, wherein the one or more auxiliary properties further include prior treatment with tamoxifen.

47. The computer-implemented method of any one of claims 39-41, wherein the particular treatment includes an immune checkpoint inhibitor, and wherein the medical condition is lung cancer, optionally wherein the immune checkpoint inhibitor is selected from the group consisting of pembrolizumab, nivolumab, cemiplimab, dostarlimab, retifanlimab, toripalimab, atezolizumab, durvalumab, avelumab, ipilimumab, tremelimumab, relatlimab, and any useful combination thereof.

48. The computer-implemented method of claim 47, wherein the one or more auxiliary properties include the one or more classifications of the biological sample, and wherein the one or more classifications include one or more biomarker statuses, optionally wherein the biomarker statuses comprise a presence, absence or level of one or more biomarker.

49. The computer-implemented method of claim 48, wherein the one or more biomarker statuses include an expression status of at least one of PD-L1 (Programmed Death Ligand 1), PD-1 (Programmed Death- 1), CTLA-4 (Cytotoxic T-Lymphocyte Associated Protein 4), LAG-3 (Lymphocyte Activation Gene-3), TIM-3 (T-cell Immunoglobulin and Mucin Domain-3), TIGIT (T-cell Immunoreceptor with Ig and ITIM domains), Tumor Mutational Burden (TMB), Microsatellite Instability -High (MSLH) / Mismatch Repair Deficiency (dMMR), Tumor-Infiltrating Lymphocytes (TILs), VISTA (V-domain Ig Suppressor of T-cell Activation), B7-H3 (CD276), B7-H4, IDO1 (Indoleamine 2,3 -dioxygenase), BTLA (B and T Lymphocyte Attenuator), and a combination thereof.

50. The computer-implemented method of any one of claims 47-49, wherein the immune checkpoint inhibitor comprises pembrolizumab, the one or more biomarker statuses includes expression status of PD-L1, and the medical condition comprises a lung cancer.

51. A computer-implemented method comprising:accessing an image of a biological sample of a patient having a medical condition, wherein the biological sample is stained using a pathology stain;generating one or more feature vectors from pixels comprising at least a portion of the image;providing the one or more feature vectors to an input layer of a neural network of a machine learning framework;processing, by the neural network, the one or more feature vectors to generate an embedding; andprocessing the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient, wherein the outcome comprises a recurrence of the medical condition, wherein the outcome comprises a determination for an early distant recurrence, wherein the machine learning regression model includes an early distant recurrence model, wherein the early distant recurrence model includes a first EDR model and a second EDR model, wherein processing the embedding includes:generating, using the one or more feature vectors, a first risk score using the first EDR model;comparing the first risk score to a threshold;determining, by the first EDR model, the recurrence responsive to the first risk score from the first EDR model being below a threshold; anddetermining, by the second EDR model, the recurrence responsive to the first risk score from the first EDR model being greater than the threshold.

52. The computer-implemented method of claim 51, wherein the outcome comprises a determination for the early distant recurrence and a late distant recurrence, and wherein the machine learning regression model includes the early distant recurrence model and a late distant recurrence model.

53. The computer-implemented method of claim 52, wherein the late distant recurrence model is invoked when the early distant recurrence indicates low risk.

54. The computer-implemented method of any one of claims 51-53, wherein the second EDR model processes sequencing data measured from the biological sample or another biological sample of the patient.

55. The computer-implemented method any one of claims 51-54, wherein the late distant recurrence model and / or the early distant recurrence model is selected from any of the models recited in any one of claims 1-54.

56. A computer-implemented method comprising:accessing an image of a biological sample of a patient having a medical condition, wherein the biological sample is stained using a pathology stain;generating one or more feature vectors from pixels comprising at least a portion of the image;providing the one or more feature vectors to an input layer of a neural network of a machine learning framework;processing, by the neural network, the one or more feature vectors to generate an embedding; andprocessing the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient, wherein the outcome comprises a determination for late distant recurrence of the medical condition, wherein the machine learning regression model includes a late distant recurrence model used to determine the recurrence based on a risk score from an early distant recurrence model being below a threshold.

57. The computer-implemented method of claim 56, wherein the early distant recurrence model uses sequencing data from the biological sample or other biological sample of the patient.

58. The computer-implemented method of claim 57, further comprising: obtaining the sequencing data; andprocessing the sequencing data by the early distant recurrence model to determine the risk score.

59. The computer-implemented method of claim 57, wherein the sequencing data is obtained using next generation sequencing, optionally wherein the NGS comprises whole genome sequencing (WGS), whole exome sequencing (WES), whole transcriptome sequencing (WTS), or a combination thereof.

60. The computer-implemented method of claim 56, wherein the early distant recurrence model uses image data of the biological sample or another biological data of the patient.

61. The computer-implemented method of claim 60, wherein the image data includes the one or more feature vectors.

62. The computer-implemented method any one of claims 56-61, further comprising:processing the embedding by an auxiliary machine learning model of the machine learning framework to obtain one or more auxiliary properties of the biological sample, the one or more auxiliary properties selected from a second group consisting of (a) the one or more classifications of the biological sample and (b) the one or more auxiliary continuous variables of the biological sample.

63. The computer-implemented method any one of claims 56-62, wherein the late distant recurrence model and / or the early distant recurrence model is selected from any of the models recited in any one of claims 1-62.

64. A computer-implemented method comprising:accessing sequencing data measured from a biological sample of a patient having a medical condition;generating one or more feature vectors from the sequencing data; providing the one or more feature vectors to an input layer of a neural network of a machine learning framework;processing, by the neural network, the one or more feature vectors to generate an embedding; andprocessing the embedding by a machine learning regression model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient, wherein the outcome comprises a recurrence of the medical condition, wherein the outcome comprises a determination for an early distant recurrence, wherein the machine learning regression model includes an early distant recurrence model, and wherein processing the embedding includes:generating, by the early distant recurrence model using the one or more feature vectors, a first risk score;comparing the first risk score to a threshold;determining a first treatment is to be administered responsive to the first risk score being above the threshold; andresponsive to the first risk score being below the threshold, generating, by a late distant recurrence model, another primary continuous variable associated with a late distant recurrence for the patient.

65. The computer-implemented method of claim 64, wherein the first treatment includes chemotherapy.

66. The computer-implemented method of claim 64 or claim 65, wherein the late distant recurrence model is selected from any of the models recited in any one of claims 1-65.

67. The computer-implemented method of any one of claims 64-66, wherein the sequencing data is obtained using next generation sequencing, optionally wherein the NGS comprises whole genome sequencing (WGS), whole exome sequencing (WES), whole transcriptome sequencing (WTS), or a combination thereof.

68. The method of any preceding claim, wherein the outcome is determined as part of a training stage performed by the machine learning framework, wherein a known outcome is used for the training stage.

69. The method of any one of claims 1-67, wherein the outcome is determined as part of an inference stage performed by the machine learning framework.

70. The computer-implemented method of claim 69, further comprising: training one or more models of the machine learning framework using training samples of patients having known outcomes.

71. The computer-implemented method of claim 70, wherein the known outcomes are for (a) a primary continuous variable and (b) one or more classifications or one or more auxiliary continuous variables.

72. The computer-implemented method of claim 71, wherein a loss function is determined using differences between predicted outcomes and the known outcomes.

73. A computer product comprising a non-transitory computer readable medium storing a plurality of instructions that, when executed, cause a computer system to perform the method of any one of the preceding claims.

74. A system comprising:the computer product of claim 73; andone or more processors configured to execute instructions stored on the non-transitory computer readable medium.

75. A system comprising means for performing any of the above methods.

76. A system comprising one or more processors configured to perform any of the above methods.

77. A system comprising modules that respectively perform the steps of any of the above methods.

78. A system comprising:a sequencing device configured to generate sequencing data by sequencing nucleic acids from a biological sample of a patient having a medical condition;first early recurrence logic configured to:generate one or more feature vectors from the sequencing data;provide the one or more feature vectors to an input layer of a neural network of a machine learning framework;process, by the neural network, the one or more feature vectors to generate an embedding; andprocess the embedding by an early distant recurrence model of the machine learning framework to obtain a primary continuous variable associated with an outcome for the patient, wherein the outcome comprises an early distant recurrence of the medical condition, and wherein processing the embedding includes:generating, by the early distant recurrence model using the one or more feature vectors, a first risk score;comparing the first risk score to a threshold;determining a first treatment is to be administered responsive to the first risk score being above the threshold; andan imaging device configured to generate an image of a biological sample of the patient, wherein the biological sample is stained using a pathology stain;late recurrence logic configured to:responsive to the first risk score being below the threshold, generate one or more other feature vectors from pixels comprising at least a portion of the image;provide the one or more other feature vectors to an input layer of another neural network of the machine learning framework;process, by the other neural network, the one or more other feature vectors to generate another embedding; andprocess the other embedding by a late distant recurrence model of the machine learning framework to obtain another primary continuous variable associated with another outcome for the patient, wherein the other outcome comprises a late distant recurrence of the medical condition.

79. The system of claim 78, wherein the sequencing includes next generation sequencing, optionally wherein the NGS comprises whole genome sequencing (WGS), whole exome sequencing (WES), whole transcriptome sequencing (WTS), or a combination thereof.

80. The system of claim 78 or claim 79, further comprising: second early recurrence logic configured to:process, before the processing of the first early recurrence logic, the other embedding or an additional embedding generated using image data by another early distant recurrence model of the machine learning framework to obtain an additional primary continuous variable associated with the early distant recurrence of the medical condition, and wherein processing the other embedding or the additional embedding includes:generating, by the other early distant recurrence model using the one or more feature vectors, a second risk score;compare the second risk score to a second threshold; anddetermine the first treatment is to be administered responsive to the second risk score being above the second threshold.

Images