Machine learning approaches for segmentation and classification of objects in digital images

Abstract

Systems, apparatuses, and methods for training and implementing machine learning models for detecting tumors in whole-slide images (WSIs) are provided. The techniques include accessing WSIs each depicting a specimen, generating ground truth labels for the WSIs by annotating each WSI to indicate whether a region on the WSI corresponds to the disease or not and labeling using an image segmentation algorithm, generating a set of patches for each WSI and training a machine learning model to classify each patch from the set of patches as diseased or healthy by minimizing a loss or error function of the machine learning model, and outputting the trained machine learning model for tumor detection.

Description

Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT MACHINE LEARNING APPROACHES FOR SEGMENTATION AND CLASSIFICATION OF OBJECTS IN DIGITAL IMAGES FIELD

[0001] The present disclosure relates to digital pathology, and in particular to machine learning techniques for the segmentation and classification of objects, such as tumor regions, within digital pathology images.BACKGROUND

[0002] Integration of artificial intelligence (Al) and digital imaging into the pathology workflow has tremendously impacted the role that pathology has historically served in the clinical setting. Advancements in Al algorithms, computer-aided diagnostic techniques, and the development of whole slide imaging (WSI) are just a few examples of how bridging traditional science with computational techniques expand the sandbox of tools available to clinicians and researchers to decipher disease pathology and advance drug discovery, translation, and clinical applications. This technological advancement is referred to as digital pathology which is the management and interpretation of pathology information in a digital environment. Scanning devices are used to generate high-resolution images of slides of biological samples, which may be stained, to generate digital images such as WSI. Application of digital pathology software enables faster processing, cheaper storage, and easy management of digital slide images increasing the opportunity' for sharing slide images across the clinical community.

[0003] In its infancy. Al-aided processing of digital images was mostly used for image-based diagnosis in radiology and cardiology. With the development of WSI, the application and benefits of Al processing greatly7improved. Compared to X-ray and MRI digital imaging, WSI provides a w ealth of information due to the large size of WSI (e.g., 100k x 100k) and increased image complexity. Image complexity is attributed to features often being in color due to various staining techniques (e.g., Hematoxylin and Eosin (H&E) and immunohistochemistry) and availability7of information at multiple scales (e.g., 4x, 20x, etc.). WSI also benefits from not necessarily being restricted by image orientation (e.g., anatomical orientation) as encountered in radiology. Accordingly, Al-aided processing offers numerous advantages in the analysis of digital pathology images, enhancing both the accuracy and efficiency of diagnostic processes. Some of the key advantages include: (i) Al algorithms can detect subtle patterns and anomalies in pathology images that may be overlooked by human eyes, leading to more accurate diagnose,Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT (ii) machine learning models can be trained to standardize interpretations, reducing variability and improving consistency in diagnostic results, (iii) Al can serve as a second opinion or decision-support tool for pathologists, helping to confirm or refine their diagnoses, (iv) Al can process and analyze large volumes of digital pathology images much faster than humans, accelerating diagnosis and treatment planning, (v) Al systems can handle large datasets and scale up to meet the increasing demand for pathology services without a proportional increase in staffing, and (vi) by automating repetitive and time-consuming tasks, Al can reduce the workload for pathologists, leading to cost savings for healthcare institutions.BRIEF SUMMARY

[0004] Disclosed herein are machine learning techniques (e g., computer-implemented methods, systems, and computer-program products) for the segmentation and classification of objects, such as tumor regions, within digital pathology images. These techniques include training convolutional neural networks (CNNs), such as Inception, ResNet. and similar machine learning models (referred to herein in general as models) using whole-slide images (WSI) with and without pathologist annotations. The models are evaluated by comparing the segmentation and classification output of the models with pathologists’ annotation (i.e., ground truth labels) using typical performance metrics such as Fl-score, sensitivity, specificity, area under the curve (AUC), etc. Once the model performance is acceptable, the models are deployed for use in one or more laboratory assays such as multiplex immunohistochemistry assays, which involve the simultaneous detection and analysis of multiple biomarkers within a single tissue sample.

[0005] In various embodiments, a computer implemented method is provided comprising accessing whole-slide images (WSIs), wherein each WSI is a WSI of a specimen associated with a disease or a specimen unassociated with the disease; generating, ground truth labels for the WSIs, wherein the generating comprises, for each WSI, obtaining, from a plurality of pathologists, annotations of the WSI from each of the plurality of pathologists, wherein the annotations indicate whether a region on the WSI corresponds to the disease or not, and generating, by inputting the annotations of the WSI from each of the plurality of pathologists into an image segmentation algorithm, a WSI with ground truth labels, wherein the ground truth labels represent probabilistically determined ground truths for each annotation; generating, from each WSI with corresponding ground truth labels, a set of patches, wherein each patch from the set of patches corresponds to a portion of the ground truth labeled WSI;Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT training, using the set of patches for each WSI, a machine learning model to classify each patch from the set of patches as diseased or healthy, wherein the training comprises iterative operations to adjust a set of model parameters to minimize a loss or error function of the machine learning model, and wherein the loss or error function is configured to measure a difference between output predictions by the machine learning model and the ground truth labels; and outputting the trained machine learning model.

[0006] In some embodiments, the image segmentation algorithm is a consensus image generation algorithm.

[0007] In some embodiments, each specimen is a tissue section originated from lung, liver, kidney, skin, bladder, breast, thyroid, lymph node, stomach, brain, blood vessels, phary nx, heart, pancreas, esophagus, muscle, or thymus of a subject.

[0008] In some embodiments, the tissue section is a bladder tissue.

[0009] In some embodiments, the disease is a cancer.

[0010] In some embodiments, the plurality of pathologists comprises at least 3 pathologists.

[0011] In some embodiments, the image segmentation algorithm performs iterative operations, wherein the iterative operations comprise: (i) generating a test segmentation of the WSI of the specimen by grouping each pixel in the WSI of the specimen based on similar properties, (ii) assigning performance parameters to each pathologist in the plurality of pathologists by comparing the annotations from the plurality of pathologists to the test segmentation, (iii) assigning each pixel in the test segmentation of the WSI to an annotation unit based on the annotations provided by the plurality of pathologists and their respective performance parameters, (iv) updating, using the image segmentation algorithm, the test segmentation based on the performance parameters of each pathologist, (v) repeating steps (ii) through (iv) until the test segmentation converges, and (vi) outputting the WSI with the ground truth labels.

[0012] In some embodiments, at least 100 individual WSIs are obtained, and wherein each WSI generates at least 300 individual patches.

[0013] In some embodiments, the machine learning model is a convolution neural network.

[0014] In some embodiments, the computer implemented method further comprises retraining of the trained machine learning model, wherein the retraining comprises: obtainingAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT patches corresponding to WSIs of a specific type, wherein the WSIs comprise positive regions and negative regions; freezing the learned sets of parameters corresponding to initial layers of the trained machine learning model and replacing a fully connected layer and an outer layer of the trained machine learning model with a new fully connected layer and a new outer layer to generate a transformed machine learning model; fine tuning, using the patches, the transformed machine learning model to classify each patch as positive or negative, wherein the fine tuning comprises performing iterative operations to adjust (i) to adjust the new fully connected layer and the new outer layer to learn specific features and (ii) to unfreeze the learned sets of parameters corresponding to the initial layers of the trained machine learning model to maximize model performance; and outputting the transformed machine learning model that classifies patches from the WSI as either positive or negative.

[0015] In some embodiments, the patches are generated by: selecting regions comprising morphological features of each WSI; dividing the selected regions into a grid of patches, wherein each patch of the grid corresponds to a portion of the WSI; and selecting, by the plurality of pathologists or a new set of a plurality of pathologists, the patches comprising tumor tissues to generate the patches for retraining the trained machine learning model.

[0016] In some embodiments, at least 100 regions are selected across at least 20 different WSIs.

[0017] In some embodiments, the patches comprising the tumor tissues consist of about 25% or more of the tumor tissues and is positive for one or more features including tumor stroma, non-invasive urothelial carcinoma, invasive urothelial carcinoma, low-grade tumors, highgrade tumors, squamous cell differentiation, glandular differentiation, or any combination thereof.

[0018] In some embodiments, the patches comprising the tumor tissues exclude regions with necrosis, extensive cautery' effect, washed off tissue, blurred view or any combination thereof.

[0019] In some embodiments, the patches comprising the tumor tissues (i) consist of about 25% or more of tumor tissue and is positive for one or more features including tumor stroma, non-invasive urothelial carcinoma, invasive urothelial carcinoma, low-grade tumors, highgrade tumors, squamous cell differentiation, glandular differentiation, or any combination thereof, and (ii) exclude regions with necrosis, extensive cautery' effect, washed off tissue, and / or blurred view.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0020] In some embodiments, the trained machine learning model is deployed into a production environment, and wherein deployment comprises: accessing a tissue sample from a subject; imaging, using a WSI scanner, the tissue sample to generate a WSI; generating, from the WSI, a set of patches, wherein each patch from the set of patches corresponds to a portion of the WSI; inputting, into the trained machine learning model, the set of patches, wherein the trained machine learning model annotates each patch in the set of patches as positive or negative; and generating a segmentation mask or a heatmap based on the patch annotations from the trained machine learning model.

[0021] In some embodiments, the deployment further comprises staining the tissue sample, wherein the stained tissue sample is being imaged using the WSI scanner.

[0022] In some embodiments, the staining is Hematoxylin and Eosin (H&E) staining.

[0023] In various embodiments, a method disclosed herein comprises: accessing a tissue sample from a subject; imaging, using a whole-slide image (WSI) scanner, the tissue sample to generate a WSI; generating, from the WSI, a set of patches, wherein each patch from the set of patches corresponds to a portion of the WSI; inputting the set of patches into a trained machine learning model, wherein the trained machine learning model annotates each patch in the set of patches as positive or negative; and generating a segmentation mask or a heatmap based on the patch annotations from the trained machine learning model.

[0024] In some embodiments, the method further comprises training a machine learning model, wherein the training comprises: accessing a training set of WSIs, wherein each WSI of the training set is a WSI of a specimen associated with a disease or a specimen unassociated with the disease; generating, ground truth labels for the training set of WSIs. wherein the generating comprises, for each WSI of the training set, obtaining, from a plurality of pathologists, annotations of the WSI from each of the plurality of pathologists, wherein the annotations indicate whether a region on the WSI corresponds to the disease or not, and generating, by inputting the annotations of the WSI from each of the plurality of pathologists into an image segmentation algorithm, a WSI with ground truth labels, wherein the ground truth labels represent probabilistically determined ground truths for each annotation; generating, from each WSI with corresponding ground truth labels, a set of patches, wherein each patch from the set of patches corresponds to a portion of the ground truth labeled WSI; training, using the set of patches for each WSI. a machine learning model to classify each patch from the set of patches as diseased or healthy, wherein the training comprises iterativeAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT operations to adjust a set of model parameters to minimize a loss or error function of the machine learning model, and wherein the loss or error function is configured to measure a difference between output predictions by the machine learning model and the ground truth labels; and outputting the trained machine learning model.

[0025] In some embodiments, the image segmentation algorithm is a consensus image generation algorithm.

[0026] In some embodiments, the image segmentation algorithm is performs iterative operations, wherein the iterative operations comprise: (i) generating a test segmentation of the WSI of the specimen by grouping each pixel in the WSI of the specimen based on similar properties, (ii) assigning performance parameters to each pathologist in the plurality of pathologists by comparing the annotations from the plurality of pathologists to the test segmentation, (iii) assigning each pixel in the test segmentation of the WSI to an annotation unit based on the annotations provided by the plurality of pathologists and their respective performance parameters, (iv) updating, using the image segmentation algorithm, the test segmentation based on the performance parameters of each pathologist, (v) repeating steps (ii) through (iv) until the test segmentation converges, and (vi) outputting the WSI with the ground truth labels.

[0027] In some embodiments, the method further comprises retraining of the trained machine learning model, wherein the retraining comprises: obtaining patches corresponding to WSIs of a specific ty pe, wherein the WSIs comprise positive regions and negative regions; freezing the learned sets of parameters corresponding to initial layers of the trained machine learning model and replacing a fully connected layer and an outer layer of the trained machine learning model with a new fully connected layer and a new outer layer to generate a transformed machine learning model; fine tuning, using the patches, the transformed machine learning model to classify each patch as positive or negative, wherein the fine tuning comprises performing iterative operations to adjust (i) to adjust the new fully connected layer and the new outer layer to leam specific features and (ii) to unfreeze the learned sets of parameters corresponding to the initial layers of the trained machine learning model to maximize model performance; and outputting the transformed machine learning model that classifies patches from the WSI as either positive or negative.

[0028] In some embodiments, the disease is a cancer.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0029] In some embodiments, the method further comprises generating a cancer status prediction for the subject based on the patch annotations and / or the segmentation mask or heatmap.

[0030] In some embodiments, the method further comprises generating a treatment plan based on the cancer status prediction and optionally administering a specific treatment to the subject based on the treatment plan.

[0031] In some embodiments, a system is provided that includes one or more processors, and a memory that is coupled to the one or more processors and stores a plurality of instructions which, when executed by the one or more processors, cause the one or more processors to perform any of the methods disclosed herein.

[0032] In some embodiments, a computer-program product is provided that is tangibly embodied in a non-transitory computer-readable memory that includes instructions which, when executed by the one or more processors, cause the one or more processors to perform any of the methods disclosed herein.

[0033] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the invention claimed. Thus, it should be understood that although the present invention has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this invention as defined by the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Aspects and features of the various embodiments will be more apparent by describing examples with reference to the accompanying drawings, in which:

[0035] FIG. 1 shows an exemplary computer environment for processing, storing, transmitting, and displaying pathology' images in accordance with various embodiments.

[0036] FIG. 2 illustrates the training and implementation of a machine learning model that processes digital pathology’ images in accordance with various embodiments.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0037] FIG. 3A shows an exemplary' digital pathology' image that illustrates how more than one pathologist's annotations may be overlay ed to generate a ground truth segmentation to be used by an image segmentation algorithm in accordance with various embodiments.

[0038] FIG. 3B shows an exemplary7digital pathology' image that illustrates the output of an image segmentation algorithm in accordance with various embodiments.

[0039] FIG. 4A shows the various layers of an exemplary' convolution neural network (CNN) in accordance with various embodiments.

[0040] FIG. 4B illustrates processing of an image via the various layers of the CNN in accordance with various embodiments.

[0041] FIG. 5 illustrates a selected region from a WSI being divided into a grid for pathologist review in accordance with various embodiments.

[0042] FIG. 6 illustrates a flowchart for training a machine learning model in accordance with various embodiments.

[0043] FIG. 7 illustrates a flowchart for using a trained machine learning model in a production environment in accordance with various embodiments.TERMS

[0044] As used herein, the articles “a” and “an” are used herein to refer to one or to more than one (i.e., at least one) of the grammatical object of the article. By way of example, an element means at least one element and can include more than one element.

[0045] As used herein, the terms “about,” “similarly,” “substantially,” and “approximately” are defined as being largely but not necessarily wholly what is specified (and include wholly what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the term “about,” “similarly,” “substantially,” or “approximately” may be substituted with “within [a percentage] of’ what is specified, where the percentage includes 0.1 percent, 1 percent, 5 percent, and 10 percent, etc. Moreover, the term terms “about,” “similarly,” “substantially,” and “approximately” are used to provide flexibility' to a numerical range endpoint by providing that a given value may be slightly above or slightly below the endpoint without affecting the desired result.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0046] As used herein, the term “and / or"’ refers to and encompasses any and all possible combinations of one or more of the associated listed items, as well as the lack of combinations were interpreted in the alternative (“or”).

[0047] As used herein, when an action is “based on” something, this means the action can be based at least in part on at least a part of the something.

[0048] As used herein, the term “cancer” refers to an abnormal state or condition characterized by rapidly proliferating cell growth. Rapidly proliferating cells may be categorized as pathologic (i.e.. characterizing or constituting a disease state), or may be categorized as non-pathologic (i.e., a deviation from normal but not associated with a disease state). In addition, cancer cells can spread locally or through the bloodstream and lymphatic system to other parts of the body. In general, a cancer will be associated with the presence of one or more tumors (i.e., abnormal cell masses). The term “tumor” is meant to include all types of cancerous growths or oncogenic processes, metastatic tissues or malignantly transformed cells, tissues, or organs, irrespective of histopathologic type or stage of invasiveness. Examples of cancer include malignancies of various organ systems, such as bladder cancers, lung cancers, breast cancers, thyroid cancers, lymphoid cancers, gastrointestinal cancers, and genito-urinary tract cancers. Cancer can also refer to adenocarcinomas, which include malignancies such as colon cancers, renal-cell carcinoma, prostate cancer and / or testicular tumors, non-small cell carcinoma of the lung, cancer of the small intestine, and cancer of the esophagus. Carcinomas are malignancies of epithelial or endocrine tissues including respiratory system carcinomas, gastrointestinal system carcinomas, genitourinary’ system carcinomas, testicular carcinomas, breast carcinomas, prostatic carcinomas, endocrine system carcinomas, and melanomas. An “adenocarcinoma” refers to a carcinoma derived from glandular tissue or in which the tumor cells form recognizable glandular structures. A “sarcoma” refers to a malignant tumor of mesenchymal derivation. “Melanoma” refers to a tumor arising from a melanocyte. Melanomas occur most commonly in the skin and are frequently observed to metastasize widely.

[0049] As used herein, the term “healthy” refers to tissue or cells that are normal, functioning as expected, and free of abnormal growth or mass formation. Healthy tissue lacks the uncontrolled cell proliferation characteristic of tumors, whether benign or malignant. In some embodiments, the term “healthy” can be used interchangeably with “non-tumor,” denoting an absence of tumor-related abnormalities.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0050] As used herein, the terms “including,” “comprising,” “having,” and variations thereof, encompass elements listed thereafter and equivalents thereof as well as additional elements. Embodiments recited as “including,” “comprising,” or “having” certain elements are also contemplated as consisting essentially of and consisting of those certain elements.

[0051] As used herein, the term “or” encompass both the inclusive sense, where either one or both of the conditions or elements can be present, and the exclusive sense, where only one of the conditions or elements can be present. In some instances, the term “or” is used interchangeably with “and / or.”

[0052] As described herein, “patient,” and “subject” are used interchangeably and refer to any single animal, more preferably a mammal (including humans and non-human animals such as dogs, cats, horses, rabbits, rats, cows, pigs, sheep, and non-human primates). Thus, the methods described herein are applicable to both human and veterinary disease. In certain embodiments, subjects are “patients,” i.e., living humans that are receiving medical care for a disease or condition. This includes persons with no defined illness who are being investigated for signs of pathology'. In some embodiments, the patient or subject may be at risk of / diagnosed with / being treated for cancer.

[0053] As used herein, the term “pixel” or “voxel” refers to the smallest individual unit of a digital image, representing a specific intensity or color value at a given spatial location. The term “pixel” can be used interchangeably with the term “voxel” in data representations.

[0054] Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. For example, if a concentration range is stated as 1% to 50%, it is intended that values such as 2% to 40%, 10% to 30%, or 1% to 3%, etc., are expressly enumerated in this specification. These are only examples of what is specifically intended, and all possible combinations of numerical values (e.g., integer, whole number, decimal, fraction, and the like) between and including the lowest value and the highest value enumerated are to be considered to be expressly stated in this disclosure.DETAILED DESCRIPTION

[0055] The ensuing description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the ensuingAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT description of the preferred exemplary' embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0056] Specific details are given in the following description to provide a thorough understanding of the embodiments. How ever, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary' detail in order to avoid obscuring the embodiments.

[0057] Also, it is noted that individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart or diagram may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0058] Publications cited herein and the material for which they are cited are hereby specifically incorporated by reference in their entireties.Introduction

[0059] Large medical centers can typically encompass an average caseload of approximately 80,000 to over 100,000 different pathology’ samples per year, with daily averages ranging from 500 to 750 tissue blocks and more than 1,500 slides. Conventionally, histologic grading is manually' performed by one or more pathologists / technicians, where a single diagnosis can take several minutes (e.g., 20 minutes) to up to several weeks (e.g., two weeks) depending on the complexity of the specimen and diagnosis. Given the large work volume, pathology analysis is inherently subjected to inter-pathologist variability, resulting in reduced diagnostic accuracy. The human brains have inherent limitations in consistently identifying subtle diagnosticAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT features or tissue patterns. While manual examination is the current standard in most cases, integration of computational approaches, such as machine learning (ML) and AL into pathology offer tremendous potential to enhance data analysis, efficiency, speed, and cost.

[0060] More specifically, Al algorithms, particularly those utilizing deep learning, can analyze vast quantities of pathology7images rapidly and with remarkable precision, identifying patterns and anomalies that might be missed by human eyes. This capability has led to earlier and more accurate diagnoses, which are crucial for effective treatment and improved patient outcomes. Improved diagnostic accuracy can also reduce the costs associated with misdiagnosis and unnecessary treatments. Furthermore, Al-powered tools can standardize diagnostic interpretations, reducing variability among pathologists and ensuring more consistent results. The integration of Al in digital pathology also supports pathologists by automating routine and time-consuming tasks, reducing the workload for pathologists, allowing pathologists to focus on more complex cases and decision-making processes, and ultimately leading to cost savings for healthcare institutions. Overall, the application of Al in digital pathology holds great promise for transforming diagnostic practices and enhancing the overall quality and cost of healthcare.

[0061] However, the implementation of Al in digital pathology is not without challenges, particularly concerning the availability of high-quality training data (i.e., training data having accurate ground truths) for supervised and semi-supervised learning. Supervised and semisupervised learning models rely on large datasets of labeled images to train algorithms effectively. These labels need to be accurate and consistent, requiring substantial input from experienced pathologists. The scarcity of such high-quality labeled data can hinder the development and performance of Al models. Additionally, the variability' in staining techniques, imaging equipment, and sample preparation across different institutions can result in non-standardized data, further complicating the training process. Ensuring the privacy and security7of patient data also presents significant challenges, as does the need for robust validation of Al models to ensure their reliability7and generalizability across diverse clinical settings. Addressing these issues is crucial for realizing the full potential of Al in digital pathology and ensuring its successful integration into clinical practice.

[0062] To address these challenges and others, disclosed herein are techniques (e.g., computer implemented methods, systems, and computer-program products) for training and deployment of models capable of performing image analysis tasks such as the segmentationAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT and classification of objects within digital pathology' images. In one aspect, a model having a convolutional neural network (CNN) architecture is trained on hundreds of whole-slide images (WSIs) resulting in thousands of patches for each WSI, where each patch represents a small region corresponding to a portion of the WSI. The WSIs are annotated by multiple pathologists to include various information or labels including regions of interest (e.g., bounding boxes, segmentation masks, and other metadata), diagnostic biomarkers or molecular markers that are relevant to the diagnosis or classification of a disease (e.g., a cancer), detailed markings of abnormalities, such as tumors, necrosis, fibrosis, or other disease-related changes, different cell types present in the tissue, such as normal cells, cancerous cells, immune cells, etc., measurements of various structures, such as the size of tumors, length of blood vessels, or density of certain cell types, or any combinations thereof.

[0063] The ground truth for the WSIs are established from multiple annotations or segmentations using a consensus image generation algorithm (e.g., Simultaneous Truth and Performance Level Estimation (STAPLE)). Multiple annotations or segmentations of a WSI are obtained, e.g., by a group of pathologists annotating the WSI to identify regions of interest, such as tumor boundaries or specific cell types, or by different computational image segmentation techniques. Given the potential variability caused by subjective interpretations, differences in expertise, or algorithmic discrepancies, the consensus image generation algorithm is applied to integrate these individual annotations or segmentations. This process results in a single, probabilistically determined ground truth that more accurately and reliably represent the actual pathology'. This consensus ground truth is then used to generate high-quality training data for training and validating the model, ensuring that the model is built on the most accurate and representative information available. This approach enhances diagnostic accuracy and consistency in digital pathology assays, addressing variability and improving the reliability’ of disease detection and classification.

[0064] In another aspect, the model is adapted from a first clinical indication (e.g., tumor identification in bladder tissue images) to a second clinical indication (e.g., tumor identification in lung tissue images) using transfer learning. Transfer learning is a powerful technique in machine learning that leverages a pre-trained model developed for one task and adapts it to perform a related but different task. In the present aspect, a model has been trained to segment and classify tumors in bladder tissue. This model has already learned to identify various features, patterns, and structures relevant to tumor detection through extensive training on bladder tissue images. To adapt this model for segmenting and classifying tumors in lung tissue,Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT transfer learning is employed. The initial layers of the pre-trained model, which capture general features such as edges, textures, and basic shapes, are retained. These features are often generic enough to be useful across different types of tissues. By keeping these layers, the model benefits from the knowledge already gained during the bladder tissue training phase.

[0065] To specialize the model for the second clinical indication, the later layers, which are more task-specific, can be fine-tuned or retrained using a smaller, labeled dataset of lung tissue images. This process involves adjusting the weights of these layers to better capture the specific characteristics and patterns of lung tumors. By doing so, the model can learn to differentiate between healthy and tumorous lung tissue while still leveraging the foundational knowledge it acquired from bladder tissue. This approach is highly efficient because it requires significantly less training data and computational resources compared to training a model from scratch. Moreover, transfer learning can lead to faster convergence and improved performance, as the model starts with a solid base of learned features, allowing it to adapt more quickly to the new task.

[0066] An advantage of using transfer learning as opposed to simply training a new model from scratch is the efficiency and resource savings it offers. Transfer learning allows a model to leverage the knowledge acquired from a pre-trained model on a related task, thus significantly reducing the amount of training data and computational resources needed. This is particularly beneficial in fields like digital pathology' where acquiring large, high-quality labeled datasets can be both time-consuming and expensive. By starting with a pre-trained model (e.g., ImageNet or a pre-trained tumor detection model) that has already learned to recognize general features, one can achieve faster convergence and often better performance with less data and fewer training iterations. This efficiency makes transfer learning a practical and powerful tool for adapting models to new but related tasks, enabling quicker deployment and more effective utilization of machine learning in real-world applications.

[0067] Consequently, in various embodiments, a computer-implement method is disclosed that comprises accessing whole-slide images (WSIs), wherein each WSI is aWSI of a specimen associated with a disease or a specimen unassociated with the disease; generating, ground truth labels for the WSIs, wherein the generating comprises, for each WSI, obtaining, from a plurality of pathologists, annotations of the WSI from each of the plurality of pathologists, wherein the annotations indicate whether a region on the WSI corresponds to the disease or not, and generating, by inputting the annotations of the WSI from each of the plurality ofAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT pathologists into an image segmentation algorithm, a WSI with ground truth labels, wherein the ground truth labels represent probabilistically determined ground truths for each annotation; generating, from each WSI with corresponding ground truth labels, a set of patches, wherein each patch from the set of patches corresponds to a portion of the ground truth labeled WSI; training, using the set of patches for each WSI, a machine learning model to classify each patch from the set of patches as diseased or healthy, wherein the training comprises iterative operations to adjust a set of model parameters to minimize a loss or error function of the machine learning model, and wherein the loss or error function is configured to measure a difference between output predictions by the machine learning model and the ground truth labels; and outputting the trained machine learning model.

[0068] The computer implemented method further comprises retraining or fine-tuning the trained machine learning model, to generate a new machine learning model that can classify patches from WSIs of a specific group (e.g., a WSI depicting tumor or necrosis, a WSI depicting a non-tumor tissue, a WSI depicting a non-bladder tissue, or the like). The retraining or fine-tuning comprises: obtaining patches corresponding to WSIs comprising positive regions (e.g., tumor tissue) and negative regions (e.g., non-tumor tissue), freezing the learned sets of parameters corresponding to the initial layers of the trained machine learning model and replacing the fully connected layer and outer layer of the trained machine learning model with a new fully connected layer and a new outer layer to generate a transformed machine learning model, fine tuning, using the patches, the transformed machine learning model to classify each patch as positive (e.g., diseased) or negative (e.g., healthy), wherein the fine tuning comprises performing iterative operations (i) to adjust the new fully connected layer and the new outer layer to leam the specific features and (ii) to unfreeze the learned sets of parameters corresponding to the initial layers of the trained machine learning model to maximize model performance: and outputting the transformed machine learning model that classifies patches from the WSI as either positive or negative.Computer Environment of Processing Pathology Images

[0069] Pathology images are high-resolution visual representations of tissue samples, often created using microscopy techniques such as whole-slide imaging (WSI). With advancements in technology, the field has transitioned toward digital pathology, which involves the digitization and computational analysis of these images. Digital pathology involves a range of tasks aimed at enhancing the accuracy and efficiency of disease diagnosis and research. TheseAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT tasks include the digital acquisition of high-resolution images from glass slides using whole slide imaging technology. Once digitized, object detection algorithms identify and locate specific structures such as cells, tissues, or pathological features within the images. Segmentation techniques then partition the images into meaningful regions, such as delineating tumor boundaries from non-tumor tissue. Classification algorithms categorize detected and segmented objects into predefined classes, such as distinguishing between normal and cancerous cells or identifying different types of tumors. These tasks may be supported by machine learning and artificial intelligence, which help automate and standardize analysis, enabling pathologists to make more precise and consistent diagnoses. Additionally, digital pathology facilitates remote consultations, data management, and integration with other digital health records, ultimately contributing to more informed therapeutic decision-making and improved patient outcomes.

[0070] FIG. 1 illustrates an exemplary computer environment 100 for processing, storing, transmitting, and displaying pathology' images in accordance with various embodiments. The computing environment 100 includes an image processing platform 150 that enables the processing and analyzing of digital image of a biological sample (e.g., pathological images such as microscopic images of tissues). The image processing platform 150 can be implemented using software only (e.g., each module of the platform is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware (e g., a medical tool to perform sample collection, a digital microscope to generate digital image, a GPU, a CPU, or the like), or using a combination of hardware and software. The computing environment 100 also includes a storage device 140, a network 160, and a client device 170. Although FIG. 1 illustrates a particular number of the components, it should be understood that any suitable number of components (e.g., multiple client devices 170) may be included in the computing environment 100. It should also be understood that the computing environment 100 may include additional components such as a camera system, an optical system, a whole slide scanner, an image management platform, or the like.A. Image Processing Platform

[0071] The image processing platform 150 may include multiple components or modules. For example, as shown in FIG. 1, the image processing platform 150 includes a sample collector 101 that collects biological samples 105, a sample processing and imaging system 110 that process the collected sample and generate digital images 135, a preprocessing moduleAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT 142, an analyzing module 145, and a diagnostic module 146. Each of the components or modules may be a solely software-based component or module, a solely hardware-based component or module, or a combination of software and hardware. Each component or module can be physically connected or integrated, or interact with each other through computing programming codes, scripts, or software. In different embodiments, the design of the digital image processing platform can be specifically tailored based on research, clinical, or commercial needs.

[0072] The sample collector 101 can be a physical device that collects and prepares biological samples 105 for imaging. For example, the sample collector 101 can include automated slide loaders or robotic arms for precise handling and positioning of samples, sample trays or carousels for organizing multiple samples, and sample preparation tools such as tissue processors, microtomes, and staining apparatus. The sample collector 101 can also incorporate sample labeling and tracking mechanisms like barcode scanners and RFID tags to maintain sample identification and metadata management. The sample collector 101 may also include controlling chips that enable environmental controls, such as temperature and humidity regulation to ensure sample integrity. The sample collector 101 can also include software or systems that facilitate the collection, preparation, tracking, and management of the biological samples 105. In some embodiments, the sample collector 101 can also be controlled by a user through a user interface and control software.

[0073] The biological samples 105 can encompass a variety of biological materials, including cell-containing liquids or tissues. Examples include but are not limited to amniotic fluid, tissue biopsies, blood, blood cells, bone marrow, fine needle biopsy samples, peritoneal fluid, plasma, pleural fluid, saliva, semen, serum, tissue homogenates, and frozen or paraffin sections of tissue. Methods for acquiring the biological samples 105 range from biofilms, aspirations, and tissue sections to swabs, blood draws, and surgical or needle biopsies. The biological samples 105 can originate from either a healthy subject or a subject afflicted with a disease, such as cancer.

[0074] Once the biological samples 105 have been collected or obtained, it enters a sample processing and imaging system 110 for converting the biological samples 105 into high-resolution digital images 135. The sample processing and imaging system 110 comprises a fixation / embedding system 115, a sectioning system 120, a staining system 125, and an imaging system 130. The sample processing and imaging system 110 prepares the biologicalAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT sample 105 for staining procedures, such as histological staining, which are useful for highlighting features of interest and enhancing contrast in sectioned tissues or cells. Staining techniques can be employed to identify specific types of cells, nucleic acids, or proteins, thereby facilitating microscopic examination. The stained sample can then be analyzed to determine the quantify and characteristics of features of interest, such as cell count, densify, expression levels, and spatial relationships among cellular structures.

[0075] The sample processing and imaging system 110 can be equipped with advanced optical components, such as high-quality objective lenses and precise light sources, as well as high-resolution cameras or image sensors, to capture detailed images of the samples. In some embodiments, the sample processing and imaging system 110 includes a motorized stage for precise movement and positioning of the slides, coupled with an autofocus system to ensure sharpness across the entire sample. Additionally, the sample processing and imaging system 110 integrates powerful control and processing software to manage image capture, real-time viewing, and initial enhancement of the images.

[0076] The sample processing and imaging system 110 can interact with the sample collector 101 in real time, facilitating a seamless workflow for sample processing and imaging. For instance, once a tissue sample is collected and prepared by the sample collector 101, it can be immediately imaged by the sample processing and imaging system 110 to capture its initial state. Following this, the sample processing and imaging system 110 performs the physical staining of the tissue sample, applying specific staining protocols to enhance the visibility of various structures within the sample. After the staining process is complete, the sample processing and imaging system 110 captures a second set of images, now highlighting the stained features. This real-time interaction between the sample collector 101 and the sample processing and imaging system 110 ensures that each step of the sample preparation and imaging is precisely coordinated, resulting in high-quality, detailed images for accurate training, analysis, and diagnosis.

[0077] A fixation / embedding system 115 fixes and / or embeds the biological sample 105 (e.g., a sample including at least part of at least one tumor). Fixation is an irreversible chemical process that preserves the biological sample’s chemical composition and cellular structure while preventing degradation. Fixatives, such as formaldehyde, neutral buffered formalin (NBF), paraformaldehyde (PF A), methanol, or Bouin’s solution, are used to harden tissues, inactivate degradative enzymes, and eliminate bacteria. The fixation process can involveAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT perfusion or immersion of the tissue sample for a predetermined time. For liquid samples like blood, the sample may be smeared onto a slide and dried before fixation, potentially omitting the embedding and sectioning steps.

[0078] Embedding involves infiltrating the fixed tissue sample with a suitable histological wax, such as paraffin, or resins like sty rene or polyethylene. Since histological wax is insoluble in water, the tissue must first be dehydrated using graded alcohols, followed by clearing with a solvent like xylene. The tissue is then embedded in warm paraffin wax, which fills the spaces previously occupied by xylene and water. Once cooled, the wax forms a hardened block that can be clamped into a microtome for precise sectioning. Deviations from this procedure can result in improper infiltration, hindering subsequent analyses.

[0079] The sectioning system 120 is used to slice the fixed and / or embedded tissue sample into thin sections using instruments such as a cryostat, microtome, vibratome, or compresstome. Sectioning produces a series of tissue slices, typically 4-5 microns thick, suitable for microscopy such as light or fluorescent microscopy. In some cases, tissues can be rapidly frozen using dry' ice or Isopentane and sectioned in a cryostat with a cold knife. Alternatively, other cooling agents like liquid nitrogen can be used. Sections intended for microscopy are generally between 4-10 micrometers thick, while those embedded in epoxy or acrylic resin can be thinner, less than 2 micrometers. The sections are then mounted on glass slides for further processing by the staining system 125. Additionally, or alternatively, the sections may first be processed by the imaging system 130 to generate unstained bright field digital images of the section prior to being processed by the staining system 125.

[0080] Staining system 125 performs various staining techniques to enhance contrast and visualize specific biological or cellular components. Staining may be conducted manually, semi-automatically, or fully automatically. In manual staining, tissue sections are exposed to one or more stains — either consecutively or concurrently — to highlight different tissue characteristics, with predefined volumes of staining agents applied for specific durations. Automated or semi-automated Stainers use slide loaders and robotic arms to handle and position tissue samples precisely, ensuring consistent and accurate reagent application. These stainers are equipped with preparation tools, including reagents and specialized software, which allow operators to configure staining parameters, monitor progress, and ensure reproducibility. They can execute protocols for various staining types, such as Hematoxylin and Eosin (H&E),Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT immunohistochemistry (IHC), and special stains, with precise control over reagent volumes, incubation times, and temperatures to enhance consistency and reduce human error.

[0081] Many staining solutions are aqueous. Thus, to stain tissue sections, the embedding agent (e.g., wax) may need to be dissolved and replaced with water (rehydration) before a staining solution is applied to a section (e.g., deparaffinization). For example, the tissue section may be sequentially passed through xylene, decreasing concentration of ethyl alcohol (from about 100% to 0%), and water. Once stained, the tissue sections may be dehydrated again and placed in xylene. The tissue section may then be mounted on microscope slides in a mounting medium dissolved in xylene. A coverslip may be placed on top to protect the sample section. The evaporation of xylene around the edges of the coverslip may dry the mounting medium and bond the coverslip firmly to the slide.

[0082] Various types of staining protocols may be used to perform the staining. One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of tissue morphology and / or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is H&E, which differentiates cellular structures by staining nuclei blue and cytoplasm pink. Additional special stains or immunohistochemical markers may be used to highlight specific proteins or cellular components. Other examples of histochemical stains include trichrome stains (e.g., Masson’s Trichrome), which are used to differentiate between collagen and muscle fibers in tissue sections, Periodic Acid-Schiff (PAS), which is used to detect polysaccharides, muco-substances, and glycoproteins in tissues, silver stains, such as Gomori’s Methenamine Silver (GMS) and Reticulin stains, which are used to highlight certain types of tissue fibers and microorganisms, and iron stains, such as the Prussian Blue stain, which are used to detect ferric iron (Fe3+) in tissues.

[0083] Another type of tissue staining is immunohistochemistry (IHC). also called “immunostaining”), which is used to visualize specific proteins or antigens within tissue sections by leveraging the principles of antibody-antigen interactions. The process begins with the fixation of tissue samples to preserve cellular structures, followed by embedding in paraffin and sectioning into thin slices placed on glass slides. These sections are then exposed to primary antibodies that specifically bind to the target antigens. A secondary antibody, conjugated to a detectable marker such as an enzyme or a fluorescent dye, is applied to bind the primaryAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT antibody. In chromogenic IHC, the enzyme catalyzes a reaction that produces a colored precipitate at the antigen site, making it visible under a light microscope. In fluorescent IHC, the fluorescent dye emits light when excited by a specific wavelength, allowing visualization under a fluorescence microscope. IHC staining is widely used in pathology7to diagnose diseases, identify specific cell ty pes, and study the distribution and localization of proteins within tissues, providing critical insights into cellular functions and disease mechanisms.

[0084] Once the stained and / or unstained tissue sections are prepared , the next step involves digitizing these slides to create whole-slide images (WSIs). Imaging system 130 scans or generates a raw digital pathology, histopathology7, or bright field digital image of the stained and / or unstained samples. In some instances, each tissue section may be mounted on a slide, which is then scanned by a whole-slide scanner, a sophisticated piece of equipment equipped with high-resolution optics and advanced sensors to create a digital image. The scanner captures bright field and / or fluorescence detailed images of the entire tissue section at various magnification levels, often ranging from 20x to 40x. The scanning process is precise to ensure that even the smallest details are accurately captured without introducing artifacts or distortions. The resulting digital images are typically large, high-resolution files stored in specialized formats, such as TIFF or proprietary formats designed for pathology7images. These digital files are then managed by imaging system 130, which further provides robust data management and archival solutions (i.e., storage device 140) to handle the large file sizes and ensure the integrity and accessibility of the data.

[0085] The scanner is equipped with a scanning mechanism, a light source, optical system, and a camera. The scanning mechanism moves the slide or the camera in a systematic pattern, capturing overlapping fields of view (FOVs) across the entire tissue section. This process is similar to stitching together multiple photographs to create a panoramic image. The scanner must ensure accurate alignment and focus for each FOV to avoid any blurring or distortion. The light source is a consistent and bright light source, such as an LED, which is used to illuminate the tissue section. The light generated by the light source passes through the tissue as it is moved by the scanning mechanism and is captured by the optical system. The optical system is high-resolution optics, including objective lenses with vary ing magnification levels (commonly 20x and 40x). The choice of magnification depends on the level of detail required for the analysis. In embodiments directed to fluorescent stains, the scanner is equipped with high-resolution optics and filters specific to the excitation and emission wavelengths of the fluorophores used. The light source is used to excite the fluorophores. A high-resolution digitalAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT camera such as CCD or CMOS sensors is used to capture the illuminated tissue image and / or fluorescence from the tissue section. These cameras are often capable of capturing images at a resolution of several gigapixels, ensuring that even the smallest cellular details are accurately recorded. The individual FOVs are stitched together to form a continuous, high-resolution WSI for each color and / or fluorescence channel. In some instance, patches, or smaller sections of the WSI, are extracted from defined regions of interest (ROIs). This extraction process involves selecting a specific region within the WSI and cropping it to create a smaller image file. The size of the patches can vary depending on the level of detail needed for analysis. Common patch sizes might range from a few hundred pixels to several thousand pixels in width and height.

[0086] The WSI and / or patches (e.g., the digital images 135) can be stored in the storage device 140, such as a server, a database, or a data repository. The digital images 135 may be stored locally, remotely, and / or in a cloud sen' er. Each of the digital images 135 may be stored in association with an identifier of a subject and a date (e.g., a date when a sample was collected and / or a date when the image was captured), or other metadata such as image identifiers, staining protocols, magnification level, operator information, and additional notes.

[0087] The digital images 135 output by the sample processing and imaging system 110 can be sent to the preprocessing module 142, analyzing module 145, and diagnostic module 146 for further analysis. The preprocessing module 142 can include preprocessing software that performs tasks such as noise reduction, contrast enhancement, image normalization, and artifact removal to ensure that the images are of the highest quality before analysis. It may also incorporate hardware accelerators (e.g., GPUs, TPUs, FPGAs, or the like) to speed up these preprocessing tasks, ensuring efficient handling of large image datasets.

[0088] In some embodiments, the preprocessing module 142 prepares images or patches for machine learning applications (e.g., training and validating a machine learning model). The preprocessing module may chop the digital images 135 into patches (or tiles) and the patches are to be used for training purposes. In some embodiments, quality checks are implemented to ensure that the images or patches are representative and of high quality (e.g., verifying that patches are not too noisy, blurred, or contain artifacts that could skew the training or analysis). The quality check may also be performed via human intervention.

[0089] In some embodiments, the preprocessing module 142 also performs training functions. Patches may also be extracted from the digital images 135 for training based onAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT extraction algorithms to ensure that different features of the digital images 135 are represented by the selected patches and further improve training efficiency. Extracted patches may be input into a selected machine learning model to train or fine-tune the machine learning model. The trained machine learning model may be used by the analyzing module 145 to perform annotation or segmentation tasks, or by the diagnostic module 146 to perform disease prediction, diagnosis, or design a personalized treatment. The training performed at the preprocessing module 142 may be a supervised training, an unsupervised learning, a semisupervised learning, or a reinforcement learning. The machine learning model may be selected from linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs), k-nearest neighbors (KNN), artificial neural network architectures like convolutional neural networks (CNNs), recurrent neural networks (RNNs). or generative adversarial network (GANs), k-means clustering, hierarchical clustering, Gaussian mixture models, principal component analysis (PCA), Q-leaming, deep Q-networks (DQNs), and policy gradient methods. Additionally, ensemble methods tike boosting (e.g., AdaBoost, Gradient Boosting Machines), bagging (e.g.. Bootstrap Aggregating), and stacking can be used to combine multiple models to enhance learning and performance.

[0090] Images or patches are further processed and analyzed using the analyzing module 145. The analyzing module 145 applies computational techniques to annotate and segment the digital images 135. The analyzing module 145 may also perform other functions such as quantitative analysis (e.g., cell counting, biomarker identification and quantification), pattern recognition (e.g., feature extraction and classification of tissues), spatial analysis (e.g., analyzing spatial relationships by reconstructing three-dimensional images), anomaly detection, and other machine learning or artificial intelligence integration.

[0091] In some embodiments, the analyzing module 145 employs one or more trained machine learning model to perform image annotation. The machine learning model may be provided by the preprocessing module 142 or may be a model stored in the storage device 140. The analyzing module 145 may take a digital image 135 output from the sample processing and imaging system or a preprocessed image from the preprocessing module 142 and input the image into the trained machine learning model to generate annotated image. The input image can be a grayscale image, or a stained image with enhanced contrast. The output image is an annotated image (or a segmentation mask or a heat map) that highlights features, structures, and / or components of the biological sample 105, enabling detailed visualization and analysis.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0092] The output image can be further processed and used in downstream applications. For example, the analyzing module 145 may employ segmentation algorithms, such as deep learning-based methods, to segment the annotated features, structures, or components from the image. In some embodiments, the features, structures or components are cells, tissues, nuclei, cytoplasm, cell membranes, proteins, enzymes, lipids, carbohydrates, glycogen, connective tissue, mitochondria, bacteria, fungi, nerve cells, fibers, and other anatomical features, and / or clinical or diagnostic biomarkers. Segmentation allows for the precise measurement and quantification of different features, structures, and components within the sample, including statistical analysis, pattern recognition, and anomaly detection. Segmentation also facilitates diagnostic and research purposes.

[0093] The diagnostic module 146 is configured to provide comprehensive and personalized diagnostic insights from the analyzed images. The diagnostic module 146 may integrate algorithms and machine learning models to interpret the data provided by the analyzing module 145, offering precise and reliable diagnostic information. It can utilize pattern recognition, anomaly detection, and predictive analytics to identify and classify various pathological conditions, aiding in the accurate diagnosis of diseases. The diagnostic module 146 can also generate detailed reports and visualizations, summarizing the findings and highlighting critical areas of concern. By leveraging the power of artificial intelligence and advanced data analysis, the diagnostic module 146 enhances the efficiency and accuracy of the diagnostic process, supporting medical professionals in making informed decisions and improving patient outcomes.B. Data and Model Storage

[0094] The storage device 140 can be served as a central repository for vast amounts of high-resolution image data and associated metadata. It can also save other medical data and models or algorithms. The storage device 140 can be a local storage that is connected to the network 160, facilitating seamless and high-speed data transfer between the storage device 140 and components of the image processing platform 150, ensuring efficient access and retrieval of data. In some embodiments, the storage device 140 is a part of the image processing platform 150. Theimage processing platform 150 can utilize the storage device 140 to archive and access raw images, processed images, and analytical results, maintaining data integrity7and availability7for various stages of image analysis and diagnostics. The image processing platform 150 can also utilize the storage device 140 to access or call a machine learning modelAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT or algorithm for data processing and analysis. Additionally, the storage device 140 interfaces with the client device 170, allowing medical professionals, researchers, and remote users to access and interact with the stored data through secure, real-time connections. The storage device 140 can also be configured to enable authorized use or access by authorized users. In some embodiments, the storage device 140 is a cloud-based storage, and may be an integral part of the network 160. In some embodiments, the computing environment 100 includes multiple storage devices, which may be associated with different components of the environment (e.g., one storage device for the image processing platform 150, and another storage device for the client device 170).C. Network

[0095] The network 160 supports data communications using a variety of available protocols including without limitation TCP / IP (transmission control protocol / Intemet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk®, and the like. Merely by way of example, the network 160 may be a local area network (LAN), networks based on Ethernet, Token-Ring, a wide-area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a public switched telephone network (PSTN), an infra-red network, a wireless network (e.g., a network operating under any of the Institute of Electrical and Electronics (IEEE) 1002.11 suite of protocols, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0096] To connect the image processing platform 150 (or any of its modules), the storage device 140, and the client device 170 to the network 1 0 (or to each other), various types of links can be utilized. These links can include wireline connections such as Digital Subscriber Line (DSL) or Data Over Cable Senice Interface Specification (DOCSIS), which provide reliable and high-speed data transfer over copper or coaxial cables. Wireless connections, such as Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX), offer the flexibility of mobility7and ease of installation without the need for physical cables. Additionally, optical links like Synchronous Optical Network (SONET) or Sy nchronous Digital Hierarchy (SDH) provide high bandwidth and long-distance data transmission capabilities, making them suitable for connecting multiple components across different locations. It is important to note that the type of link used to connect these components can vary based on specific network requirements, infrastructure availability7, and performance needs. The integration of these diverse connection methods ensures a robust and efficient data communication framework,Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT enabling seamless interaction and real-time data access across the entire computing environment 100. This versatile connectivity supports the complex workflows and data-intensive operations performed by the image processing platform 150.D. Client Device

[0097] The client device 170 is an electronic device including hardware, software, or embedded logic components or a combination of two or more such components and capable of interacting with the image processing platform 150 and the storage device 140, optionally via the network 160. The client device 170 may include various types of computing systems such as portable handheld devices, general purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems, Uinux or Uinux-like operating systems such as Google Chrome™ OS) including various mobile operating systems (e.g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices may include cellular phones, smartphones, (e.g., an iPhone), tablets (e.g., iPad®), personal digital assistants (PDAs), and the like. Wearable devices may include Google Glass® head mounted display, and other devices. The client device 170 may be capable of executing various different applications such as various Internet-related apps, communication applications (e.g.. E-mail applications, short message service (SMS) applications) and may use various communication protocols. Any suitable client device 170 can be configured to generate and output product target discovery content to a user. For example, users may use the client device 170 to execute one or more applications, which may generate one or more discovery or storage requests that may then be serviced in accordance with the teachings of this disclosure. The client device 170 provides an interface 172 (e.g., a graphical user interface) that enables a user of the client device 170 to interact with the client device 170 and other components of the computing environment 100. The client device 170 may also output information to the user via the interface 172 (e.g., displaying a report, a heatmap, or the like).

[0098] The client device 170 is capable of inputting data, generating data, and receiving data. For example, a user of the client device 170 may send out a request to annotate of a designated sample using the interface 172. The request may be sent out through the network 160 to theAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT image processing platform 150. The sample collector 101 can access the designated sample and the sample processing and imaging system 110 may capture a digital image 135 of the designated sample. After the digital image 135 being preprocessed by the preprocessing module 142, the analyzing module 145 performs image annotation based on the preprocessed image and output the annotated image to the client device 170 via the interface 172. The annotated image and related data can be saved to the storage device 140 for future analysis and diagnosis. Additional information such as patient profiles may also be retrieved from the storage device 140 to further facilitate annotation or result interpretation. For example, the annotated image and the additional information may be used together to predict if a subject where the sample is obtained has developed a genetic condition (e.g., a disorder, a disease, or a cancer). The analyzed data may be transmitted to the client device 170 via the network 160.Training and Using a Machine Learning Model with Digital Images

[0099] FIG. 2 shows a block diagram of a machine learning pipeline 200 comprising several subsystems that work together to train, validate, and implement one or more machine learning models in accordance with various embodiments. The machine learning pipeline 200 may be executed as part of the preprocessing module 142 or the analyzing module 145 of the computing environment 100 described in FIG. 1. The machine learning pipeline 200 comprises a data subsystem 205 for collecting, generating, preprocessing, and labeling of training and validation datasets 210, training and validation subsystem 215 that facilitates the training and validation of one or more machine learning algorithms 220, and inference subsystem 225 for deploying and implementing one or more trained machine learning models 230 independently or in combination with one or more other systems or services 235 for downstream processes.

[0100] As used herein, machine learning algorithms (also described herein as simply algorithm or algorithms) are procedures that are run on datasets (e.g., training and validation datasets) and perform pattern recognition on datasets, learn from the datasets, and / or are fit on the datasets. Examples of machine learning algorithms include linear and logistic regression, decision trees, artificial neural netw orks, k-means, and k-nearest neighbor. In contrast, machine learning models (also described herein as simply model or models) are the output of the machine learning algorithms and are comprised of model data and a prediction algorithm. In other words, the machine learning model is the program that is saved after running a machine learning algorithm on training data and represents the rules, numbers, and any other algorithmspecific data structures required to make inferences. For example, a linear regression algorithmAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT may result in a model comprised of a vector of coefficients with specific values, a decision tree algorithm may result in a model comprised of a tree of if-then statements with specific values, or neural network, backpropagation, and gradient descent algorithms together result in a model comprised of a graph structure with vectors or matrices of weights with specific values.Data Subsystem

[0101] Data subsystem 205 is used to collect, generate, preprocess, and label data to be used to train and validate one or more machine learning algorithms 220. The data collection can include exploring various data sources such as public datasets, private data collections, or realtime data streams, depending on a project's needs. In some instances, a data source is a public or online repository of information or examples pertinent to a general or target domain space. Many domains have publicly available datasets provided by governments, universities, or organizations. For example, many government and private entities offer datasets on healthcare, environmental data, and more through various portals. For proprietary needs, data might be available through partnerships or purchases from private companies that specialize in data aggregation. In other instances, a data source is a private repository of information or examples pertinent to a general or target domain space. For example, a data source can be the storage device 140 that stores digital WSIs generated by the computing environment 100 described in FIG. 1. Once a data source is identified, data subsystem 205 can be used to collect data through appropriate methods such as scanning histological or cytological slides using specialized scanners, downloading from online repositories, web scraping, using APIs for real-time data, creating datasets through surveys and experiments, or by running assays. The acquired raw data may be further preprocessed to generate the training and validation datasets 210.

[0102] In some instances, raw data may be generated as opposed to being collected or acquired. Data generating may comprise data synthesis and / or data augmentation. Different data synthesis and / or data augmentation techniques may be implemented by the data subsystem 205 to generate data to be used for the training and validation subsystem 215. Data synthesizing involves creating entirely new data points from scratch. This technique may be used when real data is insufficient, too sensitive to use, or when the cost and logistical barriers to obtaining more real data are too high. The synthesized data should be realistic enough to effectively train a machine learning model, but distinct enough to comply with regulations (e.g., copyright and data privacy), if necessary. Techniques using machine learning models such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) may be used to generateAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT new data examples. These models leam the distribution of real data and attempt to produce new data examples that are statistically similar but not identical. Data augmentation, on the other hand, refers to techniques used to artificially expand the size of a dataset by creating modified versions of existing data examples. The primary goal of data augmentation is to increase variation in the data in order to make the model more robust to variations it might encounter in the real world, thereby improving its ability to generalize from the training data to unseen data. This is especially common in image and speech recognition tasks but is applicable to other data types as well. For images, data augmentation may include rotations, flipping, scaling, or altering the lighting conditions. For text, data augmentation may include synonyms replacement, back translation, or sentence shuffling. For audio, data augmentation may include changes made to pitch, speed, or background noise.

[0103] Preprocessing may be implemented by the data subsystem 205, serving as a bridge between raw data acquisition and effective model training. The primary objective of preprocessing is to transform raw data into a format that is more suitable and efficient for analysis, ensuring that the data fed into machine learning algorithms is clean, consistent, and relevant. This step can be useful because raw data often comes with a variety of issues such as missing values, noise, irrelevant information, and inconsistencies that can significantly hinder the performance of a model. By standardizing and cleaning the data beforehand, preprocessing helps in enhancing the accuracy and efficiency of the subsequent analysis, making the data more representative of the underlying problem the model aims to solve.

[0104] Raw data preprocessing may comprise generating small “tiles"’ or "patches” of an original WSI. WSIs are very large, for example, a 20mm x 20mm tissue sample scanned at 40X magnification can result in an image with dimensions of 80,000 x 80,000 pixels. Further, if this image is in standard 24-bit color, the flattened version of the image would be about 19.2GB in size. Digital images of such size require very’ lengthy computational run times and multiple rounds of processing. Accordingly, the data subsystem 205 may preprocess the WSI into tiles or patches, which represent specific portions of the entire image or slide. The patches may be of any size appropriate to train a machine learning model. For example, a tile or patch can be a region of a whole-slide scan or a region of interest having (x,y) pixel dimensions (e.g., 299 x 299 pixels). An image of 1000 pixels by 1000 pixels, when divided into tiles or patches of 100 pixels by 100 pixels, would result in 100 tiles / patches, with each tile / patch containing 10,000 pixels. In some embodiments, the tiles / patches may be designed with or without overlap with neighboring tiles / patches. The WSI may be parsed into tens to hundreds to thousands ofAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT individual patches with appropriate pixel dimension and resolution quality (e.g., 299 x 299 pixels at 40X magnification ). Further, the tiles / patches may be designed with or without overlap with neighboring tiles / patches where at least one pixel is shared between the neighboring tiles / patches. In some instances, the data subsystem 205 may save the tiles / patches as individual image files or as coordinates of each individual tile / patch with respect to the original image.

[0105] Other raw data preprocessing techniques include data cleaning, normalization, feature extraction, dimensionality reduction, and the like. Data cleaning may involve removing duplicates, filling in missing values, or filtering out outliers to improve data quality. Normalization involves scaling numeric values to a common scale without distorting differences in the ranges of values, which helps prevent biases in the model due to the inherent scale of features. Feature extraction involves transforming the input data into a set of useable features, possibly reducing the dimensionality of the data in the process. For instance, in image analysis, feature reduction techniques such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA). t-Distributed Stochastic Neighbor Embedding (t-SNE), autoencoders, and feature selection can be used for simplifying images, improving model performance, and gaining insights into the underlying structure of the images. These techniques not only help in reducing the computational load on the model but also in mitigating issues like overfitting by simplifying the data without losing critical information.

[0106] In the instance that machine learning pipeline 200 is used for supervised or semisupervised learning of machine learning models, labeling techniques can be implemented as part of the data collection. The quality and accuracy of data labeling directly influence the model's performance, as labels sen e as the definitive guide that the model uses to leam the relationships between the input features and the desired output. Particularly in complex domains such as image analysis, natural language processing, or medical diagnosis, precise and consistent labeling is important because it provides the ground truth or target outcomes against which the model's predictions are compared and adjusted during training. Effective labeling ensures that the model is trained on correct and clear examples, thus enhancing its ability to generalize from the training data to real-world scenarios.

[0107] In some instances, the ground truth values (labels) are provided within the raw data. For example, when the raw data includes images, the labels may include cancer types, pathological conditions, infectious agents, tissues or cell types, grading or staining types, imageAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT processing artifacts, structure and / or anatomical objects, histological features, biomarkers expression, treatment response, pathological scores, and the like. Many different label types may be annotated to the images accessed and loaded by the data subsystem 205. For example, the labels may include cancer cell: normal, benign, likely benign, malignant, likely pathogenic or pathogenic, and specific cancer ty pes. Different structural labels may be included such as tumor margin, tumor core, surround stroma, blood vessels, and lymph nodes. In some instances, histological labels may be included such as mitotic figures, nuclei, cytoplasm, and extracellular matrix.

[0108] Labeling techniques can vary significantly depending on the type of data and the specific requirements of the project. Manual labeling, where human annotators label the data, is one method that can be used. This approach may be useful when a detailed understanding and judgment are required, such as in labeling medical images or categorizing text data where context and subtlety are important. However, manual labeling can be time-consuming and prone to inconsistency, especially with a large number of annotators. To mitigate this, semiautomated labeling tools may be used as part of data subsystem 205 to pre-label data using algorithms, which human annotators may then review and correct as needed. Another approach is active learning, a technique where the model being developed is used to label new data iteratively. The model suggests labels for new data points, and human annotators may review and adjust certain predictions such as the most uncertain predictions. This technique optimizes the labeling effort by focusing human resources on a subset of the data, e g., the most ambiguous cases, improving efficiency and label quality through continuous refinement.

[0109] In various embodiments, the images may be annotated by one or more pathologists specifically trained to detect a given disease state or morphology7such as tumor tissue in a slide. These annotations may be at the whole-slide level (e.g., annotation applies to the whole slide) or at the patch level. In the event more than one pathologist annotated the digital WSI, multiple annotated WSI are obtained that indicate regions of tumor tissue and healthy tissue. The annotations from each of the pathologist can be combined using an image segmentation algorithm to generate a WSI that displays a weighted approximation of the regions in the WSI that correspond to disease, healthy tissue, or other (regions where there is discrepancy between the pathologists). The image segmentation algorithm is a statistical method used to combine multiple annotations from different experts (e g., pathologists) into a single consensus ground truth for each annotation provided to the algorithm. It operates by iteratively estimating the most likely true segmentation of an image and the performance level of each annotator. TheAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT algorithm assumes that each annotator’s segmentation is a noisy observation of the true segmentation and models the probability that each pixel belongs to a certain class based on the annotations provided. Through this process, the image segmentation algorithm can effectively reconcile discrepancies between different annotations, assigning higher weight to more reliable annotators and producing a consensus segmentation for each annotation that reflects the collective expertise.

[0110] In digital pathology’, the image segmentation algorithm can be particularly useful for creating ground truth labels from multiple pathologist annotations of histopathological images. When pathologists annotate images to identify regions of interest, such as tumor boundaries or specific cell types, there can be considerable variability' due to subjective interpretations and differences in expertise. By applying the image segmentation algorithm, these individual annotations can be integrated into a single, probabilistically determined ground truth label for each annotation that provides a more accurate and robust representation of the true pathology. These consensus ground truth can then be used for training and validating machine learning models, ensuring that the models are based on the best possible data, and ultimately improving diagnostic accuracy and consistency in digital pathology.

[0111] In some instances, the image segmentation algorithm operates by initially generating a test segmentation by partitioning the WSI into discrete groups of pixels (e g., image segments) as the majority' vote or some other heuristic combination of the pathologists’ annotations. For example, the test segmentation may be generated based on similar properties such as gray level, color, texture, brightness, or contrast to reduce the complexity of the image. FIG. 3A shows an exemplary ground-truth WSI test segmentation that was generated to account for multiple pathologist’s annotations. The multiple annotations are combined to create a ground-truth annotation, which is then used to produce accurate training data. In some embodiments, a pixel in the ground-truth annotation is labeled as positive if it is marked as positive in every pathologist’s annotation. Conversely, a pixel is labeled as negative in the ground-truth annotation if it remains unannotated across all pathologists’ annotations. Pixels that do not meet either of these criteria are excluded from the training data.

[0112] In some embodiments, the test segmentation is generated using a consensus image generation algorithm based on performance parameters. The performance parameters (e.g., weights) for each pathologist are then initialized. These parameters can include sensitivity (true positive rate) and specificity (true negative rate) for each pathologist. Given the currentAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT estimates of the test segmentation, the probability that each pixel or voxel (or annotation unit) belongs to the test segmentation is computed based on the annotations provided by the experts and their respective performance parameters. This may be done using a Bayesian framework. The estimates of the test segmentation are then updated based on the probabilities computed. The performance parameters for each pathologist are then updated by computing the sensitivity and specificity based on how well each pathologist’s annotations agree with the updated estimate of the true annotation. The probabilities and updates are then repeated until convergence. Convergence is typically defined as the point where changes in the estimates of the true annotation and the performance parameters fall below a certain threshold. FIG. 3B shows an exemplary' output (ground-truth) image using the consensus image generation algorithm where the final segmentation is labeled with regions indicating tumor tissue, healthy tissue, and ignored regions (e.g., regions with inconsistent pathology annotations).

[0113] Once collected, generated, preprocessed, and / or labeled, the data may then be split into the training and validation datasets 210. The training and validation datasets 210 may comprise the raw data and / or the preprocessed data. The training and validation datasets 210 are typically split into at least three subsets of data: training, validation, and testing. The training set is used to fit the model, where the machine learning model learns to make inferences based on the training data. The validation set, on the other hand, is utilized to tune hyperparameters and prevent overfitting by providing a sandbox for model selection. Finally, the test set serves as a new and unseen dataset for the model, used to simulate real-world application and evaluate the final model’s performance. The process of splitting ensures that the model can perform well not just on the data it was trained on, but also on new, unseen data, thereby validating and testing its ability' to generalize.

[0114] Various techniques can be employed to split the data effectively, with each method aiming to maintain a good representation of the overall dataset in each subset. A simple random split (e.g., a 70 / 20 / 10%, 80 / 10 / 10%, or 60 / 25 / 15%) is the most straightforward approach, where examples from the data are randomly^ assigned to each of the three sets. However, more sophisticated methods may be necessary' to preserve the underlying distribution of data. For instance, stratified sampling may be used to ensure that each split reflects the overall distribution of a specific variable, particularly useful in cases where certain categories or outcomes are underrepresented. Another technique, k-fold cross-validation, involves rotating the validation set across different subsets of the data, maximizing the use of available data for training while still holding out portions for validation. These methods help in achieving moreAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT robust and reliable model evaluation and are useful in the development of predictive models that perform consistently across varied datasets.

[0115] Data subsystem 205 is also used for collecting, generating, setting, or implementing model hyperparameters 240 for the training and validation subsystem 215. The hyperparameters control the overall behavior of the models. Unlike model parameters 245 that are learned automatically during training, hyperparameters 240 are set before training begins and have a significant impact on the performance of the model. For example, in a neural network, hyperparameters include the learning rate, number of layers, number of neurons / nodes per layer, activation functions, convolution kernel width, the number of kernels for a model, the number of graph connections to make during a lookback period, and the maximum depth of a tree in a random forest among others. These settings can determine how quickly a model leams, its capacity to generalize from training data to unseen data, and its overall complexity'. Correctly setting hyperparameters is important because inappropriate values can lead to models that underfit or overfit the data. Underfitting occurs when a model is too simple to learn the underlying pattern of the data, and overfitting happens when a model is too complex, learning the noise in the training data as if it were signal.Training. Validation, and Testing

[0116] The training and validation subsystem 215 is comprised of a combination of specialized hardware and software to efficiently handle the computational demands required for training, validating, and testing a machine learning model. On the hardware side, high-performance GPUs (Graphics Processing Units) may be used for their ability to perform parallel processing, drastically speeding up the training of complex models, especially deep learning networks. CPUs (Central Processing Units), while generally slower for this task, may also be used for less complex model training or when parallel processing is less critical. TPUs (Tensor Processing Units), designed specifically for tensor calculations, provide another level of optimization for machine learning tasks. On the software side, a variety of frameworks and libraries are utilized, including TensorFlow, PyTorch, Keras, and scikit-leam. These tools offer comprehensive libraries and functions that facilitate the design, training, validation, and testing of a wide range of machine learning models across different computing platforms, whether local machines, cloud-based systems, or hybrid setups, enabling developers to focus more on model architecture and less on underlying computational details.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0117] Training is the initial phase of developing machine learning models 230 where the model learns to make predictions or decisions based on data training data provided from the training and validation datasets 210. During this phase, the model iteratively adjusts its internal model parameters 245 to achieve a preset optimization condition. In a supervised machine learning training process, the preset optimization condition can be achieved by minimizing the difference between the model output (e.g., predictions, classifications, or decisions) and the ground truth labels in the training data. In some instances, the preset optimization condition can be achieved when the preset fixed number of iterations or epochs (full passes through the training dataset) is reached. In some instances, the preset optimization condition is achieved when the performance on the validation dataset stops improving or starts to degrade. In some instances, the preset optimization condition is achieved when a convergence criterion is met, such as when the change in the model parameters falls below a certain threshold between iterations. This process, known as fitting, is fundamental because it directly influences the accuracy and effectiveness of the model.

[0118] In an exemplary training phase performed by the training and validation subsystem 215, the training subset of data is input into the machine learning algorithms 220 to find a set of model parameters 245 (e.g., weights, coefficients, trees, feature importance, and / or biases) that minimizes or maximizes an objective function (e.g., a loss function, a cost function, a contrastive loss function, a cross-entropy loss function, an Out-of-Bag (OOB) score, etc ). To train the machine learning algorithms 220 to achieve accurate predictions, "errors" (e.g., a difference between a predicted label and the ground truth label) need to be minimized. In order to minimize the errors, the model parameters can be configured to be incrementally updated by minimizing the objective function over the training phase ("optimization”). Various different techniques may be used to perform the optimization. For example, to train machine learning algorithms such as a neural network, optimization can be done using back propagation. The current error is typically propagated backwards to a previous layer, where it is used to modify the weights and bias in such a w ay that the error is minimized. The weights are modified using the optimization function. Other techniques such as random feedback, Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IF A), Hebbian learning, and the like can also be used to update the model parameters 245 in a manner as to minimize or maximize an objective function. This cycle is repeated until a desired state (e.g., a predetermined minimum value of the objective function) is reached.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0119] The training phase is driven by three primary' components: the model architecture (which defines the structure of the algorithms 220), the training data (which provides the examples from which to learn), and the learning algorithm (which dictates how the model adjusts its model parameters). The goal is for the model to capture the underlying patterns of the data without memorizing specific examples, thus enabling it to perform well on new, unseen data.

[0120] The model architecture is the specific arrangement and structure of the various components and / or layers that make up a model. In the context of a neural network, the model architecture may include the configuration of layers in the neural network, such as the number of layers, the ty pe of layers (e.g., convolutional, recurrent, fully connected), the number of neurons in each layer, and the connections between these layers. In the context of a random forest consisting of a collection of decision trees, the model architecture may include the configuration of features used by the decision trees, the voting scheme, and hyperparameters such as the number of trees in the forest, the maximum depth of each tree, the minimum number of samples required to split a node, and the maximum number of features to consider when looking for the best split. In some instances, the model architecture is configured to perform multiple tasks. For example, a first component of the model architecture may be configured to perform a feature selection function, and a second component of the model architecture may be configured to perform a feature scoring function. The different components may correspond to different algorithms or models, and the model architecture may be an ensemble of multiple components.

[0121] Model architecture also encompasses the choice and arrangement of features and algorithms used in various models, such as decision trees or linear regression. The architecture determines how input data is processed and transformed through various computational steps to produce the output. The model architecture directly influences the model's ability’ to leam from the data effectively and efficiently, and it impacts how well the model performs tasks such as classification, regression, or prediction, adapting to the specific complexities and nuances of the data it is designed to handle.

[0122] The model architecture can encompass a wide range of algorithms 220, suitable for different kinds of tasks and datatypes. Examples of algorithms 220 include, without limitation, linear regression, logistic regression, decision tree. Support Vector Machines, Naives Bayes algorithm, Bayesian classifier, linear classifier, K-Nearest Neighbors, K-Means, random forest,Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT dimensionality reduction algorithms, grid search algorithm, genetic algorithm, AdaBoosting algorithm, Gradient Boosting Machines, and Artificial Neural Networks such as convolutional neural network ('‘CNN”), an inception neural network, a U-Net, a V-Net, a residual neural network (“Resnef ’), a transform neural network, a recurrent neural network, a Generative adversarial network (GAN), or other variants of Deep Neural Networks (“DNN”) (e.g., a multilabel n-binary DNN classifier or multi-class DNN classifier). These algorithms can be implemented using various machine learning libraries and frameworks such as TensorFlow, PyTorch, Keras, and scikit-leam, which provide extensive tools and features to facilitate model building, training, validation, and testing. For example, the classification algorithm described with respect to FIGs. 4A and 4B is a set of algorithms configured as the architecture of a neural network comprised of layers of nodes, or “neurons,” which are designed to mimic the way a human brain operates.

[0123] The learning algorithm is the overall method or procedure used to adjust the model parameters 245 to fit the data. It dictates how the model learns from the data provided during training. This includes the steps or rules that the algorithm follows to process input data and make adjustments to the model's internal parameters (e.g., weights in neural networks) based on the output of the objective function. Examples of learning algorithms include gradient descent, backpropagation for neural networks, and splitting criteria in decision trees.

[0124] Various techniques may be employed by training and validation subsystem 215 to train machine learning models 230 using the learning algorithm, depending on the type of model and the specific task. For supervised learning models, where the training data includes both inputs and expected outputs (e.g., ground truth labels), gradient descent is a possible method. This technique iteratively adjusts the model parameters 245 to minimize or maximize an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.). The objective function is a method to measure how well the model’s predictions match the actual labels or outcomes in the training data. It quantifies the error between predicted values and true values and presents this error as a single real number. The goal of training is to minimize this error, indicating that the model's predictions are, on average, close to the true data. Common examples of loss functions include mean squared error for regression tasks and cross-entropy loss for classification tasks.

[0125] The adjustment of the model parameters 245 is performed by the optimization function or algorithm, which refers to the specific method used to minimize (or maximize) theAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT objective function. The optimization function is the engine behind the learning algorithm, guiding how the model parameters 245 are adjusted during training. It determines the strategy to use when searching for the best weights that minimize (or maximize) the objective function. Gradient descent is a primary example of an optimization algorithm, including its variants like stochastic gradient descent (SGD), mini-batch gradient descent, and advanced versions like Adam or RMSprop, which provide different ways to adjust learning rates or take advantage of the momentum of changes. For example, in training a neural network, backpropagation may be used with gradient descent to update the weights of the network based on the error rate obtained in the previous epoch (cycle through the full training dataset). Another technique in supervised learning is the use of decision trees, where a tree-like model of decisions is built by splitting the training dataset into subsets based on an attribute value test. This process is repeated on each derived subset in a recursive manner called recursive partitioning.

[0126] In unsupervised learning, where training data does not include labels, different techniques are used. Clustering is one method where data is grouped into clusters that maximize the similarities of data within the same cluster and maximize the differences with data in other clusters. The K-Means algorithm, for example, assigns each data point to the nearest cluster by minimizing the sum of distances between data points and their respective cluster centroids. Another technique, Principal Component Analysis (PCA), involves reducing the dimensionality of data by transforming it into a new set of variables, the principal components, which are uncorrelated and ordered so that the first few retain most of the variation present in all of the original variables. These techniques help uncover hidden structures or patterns in the data, which can be essential for feature reduction, anomaly detection, or preparing data for further supervised learning tasks.

[0127] Validating is another phase of developing machine learning models 230 where the model is checked for deficiencies in performance and the hyperparameters 240 are optimized based on validation data provided from the training and validation datasets 210. The validation data helps to evaluate the model's performance, such as accuracy, precision, recall, or Fl -score, to gauge how well the model is likely to perform in real-world scenarios. Hyperparameter optimization, on the other hand, involves adjusting the settings that govern the model's learning process (e.g.. learning rate, number of layers, size of the layers in neural networks) to find the combination that yields the best performance on the validation data. One optimization technique is grid search, where a set of predefined hyperparameter values are systematically evaluated. The model is trained with each combination of these values, and the combinationAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT that produces the best performance on the validation set is chosen. Although thorough, grid search can be computationally expensive and impractical when the hyperparameter space is large. A more efficient alternative optimization technique is random search, which samples hyperparameter combinations from a defined distribution randomly. This approach can in some instances find a good combination of hyperparameter values faster than grid search. Advanced methods like Bayesian optimization, genetic algorithms, and gradient-based optimization may also be used to find optimal hyperparameters more effectively. These techniques model the hyperparameter space and use statistical methods to intelligently explore the space, seeking hyperparameters that yield improvements in model performance.

[0128] An exemplary validation process includes iterative operations of inputting the validation subset of data into the trained algorithm(s) using a validation technique such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, Nested Cross-Validation, or the like, to fine-tune the hyperparameters and ultimately find the optimal set of hyperparameters. In some instances, a 5-fold cross-validation technique may be used to avoid overfitting the trained algorithm and / or to limit the number of selected features per split to the square-root of the total number of input features. In some instances, training dataset is split into 5 equal-size cohorts (or about equal-size), and every four of the cohorts are used to train an algorithm to generate five models (e.g., cohorts #1, 2, 3, and 4 are used to train and generate model 1, cohorts #1, 2, 3, and 5 are used to train and generate model 2, cohorts #1, 2, 4, and 5 are used to train and generate model 3, cohorts #1. 3, 4, and 5 are used to tram and generate model 4, and cohorts #2, 3, 4 and 5 are used to train and generate model 5). Each model is evaluated (or validated) using the unused cohort in the training (e.g., for model 5, cohort #1 is used for validation). The overall performance of the training can be evaluated by an average performance of the five models. K-fold cross-validation provides a more robust estimate of a model’s performance compared to a single training / validation split because it utilizes the entire dataset for both training and evaluation and reduces the variance in the performance estimate.

[0129] Once a machine learning model has been trained and validated, it undergoes a final evaluation using test data provided from the training and validation datasets 210, which is a separate subset of the data that has not been used during the training or validation phases. This step is crucial as it provides an unbiased assessment of the model's performance in simulating real-world operation. The test dataset serves as new, unseen data for the model, mimicking how the model would perform when deployed in actual use. During testing, the model’s predictionsAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT are compared against the true values in the test dataset using various performance metrics such as accuracy, precision, recall, and mean squared error, depending on the nature of the problem (classification or regression). This process helps to verify the generalizability of the model — its ability to perform well across different data samples and environments — highlighting potential issues like overfitting or underfitting and ensuring that the model is robust and reliable for practical applications. The machine learning models 230 are fully validated and tested once the output predictions have been deemed acceptable by user defined acceptance parameters. Acceptance parameters may be determined using correlation techniques such as Bland- Altman method and the Spearman's rank correlation coefficients and calculating performance metrics such as the error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.Inference Phase for Machine Learning Models

[0130] The inference subsystem 225 is comprised of various components for deploying the machine learning models 230 in a production environment (e.g., use in the image processing platform 150 as described with respect to FIG. 1). Deploying the machine learning models 230 includes moving the models from a development environment (e.g.. the training and validation subsystem 215, where it has been trained, validated, and tested), into a production environment where it can make inferences on real-world data (e.g., input data 250). This step typically starts with the model being saved after training, including its parameters and configuration such as final architecture and hyperparameters. It is then converted, if necessary, into a format that is suitable for deployment, depending on the deployment environment. For instance, a model trained in a scientific computing environment such as Python might be converted into a Java-friendly format for integration into a larger enterprise application. Deployment can be conducted on various platforms, including on-premises servers, cloud environments like AWS, Azure, Google.

[0131] Once deployed, the model is ready to receive input data 250 and return outputs (e.g., inferences 255). In some instances, the model resides as a component of a larger system or service (e.g., including additional downstream applications). In some instances, the machine learning models 230 and / or the inferences 255 can be used by the downstream applications (e.g., systems or services 235) to provide further information. For example, the inferences 255 can be used to determine whether a specific treatment should be administered to a patient. The downstream applications can be configured to generate an output 260. In some instances, theAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT output 260 comprises a report including inferences 255 and information generated by the downstream applications (e.g.. systems or services 235).

[0132] In an exemplary inference subsystem 225, the input data 250 includes images generated from one or more biological samples from a patient having been diagnosed with a disease (e.g., cancer). The images may be generated by performing whole-slide image processing (independently or as part of a digital pathology assay) on the one or more biological samples 105 collected from the patient as described with respect to FIG. 1. The one or more biological samples may be a single tissue sample or multiple tissue samples (e.g., a bladder tissue, lung tissue, tumor section, etc.) obtained from the patient.

[0133] To manage and maintain its performance, a deployed model may be continuously monitored to ensure it performs as expected over time. This involves tracking the model’s prediction accuracy, response times, and other operational metrics. Additionally, the model may require retraining or updates based on new data or changing conditions in the environment it is applied in. This can be useful because machine learning models can drift over time due to changes in the underlying data they are making predictions on — a phenomenon known as model drift. Therefore, maintaining a machine learning model in a production environment often involves setting up mechanisms for performance monitoring, regular evaluations against new test data, and potentially periodic updates and retraining of the model to ensure it remains effective and accurate in making predictions.Convolution Neural Networks

[0134] FIGs. 4A and 4B show an exemplary CNN 400 trained and deployed as described with respect to FIG. 2. In some instances, CNN 400 is a feedforward neural network where data moves from the input layer through a set of hidden layers in one direction (i.e., having no cycles contained within it). However, it should be understood that other types of neural networks could be implemented without departing from the spirit and scope of the present disclosure. CNN 400 comprises a deep learning set of algorithms that can take as input an image (e.g., a digital pathology7image or patches from a digital pathology7image generated from the image processing platform 150 described with respect to FIG. 1), assigns importance (learnable weights and biases) to various features in the image, and differentiates one aspect from the other to perform a given task such as object classification. CNNs are typically used in digital pathology7to automate and enhance the accuracy7of image classification, segmentation, and feature extraction in medical images, aiding in tasks such as tumor detection, diseaseAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT diagnosis, and cell counting. Additionally, CNNs facilitate prognostic modeling, whole-slide image analysis, and anomaly detection, thereby supporting pathologists in diagnosing diseases and planning treatments more efficiently.Architecture of CNNs

[0135] FIG. 4A illustrates the various layers of an exemplary' CNN 400, while FIG. 4B illustrates processing of an exemplary image by the CNN 400 using the various layers. As shown in FIG. 4A, the CNN 400 comprises an input layer 405, convolution-pooling layers 410, and classification-output layers 415. The input layer 405 of a CNN is the initial stage where input data (e.g., image data) is fed into the network. In some instances, the input data is digital WSIs of a specimen (may be generated by the image processing platform 150 as described with respect to FIG. 1). In other instances, the input data is a set of patches generated from an original WSI (may be generated by data subsystem 205 described with respect to FIG. 2). When image data is input into a CNN, it is typically represented as a multi-dimensional array or tensor. For instance, a color image with dimensions 32 x 32 pixels and multiple color channels (red, green, and blue) would be represented as a 32 x 32 x 3 tensor. Each element in this tensor corresponds to the pixel intensity value in the image, with separate values for each color channel in the case of colored or fluorescing images. The input layer 405 essentially takes this rayv image data and structures it in such a way that it can be efficiently processed by the subsequent layers of the CNN.

[0136] While the input layer 405 itself does not perform complex computations; its role is important in ensuring that the data is formatted correctly for the next layers. The subsequent layers, such as the convolution-pooling hidden layers 410, rely on this structured data to perform operations like convolution, which involves sliding a filter or kernel over the input image to detect features such as edges, textures, and paterns. The dimensionality and the specific arrangement of the input data are useful for these operations to be performed correctly. By accurately organizing the image data, the input layer 405 sets the stage for the network to leam and extract meaningful features that are used for various tasks such as image classification, object detection, and image segmentation.

[0137] The multiple convolution-pooling layers 410 includes one or more convolution layers 420a-n (solid boxes) that perform convolution functions followed by pooling layers 425a-n (dashed boxes) that perform pooling functions. For both the convolution layers 420a-n and the pooling layers 425a-n, with “n” representing any natural number. The convolution and poolingAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT layers work in tandem to progressively extract high-level features from the raw image data received from the input layer. The convolutional layers are designed to apply a set of learnable filters or kernels to the input data. Each filter slides over the entire image, performing an element-wise multiplication and summation to produce a feature map. This process, known as a convolution operation, helps in detecting local patterns such as edges, textures, and shapes within the image. The filters in the initial convolutional layers typically capture basic features like edges and comers, while filters in deeper layers can capture more complex patterns and structures. The convolution operation also helps in preserving the spatial relationships between pixels, which is crucial for understanding the content of the image.

[0138] After the convolution operation is performed, resulting in feature maps, an activation function is applied to introduce non-linearity into the model. This non-linearity is useful because it enables the network to leam and represent complex patterns and relationships in the data that linear operations alone cannot capture. Examples of activation functions that may be used include the linear activation function, the rectifier linear unit (ReLU), the leaky7ReLU, the sigmoid activation function, the SoftMax activation function, the hyperbolic tangent activation function, and the exponential activation function. In certain instances, the activation function used for each convolutional layer is the ReLU, which applies the function (f(x) = max(0, x)) to each element of the feature map. This means that any negative values in the feature map are set to zero, while positive values remain unchanged. In some instances, the activation function is the same across all convolution layers. In other instances, the activation function is varied across the convolution layers (e.g., one or more convolutional layers use ReLU and another one or more convolutional layers use leaky' ReLU).

[0139] The application of the activation function occurs element-wise across the entire feature map produced by the convolution operation. By introducing non-linearity7at each convolutional layer, the network can model more complex functions and is better equipped to handle diverse and intricate data. This step is useful because, without it, the model would essentially be equivalent to a single linear transformation, regardless of the number of layers, severely limiting its capacity7to solve complex tasks. Therefore, by applying activation functions after each convolutional operation, CNNs can develop a more nuanced and powerful representation of the input data, leading to more accurate and robust performance in tasks such as image recognition and classification.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0140] The convolution process generates a large amount of data that needs to be compressed by its corresponding pooling layer (e.g., convolution output 420a is compressed by pooled layer 425a) to reduce the spatial dimensions of the feature maps, which serves several purposes. Pooling operations, such as max pooling or average pooling, dow n sample the feature maps by summarizing the presence of features in patches of the feature map. For instance, max pooling takes the maximum value from a patch of the feature map, effectively reducing the size while retaining the most significant information. This not only reduces the computational load and memory requirements but also adds a degree of translational invariance, meaning the netw ork becomes more robust to slight translations of the input image. By alternating between convolutional and pooling layers, the CNN hierarchically builds a rich and abstract representation of the input image, enabling the final fully connected layers to perform high-level tasks such as classification with improved accuracy. The repeated application of convolution and pooling operations allows the network to leam and recognize increasingly complex features at different levels of abstraction, which allows for the effective processing and understanding of image data.

[0141] After the convolution-pooling layers 410, the data flows into the classification-output layers 415, which are configured to make final predictions based on the extracted features (feature maps). The classification-output layers 415 comprise three main components: a flattened matrix 430, fully connected layers 435, and an output layer 440 that provides the predicted classifications. The feature maps obtained from the final pooling layer are multidimensional arrays. To transition from these spatially structured feature maps to a format suitable for the fully connected layers 435, the feature maps are flattened into a single onedimensional vector (i.e., flattened matrix 430). This process involves taking the multidimensional array of features and converting it into a long vector that maintains the same numerical values but without any spatial dimensions. Flattening is useful because the fully connected layers 435 expect a one-dimensional input, and this transformation enables the network to leverage all the extracted features in a linear format.

[0142] Once the feature maps are flattened, the resulting vector (i.e., flattened matrix 430) is passed through the fully connected layers 435. Each neuron in a fully connected layer is connected to every neuron in the previous layer, allowing for a comprehensive integration of all the features. These layers perform linear transformations followed by the application of activation functions (such as ReLU or sigmoid), which introduce non-linearity and help in learning complex patterns. The fully connected layers 435 serve to combine the featuresAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT extracted by the convolution-pooling layers 410 in various ways to produce higher-level representations. These representations are useful as they consolidate all the information gathered from the earlier stages of the network and are tailored towards the specific task of classification. In some instances, the activation function(s) used in the fully connected layers 435 are the same or different than the activation function(s) used in the convolution layers 420a-n of the convolution-pooling layers 410.

[0143] The final layer of the classification-output layers 415 is the output layer 440, which provides the predicted classifications. In many classification tasks, this is typically a SoftMax layer, especially when dealing with multi-class classification problems. The SoftMax function converts the output scores from the final fully connected layer into probabilities that sum to one, providing a probabilistic interpretation of the predictions. Each output node in this layer corresponds to a different class label, and the node with the highest probability is often taken as the network's prediction. In other scenarios, such as binary classification, a sigmoid function might be used instead. The output layer thus translates the learned features and patterns into a final prediction, completing the process of image recognition or classification within the CNN. FIG. 4A shows an exemplary CNN 400 with a prediction problem that outputs at least three different predictions or classifications; however, this example is not meant to be limiting and other prediction problems where only one prediction is made may also be considered.

[0144] FIG. 4B illustrates processing of an exemplary’ image 450 by the CNN 400 using the various layers. As shown, initially the image 450 is obtained (e.g., obtained or acquired as described with respect to FIGs. 1 and 2). The image 450 has an image representation that can be thought of as a grid of pixels when dealing with 2D images or as a grid of voxels for 3D images. A 2D image is represented as a matrix of pixels, where each pixel has an intensity value. A 3D image, such as those used in medical imaging, is represented as a grid of voxels (volume elements), similar to pixels but in three dimensions. When dealing with images that have multiple channels for different colors, such as RGB images, the input image representation changes to account for the additional depth dimension. For example, a grayscale image has a single intensity7value per pixel, while a color image (like an RGB image) has multiple intensity7values (e.g.. 3 values for RGB image corresponding to the red, green, and blue color channels) per pixel. Several color channel options may be used in image processing such as, without limitation, Red-Green-Blue (RGB), Hue Saturation Value (HSV), cyan, magenta, yellow and key (black) (CMYK), grayscale, etc.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0145] WSI images are often very large, for example a 20mm x 20mm sample of tissue may have 80,000 x 80,000 pixels and two-color channels. Accordingly, a WSI image of this size would yield 80,000 x 80,000 x 2 = 1.28E10 weights for initial convolution. The training and use of a model using such a large image would require an excessive amount of computational power and storage space. Alternatively, the input image may be a set of smaller patches of the original WSI image. The set of patches may be of dimensions 299 x 299 x 2 = 178,802 weights, which is comparatively much more manageable in terms of computational power required. Another tool used by CNNs to overcome the computational burden of large input images is that CNNs can use parameter sharing, where nodes in a particular feature map share weights to make the computation less consuming.

[0146] The image 450 is fed into the input layer 405 and CNN 400 applies a series of functions to the image 450 using multiple hidden layers 455a-n. The hidden layers 455a-n comprise the convolution layers 420a-n, the pooling layers 425a-n, activation layers (e.g., ReLU after each convolution layer and first fully connected layer), and fully connected layers 435. The number of hidden layers 455a-n and output layer 440 define the depth of CNN 400. By having multiple hidden layers 455a-n. CNN 400 can compute complex functions by cascading simpler functions. The hidden layers 455a-n are represented as a directed acyclic graph (DAG) whose nodes 460 correspond to neurons and whose edges correspond to the links or connections 465 between them. Every node 460 within the hidden layers 455a-n is connected (via edges) to one or more other nodes 460 in the previous layer and in the next layer. This connection circuit allows for each node 460 to receive information from the layer beneath it, process that information, and send the processed information to the next layer. The connections 465 between nodes 460 have a weight associated with them. The weight represents the strength of the connection 465 and is a weighting factor (model parameter) that reflects the importance of the data (e.g., features) associated with the image 450 that CNN 400 leams during training, as described in more detail herein.

[0147] The number of nodes 460 used in hidden layers 455a-n can affect the model's overall fitting. For example, too few nodes can lead to underfitting the model and high statistical bias, while too many nodes may lead to overfitting the model, high variance, and increased training time. An appropriate number of nodes 460 to include in the hidden layers 455a-n may be determined by (i) a number between the size (e.g., number of nodes) of the input layer and the output layer, (ii) about two-thirds the size of the input layer plus the size of the output layer, (iii) less than twice the size of the input layer, or (iv) equal to the mean of the size of the inputAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT and output layers. Other factors that affect the number of nodes, and number of hidden layers, include the training cases, number of outliers, the complexity of the data that is to be learned, the type of activation function, or any combination thereof.

[0148] As shown in FIG. 4B, the nodes 460 in hidden layer 1 455(a) are each receiving multiple different inputs (e.g., three inputs) from the input layer 405 via connections 470 and thus have multiple incoming values. Each of those connections 470 has its own unique w eight associated with it. Each node 460 in hidden layer 1 455(a) can sum up the individual weights from each connection 470 to generate a weighted sum value that the receiving node is assigned. Then, each receiving node may apply functions or operations using the w eighted sum value to process the input data and generate a processed input (e g., output) that can be passed along to a new receiving node in the next layer. Essentially, the processing of a node 460 may be implemented in two steps- (1) each node 460 calculates the weighted sum of its inputs and weights, and (2) each node 460 applies functions or operations using the weighted sum value to produce the output. The nodes 460 in subsequent layers receive output of nodes 460 in the previous layer as input. There are multiple ways known in the art to determine whether an input should be passed onto the nodes 460 in the next hidden layer. One example is to use a threshold value. If the value of a node is below the threshold value, the input may not be passed to the next layer, and if the value of a node is above the threshold, the input may be passed forward. In some cases, a connection may have zero weight, in which case the node 425 is considered to not have any influence on the task at hand and thus the output will not be passed through to a node in the next layer.

[0149] The hidden layers 455a-n include convolution layers, pooling layers, and fully connected layers. The convolution layers are responsible for recognizing features in the pixels or voxels of the image 450. A convolution layer achieves this by applying a filter 480 to its input 475 to detect features from the input data (in the form of a matrix) into a feature map 485. The filter 480 is often referred to as a kernel, or a feature detector. Each filter is designed to detect specific features like edges, textures, or patterns within the image 450. The filter 480 will slide across the height and width of the input 475, based on a padding parameter and a striding parameter, and performs element-wise multiplication followed by summation, a process known as convolution operation. The padding parameter indicates how to expand the size of the input matrix by adding fake pixels (e.g., with a pixel value of 0) to the border of the matrix. Padding can help to reduce the loss of information at the borders of the input matrix (or input feature map). The striding parameter of the convolution operation refers to the numberAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT of pixels by which the filter 480 matrix moves across the input matrix. The padding and striding parameters are set either as random selection, defined by a user, or as a hyperparameter optimized during training. In the example shown in FIG. 4B, feature maps 485 are generated using a striding parameter value of 1. The filter will move across the entire width of the input matrix based on the striding parameter, then move down (also based on the striding parameter) to the next row of the image and repeat the process until the entire image has been traversed.

[0150] The result of the convolution operation is the feature map 485, which highlights the presence and location of the feature detected by that particular filter 480. Each convolution layer ty pically employs multiple filters, each generating its own feature map. For example, a convolution layer might use a set of filters to detect vertical edges, horizontal edges, and other textures within the image. As these filters convolve with the image 450, they’ produce a stack of feature maps, each corresponding to a different feature. This stack of feature maps forms a new representation of the image 450, where the spatial dimensions are reduced, but the depth (number of feature maps) is increased, reflecting the multiple features detected. The feature maps 485 are important as they serve as the input for subsequent layers, allowing the network to build a hierarchical understanding of the image 450, from simple edges in the initial convolution layers to complex objects in deeper convolution layers.

[0151] Pooling layers are interspersed between convolution layers and are configured to further process the feature maps 485. The objective of the pooling layer is to reduce the dimension of the output feature maps 485. Pooling may be done at a global level or a local level. Global level pooling involves combining the entire feature map matrix into a single value, whereas local pooling combines small clusters (e.g., pooling squares, typically of size 2x2). Multiple different pooling functions may be used for pooling, such as maximum pooling and average pooling. In maximum pooling, the largest value (e.g., the max value) for each feature map 485 (for global) or pooling square of the feature map 485 (for local) is used to generate the pooled feature map 490. For example, if the pooling square is a 2x2 square, the pooling function will take the largest value in that 2x2 square and insert it into the corresponding pooled feature map 490. For average pooling, instead of taking the largest value in the 2x2 square, the average of the values in the 2x2 square are taken and inserted into the corresponding pooled feature map 490. This process effectively down-samples the feature map. reducing its width and height but keeping the depth the same. The down-sampling helps in reducing the computational load and controlling overfitting by making the representation more compact and abstract.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0152] As the image passes through successive convolution and pooling layers, the feature maps 485 become increasingly abstract and focused on higher-level features. Early convolution layers might detect simple edges and textures, while deeper layers combine these simple features to recognize more complex patterns like shapes or even objects. Pooling layers ensure that the spatial size of the feature maps is progressively reduced, which helps in maintaining computational efficiency and reducing the risk of overfitting. This hierarchical feature extraction allows the CNN 400 to effectively leam and generalize from the data, making it particularly well-suited for image recognition tasks. Ultimately, the series of convolution and pooling operations transform the original image into a compact and information-rich representation that can be fed into fully connected layers for classification or other tasks. Thus, the convolution and pooling layers work together to progressively distill the essential features of the image, enabling the CNN 400 to perform tasks such as object detection, image classification, and the like with high accuracy.

[0153] The fully connected layers in CNN 400 play a pivotal role in the network's final decision-making process. After the hierarchical feature extraction performed by the convolutional and pooling layers, the resulting feature maps 485 are flattened into a single vector. This vector serves as the input to the fully connected layers. Each fully connected layer is comprised of a set of nodes (i.e., neurons) where each node is connected to all the nodes in the previous layer. These layers perform linear transformations followed by a non-linear activation function (like ReLU) to capture complex relationships between the features. The linear transformation involves multiplying the input vector by a weight matrix and adding a bias vector. The output of this transformation is then passed through the activation function to introduce non-linearity, enabling the network to model complex patterns and interactions within the data.

[0154] The final output layer 495 of the CNN 400 is responsible for generating the final predictions. This layer is typically a fully connected layer where the number of nodes corresponds to the number of classes in a classification task. For instance, in an image classification task with ten possible classes, the output layer would have ten nodes. The output of this layer is usually passed through an activation function suitable for classification tasks, such as the SoftMax function. The SoftMax function converts the raw output scores (logits) into probabilities that sum to one, providing a probabilistic interpretation of the network's predictions. Each node in the output layer 495 represents the probability' of the image 450 belonging to a particular class. The class with the highest probability may be chosen as theAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT network's final prediction. This combination of fully connected layers and the output layer 495 allows the CNN 400 to perform complex decision-making based on the features extracted from the image 450, ultimately enabling tasks such as image classification, object detection, and more.Training, Testing, and Validation of CNNs

[0155] With regards to training a CNN. many of the techniques described for machine learning pipeline 200 with respect to FIG. 2 apply. For example, a data subsystem (like data subsystem 205) is used to collect, generate, preprocess, and label data to be used to train, test, and validate CNN 400. Data may be generated using the image processing platform 150 described with respect to FIG. 1 (generates and stores digital WSIs). The WSIs are preprocessed as descnbed with respect to FIG. 2 by generating ‘’tiles” or ‘’patches” corresponding to a portion of a whole image, or a whole slide. The tiles / patches are labeled based on the labels provided by one or more pathologists specifically trained to detect the presence of tumor tissue in a slide. These annotations may be at the whole-slide level (e.g., annotation applies to the whole slide) or at the patch level. In some instances, the WSI and / or tiles / patches are labeled by an algorithm (e.g., STAPLE) specifically designed to take as input segmentations of digital pathology7images or a set of image patches (e.g., tissue) and output a probabilistic estimate of the true segmentation that may be used as the ground truth label and indicate features that the CNN 400 is to learn.

[0156] Once all the data is collected, preprocessed and / or labelled, the data is split into training, validating, and testing datasets. Various techniques can be employed to split the data effectively, with each method aiming to maintain a good representation of the overall dataset in each subset. A simple random split (e.g., a 70 / 20 / 10%, 80 / 10 / 10%, or 60 / 25 / 15%) is the most straightforward approach; however, more sophisticated methods may be necessary7to preserve the underlying distribution of data. For instance, stratified sampling may be used to ensure that each split reflects the overall distribution of a specific variable.

[0157] Hyperparameters for CNN 400 are also selected before training begins and have a significant impact on the performance of the model. For example, in a CNN, hyperparameters include the learning rate, number of layers, number of neurons / nodes per layer, activation functions, convolution kernel width, the number of kernels for a model, and the number of graph connections to make during a lookback period among others. These settings can determine how quickly a model learns, its capacity to generalize from training data to unseenAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT data, and its overall complexity. In some instances when a CNN is initially trained, the hyperparameters may be assigned at random. In other instances, the hyperparameters may be explicitly defined to control different aspects of the networks such as memory or cost of execution. However, additional hyperparameters may be defined to adapt a network to a specific scenario. Defining hyperparameters may be challenging, and thus a tuning process may be used for defining an optimal set of hyperparameter values for the specific scenario. The hyperparameter tuning comprises a value search process that may include selecting a combination of values for the hyperparameters from a grid of values with different hyperparameter combinations. The aforementioned value search process can either be exhaustive or leverage more intelligent optimization techniques such as Bayesian optimization or genetic algorithms. Correctly setting hyperparameters is important because inappropriate values can lead to models that underfit or overfit the data. Underfitting occurs when a model is too simple to learn the underlying pattern of the data, and overfitting happens when a model is too complex, learning the noise in the training data as if it were signal.

[0158] CNN 400 may also be trained, tested, and validated by a training and validation subsystem (e.g., the training and validation subsystem 215 described with respect to FIG. 2). Training is the initial phase of developing CNN 400 where the model learns to make predictions or decisions based on training data provided from the training and validation datasets. During this phase, the model iteratively adjusts its internal model parameters to minimize the difference between its predictions and the actual outcomes in the training data. This process, known as fitting, is fundamental because it directly influences the accuracy and effectiveness of the model. The training phase is driven by three primary components: the model architecture (which defines the structure of the algorithm(s) see FIGs. 4A and 4B), the training data (which provides the examples from which to leam), and the learning algorithm (which dictates how the model adjusts its model parameters). The goal is for the model to capture the underlying patterns of the data without memorizing specific examples, thus enabling it to perform well on new, unseen data.

[0159] The model architecture is the specific arrangement and structure of the various components and layers that make up CNN 400 (as described in detail with respect to FIGs. 4A and 4B). This includes the configuration of layers, such as the number of layers, the type of layers (e.g., convolutional, recurrent, fully connected), the number of neurons / nodes in each layer, and the connections between these layers. Model architecture also encompasses the choice and arrangement of features and algorithms used in various models. The architectureAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT determines how input data is processed and transformed through various computational steps to produce the output. The model architecture directly influences the model's ability to leam from the data effectively and efficiently, and it impacts how well the model performs tasks such as classification, or prediction, adapting to the specific complexities and nuances of the data it is designed to handle. In this example, the model architecture encompasses a CNN algorithm that is implemented using various machine learning libraries and frameworks such as TensorFlow, PyTorch, Keras, and scikit-leam, which provide extensive tools and features to facilitate model building, training, validation, and testing.

[0160] The learning algorithm is the overall method or procedure used to adjust the CNN’s parameters to fit the data. It dictates how the model leams from the data provided during training. This includes the steps or rules that the algorithm follows to process input data and adjust the model's internal parameters (e.g., weights in neural networks) based on the output of the objective function. Examples of learning algorithms include gradient descent and backpropagation. For example, CNN 400 is trained with labeled digital image data so a gradient descent learning algorithm may be used to iteratively adjusts the model parameters to minimize or maximize an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.). The objective function is a method to measure how well the model’s predictions match the actual labels or outcomes in the training data. It quantifies the error between predicted values and true values and presents this error as a single real number. The goal of training is to minimize this error, indicating that the model's predictions are, on average, close to the true data. Common examples of loss functions include mean squared error for regression tasks and cross-entropy loss for classification tasks.

[0161] The adjustment of the CNN’s parameters is performed by the optimization function or algorithm, which refers to the specific method used to minimize (or maximize) the objective function. The optimization function is the engine behind the learning algorithm, guiding how the CNN’s parameters are adjusted during training. It determines the strategy to use when searching for the best weights that minimize (or maximize) the objective function. Gradient descent is a primary example of an optimization algorithm. Another example can include combining backpropagation with gradient descent to update the weights of CNN 400 based on the error rate obtained in the previous epoch (cycle through the full training dataset).

[0162] Training a CNN can be challenging as it can be sensitive to the initial set of model parameters and configuration of the learning algorithm. A reason for this challenge is theAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT distribution of the inputs to layers deep in the network may change after each batch of data when the set of model parameters are updated. This can cause the learning algorithm to persistently chase a moving target (known as internal covariate shift). In some instances, to overcome this challenge, batch normalization is used for training CNN 400. Batch normalization may be performed by scaling the output of each hidden layer 455a-n (e.g., standardizing the activations of each input variable per batch of data) such as the activations of a node from the previous layer. Standardizing the activations of the prior layer means that assumptions the subsequent layer makes about the spread and distribution of inputs during the weight update will not change, at least not dramatically. This stabilizes the learning process and reduces the number of training epochs (speeds up training) required to train the CNN 400.

[0163] Once the set of model parameters of CNN 400 have been trained by optimizing the loss or error function, the network is able to predict one or more features from the input data (e.g., sets of patches from a WSI). In addition to tracking the training loss, the testing loss can additionally be tracked to implement early stopping, which halts training of a neural network when a clear divergence between training and testing loss is noted. The early stopping process combats overfitting, preventing scenarios in which the neural network performs extremely well on the training data at the expense of generalizing to validation data.

[0164] Validation of CNN 400 is the process of checking for deficiencies in performance and the hyperparameters using the validation dataset. The validation data helps to evaluate the model's performance, such as accuracy, precision, recall, or Fl -score, to gauge how well the model is likely to perform in real-world scenarios.

[0165] Once CNN 400 has been trained and validated, it undergoes a final evaluation using test data provided from the training and validation datasets, which is a separate subset of the data that has not been used during the training or validation phases. This step is crucial as it provides an unbiased assessment of the model's performance in simulating real-world operation. The test dataset serves as new, unseen data for CNN 400. mimicking how the model would perform when deployed in actual use. During testing, the model’s predictions are compared against the true values in the test dataset using various performance metrics such as accuracy, precision, recall, and mean squared error, depending on the nature of the problem (classification or regression). This process helps to verify the generalizability of CNN 400 — its ability to perform well across different data samples and environments — highlighting potential issues like overfitting or underfitting and ensuring that the model is robust and reliableAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT for practical applications. CNN 400 is fully validated and tested once the output predictions have been deemed acceptable by user defined acceptance parameters. Acceptance parameters may be determined using correlation techniques such as Bland-Altman method and the Spearman’s rank correlation coefficients and calculating performance metrics such as the error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc. In some embodiments, the CNN 400 is verified by pathologists using a scoring system (e.g., each pathologist would score the performance of the model using a score).

[0166] As should be understood, other training-validation mechanisms are contemplated and may be implemented. For example, CNN 400 may be trained and hyperparameters may be tuned on data from a first subset of data and the data from a second subset of data may only be used for testing and evaluating performance of the model. Moreover, although the trainingvalidation mechanisms described herein focus on training a new CNN 400, these trainingvalidation mechanisms can also be utilized to fine-tune existing CNNs trained from other datasets. For example, in some instances, a CNN might have been pre-trained using digital pathology images for one type of cancer to predict the detection of another type of cancer. In those cases, CNN 400 can be used for transfer learning and retrained / validated using new sets of digital pathology images for a second type of cancer. Further, the learned neural network parameters may be frozen (as shown in FIG. 4B) and only the last layer (e.g., the fully connected layer) of CNN 400 is retrained / validated to leam feature predictions associated with a second type of cancer.Transfer Learning of Machine Learning Models

[0167] In some embodiments, transfer learning techniques can be employed to fine-tune existing CNNs (e.g., CNN 400 described with respect to FIG. 4) that were previously trained using digital pathology’ images for one type of cancer to predict the detection of another type of cancer. Transfer learning is a machine learning method where a trained model developed for a first task (e.g., identifying cancerous regions in bladder digital pathology slides) is reused as the starting point for a model on a second task (e.g., identifying cancerous regions in different tissue digital pathology’ slides). In other words, the pre-trained model exploits the knowledge gained from a previous task to improve generalization about another task. Transfer learning differs from traditional machine learning in that it uses pre-trained models that have been used for another task to jump start the development process on a different, but related task. To achieve this, the weights that a model learned on a first task (e.g., weights learned from trainedAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT on bladder tissue slides) are transferred to a second task (e.g., training on different tissue slides, such as lung). This allows the pre-trained model to start with the patterns it learned when detecting cancer / tumors in bladder slides and apply those patterns to detecting cancer / tumors in lung slides. This “transfer” of learned information is possible because the parameters of the early and middle layers are frozen, and only the latter layers (e.g., the outer layers) are retrained. This is important because the early and middle layers of a neural network are typically responsible for detecting edges and shapes in an image, while the later layers are responsible for detecting some task-specific features.

[0168] There are several advantages that transfer learning techniques offer. For example, a pre-trained model (e.g., CNN 400 described with respect to FIG. 4) can be retrained and validated with very little new data compared to the initial amount of training data required to develop a CNN without prior training. In addition, the amount of time required to retrain and validate the CNN is significantly reduced and ty pically results in a higher performing model.

[0169] By way of example, without limitation, the following transfer learning process may be implemented for transitioning from a CNN trained to classify cancer in bladder tissue to a CNN trained to classify cancer in lung tissue. Initially, a pre-trained CNN is acquired that has previously been trained using whole-slide images of bladder tissue on a given task such as to classify regions as either tumor or tissue. The pre-trained CNN (e.g., a VGG16, ResNet, or Inception model) may have been trained, validated, and tested using the methods described above and with respect to FIG. 2 in relation to machine learning pipeline 200. In certain instances, the pre-trained CNN is similar or identical to CNN 400 described with respect to FIGs. 4A and 4B. Thereafter, the initial layers of the pre-trained model, which capture general features such as edges, textures, and simple patterns, are frozen. Freezing these layers means their weights are not updated during training on the new task. This is done to retain the learned features that are likely useful for both bladder and lung cancer classification. The final layers of the pre-trained model are then replaced with new layers that are appropriate for the lung cancer classification task. Typically, this involves replacing the fully connected layers and the output layer. For instance, if the pre-trained model has a final fully connected layer with 10 neurons (for 10 classes), it may be replaced with a fully connected layer with neurons corresponding to the number of lung cancer classes. Additionally, new fully connected la ers with ReLU activation functions and dropout for regularization may be used to replace some or all of the remaining fully connected layers.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0170] A different set of whole-slide images (i.e., images of lung tissue) is collected and preprocessed (e.g.. using the techniques described with respect to FIGs. 1 and 2). This may include resizing images to match the input size expected by the pre-trained model, normalizing pixel values, and possibly augmenting the data to increase variability. In some instances, instead of relying on a large set of fully annotated whole-slides, specific regions of the wholeslide images are used for retraining and validation of the final layers of the pre-trained CNN. The regions may be manually or automatically selected from regions of the whole-slide images displaying the largest proportions of target clinical morphological features (e g., the regions that include target clinical morphological features associated with a given disease state (e.g., tumor cells)), for example, specific physical or structural characteristics of a sample, tissue, organ, or medical image that are relevant to a clinical analysis, diagnosis, or research. The largest proportions of target clinical morphological features may be determined using a threshold such as regions having at least 50%, 60%, or 70% target clinical morphological features. As anon-limiting example, 10-1000 regions across 5-50 different whole-slide images displaying target clinical indications, with each region having 3 x 3 section of tiles, may be determined and used for transfer learning. The resulting set of region-specific images are the final lung tissue dataset to be used for fine tuning of the pretrained model.

[0171] As shown in FIG. 5, the regions of the different whole-slide images (i.e., images of lung tissue) may be split into grid-like segments, each comprising A x B sections (e.g., 3 x 3 sections), and then a pathologist(s) and / or computer program is used to select which sections include tumor cells or non-tumor cells based on inclusion and / or exclusion criteria, and those sections are then used as the specific regions for retraining and validation. Inclusion criteria may include the presence of tumor stroma, non-invasive and invasive urothelial carcinoma, low- and high-grade tumors, differentiated regions, and the like. Additional inclusion criteria may be that an individual section comprise about 25% or more tumor tissue to be considered, as shown at the bottom of FIG. 5. Exclusion criteria can include regions with necrosis, extensive cautery' effect, morphologically unacceptable (e.g., washed off tissue, blurred view, etc.), and the like. Overall, this selection process yields a significantly smaller dataset (e.g., final lung tissue dataset) compared to the large set of fully annotated whole-slides. The smaller dataset ensures that the labels or annotations are accurate. Incorrect labels can mislead the model during training and lead to poor performance. Furthermore, the smaller dataset ensures that the criteria for labeling are consistent across the dataset. Inconsistent annotations can introduce ambiguity and reduce model accuracy. Lastly, the smaller dataset ensures that theAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT images are relevant to the specific task the model is being trained for (e.g., classification of lung tissue).

[0172] For fine tuning, the pretrained model is complied with an appropriate loss function (e.g., categorical cross-entropy for multi-class classification) and optimizer (e g., Adam or SGD). The model is then trained on the final lung tissue dataset. Initially, only the new layers may be trained while keeping the pretrained layers frozen. This allows the new layers to learn task-specific features without disrupting the general features learned by the pretrained layers. After a few epochs, some of the lower layers of the pretrained model may be unfrozen and training continues with a lower learning rate. This allows the model to fine-tune the feature extraction process for the new task without causing significant weight updates that could disrupt the previously learned features. The fine-tuned model may then be evaluated on a validation set of lung tissue images to monitor performance and avoid overfitting. Use metncs such as accuracy, precision, recall, and Fl -score to assess the model’s performance, as described above and with respect to FIG. 2. In some instances, hyperparameter tuning may also be performed if necessary, adjusting learning rates, batch sizes, or network architecture to improve performance, as described above and with respect to FIG. 2. Once the fine-tuned model shows satisfactory performance on the validation set, it can be tested on a separate test set to ensure it generalizes well to new, unseen data. The fine-tuned model may then be deployed for use in a clinical setting or integrate it into a diagnostic tool for lung cancer classification (e.g.. as described in the assay or platform herein).Training a Machine Learning Model

[0173] FIG. 6 is a flowchart illustrating a process 600 for generating ground truth labeled training data for training a machine learning model to classify patches of WSI as comprising disease or non-tumor tissues (e.g., healthy tissues). The process 600 depicted in FIG. 6 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof (e.g., the intelligent selection machine). The software may be stored on a non-transitory storage medium (e.g., on a memory device). The process 600 presented in FIG. 6 and described below is intended to be illustrative and non-limiting. Although FIG. 6 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different orders, or some steps may also be performed in parallel. In some embodiments, such as the embodimentsAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT depicted in FIG. 2, the process 600 depicted in FIG. 6 may be performed by the components of the machine learning pipeline 200 described with respect to FIG. 2.

[0174] At box 605, whole-slide images (WSIs) are accessed. In some embodiments, each WSI depicts a specimen associated with a disease. In some embodiments, WSIs from healthy tissues (e.g., non-tumor samples) are also accessed. Each specimen can be a tissue section collected from a patient or subject that may be obtained and / or processed using the sample collector 101 described with respect to FIG. 1 or the machine learning pipeline 200 described with respect to FIG. 2. The tissue section may include samples from lung, liver, kidney, skin, bladder, breast, thyroid, lymph node, stomach, brain, blood vessels, pharynx, heart, pancreas, esophagus, muscle, thymus, or any other organ system from the human body. In some embodiments, the tissue section is bladder tissue that has been obtained from a subject at risk of, diagnosed with, or being treated for a bladder cancer. Accordingly, the bladder tissue section comprises regions of healthy tissue as well as regions of tumor tissue.

[0175] At box 610, ground truth labels for each WSI are generated. Initially, one or more pathologists specifically trained to detect the presence of tumor tissue on a slide annotate the bladder tissue. Typically, more than one (e.g., a plurality) pathologists independently provide their annotations for WSIs. In some embodiments, at least 3, 4, 5, 6, 7, or 8 pathologists independently provide their annotations for WSI; however, more or less pathologists may be used. The annotations may be at the whole-slide level (e.g., annotation applies to the whole slide) or at the patch level. Annotations may be provided as morphological features and / or scores and indicate regions on the WSI that correspond to healthy and disease (e.g., tumor) tissue. Example annotations provided by the plurality of pathologist include morphological features, such as cellular architecture, nuclear features, stromal changes, and necrosis; scores that quantify disease severity, including histological grades, mitotic index, or protein expression levels (e.g., Ki-67 or HER2); and indications of regions on WSIs that highlight positive or negative (e.g., tumor or non-tumor tissue), such as tumor margins, hotspots, or areas of stromal invasion.

[0176] After the annotations from each of the plurality of pathologists are obtained, the annotations are input into an image segmentation algorithm. Image segmentation is the process of partitioning an image into discrete groups of pixels (e.g., image segments) based on similar properties such as gray level, color, texture, brightness, or contrast to reduce the complexity of the image. In so doing, object detection algorithms (such as CNN 400 described with respectAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT to FIG. 4) can better learn the features / objects in the image. The image segmentation algorithm, which is a statistical model (e.g., STAPLE), may be an image segmentation algorithm that generates a WSI with ground truth labels, wherein the ground truth labels represent probabilistically determined ground truths for each annotation. To generate the WSI with ground truth labels, the image segmentation algorithm performs iterative operations including:(i) Generating a test segmentation of the WSI of the specimen by grouping each pixel in the WSI of the specimen based on similar properties. The image segmentation algorithm groups the pixels based on a majority vote or some other heuristic combination of the pathologists’ annotations. For example, the algorithm may vote pixels as “tumor” or “nontumor” and group accordingly.(ii) The image segmentation algorithm assigns performance parameters (e.g., weights) to each pathologist in the plurality of pathologists by comparing the annotations from the plurality of pathologists to the test segmentation. The performance parameters assigned to each pathologist reflects their sensitivity (true positive rate) and specificity (true negative rate) of their individual annotations as compared to the test segmentation. Thus, the more accurate the pathologist’s annotations are. the more influence their performance parameters are to assigning the pixels to either “tumor” or “non-tumor” group.(iii) The image segmentation algorithm assigns each pixel in the test segmentation of the WSI to an annotation unit (e.g., tumor, non-tumor, other) based on the annotations provided by the experts and their respective performance parameters. In other words, the probability that each pixel or voxel (or annotation unit) belongs to a particular test segmentation is computed based on the annotations provided by the experts and their respective performance parameters. This may be done using a Bayesian framework. The estimates of the test segmentation are then updated based on the probabilities computed. In addition, the performance parameters for each pathologist are also updated by computing the sensitivity and specificity based on how well each expert's annotations agree with the updated estimate of the true annotation.(iv) The image segmentation algorithm updates the test segmentation based on the performance parameters of each pathologist. The algorithm redraws the edges and boundaries of what the algorithm classifies as one feature from another (e.g., tumor versus non-tumor). The updated test segmentation displays the current weighted means of the probabilistic estimates of the segments being classified correctly. An example of a WSI test segmentationAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT generated from the weighted mean score form 8 pathologists is shown in FIG. 3A, where the annotation score is indicated by the heatmap color scale.(v) Steps (ii) through (iv) are iteratively repeated until the image segmentation algorithm generates a test segmentation that converges (i.e., does not change). Convergence is typically defined as the point where changes in the estimates of the true annotation and the performance parameters fall below a certain threshold.(vi) Once converged, the algorithm outputs a WSI with ground truth labels. The output ground truth labeled WSI depicts regions on the WSI that were unanimously classified as tumor or (non-tumor) tissue by the plurality of pathologists as well as regions labeled as unknown (or ignored) where pathologist annotations were inconsistent. An exemplary' output of a WSI with ground truth labels is shown in FIG. 3B. The ground truth labels represent probabilistic estimates of the true segmentation and a measure of the performance level represented by each segmentation.

[0177] In some embodiments, each annotated image is converted to a binary’ mask, and the set of binary masks is processed using a consensus image generation algorithm to generate the ground-truth label for the image. In some embodiments, each input mask is equally weighted. The resulting output image is a probabilistic image where each pixel value represents the probability of belonging to the annotated region. In some embodiments, each binary mask is converted into an array. Each element in the resulting array corresponds to a pixel / voxel in the mask. The consensus image generation algorithm generates a probability7array with element values ranging between 0 and 1. The consensus image generation algorithm converts the probability array into an output binary mask by thresholding the probabilities in the array, e.g.. at 0.5. That is, pixels with a probability greater than 0.5 are included in the segmented region (True or 1), while others are excluded (False or 0). This output binary mask represents the final consensus segmentation.

[0178] At box 615, a set of patches from each WSI with ground truth labels is generated. Each patch from the set of patches corresponds to a portion of the ground truth labeled WSI. In various embodiments, at least 1 ,000 individual WSI are obtained at box 605 and at least 300 individual patches from a single WSI are generated to create the set of patches. The patches may be of any appropriate (x,y) pixel dimensions (e.g., 299 x 299 pixels). In some instances, the patches may be designed with or without overlap (wherein at least one pixel is overlapped) with neighboring tiles / patches. The annotated WSI may be parsed into tens of individualAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT patches to tens of thousands of individual patches with appropriate pixel dimension and resolution quality (e.g., 299 x 299 pixels at 40X magnification ).

[0179] At box 620, the set of patches is used to train a machine learning model to classify the patch as diseased (having a specific disease or corresponding to a tumor section) or healthy (e.g., not having a specific disease or corresponding to a non-tumor section). The machine learning model may be a convolution neural network (CNN), similar to CNN 400 described with respect to FIG. 4, that leams patterns of features associated with the disease (e.g., cancer) from the training data (e.g., sets of patches) to predict whether each patch in the set of patches comprises diseased tissue or not. Training a machine learning model involves iterative operations to adjust a set of model parameters to minimize a loss or error function of the model. The loss or error function is configured to measure a difference between output predictions by the model and the ground truth labeled WSI. Because the machine learning model is a CNN, each layer has its own set of model parameters that are optimized based on that layer’s task. For example, the early and middle layers of a CNN are typically responsible for detecting edges and shapes in an image (e.g., more generic features of cancer), while the later layers are responsible for detecting some task-specific features.

[0180] At box 625, the trained machine learning model is output and deployed into a production environment where it can make inferences on real-world data. The production environment may be a clinical or pathology-based lab that is part of a medical institute that receives pathology slides, specifically bladder tissue slides. The trained model enhances data analysis, improves efficiency and speed with which pathologist can process slides, and reduces the cost associated with maintaining a pathology lab as well as the cost associated with managing the storage of the slides generated over many years.

[0181] The trained machine learning model, once deployed, can be continuously monitored to ensure it performs as expected. This involves tracking the model's prediction accuracy, response times, and other operational metrics. Additionally, the trained model may require retraining or updates based on new data or changing conditions in the environment it is applied in. This can be useful because machine learning models can drift over time due to changes in the underlying data the CNN is making predictions on — a phenomenon known as model drift.

[0182] At box 630, the trained machine learning model may be retrained using a new set of training data (e.g., non-bladder WSIs) to provide a second type of classification (e.g., to leam new feature patterns associated with a similar but different task). For example, the trainedAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT machine learning model may initially be trained to detect the bladder cancer in patches associated with WSIs of bladder tissues, and retrained to detect a non-bladder cancer (e.g., lung) in patches associated with WSIs of non-bladder tissues (e.g., lung). The process of retraining a previously trained model for a different but similar task is referred to as transfer learning. There are several benefits to adopting this practice, particularly in a pathology lab that receives numerous tissue samples from various origins. One such example is that a pretrained model can be retrained / validated with very little new data compared to the initial amount of training data required to develop a de novo model. In addition, the amount of time required to retrain / validate a model is significantly reduced and typically results in a higher performing model.

[0183] The process of retraining the trained machine learning model output at box 625 initially involves obtaining WSI patches displaying non-bladder tissue that comprise regions of tumor and non-tumor tissue. Nonlimiting examples of non-bladder tissue include lung, liver, kidney, skin, breast, thyroid, lymph node, stomach, brain, blood vessels, pharynx, heart, pancreas, esophagus, muscle, thymus, or any other organ system from the human body that is not bladder. As anon-limiting example, the non-bladder tissue can be lung tissue obtained from patients or subjects who are at risk of, have been diagnosed with, or are being treated for lung cancer. As described above, a major benefit of transfer learning is that the new model is retrained / validated with very little new data. The new training data (i.e., patches) are selected regions from WSIs that comprises morphological features of tumor (e.g., lung tumor) across the WSI. For example, at least 100 regions may be selected across at least 20 lung WSI. The selected regions are divided into a grid of patches, wherein each patch of the grid corresponds to a portion of the WSI. Then, either the same plurality7of pathologists or a different plurality7of pathologists select the patches comprising tumor tissues to generate the new set of training data (e.g., patches) for retraining the trained machine learning model.

[0184] In some embodiments, a set of selection critena is used by the plurality of pathologists to determine which grid patches should be classified as tumor and which should be classified as non-tumor. For example, the patches comprising tumor tissue consist of about 25% or more of tumor tissue and is positive for one or more features including tumor stroma, non-invasive urothelial carcinoma, invasive urothelial carcinoma, low-grade tumors, high-grade tumors, squamous cell differentiation, glandular differentiation, or any combination thereof. Additionally, or alternatively, the selected patches comprising tumor tissue exclude regions with necrosis, extensive cautery effect, washed off tissue, blurred view or any combinationAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT thereof. In so doing, a significant decrease in inter-pathology' variation is observed and the final training dataset is of high-quality’.

[0185] A key feature of transfer learning is that the learned sets of parameters corresponding to the initial layers of the trained machine learning model are initially frozen and left unchanged. In other words, the learned sets of parameters corresponding to the initial layers are transferred to the new model without any initial retraining for the new task. In addition, the fully connected layer and outer layer of the trained machine learning model are replaced with a new fully connected layer and a new outer layer to generate a transformed machine learning model. The transformed machine learning model is fine-tuned using the selected non-bladder patches. Fine tuning comprises performing iterative operations (i) to adjust the new fully connected layer and the new outer layer to leam non-bladder specific features and (ii) to unfreeze the learned sets of parameters corresponding to the initial layers of the trained machine learning model to maximize model performance.

[0186] At box 635, the retrained machine learning model is output (e.g., a model predicts whether patches are from a non-bladder WSI (e.g., lung)) to show tumor or healthy tissues. The retrained machine learning model may be deployed into the same production environment or a different production environment (e.g., a clinical or pathology-based lab in a medical institute) where it can make inferences on real-world data.Deployment of a Machine Learning Model into a Production Environment

[0187] FIG. 7 is a flowchart illustrating a process 700 for using a trained machine learning model in a production environment. The process 700 depicted in FIG. 7 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof (e.g., the intelligent selection machine). The software may be stored on a non-transitory storage medium (e.g.. on a memory device). The process 700 presented in FIG. 7 and described below is intended to be illustrative and non-limiting. Although FIG. 7 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different orders, or some steps may also be performed in parallel. In some embodiments, such as the embodiments depicted in FIG. 1 and FIG. 2, the process 700 depicted in FIG. 7 may be performed by the components of the computing environment 100 described with respect to FIG. 1 and / or the machine learning pipeline 200 described with respect to FIG. 2.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0188] At box 705, a tissue biopsy from a subject is accessed. The subject can be a subject with a known cancer, or a subject whose cancer status is unknown. The tissue section collected from the subject may be obtained and / or processed using the image processing platform 150 described with respect to FIG. 1 and / or the machine learning pipeline 200 described with respect to FIG. 2. The tissue section may include samples from lung, liver, kidney, skin, bladder, breast, thyroid, lymph node, stomach, brain, blood vessels, pharynx, heart, pancreas, esophagus, muscle, thymus, or any other organ system from the human body. In some embodiments, the tissue section comprises bladder tissues that have been obtained from a subject diagnosed with, or being treated for bladder cancer. Accordingly, the bladder tissue section comprises regions of healthy tissues as well as regions of tumor tissues.

[0189] In some instances, the tissue biopsy is processed by the image processing platform 150 described with respect to FIG. 1, where the tissue undergoes histochemical staining (e.g., H&E, Masson’s Trichrome, Periodic Acid-Schiff, Gomori’s Methenamine Silver and Reticulin stains, Prussian Blue stain, and the like) or IHC staining (single- or multi-plex staining). In various instances, the tissue is sectioned and stained using H&E methods.

[0190] At box 710, the tissue can be processed by the image processing platform 150 described with respect to FIG. 1 where staining (e.g., H&E staining) is performed. This step is optional, and whether it is performed depends on the goals of the workflow or analysis. Staining enables clear visualization of cellular and tissue morphology7, making it easier to distinguish between positive and negative (e.g., healthy and diseased) regions. The morphological details revealed by staining also enhance the quality of image data, ensuring more accurate segmentation and classification of tissue features.

[0191] In some embodiments, the staining step at box 710 is skipped to improve workflow efficiency, for example, when speed, cost, or alternative analyses are prioritized. For instance, bypassing staining allows for faster processing of tissue samples, which can be critical in timesensitive scenarios like intraoperative consultations. Moreover, unstained tissue is preserved for other downstream analyses, such as molecular or genetic testing, where staining might alter or consume the sample. By skipping this step, workflows can save on costs associated with reagents, labor, and processing time while exploring modem approaches that do not require stained slides. The decision to perform or omit staining should align with the specific objectives.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT

[0192] At box 715, the tissue sections are imaged using a scanner to generate a WSI. When multiplexed WSI are generated, each antigen marker is imaged and added to a stack of images that comprises all the individual antigen marker images. Each fluorescent image is aligned with one another to generate a single overlapping multiplexed WSI. Image alignment is achieved using specialized pixel-accurate alignment software (e.g., imageJ) that detects, matches, and aligns features between the images.

[0193] At box 720, a set of patches from the WSI are generated. Each patch from the set of patches corresponds to a portion of the WSI. The patches may be of any appropriate (x,y) pixel dimensions (e.g., 299 x 299 pixels). In some instances, the patches may be designed with or without overlap (wherein at least one pixel is overlapped) with neighboring tiles / patches. The annotated WSI may be parsed into tens to hundreds to thousands of individual patches with appropriate pixel dimension and resolution quality (e.g., 299 x 299 pixels at 40X magnification).

[0194] At box 725, the set of patches are input into a trained machine learning model (e.g., a trained CNN) to classify each patch from the set of patches as positive or negative (e.g., comprising tumor tissue or non-tumor tissue). The trained CNN may reside in production environment that is part of a clinical or pathology-based lab that is part of a medical institute that receives pathology slides. The CNN may be trained to distinguish between positive and negative features (e.g., to classify a patch as comprising tumor tissue or non-tumor tissue). For example, the CNN is trained to extract features (e.g., more complex patterns or characteristics of tumor tissue, such as irregular cell shapes, abnormal nuclear features, and disrupted tissue architecture, that cannot be identified by a human) that can classify’ a patch as comprising tumor tissue based on a probability score generated by the CNN.

[0195] At box 730, a segmentation mask and / or a heat map is generated based on the output of the CNN (e.g., the patch classifications). The segmentation mask and / or heat map can also be displayed through a GUI to enhance interpretability. The segmentation mask highlights specific regions of the image that the CNN identifies as positive (e.g., tumor tissue). The regions can be overlayed with a distinct color (e.g., red for tumor regions and green for non-tumor regions). Similarly, a heat map assigns varying intensify levels or colors to represent the probability scores calculated by the CNN for each pixel or patch within the image, with higher probabilities (e.g., those close to 1) corresponding to regions more likely to contain tumor tissue. In some embodiments, a report is also provided that includes the visualizations,Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT facilitating collaboration and supporting informative decision-making in both clinical and research settings. In some embodiments, the report further includes information related to (i) cell phenotyping in the tumor microenvironment (e.g., types of cancer cells, immune cells, and / or stroma cells), (ii) marker expression level, (iii) tumor heterogeneity (e.g., variations in marker expression within different regions of the tumor), (iv) ty pe of cancer, (v) stage of the cancer, (vi) therapeutic targets (e.g.. determine which targeted therapies are most likely to be effective), (vii) predicted treatment response, and the like. The CNN predictions and classifications can be verified by a pathologist and or other qualified personnel to ensure accurate interpretation and validation of findings. Pathologists review the annotated images, ensuring that the automated detections align with histopathological criteria and clinical relevance. The report is then given to an oncologist to assist in making more accurate and informed decisions regarding patient treatment options.Additional Considerations

[0196] Implementation of the techniques, blocks, steps and means described above can be done in various ways. For example, these techniques, blocks, steps and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described above, and / or a combination thereof.

[0197] Also, it is noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in the figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0198] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scriptingAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT language, and / or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine-readable medium such as a storage medium. A code segment or machine-executable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc.

[0199] For a firmware and / or software implementation, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example, software codes can be stored in a memorv. Memory' can be implemented within the processor or external to the processor. As used herein the term “memory ’' refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

[0200] Moreover, as disclosed herein, the term “storage medium,” “storage” or “memory ” can represent one or more memories for storing data, including read only memory (ROM), random access memory' (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other machine-readable mediums for storing information. The term “machine-readable medium” includes, but is not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage mediums capable of storing that contain or carry instruction(s) and / or data.

[0201] While the principles of the disclosure have been described above in connection with specific apparatuses and methods, it is to be clearly understood that this description is made only by way of example and not as limitation on the scope of the disclosure.

Claims

1. Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT WHAT IS CLAIMED IS:

1. A computer implemented method comprising:accessing whole-slide images (WSIs), wherein each WSI is a WSI of a specimen associated with a disease or a specimen unassociated with the disease;generating, ground truth labels for the WSIs, wherein the generating comprises, for each WSI,obtaining, from a plurality of pathologists, annotations of the WSI from each of the plurality of pathologists, wherein the annotations indicate whether a region on the WSI corresponds to the disease or not, andgenerating, by inputting the annotations of the WSI from each of the plurality of pathologists into an image segmentation algorithm, a WSI with ground truth labels, wherein the ground truth labels represent probabilistically determined ground truths for each annotation;generating, from each WSI with corresponding ground truth labels, a set of patches, wherein each patch from the set of patches corresponds to a portion of the ground truth labeled WSI;training, using the set of patches for each WSI, a machine learning model to classify7each patch from the set of patches as diseased or healthy, wherein the training comprises iterative operations to adjust a set of model parameters to minimize a loss or error function of the machine learning model, and wherein the loss or error function is configured to measure a difference between output predictions by the machine learning model and the ground truth labels; andoutputting the trained machine learning model.

2. The computer implemented method of claim 1, wherein the image segmentation algorithm is a consensus image generation algorithm .

3. The computer implemented method of claim 1, wherein each specimen is a tissue section originated from lung, liver, kidney, skin, bladder, breast, thyroid, lymph node, stomach, brain, blood vessels, phary nx, heart, pancreas, esophagus, muscle, or thymus of a subject.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT 4. The computer implemented method of claim 3, wherein the tissue section is a bladder tissue.

5. The computer implemented method of claim 1, wherein the disease is a cancer.

6. The computer implemented method of claim 1, wherein the plurality of pathologists comprises at least 3 pathologists.

7. The computer implemented method of claim 1, wherein the image segmentation algorithm performs iterative operations, wherein the iterative operations comprise:(i) generating a test segmentation of the WSI of the specimen by grouping each pixel in the WSI of the specimen based on similar properties,(ii) assigning performance parameters to each pathologist in the plurality of pathologists by comparing the annotations from the plurality' of pathologists to the test segmentation,(iii) assigning each pixel in the test segmentation of the WSI to an annotation unit based on the annotations provided by the plurality of pathologists and their respective performance parameters,(iv) updating, using the image segmentation algorithm, the test segmentation based on the performance parameters of each pathologist.(v) repeating steps (ii) through (iv) until the test segmentation converges, and (vi) outputting the WSI with the ground truth labels.

8. The computer implemented method of claim 1, wherein at least 100 individual WSIs are obtained, and wherein each WSI generates at least 300 individual patches.

9. The computer implemented method of claim 1, wherein the machine learning model is a convolution neural network.

10. The computer implemented method of claim 1 , further comprising retraining of the trained machine learning model, wherein the retraining comprises:obtaining patches corresponding to WSIs of a specific ty pe, wherein the WSIs comprise positive regions and negative regions;Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT freezing the learned sets of parameters corresponding to initial layers of the trained machine learning model and replacing a fully connected layer and an outer layer of the trained machine learning model with a new fully connected layer and a new outer layer to generate a transformed machine learning model;fine tuning, using the patches, the transformed machine learning model to classify each patch as positive or negative, wherein the fine tuning comprises performing iterative operations to adjust (i) to adjust the new fully connected layer and the new outer layer to leam specific features and (ii) to unfreeze the learned sets of parameters corresponding to the initial layers of the trained machine learning model to maximize model performance; and outputting the transformed machine learning model that classifies patches from the WSI as either positive or negative.

11. The computer implemented method of claim 10, wherein the patches are generated by¬ selecting regions comprising morphological features of each WSI;dividing the selected regions into a grid of patches, wherein each patch of the gnd corresponds to a portion of the WSI; andselecting, by the plurality of pathologists or a new set of a plurality of pathologists, the patches comprising tumor tissues to generate the patches for retraining the trained machine learning model.

12. The computer implemented method of claim 11, wherein at least 100 regions are selected across at least 20 different WSIs.

13. The computer implemented method of claim 11, wherein the patches comprising the tumor tissues consist of about 25% or more of the tumor tissues and is positive for one or more features including tumor stroma, non-invasive urothelial carcinoma, invasive urothelial carcinoma, low-grade tumors, high-grade tumors, squamous cell differentiation, glandular differentiation, or any combination thereof.

14. The computer implemented method of claim 11, wherein the patches comprising the tumor tissues exclude regions with necrosis, extensive cautery' effect, washed off tissue, blurred view or any combination thereof.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT 15. The computer implemented method of claim 11, wherein the patches comprising the tumor tissues (i) consist of about 25% or more of tumor tissue and is positive for one or more features including tumor stroma, non-invasive urothelial carcinoma, invasive urothelial carcinoma, low-grade tumors, high-grade tumors, squamous cell differentiation, glandular differentiation, or any combination thereof, and (ii) exclude regions with necrosis, extensive cautery effect, washed off tissue, and / or blurred view.

16. The computer implemented method of claim 1 , wherein the trained machine learning model is deployed into a production environment, and wherein deployment comprises:accessing a tissue sample from a subject;imaging, using a WSI scanner, the tissue sample to generate a WSLgenerating, from the WSI, a set of patches, wherein each patch from the set of patches corresponds to a portion of the WSI;inputting, into the trained machine learning model, the set of patches, wherein the trained machine learning model annotates each patch in the set of patches as positive or negative; andgenerating a segmentation mask or a heatmap based on the patch annotations from the trained machine learning model.

17. The computer implemented method of claim 1, wherein the deployment further comprises staining the tissue sample, wherein the stained tissue sample is being imaged using the WSI scanner.

18. The computer implemented method of claim 17. wherein the staining is Hematoxylin and Eosin (H&E) staining.

19. A method, comprising:accessing a tissue sample from a subject;imaging, using a whole-slide image (WSI) scanner, the tissue sample to generate a WSI;generating, from the WSI, a set of patches, wherein each patch from the set of patches corresponds to a portion of the WSI;inputting the set of patches into a trained machine learning model, wherein the trained machine learning model annotates each patch in the set of patches as positive or negative; andAttorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT generating a segmentation mask or a heatmap based on the patch annotations from the trained machine learning model.

20. The method of claim 19, further comprising training a machine learning model, wherein the training comprises:accessing a training set of WSIs, wherein each WSI of the training set is a WSI of a specimen associated with a disease or a specimen unassociated with the disease;generating, ground truth labels for the training set of WSIs, wherein the generating comprises, for each WSI of the training set,obtaining, from a plurality of pathologists, annotations of the WSI from each of the plurality of pathologists, wherein the annotations indicate whether a region on the WSI corresponds to the disease or not, andgenerating, by inputting the annotations of the WSI from each of the plurality of pathologists into an image segmentation algorithm, a WSI with ground truth labels, wherein the ground truth labels represent probabilistically determined ground truths for each annotation;generating, from each WSI with corresponding ground truth labels, a set of patches, wherein each patch from the set of patches corresponds to a portion of the ground truth labeled WSI;training, using the set of patches for each WSI, a machine learning model to classify each patch from the set of patches as diseased or healthy, wherein the training comprises iterative operations to adjust a set of model parameters to minimize a loss or error function of the machine learning model, and wherein the loss or error function is configured to measure a difference between output predictions by the machine learning model and the ground truth labels; andoutputting the trained machine learning model.

21. The method of claim 20, wherein the image segmentation algorithm is a consensus image generation algorithm .

22. The method of claim 20, wherein the image segmentation algorithm is performs iterative operations, wherein the iterative operations comprise:(i) generating a test segmentation of the WSI of the specimen by grouping each pixel in the WSI of the specimen based on similar properties,Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT (ii) assigning performance parameters to each pathologist in the plurality of pathologists by comparing the annotations from the plurality of pathologists to the test segmentation,(iii) assigning each pixel in the test segmentation of the WSI to an annotation unit based on the annotations provided by the plurality of pathologists and their respective performance parameters.(iv) updating, using the image segmentation algorithm, the test segmentation based on the performance parameters of each pathologist,(v) repeating steps (ii) through (iv) until the test segmentation converges, and (vi) outputting the WSI with the ground truth labels.

23. The method of claim 20, further comprising retraining of the trained machine learning model, wherein the retraining comprises:obtaining patches corresponding to WSIs of a specific ty pe, wherein the WSIs comprise positive regions and negative regions;freezing the learned sets of parameters corresponding to initial layers of the trained machine learning model and replacing a fully connected layer and an outer layer of the trained machine learning model with a new fully connected layer and a new outer layer to generate a transformed machine learning model;fine tuning, using the patches, the transformed machine learning model to classify each patch as positive or negative, wherein the fine tuning comprises performing iterative operations to adjust (i) to adjust the new fully connected layer and the new outer layer to leam specific features and (ii) to unfreeze the learned sets of parameters corresponding to the initial layers of the trained machine learning model to maximize model performance: and outputting the transformed machine learning model that classifies patches from the WSI as either positive or negative.

24. The method of claim 20, wherein the disease is a cancer.

25. The method of claim 24, further comprising generating a cancer status prediction for the subject based on the patch annotations and / or the segmentation mask or heatmap.Attorney Docket No.: 057618-1544087Client Reference No. : LC-2024-03-WO-PCT 26. The method of claim 25, further comprising generating a treatment plan based on the cancer status prediction, and optionally administering a specific treatment to the subject based on the treatment plan.

27. A system comprising:one or more processors; andone or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform the computer implemented method in any one of claims 1 -26.

28. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to perform the computer implemented method in any one of claims 1-26.

Images