Multiple instance learning in digital pathology

Abstract

Systems, apparatuses, and methods are provided for generating patch-level predictions in whole slide images (WSIs) using classification models trained with attention and self-attention mechanisms. A WSI is divided into patches, and each patch is processed by a feature extraction model to obtain features. Attention-based aggregation assigns weights to patch features during training, enabling the classification model to generate patch-level predictions for disease-associated or non-disease-associated target classes during inference, including artifacts. A patch is classified as positive for a target class if its predicted probability exceeds a threshold, and negative otherwise. Training uses multiple instance learning and attention-derived data to optimize model performance. This approach supports granular and interpretable outputs for diagnostic and quality assurance applications.

Description

MULTIPLE INSTANCE LEARNING IN DIGITAL PATHOLOGYCROSS-REFERENCES TO RELATED APPLICATION

[0001] The present application claims priority and benefit from U.S. Provisional Application No. 63 / 745,515, filed January 15, 2025, the entire contents of which are incorporated herein by reference for all purposes.FIELD

[0002] The present disclosure relates to digital pathology, and in particular to identifying pathology structures and features from digital images using multiple instance learning techniques.BACKGROUND

[0003] Integration of artificial intelligence (Al) and digital imaging into the pathology workflow has tremendously impacted the role that pathology has historically served in the clinical setting. Advancements in Al algorithms, computer-aided diagnostic techniques, and the development of whole slide imaging (WSI) are just a few examples of how bridging traditional science with computational techniques expand the sandbox of tools available to clinicians and researchers to decipher disease pathology and advance drug discovery, translation, and clinical applications. This technological advancement is referred to as digital pathology which is the management and interpretation of pathology information in a digital environment. Scanning devices are used to generate high resolution images of slides of biological samples, which may be stained, to generate digital images such as WSI. Application of digital pathology software enables faster processing, cheaper storage, and easy management of digital slide images increasing the opportunity for sharing slide images across the clinical community. In digital pathology, analysis may be applied to a wide range of features, including disease-associated findings such as tumors or abnormal tissue, as well as non-disease-associated elements such as artifacts, background, or normal tissue structures. Quality assurance tasks, such as detection of air bubbles, tissue folds, staining irregularities, debris, and other slide preparation features, are also addressed within digital pathology workflows.

[0004] In its infancy, Al-aided processing of digital images was mostly used for image -based diagnosis in radiology and cardiology. With the development of WSI, the application and benefits of Al processing greatly improved. Compared to X-ray imaging and Magnetic resonance imaging (MRI), WSI provides a wealth of information due to the large size of WSI (e.g., 100k x 100k) and increased image complexity. Image complexity is attributed to features often being in color due to various staining techniques (e.g., H&E and immunohistochemistry), availability of information at multiple scales (e.g. 4x, 20x, etc.), and multiple z-stack levels generating different views of the image depending on the plane of focus. WSI also benefits from not necessarily being restricted by image orientation (e.g., anatomical orientation) as encountered in radiology. Accordingly, Al-aided processing offers numerous advantages in the analysis of digital pathology images, enhancing both the accuracy and efficiency of diagnostic processes. Some of the key advantages include: (i) Al algorithms can detect subtle patterns and anomalies in pathology images that may be overlooked by human eyes, leading to more accurate diagnose, (ii) machine learning models can be trained to standardize interpretations, reducing variability and improving consistency in diagnostic results, (iii) Al can serve as a second opinion or decisionsupport tool for pathologists, helping to confirm or refine their diagnoses, (iv) Al can process and analyze large volumes of digital pathology images much faster than humans, accelerating diagnosis and treatment planning, (v) Al systems can handle large datasets and scale up to meet the increasing demand for pathology services without a proportional increase in staffing, and (vi) by automating repetitive and time-consuming tasks, Al can reduce the workload for pathologists, leading to cost savings for healthcare institutions.BRIEF SUMMARY

[0005] Disclosed herein are machine learning techniques (e.g., computer-implemented methods, systems, and computer-program products) for (i) extracting features from whole slide digital pathology images, and (ii) using attention-based and / or self-attention-based models to identify and prioritize features most relevant for classification. The described approach supports training of a multiple instance learning model to classify regions (e.g., patches) within a whole slide digital pathology image as positive or negative for any target class. The target class may include a disease-associated feature (such as tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence ofinfiltrating immune cells) or a non-disease-associated feature (such as artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content). These techniques include utilizing convolutional neural networks (CNNs), such as Inception, residual neural network (ResNet), and similar models to identify features from patches of whole slide images (WSI) to be used as input into attention-based or self-attentionbased models. The attention and / or self-attention models identify which of the extracted features should be prioritized for classification tasks, enabling flexible application to both diagnostic and quality assurance workflows. Once feature prioritization is established, machine learning models are trained using multiple instance learning training techniques to predict a label for a bag of patches, where the bag corresponds to a set of patches from a WSI labeled for the presence or absence of a target class. This technique is applicable to a wide range of medical images, including pathology slides and digital WSIs, particularly where regional annotations are lacking. Performance of the models can be evaluated during training by comparing ground truth bag labels to predicted bag labels using metrics such as Fl -score, sensitivity, specificity, and area under the curve (AUC). Once model performance is acceptable, the models are deployed for use in diagnostic, quality assurance, or research environments.

[0006] Because the machine learning models are trained on data where each feature is assigned a corresponding attention weight, the model parameters are tuned to prioritize features most relevant for classification of the chosen target class. When deployed into a production environment, the models make predictions at the patch level, providing spatially resolved outputs such as heatmaps or binary masks for any target class of interest. By training with an attentionbased or self-attention-based multiple instance learning technique, the resulting models deliver more precise, granular, and interpretable predictions during inference compared to approaches that are limited to slide-level or region-aggregated outputs. This enables more accurate and transparent clinical review, quality control, and reporting in digital pathology workflows.

[0007] In various embodiments, a computer-implemented method is provided, comprising obtaining, from a whole slide image (WSI) of a specimen, a set of patches; extracting, using a feature extraction model, a set of features from each patch in the set of patches; generating, using a classification model, a patch-level prediction for each patch in the set of patches, wherein the classification model outputs a predicted probability for each patch, and wherein the predictedprobability is based on attention weighted features used to train the classification model; and outputting the patch-level prediction for each patch, wherein: the patch-level prediction is positive for a target class when the predicted probability of the patch is outside of a predetermined threshold, and the patch-level prediction is negative for the target class when the predicted probability of the patch is within the predetermined threshold.

[0008] In some embodiments, the target class comprises a disease-associated feature or a nondisease-associated feature.

[0009] In some embodiments, the disease-associated feature comprises at least one of: a tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells.

[0010] In some embodiments, the non-disease-associated feature comprises at least one of: an artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content.

[0011] In some embodiments, the classification model is trainable with feature tensors corresponding to patches input to an embedding aggregator, the embedding aggregator comprising a self-attention mechanism and an attention mechanism, wherein attention scores are generated with respect to the feature tensors and wherein model parameters are adjustable based on a comparison between predicted outputs and reference labels.

[0012] In some embodiments, the embedding aggregator is configured such that any patch in the set of patches is able to attend to all other patches in the set to support global context modeling.

[0013] In some embodiments, the computer- implemented method further comprises obtaining slides of the specimen, wherein the slides are stained using a histological staining technique, and examining the slides for pathological features associated with the disease.

[0014] In some embodiments, examining the slides comprises (i) identifying one or more regions of interest within the slides; (ii) segmenting the slides into regions associated with a plurality of tissue types or pathological features; (iii) detecting and quantifying morphological features selected from the group consisting of: cell nuclei, cell membranes, cytoplasm,extracellular matrix, tissue architecture, or vascular structures; (iv) detecting and classifying the pathological features selected from the group consisting of: tumor regions, necrotic regions, fibrotic regions, areas of abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or infiltrating immune cells; (v) detecting and classifying non-disease-associated features selected from the group consisting of: artifacts, air bubbles, tissue folds, staining irregularities, debris, background regions, normal tissue structures, or regions lacking diagnostic content; (vi) quantifying an extent or severity of the pathological features; and / or (vii) generating a statistical report on a distribution, frequency, or area of the pathological features within the slides.

[0015] In some embodiments, the histological staining technique is hematoxylin and eosin staining.

[0016] In some embodiments, the specimen comprises cells, tissue, or biological fluid.

[0017] In some embodiments, the specimen is a tissue sample.

[0018] In some embodiments, the specimen is collected from a tissue sample suspected of disease positive.

[0019] In some embodiments, the target class is a cancer-positive class.

[0020] In some embodiments, the cancer is a pancreatic, kidney, renal, pelvic, colorectal, stomach, thymic, head and neck, mesothelial, prostate, cervical, thyroid, adrenal, testicular, breast, uterine, bone, esophageal, lung, liver, bile duct, ovarian, bladder, nervous system, or skin cancer.

[0021] In some embodiments, the set of patches comprises at least 50 patches from the WS1.

[0022] In some embodiments, the feature extraction model is a convolution neural network.

[0023] In some embodiments, the set of features is related to pathological features, edges, comers, patterns detected in the patch, or any combination thereof associated with a disease negative specimen, a disease positive specimen, or both.

[0024] In some embodiments, the set of features is stored in a specific data structure.

[0025] In some embodiments, the specific data structure is a tensor having a size of at least 400.

[0026] In some embodiments, the classification model is a support vector machine, a decision tree, a random forest model, logistic regression model, a neural network, an attention-based neural network, or a k-nearest neighbor model.

[0027] In some embodiments, the classification model was trained on training data comprising attention-weighted features generated from an attention mechanism, a self-attention mechanism, or both.

[0028] In some embodiments, the training data comprises at least 1,000 sets of patches, wherein each set of patches is generated from a unique WSI with a global label.

[0029] In some embodiments, the classification model was trained using multiple instance learning (MIL) techniques

[0030] In some embodiments, the predicted probabilities for the set of patches are stored in an array representation of the WSI.

[0031] In some embodiments, the computer-implemented method further comprises generating, using the patch-level prediction for each patch in the set of patches, a binary mask of the WSI, wherein the binary mask indicates regions in the WSI that are predicted to be either positive or negative for the target class.

[0032] In some embodiments, the computer-implemented method further comprises generating, using the predicted probability for each patch in the set of patches, a heatmap of the WSI.

[0033] In some embodiments, the computer-implemented method further comprises generating a slide-level prediction for the whole slide image based on an aggregation of the patch-level predictions.

[0034] In some embodiments, the computer-implemented method further comprises displaying, on a graphical user interface, the WSI with the patch-level predictions overlayed on the WSI.

[0035] In various embodiments, a computer-implemented method is provided comprising: obtaining from a plurality of whole slide images (WSIs), sets of patches, wherein each set of patches corresponds to a WSI of the plurality of WSIs, and wherein each WSI is labeled with a target class; extracting, using a feature extraction model, a set of features for each patch in the sets of patches; inputting the sets of the extracted features for the sets of patches into an aggregator, wherein the aggregator comprises: (i) an attention mechanism that takes a set of extracted features as input and generates a first set of attention scores, (i) a self-attention mechanism that takes a set of extracted features as input and generates a second set of attention scores, and / or (iii) an attention-transformer mechanism that takes transformed features of a set of extracted features as input and generates a third set of attention scores; generating, by taking the product of one or more attention scores and corresponding set of extracted features for each patch in the set of patches, attention-weighted feature vectors; aggregating the attention-weighted feature vectors into a single representation to generate bag features; generating, by a classification model using the bag features, a predicted bag label for each bag, wherein each bag correspond to a set of patches of the sets of patches, and wherein each bag corresponds to the set of patches from a single WSI and a ground truth label for each bag indicates a presence or absence of the target class in the corresponding WSI; performing iterative operations to adjust model parameters of the aggregator and / or the classification model to minimize a loss or error function, wherein the loss or error function is configured to measure a difference between the predicted bag label and a ground truth label for the target class; and outputting the aggregator and / or the classification model to be deployed into a production environment.

[0036] In some embodiments, the target class comprises a disease-associated feature or a nondisease-associated feature.

[0037] In some embodiments, the disease-associated feature comprises at least one of: a tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells.

[0038] In some embodiments, the non-disease-associated feature comprises at least one of: an artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content.

[0039] In some embodiments, each set of extracted features is stored in a specific data structure.

[0040] In some embodiments, the specific data structure is a tensor having a size of at least 400.

[0041] In some embodiments, the plurality of WSIs comprises at least 1,000 WSIs.

[0042] In some embodiments, the feature extraction model is a convolution neural network.

[0043] In some embodiments, each set of the extracted features is stored in a tensor having a size of at least 400.

[0044] In some embodiments, the self-attention mechanism in (ii) generates the transformed features of each set of extracted features.

[0045] In some embodiments, each bag includes a different number of patches.

[0046] In some embodiments, each bag comprises at least 50 patches generated from a single WSI.

[0047] In some embodiments, the aggregator is an embedding aggregator and / or the aggregator is configured such that any patch in a set of patches is able to attend to all other patches in the same set.

[0048] In some embodiments, a system is provided that includes one or more processors, and a memory (e.g., a non-transitory computer readable medium) that is coupled to the one or more processors and stores a plurality of instructions which, when executed by the one or more processors, cause the one or more processors to perform part or all actions or operations in one or more methods or processes disclosed herein.

[0049] In some embodiments, a non-transitory computer readable storage medium is provided comprising computer program instruction that, when executed by one or more processors, cause the one or more processors to perform part or all actions or operations in one or more methods or processes disclosed herein.

[0050] In some embodiments, a computer-program product is provided that is tangibly embodied in a non-transitory computer-readable memory that includes instructions which, whenexecuted by the one or more processors, cause the one or more processors to perform part or all actions or operations in one or more methods or processes disclosed herein.

[0051] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the disclosure. Thus, it should be understood that although the present application has been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this disclosure as defined by the appended claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings illustrate certain embodiments of the technology and are not limiting. For clarity and ease of illustration, the drawings are not made to scale and, in some instances, various aspects may be shown exaggerated or enlarged to facilitate an understanding of particular embodiments.

[0053] FIG. 1 shows an exemplary system for generating digital pathology images in accordance with various embodiments.

[0054] FIG. 2 illustrates the training and implementation of a machine learning model that processes digital pathology images in accordance with various embodiments.

[0055] FIG. 3 A shows the various layers of an exemplary convolution neural network (CNN) in accordance with various embodiments.

[0056] FIG. 3B illustrates processing of an image via the various layers of the CNN in accordance with various embodiments.

[0057] FIG. 4A illustrates an exemplary attention mechanism to generate bag features in accordance with various embodiments.

[0058] FIG. 4B shows an exemplary self-attention mechanism to generate bag features in accordance with various embodiments.

[0059] FIG. 5A illustrates an instance-based multiple instance learning (IMIL) framework to provide bag predictions in accordance with various embodiments.

[0060] FIG. 5B illustrates an embedding-based multiple instance learning (EMIL) framework to provide bag predictions in accordance with various embodiments.

[0061] FIG. 6 illustrates a multiple instance learning (MIL) training technique in accordance with various embodiments.

[0062] FIG. 7 illustrates an attention-based multiple instance learning (ABMIL) framework in accordance with various embodiments.

[0063] FIG. 8 illustrates an exemplary process of generating a heat map and a mask map from an input WSI using a trained ABMIL classifier in accordance with various embodiments

[0064] FIG. 9A illustrates a self-attention based multiple instance learning (SAB MIL) framework in accordance with various embodiments.

[0065] FIG. 9B shows an exemplary embedding aggregator used in a SAB MIL framework in accordance with various embodiments.

[0066] FIG. 10 illustrates a flowchart for training a machine learning model in accordance with various embodiments.

[0067] FIG. 11 illustrates a flowchart for using a trained machine learning model in a deployed environment in accordance with various embodiments.

[0068] FIG. 12 illustrates an exemplary preprocessing for patch tiling in accordance with various embodiments.

[0069] FIG. 13 illustrates the interpretability and performance of the disclosed techniques for patch-level classification in digital pathology in accordance with various embodiments.DETAILED DESCRIPTION

[0070] The ensuing description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the ensuing description of the preferred exemplary embodiments will provide those skilled in the art with anenabling description for implementing various embodiments. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0071] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order to avoid obscuring the embodiments.

[0072] Also, it is noted that individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart or diagram may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed, but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0073] Publications cited herein and the material for which they are cited are hereby specifically incorporated by reference in their entireties.I. INTRODUCTION

[0074] Large medical centers typically have an average caseload of approximately 80,000 to over 100,000 different pathology samples per year, with daily averages ranging from 500 to 750 tissue blocks and more than 1,500 slides. Conventionally, histologic grading is manually performed by one or more pathologists / technicians, where a single diagnosis can take several minutes (e.g., 20 minutes) to up to several weeks (e.g., two weeks) depending on the complexity of the specimen and diagnosis. Given the large work volume, pathology analysis is inherently subjected to inter-pathologist variability. These trends are further exasperated as the size of theimage increases, making it even more challenging for pathologists to detect subtle morphological features or important pattems / trends associated with sets of similar tissue pathologies that pathologists may infrequently encounter over many years. In addition to tumor identification, accurate detection of artifacts such as air bubbles, tissue folds, staining irregularities, and other slide preparation features remains challenging, as these may influence diagnostic interpretation and quality review. Difficulty in consistently recognizing both pathology-related and nonpathology-related features, including artifacts and typical tissue structures, can impact diagnostic outcomes and workflow efficiency.

[0075] Accordingly, it is impossible for the human mind to practically process, consistently produce accurate diagnosis, and / or detect subtle pattems / trends over the course of a few days. While manual examination is the current standard, integration of computational approaches, such as machine learning (ML) and Al, into pathology offer tremendous potential to provide assistance to pathologists as well as enhance data analysis, efficiency, speed, and cost.Computer-implemented methods that prioritize diagnostically relevant regions and features, rather than processing all image data uniformly, can reduce computation time and memory demands during both training and inference, enabling more rapid analysis of large datasets and supporting scalable digital pathology workflows.

[0076] More specifically, Al algorithms, particularly those utilizing deep learning, can analyze vast quantities of pathology images rapidly and with remarkable precision. These algorithms can identify patterns, anomalies, and subtle features that may not be readily detected by human observers. In practice, ALassisted pathology has been associated with improvements in the accuracy and timeliness of diagnoses, which can have a significant impact on treatment decisions and patient outcomes. Increased diagnostic accuracy may also help reduce costs related to misdiagnosis and unnecessary treatments. In addition, Al-powered tools can help standardize diagnostic interpretations, minimizing variability among pathologists and promoting more consistent results. The use of Al in digital pathology can also automate repetitive and timeintensive tasks, thereby reducing the workload for pathologists and enabling them to concentrate on more complex cases and decision-making processes. Al-based systems are also being considered for applications beyond disease detection, such as the identification of artifacts including air bubbles, tissue folds, and staining irregularities, which can impact overalldiagnostic quality and workflow. These developments have the potential to improve both the quality and cost-efficiency of healthcare.

[0077] However, the application of Al in digital pathology is not without challenges, particularly concerning the availability of high-quality training data (e.g., training data having accurate ground truths) for supervised and semi-supervised learning. The effectiveness and reliability of machine learning models are directly linked to the quality of the training data provided. Obtaining such data is difficult for several reasons. Manual annotation to establish ground truths requires significant time and effort, and is susceptible to inconsistencies and inaccuracies caused by human error. In domains such as medicine, the necessary expertise for accurate labeling is limited and acquiring expert-labeled datasets can be costly. Even when labeled data is available, ensuring that the dataset is representative and unbiased remains a challenge, as imbalanced or biased datasets may reduce model performance and hinder the generalizability of results.

[0078] Beyond data constraints, traditional computational approaches, including established multiple instance learning (MIL) techniques, face limitations in their ability to produce finegrained and interpretable predictions or to flexibly model the complex spatial relationships present in whole slide images. Many of these approaches do not adequately capture long-range dependencies or nuanced contextual information that spans across an entire digital slide.Conventional MIL approaches, including those that use block sparse or cluster-based selfattention, typically aggregate information within pre-defined or learned clusters. This aggregation limits the model’s ability to consider relationships at the scale of the whole slide image, resulting in less detailed and less interpretable predictions at the patch or region level. As a result, weakly supervised and semi-supervised learning strategies in digital pathology have primarily produced global or slide-level labels, rather than labels for individual patches, because generating high-quality regional annotations remains labor-intensive, time-consuming, and susceptible to error. These challenges are compounded when attempting to generate outputs that provide detailed, region-specific information, such as heatmaps or masks for quality control and diagnostic support. These challenges highlight the need for computational frameworks that support efficient and scalable use of available training data, while also enabling more flexible and globally contextual modeling of spatial relationships within pathology images.

[0079] To address these challenges and others, disclosed herein are techniques (e.g., computer-implemented methods, systems, computer readable storage media, and computer-program products) for training and deployment of attention and self-attention MIL models that classify patches (e.g., small regions corresponding to a portion of a whole image, or a whole slide image) of a WSI as containing relevant features. The classification for each patch is provided as positive or negative for a target class, where the target class may comprise a disease-associated feature or a non-disease-associated feature. Examples of disease-associated features include tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells. Examples of non-disease-associated features include artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content. The disclosed techniques leverage advanced transformer-based self-attention mechanisms and attention based embedding aggregators to enable global context modeling across all patches within a bag. The embedding aggregator may be configured such that any patch in the set of patches is able to attend to all other patches in the set, supporting global context modeling.

[0080] The disclosed approach allows the model to capture long-range dependencies and nuanced relationships throughout the entire slide, resulting in more comprehensive and informative feature representations. This capability supports not only clinical diagnosis but also quality assurance by identifying artifacts that may interfere with interpretation. The disclosed techniques also allow for automated generation of predicted annotations for regions of interest within the WSI, reducing the need for extensive manual review or labeling by human experts. In particular, the disclosed approach provides increased granularity and improved interpretability by generating patch-level predictions for each region of the whole slide image. This enables visualization and review of model outputs at a fine spatial resolution, allowing pathologists and users to understand which regions contributed most to a given classification or quality control assessment. Compared to approaches limited to slide-level or cluster-based predictions, this patch-level output supports more transparent, auditable, and clinically useful results. The extracted features may be stored in a specific data structure such as a tensor and may include features related to pathological findings, edges, comers, patterns, diagnostic biomarkers, molecular markers, or any features relevant to the classification of the target class. This approach also improves computer system performance by prioritizing analysis of the most relevant regionsand features, which reduces overall computation time and memory usage during both training and inference. As a result, the methods support more efficient and scalable digital pathology workflows.

[0081] In an exemplary embodiment, a computer-implemented method is disclosed that comprises obtaining, from a whole slide image (WSI) of a specimen, a set of patches; extracting, using a feature extraction model, a set of features from each patch in the set of patches; generating, using a classification model, a patch-level prediction for each patch in the set of patches, wherein the classification model outputs a predicted probability for each patch, and wherein the predicted probability is based on attention weighted features used to train the classification model; and outputting the patch-level prediction for each patch, wherein: the patchlevel prediction is positive for a target class when the predicted probability of the patch is outside of a predetermined threshold, and the patch-level prediction is negative for the target class when the predicted probability of the patch is within the predetermined threshold. The target class may comprise a disease-associated feature or a non-disease-associated feature, including but not limited to tumor, necrosis, artifact, tissue fold, or normal tissue structure.

[0082] In another exemplary embodiment, a computer-implemented method is disclosed comprising: obtaining, from a plurality of whole slide images (WSIs), sets of patches, wherein each set of patches corresponds to a WSI and each WSI is labeled with a target class; extracting, using a feature extraction model, a set of features for each patch in the sets of patches; inputting the sets of the extracted features for the sets of patches into an aggregator, wherein the aggregator comprises one or more of: (i) an attention mechanism that takes a set of extracted features as input and generates a first set of attention scores, (ii) a self-attention mechanism that takes a set of extracted features as input and generates a second set of attention scores, or (iii) an attention-transformer mechanism that takes transformed features as input and generates a third set of attention scores; generating, by taking the product of one or more attention scores and the corresponding set of extracted features for each patch, attention-weighted feature vectors; aggregating the attention-weighted feature vectors into a single representation to generate bag features; generating, by a classification model using the bag features, a predicted bag label for each bag, wherein each bag corresponds to the set of patches from a single WSI and a ground truth label for each bag indicates a presence or absence of the target class in the correspondingWSI; performing iterative operations to adjust model parameters of the aggregator or the classification model to minimize a loss or error function, wherein the loss or error function measures a difference between the predicted bag label and the ground truth label for the target class; and outputting the aggregator or the classification model for deployment in a production environment.

[0083] The attention-based (or self-attention-based) model outputs weighted attention scores for each input feature from the sets of patches extracted by the feature extraction model. The use of transformer-based self-attention or embedding aggregator architectures allows any patch in the set of patches to attend to all other patches in the same set, enabling global context modeling and holistic representation across the entire slide. The resulting attention-weighted features are aggregated to form bag features that holistically represent the content of the slide, capturing both local and global relationships among pathological and non-pathological features. These bag features are then input into a classification model that predicts whether the bag is positive or negative for a target class (such as a disease-associated feature or a non-disease-associated feature, including artifacts or normal tissue regions) based on the aggregated weighted features. This approach eliminates the need for one-to-one patch-level labels during training, reducing annotation burden and making use of weakly supervised data while still supporting high-resolution, interpretable predictions. By supporting patch-level predictions, the present approach provides a higher degree of granularity and interpretability compared to conventional approaches that generate only slide-level or region-aggregated outputs. Patch-level predictions enable generation of heatmaps, binary masks, and overlays that visually indicate the specific regions that contributed to the classification, thereby supporting clinical review, audit, and quality control. Obtaining WSI labels in this way is considerably easier, more consistent, and significantly more cost effective compared to obtaining region specific annotations within the WSIs manually generated by an expert (e.g., pathologist).

[0084] Moreover, the disclosed attention mechanisms provide several important advantages for digital pathology analysis: (i) they introduce feature prioritization by assigning calculated attention weights to each input feature, allowing the model to quantify the relative importance of different regions within a whole slide image; (ii) they enable the model to suppress irrelevant noise and focus computational resources on diagnostically meaningful content; (iii) they supportselective emphasis of specific features or regions, rather than treating all input patches equally, which leads to more efficient learning and improved predictive performance; and (iv) they enhance interpretability by providing explicit insight into which parts of the input data most influence the model’s classification decisions. Additionally, these mechanisms support the identification of both disease-associated features and non-disease-associated features, such as artifacts or normal tissue, as part of the target class prediction.

[0085] Beyond these advantages, the disclosed techniques use transformer-based self-attention and embedding aggregator architectures to enable the model to capture complex, long-range dependencies and contextual relationships among all patches in a slide, rather than being limited to local clusters. This global context modeling supports flexible, data-driven aggregation of patch-level features and enables interpretable, high-resolution patch-level predictions at inference for any target class of interest, including both pathological and non-pathological findings, even when trained only with slide-level (bag-level) labels. As a result, the disclosed frameworks address limitations of prior MIL approaches by providing both improved diagnostic granularity and greater clinical interpretability.

[0086] MIL training techniques are used to train an attention-based ML model. MIL training is a type of weak supervised learning where instead of receiving a set of individually labeled instances, a set of labeled “bags,” each containing many unlabeled instances, is received. This training technique is particularly useful in situations where obtaining high quality labeled data is challenging, such as in digital pathology. In this particular case, the attention-based ML model is trained on sets of bags, where the bags contain sets of unlabeled patches whose features have been assigned a weighted attention. The attention-based ML model predicts labels for the bags as positive or negative for a target class (e.g., tumor, artifact, or other region of interest) based on the summary of weighted patch features, where the predicted bag labels correspond to the global label of the WSI. Other weak supervised learning training may be used without departing from the scope of the present disclosure.

[0087] By training the ML models on attention derived training data, the model parameters are tuned to give more weight / prioritization to certain features when making a classification. As a result, deployment of these models into a production environment (e.g., a diagnostic laboratory) allows for patch level classifications as opposed to global / slide level predictions made duringtraining. The final models are thus capable of producing precise patch-level predictions at inference, supporting a variety of target classes, including both disease-associated and nondisease-associated features, and improving the resolution and interpretability of outputs compared to approaches that are limited to slide-level classification.II. TERMS

[0088] As used herein, the articles “a” and “an” are used herein to refer to one or to more than one (i.e. at least one) of the grammatical object of the article. By way of example, an element means at least one element and can include more than one element.

[0089] As used herein, the terms “about,” “similarly,” “substantially,” and “approximately” are defined as being largely but not necessarily wholly what is specified (and include wholly what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the term “about,” “similarly,” “substantially,” or “approximately” may be substituted with “within [a percentage] of’ what is specified, where the percentage includes 0.1 percent, 1 percent, 5 percent, and 10 percent, etc. Moreover, the term terms “about,” “similarly,” “substantially,” and “approximately” are used to provide flexibility to a numerical range endpoint by providing that a given value may be slightly above or slightly below the endpoint without affecting the desired result.

[0090] As used herein, the term “artifact” refers to any visual element or region in a pathology image that does not reflect underlying biological or pathological structures, but arises from slide preparation, staining, imaging, or scanning processes. Artifacts may include, but are not limited to, air bubbles, tissue folds, staining irregularities, debris, or background regions.

[0091] As used herein, the term “bag” refers to a collection or set of patches, typically all extracted from a single whole slide image (WSI), that is used as a unit of analysis in multiple instance learning (MIL) frameworks. Each bag may be assigned a label corresponding to the presence or absence of a target class within the originating WSI.

[0092] As used herein, when an action is “based on” something, this means the action can be based at least in part on at least a part of the something.

[0093] The use herein of the terms including, comprising, or having, and variations thereof, is meant to encompass the elements listed thereafter and equivalents thereof as well as additional elements. Embodiments recited as including, comprising, or having certain elements are also contemplated as consisting essentially of and consisting of those certain elements. As used herein, the term “and / or” refers to and encompasses any and all possible combinations of one or more of the associated listed items, as well as the lack of combinations were interpreted in the alternative (or).

[0094] Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. For example, if a concentration range is stated as 1% to 50%, it is intended that values such as 2% to 40%, 10% to 30%, or 1% to 3%, etc., are expressly enumerated in this specification. These are only examples of what is specifically intended, and all possible combinations of numerical values (e.g., integer, whole number, decimal, fraction, and the like) between and including the lowest value and the highest value enumerated are to be considered to be expressly stated in this disclosure.

[0095] As used herein, the term “embedding aggregator” refers to a computational module or mechanism that receives feature vectors (embeddings) from patches and combines them into a bag-level representation using one or more operations, such as pooling, attention, self-attention, or transformer-based methods. An embedding aggregator may be configured such that any patch in a set of patches is able to attend to all other patches in the set, thereby supporting global context modeling.

[0096] As described herein, the terms “non-tumor tissue / cell(s)” and “non-cancerous tissue / cell(s)” are used interchangeably and refer to tissues or cells that are not tumor or cancerous. Examples of non-tumor tissues / cells include, but are not limited to, pathologically healthy / normal cells, immune cells, non-malignant cells, and cells / tissues displaying abnormal pathology that is not related to cancer (e.g., inflammation, increase inflammatory cells, increase in apoptotic cells, and the like).

[0097] As described herein, the terms “non-tumor tissue features,” “non-cancerous tissue features,” “non-tumor associate features” are used interchangeably and indicate pathologyfeatures indicative of non-tumor tissue according to the specific tissue type. Pathological features associated with non-tumor tissue may include any combination of the following examples, (i) Uniformity of cell size, shape, and appearance as well as cellular organization. Cells are uniform in size, shape, and appearance, without significant variation (e.g., pleomorphism). Moreover, cells are organized in a manner consistent with the specific tissue type (e.g., orderly layers of cells in epithelial tissue), (ii) Cell structure is consistent with the specific tissue type such as nucleus-to-cytoplasm ratio, cell shape regularity, evenly distributed chromatin, clearly defined nucleolus, and cytoplasm appearance (e.g., eosinophilic cytoplasm in muscle cells), (iii) Lack of or absence of abnormal cellular activity. Cellular activity is indicated by rates of cell division measured by the presence and structure of mitotic figures. In other words, the amount of mitotic figures present is appropriate for the cell type (e.g., epithelial tissues, blood cells, and bone marrow have a higher rate of mitosis compared to other tissue systems), (iv) The tissue architecture is intact, meaning that tissue structures like glands, ducts, and supportive stroma are intact and well-organized without any disruption or distortion, (v) The extracellular matrix appears normal, with appropriate amounts of collagen, elastin, and other matrix components without excessive fibrosis or scarring, (vi) Adequate vascularization for the specific tissue type. A non-tumor tissue may display blood vessels present and appropriately distributed without evidence of abnormal blood vessel proliferation (e.g., angiogenesis) or damage, (vii) Connective tissue should be properly structured and provide adequate support and separation for different tissue types. There are limited to no signs of edema, fibrosis, or other connective tissue abnormalities, (viii) Apoptotic cells, if present, are in small numbers and appear as part of normal tissue turnover, or if present are not attributed to cancer, (ix) Lack of or no presence of pathogens (e.g., bacteria, viruses, fungi) that indicate infection or disease.

[0098] As used herein, the term “patch” refers to a small region or sub-image extracted from a WSI, which may be defined by specific dimensions or coordinates, and which serves as the basis for feature extraction and classification. As used herein, “patch-level prediction” refers to a classification or probability output that is generated for each individual patch, indicating whether that patch is predicted to be positive or negative for a target class. As used herein, “bag-level prediction” refers to a classification or probability output that is generated for a set of patches (bag), typically corresponding to the overall presence or absence of a target class in the originating WSI.

[0099] The term “substantial” or “substantially,” as used in the context of a worsening or improvement in a condition (e.g., a disease) refers to a change of at least 20%. Exemplary embodiments include at least: 20%, 25%, 30%, 35%, 40%, 45%, 50%, 55%, 60%, 65%, 70%, 75%, 80%, 85%, 90%, 91%, 92%, 93%, 94%, 95%, 96%, 97%, 98%, or 99%, as compared to an initial or baseline measure.

[0100] As used herein, the term “target class” refers to any category or type of feature, structure, or region to be identified or classified within a whole slide image, including but not limited to disease-associated features (such as tumors, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells) and non-disease-associated features (such as artifacts, air bubbles, tissue folds, staining irregularities, debris, background regions, normal tissue structure, or regions lacking diagnostic content).

[0101] As described herein, the terms “tumor tissue features,” “cancerous tissue features,” “tumor associate features” are used interchangeably and indicate pathology features indicative of tumor tissue according to the specific tissue type. Descriptions using “abnormal,” “atypical,” “increase / decrease / reduction,” etc. are all with respect to the non-cancerous state of the specific tissue type. Pathological features associated with tumor tissue may include any combination of the following examples, (i) Cellular atypia including pleomorphism (e.g., significant variation in cell size), nuclear atypia (e.g., abnormally large, irregularly shaped nuclei with uneven chromatin distribution and prominent nucleoli), and / or hyperchromasia (e.g., darkly stained nuclei due to increased DNA content), (ii) Increased mitotic activity indicated by an elevated number of cells undergoing division. Additionally, abnormal mitotic figures characterized by tripolar, or multipolar mitoses also indicate abnormal mitotic activity, (iii) Loss / reduced of normal tissue architecture as indicated by disorganized cell growth patterns (e.g., cell arrangement and organization). Also, signs of invasion where cancer cells are invading and destroying surrounding normal tissues, (iv) The nuclear-cytoplasmic ratio is increased (e.g., larger nuclei relative to the amount of cytoplasm) compared to the non-tumor state of the specific tissue, (v) Infiltrative growth such as invasion of cancer cells into adjacent structures (e.g., tissues, blood vessels, and lymphatic system) and desmoplasia (e.g., reactive fibrous tissue response to invasive cancer cells), (vi) Loss / reduction of differentiation (e.g., anaplasia) where the cells lack expecteddifferentiation patterns and instead appear undifferentiated. Also, the loss / reduction of specialized functions associated with the specific tissue type, (vii) Necrosis, or areas of cell death, due to rapid growth outpacing the blood supply, (viii) Angiogenesis indicating the formation of new blood vessels within the tumor to supply nutrients to rapidly growing cancer cells.III. GENERATION OF DIGITAL PATHOLOGY IMAGES

[0102] Digital pathology involves a range of tasks aimed at enhancing the accuracy and efficiency of disease diagnosis and research. These tasks include the digital acquisition of high-resolution images from glass slides using whole slide imaging technology. Once digitized, object detection algorithms identify and locate specific structures such as cells, tissues, or pathological features within the images. Segmentation techniques then partition the images into meaningful regions, such as delineating tumor boundaries from non-tumor tissue. Classification algorithms categorize detected and segmented objects into predefined classes, such as distinguishing between non-cancerous and cancerous cells or identifying different types of tumors. In addition to disease-associated features, digital pathology workflows may also involve the identification and classification of artifacts (such as air bubbles, tissue folds, staining irregularities, and debris) and normal tissue structures, which are important for quality assurance and assessment of slide preparation. These tasks may be supported by machine learning and artificial intelligence, which help automate and standardize analysis, enabling pathologists to make more precise and consistent diagnoses. Additionally, digital pathology facilitates remote consultations, data management, and integration with other digital health records, ultimately contributing to more informed therapeutic decision-making and improved patient outcomes.Whole Slide Image Preparation

[0103] FIG. 1 illustrates an exemplary system 100 designed to generate digital pathology images from a biological specimen 105 obtained from a subject, such as a patient. The specimen 105 can encompass a variety of biological materials, including cell-containing liquids or tissues. Examples include but are not limited to amniotic fluid, tissue biopsies, blood, blood cells, bone marrow, fine needle biopsy samples, peritoneal fluid, plasma, pleural fluid, saliva, semen, serum, tissue homogenates, and frozen or paraffin sections of tissue. Methods for acquiring these specimens 105 range from biofilms, aspirations, and tissue sections to swabs, blood draws, andsurgical or needle biopsies. The specimen 105 can originate from either a healthy subject, a subject afflicted with a non-cancer disease, or a subject afflicted with cancer.

[0104] As described herein, “patient,” and “subject” are used interchangeably and refer to any single animal, more preferably a mammal (including humans and non-human animals such as dogs, cats, horses, rabbits, rats, cows, pigs, sheep, and non-human primates). Thus, the methods described herein are applicable to both human and veterinary disease. In certain embodiments, subjects are “patients,” i.e., living humans that are receiving medical care for a disease or condition. This includes persons with no defined illness who are being investigated for signs of pathology. In some embodiments, the patient or subject may be at risk of / diagnosed with / being treated for cancer.

[0105] Cancer refers to an abnormal state or condition characterized by rapidly proliferating cell growth. Rapidly proliferating cells may be categorized as pathologic (e.g., characterizing or constituting a disease state), or may be categorized as non-patho logic (e.g., a deviation from normal but not associated with a disease state). In addition, cancer cells can spread locally or through the bloodstream and lymphatic system to other parts of the body. In general, a cancer will be associated with the presence of one or more tumors (e.g., abnormal cell masses). The term “tumor” is meant to include all types of cancerous growths or oncogenic processes, metastatic tissues or malignantly transformed cells, tissues, or organs, irrespective of histopathologic type or stage of invasiveness. Examples of cancer include malignancies of various organ systems, such as bladder cancers, lung cancers, breast cancers, thyroid cancers, lymphoid cancers, gastrointestinal cancers, and genito-urinary tract cancers. Cancer can also refer to adenocarcinomas, which include malignancies such as colon cancers, renal-cell carcinoma, prostate cancer and / or testicular tumors, non-small cell carcinoma of the lung, cancer of the small intestine, and cancer of the esophagus. Carcinomas are malignancies of epithelial or endocrine tissues including respiratory system carcinomas, gastrointestinal system carcinomas, genitourinary system carcinomas, testicular carcinomas, breast carcinomas, prostatic carcinomas, endocrine system carcinomas, and melanomas. An “adenocarcinoma” refers to a carcinoma derived from glandular tissue or in which the tumor cells form recognizable glandular structures. A “sarcoma” refers to a malignant tumor of mesenchymal derivation. “Melanoma” refers to atumor arising from a melanocyte. Melanomas occur most commonly in the skin and are frequently observed to metastasize widely.

[0106] Once the specimen 105 has been obtained, it enters a sample processing and image system 110. The sample processing and image system 110 comprises a fixation / embedding system 115, a sectioning system 120, a staining system 125, and an imaging system 130. The sample processing and image system 110 prepares the specimen 105 for staining procedures, such as histological staining, which are useful for highlighting features of interest and enhancing contrast in sectioned tissues or cells. Staining techniques can be employed to identify specific types of cells, nucleic acids, or proteins, thereby facilitating microscopic examination. The stained sample can then be analyzed to determine the quantity and characteristics of features of interest, such as cell count, density, expression levels, and spatial relationships among cellular structures.

[0107] A fixation / embedding system 115 fixes and / or embeds the specimen 105 (e.g., a sample including at least part of at least one tumor). Fixation is an irreversible chemical process that preserves the sample's chemical composition and cellular structure while preventing degradation. Fixatives, such as formaldehyde, neutral buffered formalin (NBF), paraformaldehyde (PF A), methanol, or Bouin’s solution, are used to harden tissues, inactivate degradative enzymes, and eliminate bacteria. The fixation process can involve perfusion or immersion of the tissue sample for a predetermined time. For liquid samples like blood, the sample may be smeared onto a slide and dried before fixation, potentially omitting the embedding and sectioning steps.

[0108] Embedding involves infiltrating the fixed tissue sample with a suitable histological wax, such as paraffin, or resins like styrene or polyethylene. Since histological wax is insoluble in water, the tissue must first be dehydrated using graded alcohols, followed by clearing with a solvent like xylene. The tissue is then embedded in warm paraffin wax, which fills the spaces previously occupied by xylene and water. Once cooled, the wax forms a hardened block that can be clamped into a microtome for precise sectioning. Deviations from this procedure can result in improper infiltration, hindering subsequent analyses.

[0109] The sectioning system 120 is used to slice the fixed and / or embedded tissue sample into thin sections using instruments such as a cryostat, microtome, vibratome, or compresstome. Sectioning produces a series of tissue slices, typically 4-5 microns thick, suitable for microscopysuch as light or fluorescent microscopy. In some cases, tissues can be rapidly frozen using dry ice or Isopentane and sectioned in a cryostat with a cold knife. Alternatively, other cooling agents like liquid nitrogen can be used. Sections intended for microscopy are generally between 4-10 micrometers thick, while those embedded in epoxy or acrylic resin can be thinner, less than 2 micrometers. The sections are then mounted on glass slides for further processing by the staining system 125. Additionally or alternatively, the sections may first be processed by the imaging system 130 to generate unstained bright field digital images of the section prior to being processed by the staining system 125.Staining Methods

[0110] Staining system 125 can implement various staining methods to enhance contrast and visualize specific cellular components. In some instances, the staining is performed manually. Staining can include exposing an individual section of the tissue to one or more different stains (e.g., consecutively or concurrently) to express different characteristics of the tissue. For example, each section may be exposed to a predefined volume of a staining agent for a predefined period of time. In other instances, the staining is performed semi-automatically or automatically by Stainers. The Stainers can be programmed with specific protocols for various types of staining, such as Hematoxylin and Eosin (H&E), immunohistochemistry (IHC), and special stains. The Stainers control parameters such as reagent volumes, incubation times, and temperatures, ensuring reproducibility and reducing human error.

[0111] Many staining solutions are aqueous. Thus, to stain tissue sections, the embedding agent (e.g., wax) may need to be dissolved and replaced with water (rehydration) before a staining solution is applied to a section (e.g., deparaffinization). For example, the tissue section may be sequentially passed through xylene, decreasing concentration of ethyl alcohol (from about 100% to 0%), and water. Once stained, the tissue sections may be dehydrated again and placed in xylene. The tissue section may then be mounted on microscope slides in a mounting medium dissolved in xylene. A coverslip may be placed on top to protect the sample section. The evaporation of xylene around the edges of the coverslip may dry the mounting medium and bond the coverslip firmly to the slide.

[0112] Various types of staining protocols may be used to perform the staining. One exemplary type of tissue staining is histochemical staining, which uses one or more chemicaldyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of tissue morphology and / or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is H&E, which differentiates cellular structures by staining nuclei blue and cytoplasm pink. Additional special stains or immunohistochemical markers may be used to highlight specific proteins or cellular components. Other examples of histochemical stains include trichrome stains (e.g., Masson’s Trichrome), which are used to differentiate between collagen and muscle fibers in tissue sections, Periodic Acid-Schiff (PAS), which is used to detect polysaccharides, muco-substances, and glycoproteins in tissues, silver stains, such as Gomori’s Methenamine Silver (GMS) and Reticulin stains, which are used to highlight certain types of tissue fibers and microorganisms, and iron stains, such as the Prussian Blue stain, which are used to detect ferric iron (Fe3+) in tissues.

[0113] Another type of tissue staining is immunohistochemistry (IHC), also called “immunostaining”), which is used to visualize specific proteins or antigens within tissue sections by leveraging the principles of antibody-antigen interactions. The process begins with the fixation of tissue samples to preserve cellular structures, followed by embedding in paraffin and sectioning into thin slices placed on glass slides. These sections are then exposed to primary antibodies that specifically bind to the target antigens. A secondary antibody, conjugated to a detectable marker such as an enzyme or a fluorescent dye, is applied to bind the primary antibody. In chromogenic IHC, the enzyme catalyzes a reaction that produces a colored precipitate at the antigen site, making it visible under a light microscope. In fluorescent IHC, the fluorescent dye emits light when excited by a specific wavelength, allowing visualization under a fluorescence microscope. IHC staining is widely used in pathology to diagnose diseases, identify specific cell types, and study the distribution and localization of proteins within tissues, providing critical insights into cellular functions and disease mechanisms.Imaging and Digitization

[0114] Once the stained and / or unstained tissue sections are prepared, the next step involves digitizing these slides to create whole slide images. In some instances, each tissue section may be mounted on a slide, which is then scanned by a whole slide scanner, a sophisticated piece of equipment equipped with high-resolution optics and advanced sensors to create a digital image.Imaging system 130 comprises a whole slide scanner and may scan the slide tile by tile keeping the image in focus to ensure the same resolution across all tiles. After, scanned slide tiles are stitched together using post-processing tools within the imaging system 130 to generate a WSI. The generated WSI corresponds to the type of staining method applied to the slide (e.g., H&E, fluorescence, bright field, and the like). Imaging system 130 can also scan the slide at different magnifications (e.g., no magnification, 4x magnification, lOx magnification, 20x magnification, and / or 40x magnification, etc.). Higher magnification allows for detailed cellular- level information to be captured, while lower magnification captures less detailed but broader structural features covering a larger area. Accordingly, the scanning process is precise enough to ensure that even the smallest details are accurately captured without introducing artifacts or distortions. The imaging system 130 may also be configured to detect and account for preparation artifacts, imaging artifacts, or background regions in the resulting images. The resulting digital images are typically large (e.g., 10 billion plus pixels for a 40x WSI that is several GB in size), high-resolution files stored in specialized formats, such as TIFF or proprietary formats designed for pathology images. In some instances, the WSI may be stored in unsigned integer numbers (uint) in 8-bit RBG format. In other instances, the WSI may be stored as a pyramidal image (e.g., .svs images), where the same image is stored in different magnifications. These digital files are then managed by imaging system 130, which further provides robust data management and archival solutions (e.g., storage device 140) to handle the large file sizes and ensure the integrity and accessibility of the data.

[0115] The scanner is equipped with a scanning mechanism, a light source, optical system, and a camera. The scanning mechanism moves the slide or the camera in a systematic pattern, capturing overlapping fields of view (FOVs) across the entire tissue section. This process is similar to stitching together multiple photographs to create a panoramic image. The scanner must ensure accurate alignment and focus for each FOV to avoid any blurring or distortion. The light source is a consistent and bright light source, such as an LED, which is used to illuminate the tissue section. The light generated by the light source passes through the tissue as it is moved by the scanning mechanism and is captured by the optical system. The optical system is high-resolution optics, including objective lenses with varying magnification levels (commonly 20x and 40x). The choice of magnification depends on the level of detail required for the analysis. In embodiments directed to fluorescent stains, the scanner is equipped with high-resolution opticsand filters specific to the excitation and emission wavelengths of the fluorophores used. The light source is used to excite the fluorophores. A high-resolution digital camera such as CCD or CMOS sensors is used to capture the illuminated tissue image and / or fluorescence from the tissue section. These cameras are often capable of capturing images at a resolution of several gigapixels, ensuring that even the smallest cellular details are accurately recorded. The individual FOVs are stitched together to form a continuous, high-resolution WSI for each color and / or fluorescence channel. In some instance, patches, or smaller sections of the WSI, are extracted from defined regions of interest (ROIs). This extraction process involves selecting a specific region within the WSI and cropping it to create a smaller image file. The size of the patches can vary depending on the level of detail needed for analysis. Common patch sizes might range from a few hundred pixels to several thousand pixels in width and height.

[0116] The WSI and / or patches (digital images 135) are then stored in the storage device 140, such as a server, a database, or a data repository. The digital images 135 may be stored locally, remotely, and / or in a cloud server. Each digital image 135 may be stored in association with an identifier of a subject and a date (e.g., a date when a sample was collected and / or a date when the image was captured).Analysis

[0117] During and / or after image acquisition, the digital images 135 may be processed and / or transmitted to another system for processing and analysis (e.g., analysis system 145 associated with a pathologist, an automated or semi-automated image analysis system, or a machine learning training and deployment system, as described in further detail herein). The analysis system 145 may include algorithms such as UPath, QUPath, Visiopharm, or InForm used to separate the overlapping color or fluorescence signals from different floors or fluorophores. This process deconvolves the composite image into individual channels, each representing a specific color or fluorophore. The analysis system 145 may further include software tools, algorithms, and models such as the machine learning models described herein to analyze the images. These tools, algorithms, and models offer functionalities such as: object detection for identifying specific cellular or subcellular structures, segmentation for partitioning the image into meaningful regions, such as delineating different tissue components, and classification for categorizing detected objects based on various characteristics such as their fluorescence intensityand spatial relationships. Analysis may also include detection and classification of artifacts or background regions, which may be excluded from downstream analysis or flagged for quality review. The software tools can also be used to automatically annotate regions of interest (ROIs) based on predefined criteria, and or facilitate pathologists review, annotation, and / or verification of automated annotations to ensure accuracy and clinical relevance. The images and analysis results can be easily shared with other healthcare professionals for multidisciplinary review and decision-making.IV. TRAINING AND USING A MACHINE LEARNING MODEL WITH DIGITAL IMAGES

[0118] FIG. 2 shows a block diagram of a machine learning pipeline 200 comprising several subsystems that work together to train, validate, and implement one or more machine learning models in accordance with various embodiments. The machine learning pipeline 200 may be executed as part of the analysis system 145 of the system 100 described in FIG. 1. The machine learning pipeline 200 comprises a data subsystem 205 for collecting, generating, preprocessing, and labeling of training and validation datasets 210, training and validation subsystem 215 that facilitates the training and validation of one or more machine learning algorithms 220, and inference subsystem 225 for deploying and implementing one or more trained machine learning models 230 independently or in combination with one or more other systems or services 235 for downstream processes.

[0119] As used herein, machine learning algorithms (also described herein as simply algorithm or algorithms) are procedures that are run on datasets (e.g., training and validation datasets) and perform pattern recognition on datasets, learn from the datasets, and / or are fit on the datasets. Examples of machine learning algorithms include linear and logistic regression, decision trees, artificial neural networks, k-means, and k-nearest neighbor. In contrast, machine learning models (also described herein as simply model or models) are the output of the machine learning algorithms and are comprised of model data and a prediction algorithm. In other words, the machine learning model is the program that is saved after running a machine learning algorithm on training data and represents the rules, numbers, and any other algorithm-specific data structures required to make inferences. For example, a linear regression algorithm may result in a model comprised of a vector of coefficients with specific values, a decision tree algorithm mayresult in a model comprised of a tree of if-then statements with specific values, or neural network, backpropagation, and gradient descent algorithms together result in a model comprised of a graph structure with vectors or matrices of weights with specific values.Data Subsystem

[0120] Data subsystem 205 is used to collect, generate, preprocess, and label data to be used to train and validate one or more machine learning algorithms 220. The data collection can include exploring various data sources such as public datasets, private data collections, or real-time data streams, depending on a project’s needs. In some instances, a data source is a public or online repository of information or examples pertinent to a general or target domain space. Many domains have publicly available datasets provided by governments, universities, or organizations. For example, many government and private entities offer datasets on healthcare, environmental data, and more through various portals. For proprietary needs, data might be available through partnerships or purchases from private companies that specialize in data aggregation. In other instances, a data source is a private repository of information or examples pertinent to a general or target domain space. For example, a data source can be the storage device 140 that stores digital WSI generated by system 100 described in FIG. 1. Once a data source is identified, data subsystem 205 can be used to collect data through appropriate methods such as downloading from online repositories, web scraping, using APIs for real-time data, creating datasets through surveys and experiments, or by running assays. The acquired raw data may be further preprocessed to generate the training and validation datasets 210.

[0121] In some instances, raw data may be generated as opposed to being collected or acquired. Data generating may comprise data synthesis and / or data augmentation. Different data synthesis and / or data augmentation techniques may be implemented by the data subsystem 205 to generate data to be used for the training and validation subsystem 215. Data synthesizing involves creating entirely new data points from scratch. This technique may be used when real data is insufficient, too sensitive to use, or when the cost and logistical barriers to obtaining more real data are too high. The synthesized data should be realistic enough to effectively train a machine learning model, but distinct enough to comply with regulations (e.g., copyright and data privacy), if necessary. Techniques such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) may be used to generate new data examples. These modelslearn the distribution of real data and attempt to produce new data examples that are statistically similar but not identical. Data augmentation, on the other hand, refers to techniques used to artificially expand the size of a dataset by creating modified versions of existing data examples. The primary goal of data augmentation is to increase variation in the data in order to make the model more robust to variations it might encounter in the real world, thereby improving its ability to generalize from the training data to unseen data. This is especially common in image and speech recognition tasks but is applicable to other data types as well. For images, data augmentation may include rotations, flipping, scaling, or altering the color / lighting conditions. For text, data augmentation may include synonyms replacement, back translation, or sentence shuffling. For audio, data augmentation may include changes made to pitch, speed, or background noise.

[0122] Preprocessing may be implemented using data subsystem 205, serving as a bridge between raw data acquisition and effective model training. The primary objective of preprocessing is to transform raw data into a format that is more suitable and efficient for analysis, ensuring that the data fed into machine learning algorithms is clean, consistent, and relevant. This step can be useful because raw data often comes with a variety of issues such as missing values, noise, irrelevant information, and inconsistencies that can significantly hinder the performance of a model. By standardizing and cleaning the data beforehand, preprocessing helps in enhancing the accuracy and efficiency of the subsequent analysis, making the data more representative of the underlying problem the model aims to solve.

[0123] An example of preprocessing, particularly for scanned WSI, is inter- WSI variability. Sources that can cause inter-WSI variability include the use of different scanners, different staining processes, tissue thickness, magnification, etc. These differences in experimental and digitization of WSIs can result in artifacts like blurring, folding, and color variation between WSIs having the same tissue or tumor type. Artifacts arising from slide preparation, staining, or imaging such as air bubbles, tissue folds, staining irregularities, or background are a common challenge in digital pathology datasets and may be present in both disease and non-disease regions. Although these artifacts are not associated with the presence of tumor tissue, a ML model could extract these variations as features critical for tumor classification. To overcome this, data subsystem 205 can apply techniques that correct color variations, differences in imagesizes / magnification, and / or signal normalization. To address stain variation or color variation, data subsystem 205 may apply grayscale conversion, color normalization, and / or color augmentation (application of random hue, saturation, brightness, and contrast) to the WSIs. Data subsystem 205 may apply a quality control filter that removes low quality WSIs (e.g., blurry, folded tissue) from the acquired images. Additionally or alternatively, a filter for magnification level may also be applied so that only high level resolution images are included in the final data set.

[0124] Raw data preprocessing may comprise generating small ‘tiles’ or ‘patches’ of an original WSI (see e.g., FIG. 12). Digitization of pathology slides produce WSIs reaching gigapixel sizes. For example, an image of a 20mm x 20mm sample of tissue scanned at 40X objective power may have 80,000 x 80,000 pixels. Further, if this image is in standard 24-bit color, the flattened version of the image would be about 19.2GB in size. Digital images of such size require very lengthy computational run times and multiple rounds of processing.Accordingly, data subsystem 205 may preprocess the WSI into ‘tiles’ or ‘patches’, which are small regions corresponding to a portion of a whole image, or a whole slide. The patches may be of any size appropriate to train a machine learning model. For example, a tile or patch can be a region of a whole slide scan or a region of interest having appropriate pixel dimension (x,y) and resolution quality (e.g., no magnification, 4x magnification, lOx magnification, 20x magnification, and / or 40x magnification, etc.). Moreover, the degree of overlap allowed between patches may also be controlled for. In various embodiments, the size of the patches, resolution quality, and overlap may be adjustable parameters of the model and dependent on the type of sample (cell, tissue, fluid) present on the slide. In addition to patch generation, manual or computational tools (e.g., segmentation tools) may also be applied to the WSI to distinguish between tissue patches and non-tissue / background patches. Manual removal of background can include outlining the tissue or applying a box around the tissue that minimizes the amount of background. More advanced tools, like segmentations, may apply filters and thresholds to the WSI, where any patch below the threshold (chosen arbitrarily) is removed. In so doing, the amount of computational power needed during training and / or inference is reduced as only patches with sample on them are used for analysis. In some instances, data subsystem 205 may save the patches as individual image files or as coordinates of each individual patch with respect to the original image.

[0125] Other raw data preprocessing techniques include data cleaning, normalization, feature extraction, dimensionality reduction, and the like. Data cleaning may involve removing duplicates, filling in missing values, or filtering out outliers to improve data quality.Normalization involves scaling numeric values to a common scale without distorting differences in the ranges of values, which helps prevent biases in the model due to the inherent scale of features. Feature extraction involves transforming the input data into a set of useable features, possibly reducing the dimensionality of the data in the process. For instance, in image analysis, feature reduction techniques such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), t-Distributed Stochastic Neighbor Embedding (t-SNE), autoencoders, and feature selection can be used for simplifying images, improving model performance, and gaining insights into the underlying structure of the images. These techniques not only help in reducing the computational load on the model but also in mitigating issues like overfitting by simplifying the data without losing critical information.

[0126] In the instance that machine learning pipeline 200 is used for supervised or semisupervised learning of machine learning models, labeling techniques can be implemented as part of the data collection. The quality and accuracy of data labeling directly influence the model's performance, as labels serve as the definitive guide that the model uses to learn the relationships between the input features and the desired output. Particularly in complex domains such as image recognition, natural language processing, or medical diagnosis, precise and consistent labeling is important because it provides the ground truth or target outcomes against which the model's predictions are compared and adjusted during training. Effective labeling ensures that the model is trained on correct and clear examples, thus enhancing its ability to generalize from the training data to real-world scenarios.

[0127] In some instances, the ground truth values (labels) are provided within the raw data. For example, when the raw data includes images, the labels may be at the global (i.e., whole slide) level or at the region-specific level. Examples of global level ground truth labels may include cancer presence, cancer types, pathological conditions, presence of infectious agents, tissues or cell types, grading or staining types, presence or not of histological features, treatment response, pathological scores, normal, benign, likely benign, malignant, likely pathogenic or pathogenic, and the like, where no further annotations indicating where the regions of interest are located. Onthe other hand, region-specific ground truth labeling may include regions of image processing artifacts, structure and / or anatomical objects, histological features, biomarkers expression, cancer cells, non-cancerous cells, tumor margin, tumor core, surrounding stroma, blood vessels, lymph nodes, mitotic figures, nuclei, cytoplasm, and extracellular matrix. Labeling may also be performed for non-disease-associated features such as artifacts, background, or normal tissue, supporting quality control and artifact detection applications. In various embodiments, the data includes WSIs comprising global level ground truth labels that are provided by an expert pathologist.

[0128] Labeling techniques can vary significantly depending on the type of data and the specific requirements of the project. Manual labeling, where human annotators label the data, is one method that can be used. This approach is useful when a detailed understanding and judgment are required, such as in labeling medical images or categorizing text data where context and subtlety are important. However, manual labeling is time-consuming and prone to inconsistency, especially with many annotators. To mitigate this, semi-automated labeling tools may be used as part of data subsystem 205 to pre-label data using algorithms, which human annotators may then review and correct as needed. Another approach is active learning, a technique where the model being developed is used to label new data iteratively. The model suggests labels for new data points, and human annotators may review and adjust certain predictions such as the most uncertain predictions. This technique optimizes the labeling effort by focusing human resources on a subset of the data, e.g., the most ambiguous cases, improving efficiency and label quality through continuous refinement.

[0129] Once collected, generated, preprocessed, and / or labeled, the data may then be split into the training and validation datasets 210. The training and validation datasets 210 may comprise the raw data and / or the preprocessed data. The training and validation datasets 210 are typically split into at least three subsets of data: training, validation, and testing. The training set is used to fit the model, where the machine learning model learns to make inferences based on the training data. The validation set, on the other hand, is utilized to tune hyperparameters and prevent overfitting by providing a sandbox for model selection. Finally, the test set serves as a new and unseen dataset for the model, used to simulate real-world application and evaluate the final model’s performance. The process of splitting ensures that the model can perform well not juston the data it was trained on, but also on new, unseen data, thereby validating and testing its ability to generalize.

[0130] Various techniques can be employed to split the data effectively, with each method aiming to maintain a good representation of the overall dataset in each subset. A simple random split (e.g., a 70 / 20 / 10%, 80 / 10 / 10%, or 60 / 25 / 15%) is the most straightforward approach, where examples from the data are randomly assigned to each of the three sets. However, more sophisticated methods may be necessary to preserve the underlying distribution of data. For instance, stratified sampling may be used to ensure that each split reflects the overall distribution of a specific variable, particularly useful in cases where certain categories or outcomes are underrepresented. Another technique, k-fold cross-validation, involves rotating the validation set across different subsets of the data, maximizing the use of available data for training while still holding out portions for validation. These methods help in achieving more robust and reliable model evaluation and are useful in the development of predictive models that perform consistently across varied datasets.

[0131] Data subsystem 205 is also used for collecting, generating, setting, or implementing model hyperparameters 240 for the training and validation subsystem 215. The hyperparameters control the overall behavior of the models. Unlike model parameters 245 that are learned automatically during training, hyperparameters 240 are set before training begins and have a significant impact on the performance of the model. For example, in a neural network, hyperparameters include the learning rate, number of layers, number of neurons / nodes per layer, activation functions, convolution kernel width, the number of kernels for a model, the number of graph connections to make during a lookback period, and the maximum depth of a tree in a random forest among others. These settings can determine how quickly a model learns, its capacity to generalize from training data to unseen data, and its overall complexity. Correctly setting hyperparameters is important because inappropriate values can lead to models that underfit or overfit the data. Underfitting occurs when a model is too simple to learn the underlying pattern of the data, and overfitting happens when a model is too complex, learning the noise in the training data as if it were signal.Training, Validation, and Testing

[0132] The training and validation subsystem 215 is comprised of a combination of specialized hardware and software to efficiently handle the computational demands required for training, validating, and testing a machine learning model. On the hardware side, high-performance GPUs (Graphics Processing Units) may be used for their ability to perform parallel processing, drastically speeding up the training of complex models, especially deep learning networks. CPUs (Central Processing Units), while generally slower for this task, may also be used for less complex model training or when parallel processing is less critical. TPUs (Tensor Processing Units), designed specifically for tensor calculations, provide another level of optimization for machine learning tasks. On the software side, a variety of frameworks and libraries are utilized, including TensorFlow, PyTorch, Keras, and scikit-leam. These tools offer comprehensive libraries and functions that facilitate the design, training, validation, and testing of a wide range of machine learning models across different computing platforms, whether local machines, cloud-based systems, or hybrid setups, enabling developers to focus more on model architecture and less on underlying computational details.

[0133] Training is the initial phase of developing machine learning models 230 where the model learns to make predictions or decisions based on training data provided from the training and validation datasets 210. During this phase, the model iteratively adjusts its internal model parameters 245 to achieve a preset optimization condition. In a supervised machine learning training process, the preset optimization condition can be achieved by minimizing the difference between the model output (e.g., predictions, classifications, or decisions) and the ground truth labels in the training data. In some instances, the preset optimization condition can be achieved when the preset fixed number of iterations or epochs (full passes through the training dataset) is reached. In some instances, the preset optimization condition is achieved when the performance on the validation dataset stops improving or starts to degrade. In some instances, the preset optimization condition is achieved when a convergence criterion is met, such as when the change in the model parameters falls below a certain threshold between iterations. This process, known as fitting, is fundamental because it directly influences the accuracy and effectiveness of the model.

[0134] In an exemplary training phase performed by the training and validation subsystem 215, the training subset of data is input into the machine learning algorithms 220 to find a set of model parameters 245 (e.g., weights, coefficients, trees, feature importance, and / or biases) that minimizes or maximizes an objective function (e.g., a loss function, a cost function, a contrastive loss function, a cross-entropy loss function, an Out-of-Bag (OOB) score, etc.). To train the machine learning algorithms 220 to achieve accurate predictions, “errors” (e.g., a difference between a predicted label and the ground truth label) need to be minimized. In order to minimize the errors, the model parameters can be configured to be incrementally updated by minimizing the objective function over the training phase ("optimization”). Various different techniques may be used to perform the optimization. For example, to train machine learning algorithms such as a neural network, optimization can be done using back propagation. The current error is typically propagated backwards to a previous layer, where it is used to modify the weights and bias in such a way that the error is minimized. The weights are modified using the optimization function. Other techniques such as random feedback, Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IF A), Hebbian learning, and the like can also be used to update the model parameters 245 in a manner as to minimize or maximize an objective function. This cycle is repeated until a desired state (e.g., a predetermined minimum value of the objective function) is reached.

[0135] The training phase is driven by three primary components: the model architecture (which defines the structure of the algorithm(s) 220), the training data (which provides the examples from which to learn), and the learning algorithm (which dictates how the model adjusts its model parameters). The goal is for the model to capture the underlying patterns of the data without memorizing specific examples, thus enabling it to perform well on new, unseen data.

[0136] The model architecture is the specific arrangement and structure of the various components and / or layers that make up a model. In the context of a neural network, the model architecture may include the configuration of layers in the neural network, such as the number of layers, the type of layers (e.g., convolutional, recurrent, fully connected), the number of neurons in each layer, and the connections between these layers. In the context of a random forest consisting of a collection of decision trees, the model architecture may include the configuration of features used by the decision trees, the voting scheme, and hyperparameters such as thenumber of trees in the forest, the maximum depth of each tree, the minimum number of samples required to split a node, and the maximum number of features to consider when looking for the best split. In some instances, the model architecture is configured to perform multiple tasks. For example, a first component of the model architecture may be configured to perform a feature selection function, and a second component of the model architecture may be configured to perform a feature scoring function. The different components may correspond to different algorithms or models, and the model architecture may be an ensemble of multiple components.

[0137] Model architecture also encompasses the choice and arrangement of features and algorithms used in various models, such as decision trees or linear regression. The architecture determines how input data is processed and transformed through various computational steps to produce the output. The model architecture directly influences the model's ability to learn from the data effectively and efficiently, and it impacts how well the model performs tasks such as classification, regression, or prediction, adapting to the specific complexities and nuances of the data it is designed to handle.

[0138] The model architecture can encompass a wide range of algorithms 220 suitable for different kinds of tasks and data types. Examples of algorithms 220 include, without limitation, linear regression, logistic regression, decision tree, Support Vector Machines, Naives Bayes algorithm, Bayesian classifier, linear classifier, K-Nearest Neighbors, K-Means, random forest, dimensionality reduction algorithms, grid search algorithm, genetic algorithm, AdaBoosting algorithm, Gradient Boosting Machines, and Artificial Neural Networks such as convolutional neural network (“CNN”), an inception neural network, a U-Net, a V-Net, a residual neural network (“Resnet”), a transformer neural network, a recurrent neural network, a Generative adversarial network (GAN), or other variants of Deep Neural Networks (“DNN”) (e.g., a multilabel n-binary DNN classifier or multi-class DNN classifier). These algorithms can be implemented using various machine learning libraries and frameworks such as TensorFlow, PyTorch, Keras, and scikit-leam, which provide extensive tools and features to facilitate model building, training, validation, and testing. For example, the classification algorithm described with respect to FIGS. 3A and 3B is a set of algorithms configured as the architecture of a neural network comprised of layers of nodes, or "neurons," which are designed to mimic the way a human brain operates.

[0139] The learning algorithm is the overall method or procedure used to adjust the model parameters 245 to fit the data. It dictates how the model learns from the data provided during training. This includes the steps or rules that the algorithm follows to process input data and make adjustments to the model's internal parameters (e.g., weights in neural networks) based on the output of the objective function. Examples of learning algorithms include gradient descent, backpropagation for neural networks, and splitting criteria in decision trees.

[0140] Various techniques may be employed by training and validation subsystem 215 to train machine learning models 230 using the learning algorithm, depending on the type of model and the specific task. For supervised learning models, where the training data includes both inputs and expected outputs (e.g., ground truth labels), gradient descent is a possible method. This technique iteratively adjusts the model parameters 245 to minimize or maximize an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.). The objective function is a method to measure how well the model’s predictions match the actual labels or outcomes in the training data. It quantifies the error between predicted values and true values and presents this error as a single real number. The goal of training is to minimize this error, indicating that the model's predictions are, on average, close to the true data. Common examples of loss functions include mean squared error for regression tasks and cross-entropy loss for classification tasks.

[0141] The adjustment of the model parameters 245 is performed by the optimization function or algorithm, which refers to the specific method used to minimize (or maximize) the objective function. The optimization function is the engine behind the learning algorithm, guiding how the model parameters 245 are adjusted during training. It determines the strategy to use when searching for the best weights that minimize (or maximize) the objective function. Gradient descent is a primary example of an optimization algorithm, including its variants like stochastic gradient descent (SGD), mini-batch gradient descent, and advanced versions like Adam or RMSprop, which provide different ways to adjust learning rates or take advantage of the momentum of changes. For example, in training a neural network, backpropagation may be used with gradient descent to update the weights of the network based on the error rate obtained in the previous epoch (cycle through the full training dataset). Another technique in supervised learning is the use of decision trees, where a tree-like model of decisions is built by splitting the trainingdataset into subsets based on an attribute value test. This process is repeated on each derived subset in a recursive manner called recursive partitioning.

[0142] In unsupervised learning, where training data does not include labels, different techniques are used. Clustering is one method where data is grouped into clusters that maximize the similarities of data within the same cluster and maximize the differences with data in other clusters. The K-Means algorithm, for example, assigns each data point to the nearest cluster by minimizing the sum of distances between data points and their respective cluster centroids. Another technique, Principal Component Analysis (PCA), involves reducing the dimensionality of data by transforming it into a new set of variables, the principal components, which are uncorrelated and ordered so that the first few retain most of the variation present in all of the original variables. These techniques help uncover hidden structures or patterns in the data, which can be essential for feature reduction, anomaly detection, or preparing data for further supervised learning tasks.

[0143] Validating is another phase of developing machine learning models 230 where the model is checked for deficiencies in performance and the hyperparameters 240 are optimized based on validation data provided from the training and validation datasets 210. The validation data helps to evaluate the model's performance, such as accuracy, precision, recall, or Fl -score, to gauge how well the training is ongoing, for example, by monitoring if an underfitting or overfitting is occurring. Hyperparameter optimization, on the other hand, involves adjusting the settings that govern the model's learning process (e.g., learning rate, number of layers, size of the layers in neural networks) to find the combination that yields the best performance on the validation data. One optimization technique is grid search, where a set of predefined hyperparameter values are systematically evaluated. The model is trained with each combination of these values, and the combination that produces the best performance on the validation set is chosen. Although thorough, grid search can be computationally expensive and impractical when the hyperparameter space is large. A more efficient alternative optimization technique is random search, which samples hyperparameter combinations from a defined distribution randomly. This approach can in some instances find a good combination of hyperparameter values faster than grid search. Advanced methods like Bayesian optimization, genetic algorithms, and gradientbased optimization may also be used to find optimal hyperparameters more effectively. Thesetechniques model the hyperparameter space and use statistical methods to intelligently explore the space, seeking hyperparameters that yield improvements in model performance.

[0144] An exemplary validation process includes iterative operations of inputting the validation subset of data into the trained algorithm(s) using a validation technique such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, Nested Cross-Validation, or the like, to fine-tune the hyperparameters and ultimately find the optimal set of hyperparameters. In some instances, a 5-fold cross-validation technique may be used to avoid overfitting the trained algorithm and / or to limit the number of selected features per split to the square-root of the total number of input features. In some instances, training dataset is split into 5 equal-size cohorts (or about equal-size), and every four of the cohorts are used to train an algorithm to generate five models (e.g., cohorts #1, 2, 3, and 4 are used to train and generate model 1, cohorts #1, 2, 3, and 5 are used to train and generate model 2, cohorts #1, 2, 4, and 5 are used to train and generate model 3, cohorts #1, 3, 4, and 5 are used to train and generate model 4, and cohorts #2, 3, 4 and 5 are used to train and generate model 5). Each model is evaluated (or validated) using the unused cohort in the training (e.g., for model 5, cohort #1 is used for validation). The overall performance of the training can be evaluated by an average performance of the five models. K-fold cross-validation provides a more robust estimate of a model’s performance compared to a single training / validation split because it utilizes the entire dataset for both training and evaluation and reduces the variance in the performance estimate.

[0145] Once a machine learning model has been trained and validated, it undergoes a final evaluation using test data provided from the training and validation datasets 210, which is a separate subset of the data that has not been used during the training or validation phases. This step is crucial as it provides an unbiased assessment of the model's performance in simulating real- world operation. The test dataset serves as new, unseen data for the model, mimicking how the model would perform when deployed in actual use. During testing, the model’s predictions are compared against the true values in the test dataset using various performance metrics such as accuracy, precision, recall, Fl, AUC, and mean squared error, depending on the nature of the problem (classification or regression). This process helps to verify the generalizability of the model — its ability to perform well across different data samples and environments — highlighting potential issues like overfitting or underfitting and ensuring that the model is robust and reliablefor practical applications. The machine learning models 230 are fully validated and tested once the output predictions have been deemed acceptable by user defined acceptance parameters. Acceptance parameters may be determined using correlation techniques such as Bland- Altman method and the Spearman’s rank correlation coefficients and calculating performance metrics such as the error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.In ference Phase for Machine Learning Models

[0146] The inference subsystem 225 is comprised of various components for deploying the machine learning models 230 in a production environment (e.g., use in a laboratory assay as described with respect to FIG. 1). Deploying the machine learning models 230 includes moving the models from a development environment (e.g., the training and validation subsystem 215, where it has been trained, validated, and tested), into a production environment where it can make inferences on real-world data (e.g., input data 250). This step typically starts with the model being saved after training, including its parameters and configuration such as final architecture and hyperparameters. It is then converted, if necessary, into a format that is suitable for deployment, depending on the deployment environment. For instance, a model trained in a scientific computing environment such as Python might be converted into a Java- friendly format for integration into a larger enterprise application. Deployment can be conducted on various platforms, including on-premises servers, cloud environments like AWS, Azure, Google.

[0147] Once deployed, the model is ready to receive input data 250 and return outputs (e.g., inferences 255). In some instances, the model resides as a component of a larger system or service (e.g., including additional downstream applications). In some instances, the models 230 and / or the inferences 255 can be used by the downstream applications to provide further information. For example, the inferences 255 can be used to aid qualified personnel (e.g., oncologists) to help diagnosis and / or determine whether treatment should be administered to a patient. In some instances, the inferences 255 can be used to aid qualified personnel to determine a specific type of treatment to administer to a patient based on the inference results. The downstream applications can be configured to generate an output 260. In some instances, the output 260 comprises a report including inferences 255 and information generated by the downstream applications. In some instances, the outputs may include visualizations, such asheatmaps or overlays, indicating patch-level predictions for disease-associated or non-disease-associated features, including artifacts, to support clinical review and quality assurance.

[0148] In an exemplary inference subsystem 225, the input data 250 includes images generated from one or more biological samples from a patient having been diagnosed a disease (e.g., cancer). The images may be generated by performing whole slide image processing (independently or as part of a digital pathology assay) on the one or more biological samples collected from the patient as described with respect to FIG. 1. The one or more biological samples may be a single or multiple tissue samples (e.g., a bladder tissue, lung tissue, tumor section, etc.) obtained from the patient. In some instances, the input data may comprise slides or regions for which the primary task is quality assurance, artifact detection, or classification of non-disease-associated features.

[0149] To manage and maintain its performance, a deployed model may be continuously monitored to ensure it performs as expected over time. This involves tracking the model’s prediction accuracy, response times, and other operational metrics. Additionally, the model may require retraining or updates based on new data or changing conditions in the environment in which it is applied. This can be useful because machine learning models can drift over time due to changes in the underlying data they are making predictions on — a phenomenon known as model drift. Therefore, maintaining a machine learning model in a production environment often involves setting up mechanisms for performance monitoring, regular evaluations against new test data, and potentially periodic updates and retraining of the model to ensure it remains effective and accurate in making predictions.V. CONVOLUTION NEURAL NETWORKS

[0150] FIGS. 3A and 3B show an exemplary CNN 300 that can be trained and deployed as described with respect to FIG. 2. In some instances, CNN 300 is a feedforward neural network where data moves from the input layer through a set of hidden layers in one direction (e.g., having no cycles contained within it). However, it should be understood that other types of neural networks (e.g., ResNet), such as those with a cyclic architecture, could be implemented without departing from the spirit and scope of the present disclosure. CNN 300 comprises a deep learning set of algorithms that can take as input an image (e.g., a digital pathology image orpatches from a digital pathology image generated from system 100), assigns importance (learnable weights and biases) to various features in the image, and differentiates one aspect from the other to perform a given task such as object classification. CNNs are typically used in digital pathology to automate and enhance the accuracy of image classification, segmentation, and feature extraction in medical images, aiding in tasks such as tumor detection, disease diagnosis, and cell counting. Additionally, CNNs can be used to identify and classify nondisease-associated features, such as artifacts (including air bubbles, tissue folds, staining irregularities, and debris) and normal tissue structures, supporting both diagnostic and quality assurance workflows. CNNs facilitate prognostic modeling, whole slide image analysis, and anomaly detection, thereby supporting pathologists in diagnosing diseases and planning treatments more efficiently.Architecture of CNNs

[0151] FIG. 3A illustrates the various layers of an exemplary CNN 300, while FIG. 3B illustrates processing of an exemplary image by the CNN 300 using the various layers. As shown in FIG. 3 A, the CNN 300 comprises an input layer 305, convolution-pooling layers 310, and classification-output layers 315. The input layer 305 of a CNN is the initial stage where input data (e.g., raw image data and / or preprocessed image data) are fed into the network. In some instances, the input data is digital WSls of a specimen (may be generated by system 100 as described with respect to FIG. 1) with a global label of either “non-tumor” or “disease” (e.g., cancer). In other instances, the input data is a set of patches generated from an original WS1 (may be generated by data subsystem 205, described with respect to FIG. 2) with a global label of either “non-tumor” or “disease” (e.g., cancer). When labeled as “disease,” the specific regions indicative of said disease are not indicated. In other words, there are no annotations at the region / sample level to indicate where in the WS1 or patch the disease is located and / or what features of the disease are present. Additionally, the input data may comprise one or more magnifications of the digital image. For example, an input WS1 or patch may comprise no magnification, 4x magnification, lOx magnification, 20x magnification, and / or 40x magnification, etc. Prior to being input into CNN 300, the raw image data comprising a set of patches from a globally labeled WS1 are converted into tensors (e.g., multi-dimensional array). For instance, a color image with dimensions 32x32 pixels and multiple color channels (red,green, and blue) would be represented as a 32x32x3 tensor. Each element in this tensor corresponds to the pixel intensity value in the image, with separate values for each color channel. As such, CNN 300 receives the raw image data as a tensor so that the raw image can be efficiently processed by the subsequent layers of the CNN.

[0152] While the input layer 305 itself does not perform complex computations; its role is important in ensuring that the data is formatted correctly for the next layers. The subsequent layers, such as the convolution-pooling layers 310, rely on this structured data to perform operations like convolution, which involves sliding a filter or kernel over the input tensor to detect features related to edges, textures, and patterns from the original image. The dimensionality and the specific arrangement of the input data are useful for these operations to be performed correctly. By accurately organizing the image data (e.g., into tensors), the input layer 305 sets the stage for the network to learn and extract meaningful features that are used for various tasks such as image classification, object detection, and image segmentation.

[0153] The multiple convolution-pooling layers 310 includes one or more convolution layers 320a-n (solid boxes) that perform convolution functions followed by pooling layers 325a-n (dashed boxes) that perform pooling functions. For both the convolution layers 320a-n and the pooling layers 325a-n, ‘n’ represents any natural number. The convolution and pooling layers work in tandem to progressively extract high-level features from the input tensor data received from the input layer. The convolutional layers are designed to apply a set of learnable filters or kernels to the input tensor data. Each filter slides over the entire tensor, performing an element-wise multiplication and summation to produce a feature map (e.g., numerical tensor). This process, known as a convolution operation, helps in detecting local patterns related to edges, textures, and shapes within the image. The filters in the initial convolutional layers capture basic features related to edges and comers. Thus, the resulting feature maps (e.g., numerical tensors) represent the features (e.g., edges and comers) extracted from original image. On the other hand, filters in deeper layers capture more complex patterns and structures, which produce more abstract numerical tensor representations. The convolution operation also helps in preserving the spatial relationships between pixels, which is crucial for understanding the content of the image.

[0154] After the convolution operation is performed, resulting in feature maps, an activation function is applied to introduce non-linearity into the model. This non-linearity is useful becauseit enables the network to learn and represent complex patterns and relationships in the data that linear operations alone cannot capture. Examples of activation functions that may be used include the linear activation function, the rectifier linear unit (ReLU), the leaky ReLU, the sigmoid activation function, the SoftMax activation function, the hyperbolic tangent activation function, and the exponential activation function. In certain instances, the activation function used for each convolutional layer is the ReLU, which applies the function (f(x) = max(0, x)) to each element of the feature map. This means that any negative values in the feature map are set to zero, while positive values remain unchanged. In some instances, the activation function is the same across all convolution layers. In other instances, the activation function is varied across the convolution layers (e.g., one or more convolutional layers use ReLU and another one or more convolutional layers use leaky ReLU).

[0155] The application of the activation function occurs elementwise across the entire feature map produced by the convolution operation. By introducing non-linearity at each convolutional layer, the network can model more complex functions and is better equipped to handle diverse and intricate data. This step is useful because, without it, the model would essentially be equivalent to a single linear transformation, regardless of the number of layers, severely limiting its capacity to solve complex tasks. Therefore, by applying activation functions after each convolutional operation, CNNs can develop a more nuanced and powerful representation of the input data, leading to more accurate and robust performance in tasks such as image recognition and classification.

[0156] The convolution process generates a large amount of data that needs to be compressed by its corresponding pooling layer (e.g., convolution output 320a is compressed by pooling layer 325a) to reduce the spatial dimensions of the feature maps, which serves several purposes. Pooling operations, such as max pooling or average pooling, down sample the features in the feature maps. For instance, max pooling takes the maximum value from the feature map of a patch, effectively reducing the size while retaining the most significant information. This not only reduces the computational load and memory requirements but also adds a degree of translational invariance, meaning the network becomes more robust to slight translations of the input image. By alternating between convolutional and pooling layers, the CNN hierarchically builds a rich and abstract representation of the input image, enabling the final fully connectedlayers to perform high-level tasks such as classification with improved accuracy. The repeated application of convolution and pooling operations allows the network to learn and recognize increasingly complex features at different levels of abstraction, which allows for the effective processing and understanding of image data.

[0157] After the convolution-pooling layers 310, the data flows into the classification-output layers 315, which are configured to make final predictions based on the extracted features (feature maps). The classification-output layers 315 comprise three main components: a flattened vector 330, fully connected layers 335, and an output layer 340 that provides the predicted classifications. The feature maps obtained from the final pooling layer are multi-dimensional arrays (e.g., numerical tensors). To transition from these spatially structured feature maps to a format suitable for the fully connected layers 335, the feature maps are flattened into a single one-dimensional vector (e.g., flattened vector 330). This process involves taking the multidimensional array of features and converting it into a long vector that maintains the same numerical values but without any spatial dimensions. Flattening is useful because the fully connected layers 335 expect a one-dimensional input, and this transformation enables the network to leverage all the extracted features in a linear format.

[0158] Once the feature maps are flattened, the resulting outputs (e.g., flattened vector 330) are passed through the fully connected layers 335. Each neuron in a fully connected layer is connected to every neuron in the previous layer, allowing for a comprehensive integration of all the features. These layers perform linear transformations followed by the application of activation functions (such as ReLU or sigmoid), which introduce non-linearity and help in learning complex patterns. The fully connected layers 335 serve to process the features extracted by the convolution-pooling layers 310 in various ways to produce higher-level representations. These representations are useful as they consolidate all the information gathered from the earlier stages of the network and are tailored towards the specific task of classification. In some instances, the activation function(s) used in the fully connected layers 335 are the same or different than the activation function(s) used in the convolution layers 320a-n of the convolutionpooling layers 310.

[0159] The final layer of the classification-output layers 315 is the output layer 340, which provides the predicted classifications. In many classification tasks, this is typically a SoftMaxlayer, especially when dealing with multi-class classification problems. The SoftMax function converts the output scores from the final fully connected layer into positive values that sum to one, providing a probabilistic interpretation of the predictions. Each output node in this layer corresponds to a different class label, and the node with the highest probability is often taken as the network's prediction. In other scenarios, such as binary classification, a sigmoid function might be used instead. The output layer 340 thus translates the learned features and patterns into a final prediction, completing the process of image recognition or classification within the CNN. FIG. 3 A shows an exemplary CNN 300 with a prediction problem that outputs at least three different predictions or classifications through the output layer 340; however, this example is not meant to be limiting and other prediction problems where only one prediction is made may also be considered.

[0160] FIG. 3B illustrates processing of an exemplary image 350 by the CNN 300 using the various layers. As shown, initially the image 350 is obtained (e.g., obtained or acquired as discussed with respect to FIGS. 1 and 2). The image 350 has an image representation that can be thought of as a grid of pixels when dealing with 2D images or as a grid of voxels for 3D images. A 2D image is represented as a matrix of pixels, where each pixel has an intensity value. A 3D image, such as those used in medical imaging, is represented as a grid of voxels (volume elements), similar to pixels but in three dimensions. When dealing with images that have multiple channels for different colors, such as RGB images, the input image representation changes to account for the additional depth dimension. For example, a grayscale image has a single intensity value per pixel, while a color image (like an RGB image) has multiple intensity values (e.g., 3 values for RGB image corresponding to the red, green, and blue color channels) per pixel. Several color channel options may be used in image processing such as, without limitation, Red-Green-Blue (RGB), Hue Saturation Value (HSV), cyan, magenta, yellow and key (black) (CMYK), grayscale, etc.

[0161] WSIs are often very large in size. For example, a 20mm x 20mm sample of tissue may have 80,000 x 80,000 pixels and two-color channels. Accordingly, the WSI would yield 80,000 x 80,000 x 2 = 1.28E10 inputs for initial convolution. The training and use of a model (e.g., CNN) using such a large image would require an excessive amount of computational power and storage space. As such, the original WSI is often tiled into patches to reduce the computational demand.For example, the input data to a CNN may be a set of patches corresponding to the original WSI. The set of patches may be of dimensions 299 x 299 x 2 = 178,802 pixels, which is comparatively much more manageable in terms of computational power and memory required. Another tool used by CNNs to overcome the computational burden of larger input is that CNNs use parameter sharing, where nodes in a particular feature map share weights to make the computation less consuming.

[0162] The image 350 is fed into the input layer 305 and CNN 300 applies a series of functions to the image 350 using multiple hidden layers 355a-n. The hidden layers 355a-n comprise the convolution layers 320a-n, the pooling layers 325a-n, activation layers (e.g., ReLU after each convolution layer and first fully connected layer), and fully connected layers 335. The number of hidden layers 355a-n and output layer 340 define the depth of CNN 300. By having multiple hidden layers 355a-n, CNN 300 can compute complex functions by cascading simpler functions. The hidden layers 355a-n are represented as a directed acyclic graph (DAG) whose nodes 360 correspond to neurons and whose edges correspond to the links or connections 365 between them. Every node 360 within the hidden layers 355a-n is connected (via edges) to one or more other nodes 360 in the previous layer and in the next layer. This connection circuit allows for each node 360 to receive information from the layer beneath it, process that information, and send the processed information to the next layer. The connections 365 between nodes 360 have a weight associated with them. The weight represents the strength of the connection 365 and is a weighting factor (model parameter) that reflects the importance of the data (e.g., features) associated with the image 350 that CNN 300 learns during training, as described in more detail herein.

[0163] The number of nodes 360 used in hidden layers 355a-n can affect the model's overall fitting. For example, too few nodes can lead to underfitting the model and high statistical bias, while too many nodes may lead to overfitting the model, high variance, and increased training time. An appropriate number of nodes 360 to include in the hidden layers 355a-n is dependent on the model architecture and options to determine node number may include but are not limited to (i) a number between the size (e.g., number of nodes) of the input layer and the output layer, (ii) about two-thirds the size of the input layer plus the size of the output layer, (iii) less than twice the size of the input layer, or (iv) equal to the mean of the size of the input and output layers.Other factors that can affect the number of nodes, and number of hidden layers, include the training cases, number of outliers, the complexity of the data that is to be learned, the type of activation function, or any combination thereof.

[0164] As shown in FIG. 3B, the nodes 360 in hidden layer 1 355(a) are each receiving multiple different inputs (e.g., three inputs) from the input layer 305 via connections 370, and thus have multiple incoming values. Each of those connections 370 has its own unique weight associated with it. Each node 360 in hidden layer 1 355(a) can sum up the individual weights from each connection 370 to generate a weighted sum value that the receiving node is assigned. Then, each receiving node may apply functions or operations using the weighted sum value to process the input data and generate a processed input (e.g., output) that can be passed along to a new receiving node in the next layer. Essentially, the processing of a node 360 may be implemented in three steps- (1) each node 360 calculates the weighted sum of its inputs and weights, (2) add bias to the weighted sum output, and (3) each node 360 applies functions or operations (e.g., activation functions) using the weighted sum value to produce the output. The nodes 360 in subsequent layers receive the output of nodes 360 in the previous layer as input. Whether or not an input is passed onto the nodes 360 in the next hidden layer may be dependent on the functions or operations used to produce the output. By way of example, a thresholding function may be used. If the value of a node is below the threshold value, the input may not be passed to the next layer, and if the value of a node is greater than or equal to the threshold, the input may be passed forward. In some cases, a connection may have zero weight, in which case the node 325 is considered to not have any influence on the task at hand and thus the output will not be passed through to a node in the next layer. In other instances, a threshold is not used, and the nodes 360 in subsequent layers receive output of nodes 360 in the previous layer as input regardless of the weighted sum value.

[0165] The hidden layers 355a-n include convolution layers, pooling layers, and fully connected layers. The convolution layers are responsible for recognizing features in the pixels or voxels of the image of the input layer 305. A convolution layer achieves this by applying one or more filters 380 to its input 375 to detect features from the input data (in the form of a matrix) into a feature map 385. The filters 380 are often referred to as kernels, or feature detectors, where each filter applied to the input data is specialized to detect certain features like edges, textures, orpatterns within the image of the input layer 305. Although each filter 380 is specialized to detect different features, it is possible for different filters to detect similar / overlapping features. The filter 380 will slide across the height and width of the input 375, based on a padding parameter and a striding parameter, and performs element-wise multiplication followed by summation, a process known as convolution operation. The padding parameter indicates how to expand the size of the input matrix by adding fake pixels to the border of the matrix, for example Os.Padding can help to reduce the loss of information at the borders of the input matrix (or input feature map). The striding parameter of the convolution operation refers to the number of pixels by which the filter 380 matrix moves across the input matrix. The padding and striding parameters are set either as random selection, defined by a user, or as a hyperparameter optimized during training. In the example shown in FIG. 3B, feature maps 385 are generated using a striding parameter value of 1. The filter will move across the entire width of the input matrix based on the striding parameter, then move down (also based on the striding parameter) to the next row of the image and repeat the process until the entire image has been traversed.

[0166] Pooling layers are interspersed between convolution layers and are configured to further process the feature maps 385. The objective of the pooling layer is to reduce the dimension of the output feature maps 385. Pooling may be done at a global level or a local level. Global level pooling involves combining the entire feature map matrix into a single value, whereas local pooling combines small clusters (e.g., pooling squares, typically of size 2x2). Multiple different pooling functions may be used for pooling, such as maximum pooling and average pooling. In maximum pooling, the largest value (e.g., the max value) for each feature map 385 (for global) or pooling square of the feature map 385 (for local) is used to generate the pooled feature map 390. For example, if the pooling square is a 2x2 square, the pooling function will take the largest value in that 2x2 square and insert it into the corresponding pooled feature map 390. For average pooling, instead of taking the largest value in the 2x2 square, the average of the values in the 2x2 square are taken and inserted into the corresponding pooled feature map 390. This process effectively down-samples the feature map, reducing its width and height but keeping the depth the same. The down-sampling helps in reducing the computational load and controlling overfitting by making the representation more compact and abstract.

[0167] As the image passes through successive convolution and pooling layers, the feature maps 385 become increasingly abstract and focused on higher-level features. Early convolution layers might detect simple features (e.g., edges and textures), while deeper layers combine these simple features to recognize more complex patterns like shapes or even objects. Pooling layers ensure that the spatial size of the feature maps is progressively reduced, which helps in maintaining computational efficiency and reducing the risk of overfitting. This hierarchical feature extraction allows the CNN 300 to effectively learn and generalize from the data, making it particularly well-suited for image recognition tasks. Ultimately, the series of convolution and pooling operations transform the original image into a compact and information-rich representation that can be fed into fully connected layers for classification or other tasks. Thus, the convolution and pooling layers work together to progressively distill the essential features of the image, enabling the CNN 300 to perform tasks such as object detection, image classification, and the like.

[0168] The fully connected layers in CNN 300 play a pivotal role in the network's final decision-making process. After the hierarchical feature extraction is performed by the convolutional and pooling layers, the resulting feature maps 385 are flattened into a single vector. This vector serves as the input to the fully connected layers. Each fully connected layer is comprised of a set of nodes (e.g., neurons) where each node is connected to all the nodes in the previous layer. These layers perform linear transformations followed by a non-linear activation function (like ReLU) to capture complex relationships between the features. The linear transformation involves multiplying the input vector by a weight matrix and adding a bias vector. The output of this transformation is then passed through the activation function to introduce nonlinearity, enabling the network to model complex patterns and interactions within the data.

[0169] The final output layer 395 of the CNN 300 is responsible for generating the final predictions. This layer is typically a fully connected layer where the number of nodes corresponds to the number of classes in a classification task. For instance, in an image classification task with ten possible classes, the output layer would have ten nodes. The output of this layer is usually passed through an activation function suitable for classification tasks, such as the SoftMax function. The SoftMax function converts the raw output scores (logits) into positive probabilities that sum to one, providing a probabilistic interpretation of the network's predictions.Each node in the output layer 395 represents the probability of the image of the input layer 305 belonging to a particular class. The class with the highest probability may be chosen as the network's final prediction. This combination of fully connected layers and the output layer 395 allows the CNN 300 to perform complex decision-making based on the features extracted from the image 350, ultimately enabling tasks such as image classification, object detection, and more.Training, Testing, and Validation ofCNNs

[0170] With regards to training a CNN, many of the techniques described for machine learning pipeline 200 with respect to FIG. 2 apply. For example, a data subsystem (like data subsystem 205) is used to collect, generate, preprocess, and label data to be used to train, test, and validate CNN 300. Data may be generated by system 100 (from FIG. 1) that generates and stores digital whole slide pathology images. Further, the WSI generated from system 100 can be processed by a data subsystem (described with respect to FIG. 2) to generate ‘tiles’ or ‘patches’ corresponding to portions of the WSI. In some embodiments, the WSI may include a global / slide level annotation indicating if the specimen in the WSI is non-tumor or diseased (e.g., cancerous). The global annotation may be provided by one or more pathologists specifically trained to detect the presence or absence of tumor tissue on a slide. In this scenario, the tiles / patches are unlabeled as annotations indicating regions of interest are not provided. The unlabeled patches are used to train CNN 300 to identify features within the patches to distinguish regions of non-tumor tissue from regions of diseased tissue. In other instances, the WSI is labeled by an algorithm (e.g., STAPLE) specifically designed to take as input, multiple pathologist annotations of digital pathology images (e.g., tissue) and output a probabilistic estimate of the true ground truth segmentation that may be used to label regions in the WSI that are non-tumor, diseased (e.g., cancerous), or undefined. As a result, the tiles / patches are labeled (e.g., non-cancerous or cancerous). In this scenario, CNN 300 can learn, and extract features specifically associated with non-tumor tissue. Or conversely, features specifically associated with diseased tissue, such as diagnostic biomarkers or molecular markers that are relevant to the diagnosis or classification of the disease, detailed markings of abnormalities, such as tumors, necrosis, fibrosis, or other disease-related changes, different cell types present in the tissue, such as cancerous cells, immune cells, etc., measurements of various structures, such as the size of tumors, length of blood vessels, or density of certain cell types, or any combinations thereof. Training may also beperformed for detection of non-disease-associated features, such as artifacts or normal tissue, using corresponding annotations where available, enabling the CNN to support quality assurance and artifact identification applications alongside disease detection.

[0171] Once all the data is collected, preprocessed and / or labelled, the data is split into training, validating, and testing datasets. Various techniques can be employed to split the data effectively, with each method aiming to maintain a good representation of the overall dataset in each subset. A simple random split (e.g., a 70 / 20 / 10%, 80 / 10 / 10%, or 60 / 25 / 15%) is the most straightforward approach; however, more sophisticated methods may be necessary to preserve the underlying distribution of data. For instance, stratified sampling may be used to ensure that each split reflects the overall distribution of a specific variable.

[0172] Hyperparameters for CNN 300 are also selected before training begins and have a significant impact on the performance of the model. For example, in a CNN, hyperparameters include the learning rate, number of layers, number of neurons / nodes per layer, activation functions, convolution kernel width, the number of kernels per convolutional layer, and the number of graph connections to make during a lookback period among others. These settings can determine how quickly a model learns, its capacity to generalize from training data to unseen data, and its overall complexity. In some instances when a CNN is initially trained, the hyperparameters may be assigned at random. In other instances, the hyperparameters may be explicitly defined to control different aspects of the networks such as memory or cost of execution. However, additional hyperparameters may be defined to adapt a network to a specific scenario. Defining hyperparameters may be challenging, and thus a tuning process may be used for defining an optimal set of hyperparameter values for the specific scenario. The hyperparameter tuning comprises a value search process that may include selecting a combination of values for the hyperparameters from a grid of values with different hyperparameter combinations. The aforementioned value search process can either be exhaustive or leverage more intelligent optimization techniques such as Bayesian optimization or genetic algorithms. Correctly setting hyperparameters is important because inappropriate values can lead to models that underfit or overfit the data. Underfitting occurs when a model is too simple to learn the underlying pattern of the data, and overfitting happens when a model is too complex,learning the noise in the training data as if it were signal, and fail to predict correctly on unseen data.

[0173] CNN 300 may also be trained, tested, and validated by a training and validation subsystem (e.g., the training and validation subsystem 215 described with respect to FIG.2). Training is the initial phase of developing CNN 300 where the model learns to make predictions or decisions based on training data provided from the training and validation datasets. During this phase, the model iteratively adjusts its internal model parameters to minimize the difference between its predictions and the actual outcomes in the training data. This process, known as fitting, is fundamental because it directly influences the accuracy and effectiveness of the model. The training phase is driven by three primary components: the model architecture (which defines the structure of the algorithm(s) see FIG. 3A and 3B), the training data (which provides the examples from which to learn), and the learning algorithm (which dictates how the model adjusts its model parameters). The goal is for the model to capture the underlying patterns of the data without memorizing specific examples, thus enabling it to perform well on new, unseen data.

[0174] The model architecture is the specific arrangement and structure of the various components and layers that make up CNN 300 (as described in detail with respect to FIGS 3 A and 3B). This includes the configuration of layers, such as the number of layers, the type of layers (e.g., convolutional, recurrent, fully connected), the number of neurons / nodes in each layer, and the connections between these layers. Model architecture also encompasses the choice and arrangement of features and algorithms used in various models. The architecture determines how input data is processed and transformed through various computational steps to produce the output. The model architecture directly influences the model's ability to learn from the data effectively and efficiently, and it impacts how well the model performs tasks such as classification, or prediction, adapting to the specific complexities and nuances of the data it is designed to handle. In this example, the model architecture encompasses a CNN algorithm that is implemented using various machine learning libraries and frameworks such as TensorFlow, PyTorch, Keras, and scikit-leam, which provide extensive tools and features to facilitate model building, training, validation, and testing.

[0175] The learning algorithm is the overall method or procedure used to adjust the CNN’s parameters to fit the data. It dictates how the model learns from the data provided duringtraining. This includes the steps or rules that the algorithm follows to process input data and adjust the model's internal parameters (e.g., weights in neural networks) based on the output of the objective function. Examples of learning algorithms include gradient descent and backpropagation. For example, CNN 300 is trained with labeled digital image data so a gradient descent learning algorithm may be used to iteratively adjusts the model parameters to minimize or maximize an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.). The objective function is a method to measure how well the model’s predictions match the actual labels or outcomes in the training data. It quantifies the error between predicted values and true values and presents this error as a single real number. The goal of training is to minimize this error, indicating that the model's predictions are, on average, close to the true data. Common examples of loss functions include mean squared error for regression tasks and crossentropy loss for classification tasks.

[0176] The adjustment of the CNN’s parameters is performed by the optimization function or algorithm, which refers to the specific method used to minimize (or maximize) the objective function. The optimization function is the engine behind the learning algorithm, guiding how the CNN’s parameters are adjusted during training. It determines the strategy to use when searching for the best weights that minimize (or maximize) the objective function. Gradient descent is a primary example of an optimization algorithm. Another example can include combining backpropagation with gradient descent to update the weights of CNN 300 based on the error rate obtained in the previous epoch (cycle through the full training dataset).

[0177] Training a CNN can be challenging as it can be sensitive to the initial set of model parameters and configuration of the learning algorithm. A reason for this challenge is the distribution of the inputs to layers deep in the network may change after each batch of data when the set of model parameters are updated. This can cause the learning algorithm to persistently chase a moving target (known as internal covariate shift). In some instances, to overcome this challenge, batch normalization is used for training CNN 300. Batch normalization may be performed by scaling the output of each hidden layer 355a-n (e.g., standardizing the activations of each input variable per batch of data) such as the activations of a node from the previous layer. Standardizing the activations of the prior layer means that assumptions the subsequent layer makes about the spread and distribution of inputs during the weight update will not change,at least not dramatically. This stabilizes the learning process and reduces the number of training epochs (speeds up training) required to train the CNN 300.

[0178] Once the set of model parameters of CNN 300 have been trained by optimizing the loss or error function, the network is able to predict one or more features from the input data (e.g., sets of patches from a WSI). In addition to tracking the training loss, the testing loss can additionally be tracked to implement early stopping, which halts training of a neural network when a clear divergence between training and testing loss is noted. The early stopping process combats overfitting, preventing scenarios in which the neural network performs extremely well on the training data at the expense of generalizing to validation data.

[0179] Validation of CNN 300 is the process of checking for deficiencies in performance and the hyperparameters using the validation dataset. The validation data helps to evaluate the model's performance during the training phase and ensures that it is general. Once training is complete the model performance can be evaluated using the test dataset, using measurements such as accuracy, precision, recall, or Fl-score, to gauge how well the model is likely to perform in real-world scenarios.

[0180] Once CNN 300 has been trained and validated, it undergoes a final evaluation using test data provided from the training and validation datasets, which is a separate subset of the data that has not been used during the training or validation phases. This step is crucial as it provides an unbiased assessment of the model's performance in simulating real-world operation. The test dataset serves as new, unseen data for CNN 300, mimicking how the model would perform when deployed in actual use. During testing, the model’s predictions are compared against the true values in the test dataset using various performance metrics such as accuracy, precision, recall, and mean squared error, depending on the nature of the problem (classification or regression). This process helps to verify the generalizability of CNN 300 — its ability to perform well across different data samples and environments — highlighting potential issues like overfitting or underfitting and ensuring that the model is robust and reliable for practical applications. CNN 300 is fully validated and tested once the output predictions have been deemed acceptable by user defined acceptance parameters. Acceptance parameters may be determined using correlation techniques such as Bland-Altman method and the Spearman’s rank correlation coefficients andcalculating performance metrics such as the error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.

[0181] As should be understood, other training-validation mechanisms are contemplated and may be implemented. For example, CNN 300 may be trained and hyperparameters may be tuned on data from a first subset of data and the data from a second subset of data may only be used for testing and evaluating performance of the model. Moreover, although the training-validation mechanisms described herein focus on training a new CNN 300, these training-validation mechanisms can also be utilized to fine-tune existing CNNs trained from other datasets. For example, in some instances, a CNN might have been pre -trained using digital pathology images for one type of cancer to predict the detection of another type of cancer. In those cases, CNN 300 can be used for transfer learning and retrained / validated using new sets of digital pathology images for a second type of cancer. Further, the learned neural network parameters may be frozen (as shown in FIG. 3B) and only the last layer (e.g., the fully connected layer) of CNN 300 is retrained / validated to learn feature predictions associated with a second type of cancer.VI. ARCHITECTURE OF ATTENTION AND SELF -ATTENTION BASED MODELS

[0182] Attention models (e.g., attention and self-attention models) are a class of deep learning mechanisms that allow neural networks to focus on specific parts of the input sequence when making decisions. Not only do these mechanisms improve overall model performance on tasks, but in the case of self-attention mechanisms, the capability to focus on specific parts of the input also allows the model process sequences in parallel, handle long-range dependencies. Feature prioritization is based on a calculated attention weight given to each input feature by an attention module. The larger the attention weight, the more contribution (e.g., higher importance) the feature has. Attention and self-attention models can be applied to features representing a wide variety of target classes, including disease-associated features and non-disease-associated features such as artifacts, background, or normal tissue, thus enabling both diagnostic and quality assurance applications.

[0183] FIG. 4 illustrates the different architectures of an attention mechanism (FIG. 4A) compared to a self-attention mechanism (FIG. 4B). Attention mechanisms focus on specific parts of the input data when computing attention scores for each input token. In other words, eachattention weight determined for each input token is independent (e.g., not influenced) from the other input tokens. Self-attention mechanisms, on the other hand, capture pairwise (e.g., contextualized) relationships and interactions between input tensors. This results in a more complex and richer representation that captures dependencies between input tensors, compared to attention mechanisms. This also means that the weight given to each input tensor is dependent (e.g., influenced) on the weighted contributions of all other input tensors.

[0184] Starting with FIG. 4A, the attention mechanism uses fully connected layers to compute an attention score for each patch independently. The input data 405 may include natural language phrases, raw image data, audio extractions and the like. In some instances, input data 405 are the features extracted from the patches corresponding to the flattened vectors 330 output from the final pooling layer of a feature extractor model (e.g., CNN 300 described in FIG. 3). Input data 405 may be stored in a data structure, such as a multi-dimensional array (e.g., a tensor) to allow flexible and efficient data manipulations. As illustrated in FIG. 4A, input data 405 may comprise a tensor for each patch 410 that was processed by the feature extraction model, where each patch tensor stores all the extracted features identified by the feature extraction model for that specific patch. In some embodiments, extracted features may also represent non-disease-associated elements, including artifacts (such as air bubbles, tissue folds, staining irregularities, or debris), background, or normal tissue structure, supporting a broad range of classification tasks. The extracted features may correspond to biological / pathological features detected in the patch such as necrosis, fibrosis, carcinoma, invasive, ductal, and the like. Additionally or alternatively, the extracted features may correspond to non-biological / pathological features such as edges, comers, and other patterns detected in the patch. The extracted features can be represented numerically within the tensor, where the numeric value can correspond to one (or more) extracted feature. The input data 405 (e.g., tensors) may be of various sizes, such as about 100, about 500, about 1,000 about 2,000, about 2,500, or more in size. As illustrated in FIG. 4A, the input data for a first patch (e.g., hi) may have the structure hi = [hi.i, hi.2, ... hi.x], where each element (e.g., hi.i) corresponds to a numerical value and ‘x’ indicates the last element, and the thus the size of the tensor.

[0185] The input data 405 (e.g., tensor) is fed into the attention layer 415 to compute attention scores 420 (e.g., alra2, ... , aBas shown in FIG. 4a). As used herein, the term “score” may beused interchangeably with “attention score” or “alignment score.” The attention score 420 may be calculated by applying a linear transformation through a fully connected network to the input data 405. More specifically, the input data 405 is multiplied by weights (w) and added to a bias or biases that are learned during the training of the attention layer 415. Although FIG. 4A only shows one fully connected layer, the number of hidden layers used in the attention layer 415 is a tunable parameter that can be increased. The attention scores 420 are passed through a SoftMax function for normalization. The SoftMax function converts the attention scores 420 into positive probabilities (e.g., attention weights) that sum to 1. The normalized attention scores are multiplied to their corresponding input 405 (e.g., al[hi.i,hi.2, ... hi.x]) to generate attention weighted feature vectors 425, which indicate the importance of each output and corresponding feature. As nonlimiting examples, a normalized attention score of 0.2 indicates low importance; a normalized attention score of 0.6 indicates moderate importance; a normalized attention score of 0.9 indicates high importance; and a normalized attention score of 0.95 indicates very high importance. Finally, the attention weighted feature vectors 425 are aggregated using an aggregation function (e.g., weighted sums, weighted averages, etc.) to generate an aggregated vector 430. The aggregated vector 430 represents a context-aware combination of the input data 405.

[0186] With respect to FIG. 4B illustrating a self-attention mechanism, much of the description for attention mechanism FIG. 4A is the same with minor differences. Input data 405 comprises the features extracted from the patches 410 corresponding to the flattened vectors 330 output from the final pooling layer of a feature extractor model (e.g., CNN 300 described in FIG. 3). Input data 405 may be stored in a data structure, such as a multi-dimensional array (e.g., a tensor) that stores all the extracted features (e.g., biologicaFpathological features and / or edges, comers, and other patterns detected in the patch) identified by the feature extraction model for that specific patch. The extracted features can be represented numerically within the tensor, where the numeric value can correspond to one (or more) extracted feature. The input data 405 (e.g., tensors) may be of various sizes, such as about 100, about 500, about 1,000 about 2,000, about 2,500, or more in size. As illustrated in FIG. 4B, the input data (e.g., tensor) for a first patch (e.g., hi) may have the structure hi = [hi.i, hi.2, ... hi.x], where each element (e.g., hi.i) corresponds to a numerical value and ‘x’ indicates the last element, which is the size of the tensor.

[0187] The input data 405 (e.g., tensors) are input into the embedding layer 415 of the selfattention mechanism to generate three different embedding vectors 418 (e.g., a query vector (q), a key vector (k), and a value vector (v)) using a fully connected layer for each tensor. More specifically, the input data 405 is multiplied by weight matrices (e.g., a key weight matrix (wk), a query weight matrix (qk), and a value weight matrix (wv)) that are learned during the training of the embedding layer 415.

[0188] The “key,” “query,” and “value” vectors are components used in the mechanism of attention. They facilitate the computation of the attention scores 420 and aggregating information from different parts of the input. The query vectors represent the information the model wants to focus on or retrieve from the input data 405. The key vectors capture the information that is used to determine the importance or relevance of each element (e.g., feature) in the input data 405. For example, the key vectors can be used to compute how relevant each element in the input data 405 is to the query vectors. The relevance (e.g., attention scores) is often calculated using a dot product or another similarity measure between the query and key vectors, depending on the type of attention mechanism being used (e.g., self-attention (dot-product) mechanisms or globaFlocal (additive) attention mechanisms). The value vectors are also a projection of the input data and associated with each element (e.g., feature) in the input sequence, just like the key vector. The value vectors can store the information that is used to update the representation of the query and the key. These value vectors are weighted by the attention scores (computed from the query -key interaction) to determine how much each element contributes to the final output. The attention scores 420 can be computed based on the query and key vectors, and used to weigh the value vectors to generate the attention weighted feature vectors 425. Higher attention scores may mean that the corresponding values are more important for the output.

[0189] To compute the attention for each feature in an input tensor against every other feature in other input tensors, a variety of similarity metric functions (e.g., transformation functions f(x)) may be applied to the query and key vectors. Examples of similarity metric functions include dot product, cosine similarity, scaled dot product, additive, and the like. In various embodiments, the dot product between the query and key vectors is used to calculate the attention scores 420. The attention mechanism can store the attention scores 420 as various data structures such as a single value, a ID vector, a 2D matrix / array, etc. For example, the dot product between kl and ql forthe first patch tensor hl may be stored as a single attention score. As another example, the dot product between the query vector corresponding to the input tensor for the patch currently being processed (e.g., the second patch (q2)) may be multiplied by the key vectors for every other input tensor (including itself). In other words, the dot product between the query vector corresponding to the input tensor for the second patch (q2) is multiplied by the key vectors for (i) the first patch (kl), (ii) the second patch (k2), and (iii) the z'thpatch (kz). In this instance, all the attention scores of the second patch (q2) compared to other patches (kl, k2, ... ,kB) may be stored in a ID vector.

[0190] As illustrated in FIG. 4B, the attention scores 420 may be influenced by (e.g., dependent) the features in the second patch. The attention scores 420 can then be normalized a SoftMax function to generate the attention weights. The attention weights are multiplied by each patches’ corresponding value vector to generate the attention weighted feature vectors 425. Finally, the attention weighted feature vectors 425 are aggregated using an aggregation function (e.g., weighted sums, weighted averages, etc.) to generate an aggregated vector 430. The aggregated vector 430 can represent a summary of the features (in patch 2) incorporating all correlated information of others patches features.VII. MULTIPLE INSTANCE LEARNING

[0191] Multiple instance learning (MIL) is a type of supervised learning where instead of receiving a set of individually labeled instances, a machine learning model (e.g., a classifier) receives a set of labeled “bags”, each containing many unlabeled instances. A bag is positively labeled if at least one instance in the bag is positive and is negatively labeled if all instances in the bag are negative. There are two main types of MIL frameworks: Instance-based MIL (IMIL) and embedding-based MIL (EMIL) which are illustrated in FIG. 5A and 5B, respectively. In both methods, features are extracted from each patch using a feature extractor model (e.g., CNN 300 described with respect to FIG. 3), where each patch originated from a WSI. The extracted features are then used as input. IMIL (FIG. 5A) uses a classifier (fully connected layer) applied to each patch feature independently to make patch-level predictions. Then an aggregator is used to get the bag level prediction by pooling all patch predictions. Here the aggregator is a pooling method like max pooling, mean pooling, or the use of a fully connected layer. On the other hand, EMIL (FIG. 5B) uses directly an aggregator to aggregate all patch features to make features representing the bag. Then a classifier (fully connected layer) is applied to this bag features toget bag level prediction. With IMIL, instance-level prediction can be obtained, even if the process is not optimizing the instance-level prediction as there is no instance-level ground truth. With EMIL, instance level prediction is not computed, but it shows better performance than IMIL for bag level prediction. During the training, the loss between bag-level predictions and ground truths are computed using the objective function as described with respect to FIG. 2. MIL may be applied to train models to recognize both disease-associated features (such as tumor, necrosis, or abnormal tissue) and non-disease-associated features (such as artifacts, background, or normal tissue), supporting a wide range of diagnostic and quality assurance tasks.

[0192] As illustrated in FIG. 6, medical images and / or digital WSI processing are ideal MIL tasks since a WSI or portions of the WSI can be treated as a bag. Briefly, the medical images and / or digital WSI may be generated using system 100 as described with respect to FIG. 1. The global label of a WSI may be determined by one or more pathologists, where a WSI is labeled as tumor positive 605 if it contains a tumor, but no other information about the tumor region such as localization or quantification is provided. Conversely, a WSI is labeled as tumor negative (nontumor) 610 if it contains no tumor regions. The tumor positive 605 and tumor negative 610 WSIs can be tiled into patches using a similar system as the data subsystem 205 described with respect to FIG. 2. In addition, tissue segmentation processing may also be done to distinguish between tissue patches and non-tissue / background patches, where background patches may be removed from analysis. Patches 615 may be randomly sampled from each tiled WSI to a predetermined number based on the training needs of the machine learning model. Patches 615 randomly sampled from a tumor positive 605 WSI contain at least one tumor patch, although which patch is tumor positive is unknown. The randomly selected patches 615 are fed into a MIL framework 620 to generate a bag-level prediction 625 (e.g., tumor positive or tumor negative). The bag-level prediction 625 is compared to the WSI global label (e.g., ground truth label) to compute a loss, using an objective function, and update the MIL framework 620. When computing the loss, the goal is to minimize the difference between the bag prediction and the ground truth label. In some embodiments, the global label may indicate the presence or absence of a non-disease-associated feature, such as an artifact, and bag-level training can be used to enable detection of such features in addition to disease-associated features.Attention Multiple Instance Learning Framework

[0193] The addition of an attention mechanism to MIL training techniques provides a machine learning model with information (e.g., weights) pertaining to which features from instances (e.g., patches) in the bag are the most relevant to the bag’s label. Specifically, the attention mechanism generates weighted attention scores for each feature identified in a bag of patches. The weighted features are aggregated together using an aggregation function (e.g., weighted sums, weighted averages, etc.) and classified to generate a predicted label for that bag of patches. Accordingly, the machine learning model learns to generate bag-level labels based on the weighted attention scores associated with each feature. There are several advantages to incorporating an attention mechanism into a MIL framework include (i) the interpretability of the model is greatly improved as using weighted instances provides insight into which instances were used to generate the bag’s label, and (ii) the performance of the machine learning model is improved by using weighted instances as opposed to traditional MIL that treats all instances equally. Unlike approaches that restrict attention computations to predefined clusters or local regions, the disclosed framework allows any patch in a set to attend to all other patches in the set. This global context modeling supports the aggregation of information from diverse regions of the slide, enabling a more comprehensive and interpretable representation of both pathological and non-pathological features.

[0194] FIG. 7 shows an exemplary illustration of an attention-based MIL (ABMIL) framework 700 for generating a predicted bag label for a set of patches. The ABMIL framework 700 includes three main parts: a feature extraction process 705 to identify features from sets of patches, an attention mechanism 710 to weigh and aggregate the features, and a multiple instance learning process (MIL) 715 to predict diagnostic labels for bags of patches based on the weighted aggregated features. The feature extraction process 705 performs the process of extracting features from sets of patches generated from globally labeled WSIs. A data subsystem (like data subsystem 205 described with respect to FIG. 2) may be used to generate the patches 720 and divide them into sets if necessary.

[0195] In various embodiments, at least 1,000 individual WSI are used and at least 50 individual patches from a single WSI are generated to create the set of patches. The patches may be of any appropriate (x,y) pixel dimensions (e.g., 299 x 299 pixels) and resolution quality (e.g.,40x magnification). In some instances, the patches may be designed with or without overlap (wherein at least one pixel is overlapped) with neighboring tiles / patches. In various embodiments, the number of WSIs and the number of patches generated from a single whole side image is a parameter that is tuned during the training of the feature extractor model. In various embodiments, a single WSI corresponds to a single bag, where the bag contains multiple patches (e.g., 50-400 patches) from the WSI (e.g., 1 WSI generates 1 bag). In other embodiments, a WSI may be used to generate multiple bags, where each bag may contain multiple patches (e.g., 50-400 patches) from the WSI (e.g., 1 WSI generates 2, 3, 4, 5, or more bags). In various embodiments, the feature extraction process 705, and thus downstream processing (e.g., attention mechanism 710 and MIL 715), is iteratively repeated based on the number of individual WSIs that are available to generate a sufficient number of bag features 750 to train the ABMIL classifier 755. For example, if at least 1,000 individual WSI are available, at least 1,000 iterations of feature extraction and attention generation are performed to generate at least 1,000 bags for training the ABMIL classifier 755.

[0196] During training, the attention mechanism 710 and aggregator are used to assign weights (attention weights 740) to features extracted from each patch (feature tensors 730) and aggregate them into bag features 750, which are then used to train the ABMIL classifier 755 to predict baglevel predictions 760. The attention mechanism 710 and aggregator are only applied during the training phase to optimize the classification model for bag-level prediction. During inference, the trained classification model (e.g., the ABMIL classifier 755) is applied directly to patch features (e.g., the feature tensors 730) to generate patch-level predictions. The attention mechanism 710 and aggregator are not used during inference for generating patch-level predictions. While attention weights 740 may optionally be computed for interpretability or visualization, these scores are independent of the patch-level predictions and are not used for classification at inference.

[0197] The sets of patches 720 are input into the feature extractor 725 that can be a trained machine learning model (e.g., CNN 300 described with respect to FIG. 3) used to identify meaningful features in WSIs and / or patches. The extracted features from each patch in the set of patches 720 are stored in a data structure like a multi-dimensional array (e.g., a feature tensor 730. The extracted features can be biological / pathological features and / or edges, comers, andother patterns detected in the patch and of various sizes (e.g., about 100, about 500, about 1,000 about 2,000, about 2,500, or more in size). The extracted features can be represented numerically within the feature tensor 730, where the numeric value can correspond to one (or more) extracted feature. In addition to features related to pathology, the extracted features may represent nondisease-associated elements, such as artifacts, background, or normal tissue structure, supporting both diagnostic and quality control applications. The disclosed ABMIL framework supports extraction and aggregation of both disease-associated and non-disease-associated features, enabling flexible application to a wide range of diagnostic and quality control tasks. In addition to features related to pathology, the extracted features may represent non-disease-associated elements, such as artifacts, background, or normal tissue structure, supporting both diagnostic and quality control applications. The ABMIL framework supports extraction and aggregation of both disease-associated and non-disease-associated features, enabling flexible application to a wide range of diagnostic and quality control tasks.

[0198] The feature tensors 730 are input into the attention mechanism 710, which includes much of the same architecture described with respect to FIG. 4A. Briefly, the feature tensors 730 are input into attention layer 735 to generate attention scores for each feature tensor 730. As previously described, attention scores generated by attention mechanisms are calculated independently from the other features stored in the other patch tensors. The attention scores are passed through a SoftMax function for normalization to generate attention weights (a) 740 for each feature tensor 730. The SoftMax function converts the attention scores into positive values between 0-1 that sum to one, allowing the resulting attention weights 740 to be interpreted as probabilities. The attention weights 740 are multiplied by their corresponding feature tensor 730, generating attention weighted vectors for each patch. For example, the attention weighted vector for the first patch may look like: Vhl=where a is the attention weight and ‘h’ is a feature in the attention weighted vector of the first patch. The attention mechanism 710 also applies an attention pooling mechanism to aggregate the attention weighted vectors for each patch forming an aggregated vector. The aggregated vector represents a context-aware combination of the set of patches 720 and represents bag features 750.

[0199] The bag features 750 can be thought of as a weighted summary of all features stored in the feature tensors 730 composing the bag. The bag features 750 are used to train a fullyconnected ABMIL classifier 755 to make bag-level predictions 760. Training involves iteratively calculating an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.) based on the bag-level prediction and the ground truth bag label (e.g., the global label from the WSI the sets of patches 720 are generated from), where the goal is to minimize the difference between the two values. In some instances, backpropagation may be used to update the model parameters, including those of the attention mechanism, to minimize the loss. Model performance is evaluated based on performance metrics such as accuracy, precision, recall, and Fl -score at the bag level.

[0200] Once the ABMIL classifier 755 has been trained and validated, the final model is tested on a set of data it has never seen before to assess its performance on real-world scenarios. In the ABMIL framework 700, attention scores are calculated independently for each patch by the attention layer 735, without considering interactions between patches at this stage. The attention weights 740 generated for each patch are then used during the aggregation step, which combines the feature tensors from all patches in the bag into a single bag feature by taking a weighted sum. This aggregation step accounts for contributions from all patches when forming the bag feature for multiple instance learning. In this way, while the attention for each patch is computed independently, the aggregation step incorporates information from all patches in the bag, enabling the detection of subtle or spatially dispersed features, including artifacts, which may not be identified by aggregation methods limited to local or cluster-based groupings.

[0201] FIG. 8 illustrates how a trained ABMIL classifier model may be used in inference phase 800. During inference, a WSI 805 (either globally labeled or unlabeled) may be tiled to generate patches 810 that are input into a feature extractor 815. A data subsystem (like data subsystem 205 described with respect to FIG. 2) may be used to generate the patches. The feature extractor 815 may be a trained machine learning model (e.g., CNN 300 described with respect to FIG. 3) used to identify meaningful features in WSIs and / or patches. The extracted features from each patch 810 are stored in a feature tensor 820 and represented as numeric values corresponding to one (or more) extracted feature. Each feature tensor 820 is input directly into the trained ABMIL classifier 825, which generates a patch-level prediction for each patch.During inference, the attention mechanism and aggregator are not used to compute patch-level predictions. The classification model is applied directly to each patch’s features to obtain thepatch- level prediction. While attention scores (e.g., weights that may be assigned to each patch feature by an attention mechanism) may optionally be computed for interpretability or visualization, these scores are independent of the patch prediction and are not used for classification at inference. This approach enables efficient and interpretable patch-level classification.

[0202] The ABMIL classifier 825 outputs logits, or raw, unnormalized scores that are used in classification tasks to determine the probabilities of different classes (representing the likelihood that each patch belongs to a specific target class). For example, the ABMIL classifier 825 may have 10 possible classes and will thus output 10 logits, one for each class. As another example, the ABMIL classifier 825 may have 2 possible classes (e.g., tumor or not tumor) and will accordingly output 2 logits. The values of these logits can range between [-infinite, +infinite]. To make these values more interpretable, an activation function may be applied to transform the logits. Examples of activation functions that may be used include the linear activation function, the rectifier linear unit (ReLU), the leaky ReLU, the sigmoid activation function, the SoftMax activation function, the hyperbolic tangent activation function, and the exponential activation function. In some instances, the activation function is the SoftMax function which converts the logits into a positive probability distribution that sums to 1. By way of example, and not to be limiting, the SoftMax function may output a value such as [0.1, 0.9] for a patch where the first value is linked to the tumor label, and second value corresponds to the non-tumor label. This output is interpreted to mean that the ABMIL classifier 825 predicted the patch as 10% tumor, and 90% non-tumor. The ABMIL classifier 825 may further apply a threshold value (e.g., 0.5) where if the tumor probability is less than the threshold, the patch is classified as non-tumor.

[0203] The patch-level prediction probabilities produced by the ABMIL classifier 825 can be used to generate high-resolution, spatially resolved outputs, such as heatmaps, that visually indicate the probability of each patch for a target class. Attention scores, if computed, may also be visualized as heatmaps or overlays for interpretability, but are not used for generating patch predictions during inference and are typically less distinct and less clinically interpretable than the prediction probabilities. In this context, “prediction probabilities” and “attention scores” may be visualized in similar ways, but only the prediction probabilities may be used for classification during inference, and are therefore preferred for clinical review and quantitative assessment.

[0204] After the ABMIL classifier 825 outputs patch predictions 830, which may be further processed to generate images (e.g., heatmaps, binary masks, and the like) that are used by a pathologist to compare the model predictions to the original WSI to make a diagnosis and generate a report. The individual patch predictions may be stored in a data structure like an array. Post processing outputs 835 may include a heatmap (top image) and / or a binary mask (bottom image). A heatmap may be generated by mapping the predicted probabilities (e.g., probabilities output from SoftMax) or attention scores for each patch to a color scale, where regions of high predicted probability or high attention for a target class appear as visually distinct zones, providing an intuitive overview of model focus and confidence across the slide. For example, red may correspond to high tumor predicted probability patches (e.g., probability value close to 1), blue may indicate low tumor predicted probability patches (e.g., close to 0), and all other colors correspond to intermediate predicted probability values. A binary mask can be generated by thresholding the predicted probabilities: patches with predicted probabilities above the threshold are considered as tumor, while those below the threshold are considered non-tumor. In the binary mask example shown in FIG. 8, the white areas correspond to predicted tumor patches and the black areas to predicted non-tumor patches. The ability to generate and visualize patch-level heatmaps and binary masks allows for enhanced interpretability and more informed clinical review or quality assurance, as users can directly see which regions of the slide contributed most to the prediction.

[0205] After post processing, the individual patches may be “stitched” together to generate the WSI images illustrated in FIG. 8 using computational tools and libraries known in the art, for example, ImageJ / FIJI, OpenCV, PIL (Pillow), or NumPy. As patches are generated (for example by data subsystem 205 described with respect to FIG. 2), metadata that includes the patch position (e.g., (x, y) coordinates) are also generated and stored. An exemplary stitching process may include the following steps. Initially, a blank canvas that matches the dimensions of the original whole slide image is generated. Then, the patches and their corresponding values (e.g., tumor predicted probability value, binary 0 or 1 tumor classified value, etc.) are placed within the canvas according to their coordinates using the pixel data of the patch to alignment. If patches overlap, the overlapping regions are merged using any variety of techniques like averaging the pixel values at overlapping regions, or blending techniques like feathering may be used. Finally, the stitched image may undergo some quality control (e.g., artifact removal) and adjustmentsbefore the final stitched WSI is output. The final stitched WSI may be output as an array based on the patches’ positions in the original WSI and its corresponding values (e.g. model prediction outputs) and transformed into visual images.Self-Attention Multiple Instance Learning Framework

[0206] As opposed to attention-based MIL which calculates weights independently, selfattention MIL captures pairwise (e.g., contextualized) relationships and interactions between instances within a bag. In other words, each instance attends to all other instances within the bag, computing attention scores for these interactions. This results in a more complex and richer representation that captures dependencies between instances, compared to attention-based MIL. This also means that the weight given to each instance is dependent on the weighted contributions of all other instances in the bag. In the disclosed framework, self-attention is leveraged not only to enable aggregation of information from all patches within a bag but also to support flexible architectures where different combinations of attention and self-attention are used in generating feature representations for multiple instance learning.

[0207] FIG. 9A shows an exemplary illustration of a self-attention-based MIL (SAB MIL) framework 900 for generating a predicted bag label for a set of patches. The SABMIL framework 900 includes three main parts: a feature extraction process 905 to identify features from images, an embedding aggregator 910 that computes attention scores and aggregates the features, and a multiple instance learning process 915 to predict diagnostic labels for bags of patches based on the weighted aggregated features. The feature extraction process 905 extracts features from sets of patches generated from globally labeled WSIs. A data subsystem (like data subsystem 205 described with respect to FIG. 2) may be used to generate the patches 920 and randomly select patches to be included in the set of patches 920. In various embodiments, the feature extraction process 905, and thus downstream processing (e.g., embedding aggregator 910 and MIL 915), is iteratively repeated based on the number of individual WSIs that are available to generate a sufficient number of bag features 965 to train the AB MIL classifier 970. For example, if at least 1,000 individual WSI are available, at least 1,000 iterations of feature extraction and attention generation are performed to generate at least 1,000 bags for training the AB MIL classifier 970.

[0208] The sets of patches 920 are input into the feature extractor 925 which outputs the identified features into a data structure, such as a multi-dimensional array (e.g., a feature tensor 930). The feature extractor 925 may be a trained machine learning model (e.g., CNN 300 described with respect to FIG. 3) used to identify meaningful features in WSIs and / or patches. The extracted features can be biological / pathological features and / or edges, comers, and other patterns detected in the patch and of various sizes (e.g., about 100, about 500, about 1,000 about 2,000, about 2,500, or more in size). The extracted features can be represented numerically within the feature tensor 730, where the numeric value can correspond to one (or more) extracted feature. The feature tensors 930 correspond to a specific patch and store the extracted features identified by the feature extractor 925. For example, the tensor represented as “hi” correspond to a first patch in the set of patches 920 and stores the extracted features from the first patch. The feature tensors 930 represent characteristics of the patches that can be used to distinguish nontumor tissue from abnormal (e.g., cancerous or tumor positive) tissues.

[0209] The feature tensors 930 are input into the embedding aggregator 910 (see FIG. 9B) which includes a self-attention mechanism 935 and an attention layer 940. The self-attention mechanism 935 includes much of the same architecture described with respect to FIG. 4B, while the attention layer 940 includes much of the same architecture described with respect to FIG. 4A. As illustrated in FIG. 9B, the feature tensors 930 are input into the embedding layer of the selfattention mechanism 935 to generate three different embedding vectors (query vector (q), key vector (k), and value vector (v)) (e.g., attention vectors 945) using a fully connected layer for each tensor. As previously described, in self-attention-based mechanisms, the attention for each feature in the feature tensor is influenced by every other feature in the other feature tensors. Thus, the dot-product is calculated between the query of the tensor being currently processed, and the key of every other tensor (including itself). As illustrated in FIG. 9B, the query vector corresponding to the tensor for the second patch (q2) is multiplied by the key vectors for (i) the first patch (kl), (ii) the second patch (k2), and (iii) the Bthpatch (kB). The resulting attention weights 950 (stored as individual numerical values) for each patch are all influenced by the features in the second patch. The attention weights 950 are normalized using the SoftMax function and the normalized attentions are multiplied by each patches’ corresponding value vector to generate attention-weighted vectors 980. The attention-weighted vectors 980 are aggregated into a single vector representation that represents a summary of the features (in patch2) incorporating all correlated information of others patches features. As illustrated in FIG. 9B, the aggregation of the attention-weighted vectors 980 generate the transformed feature of patch 2 ( / I2). In other words, the output of the aggregation function is transformed features 955 with weighted attention scores that are influenced by the features from other patches.

[0210] A distinguishing aspect of the disclosed approach, as illustrated by path (b) in FIG. 9B, is the application of attention mechanisms directly to the original feature tensors, rather than relying solely on transformed features. In this configuration, attention scores are computed with respect to the input feature tensors, and these scores are used to generate attention-weighted feature vectors for each patch. This structure preserves the contextual relationships established by the self-attention mechanism and enables patch-level interpretability and granularity in the outputs. In this SABMIL configuration, the attention score 960 is computed with respect to the input feature tensors 930, capturing global context and integrating information from all other patches in the set. However, only the attention score leverages this global context, while the original feature tensors 930 preserve the individual information for each patch. This separation allows the model to maintain distinct patch-level characteristics and avoid homogenizing the outputs, which can occur if only transformed features are used (as in SA-AbMILP, where transformed features 955 mix information from all patches and can result in all patches being classified as either positive or negative). As a result, SABMIL provides improved performance and better patch-level separability, because it benefits from global context through the attention mechanism while retaining unique information for each patch via the original features tensors 930. Unlike approaches that restrict attention computation to local or cluster-based aggregations, this method allows any patch within the set to attend to all other patches, supporting global context modeling and comprehensive identification of both disease-associated and non-disease-associated features, such as artifacts and background regions. Additionally, by applying attention directly to the original feature tensors, the approach can reduce unnecessary intermediate transformations and computations, resulting in improved computational efficiency in both memory usage and processing time, particularly during inference and post -processing.

[0211] The transformed features 955 are used as input into the attention layer 940, which processes the transformed features 955 in the same way as described with respect to FIG. 4A. In so doing, the attentions 960 generated by the attention layer 940 are calculated independently foreach transformed feature 955, meaning the attention weights only correspond to the transformed features 955 of a single patch, not across patches.

[0212] There are two approaches that may be taken to generate the bag features 965. The standard method (e.g., self-attention attention-based MIL pooling or SA-AbMILP(a)) involves taking the product of attentions 960 and the transformed features 955, which outputs weighted feature vectors for each feature. An attention-based pooling function (e.g., an aggregation function) is applied to each feature vector generating an average attention- weighted vector of all the input features. However, when using the SA-AbMILP(a) model, the attention-based classification model gives very low patch level classification performance as the features are influenced by the weighted contributions of all other feature tensors in the self-attention module. Accordingly, when the ABMIL classifier 970 is used in inference, the predicted patch-level labels are the same as the global WSI label, regardless if the patch is tumor positive or tumor negative. To overcome this and obtain predicted patch-level labels, an alternative approach (e.g., SABMIL(b)) is used to instead take the product of the attentions 960 and the feature tensors 930. In so doing, the ABMIL classifier 970 uses feature tensors 930 which are not influenced by others patch features. Accordingly, the aggregated vectors generated using the SABMIL(b) method are used as the bag features 965 to train an ABMIL classifier 970 (see FIG. 9A).

[0213] This alternative configuration, corresponding to path (b) in FIG. 9B, supports patchlevel predictions with increased granularity and interpretability, even when only bag-level labels are available during training. By applying the attention mechanism directly to the feature tensors, the model generates attention weights that reflect the contribution of each patch to the overall bag label. This configuration enables downstream generation of heatmaps, binary masks, or overlays that indicate which regions of the slide are most relevant for a given classification, supporting clinical review, audit, and quality assurance. The ability to produce patch-level outputs in this manner is not achieved by methods that restrict attention computation to transformed features or local clusters.

[0214] Each of the bag features 965 can be thought of as a weighted summary of all the features extracted from the patches making up the bag. The bag features 965 are used to train a fully connected ABMIL classifier 970 to make bag-level predictions 975. Training involves iteratively calculating an objective function (e.g., a loss function, a cost function, a contrastiveloss function, etc.) based on the bag- level prediction and the ground truth bag label (e.g., the global label from the WSI the sets of patches 920 are generated from), where the goal is to minimize the difference between the two values. In some instances, backpropagation may be used to update the model parameters, including those of the attention mechanism, to minimize the loss. Model performance is evaluated based on performance metrics such as accuracy, precision, recall, and Fl -score at the bag level.

[0215] Once the AB MIL classifier 970 has been trained and validated, the final model is tested on a set of data it has never seen before to assess its performance on real-world scenarios. The inference phase for the SABMIL(b) trained AB MIL classifier 970 is the same as what is shown in FIG. 8. Briefly, tensor patches comprising their extracted features are directly input into the trained AB MIL classifier to obtain patch-level predictions (e.g., tumor or non-tumor). The raw logit predictions can be normalized using an activation function, such as SoftMax, to generate tumor predicted probabilities for each patch that can be used to generate a heatmap during post processing. Moreover, the tumor predicted probabilities can be compared to a threshold value for classifying the patch as tumor or non-tumor and used to generate a binary mask of the original WSI. During post processing, the individual patches can be stitched together to generate a stitched WSI that is stored as an array for further downstream processing.

[0216] During inference, as illustrated in FIG. 9A and FIG. 9B (see also FIG. 8), the attention mechanism and self-attention mechanism are not used to compute patch-level predictions.Instead, the trained AB MIL classifier 970 (e.g., corresponding to 830 in FIG. 8) receives, for each whole slide image, a set of feature tensors 930 (e.g., corresponding to 820 in FIG. 8) corresponding to patches generated by the feature extraction process 905. The AB MIL classifier 970 is applied directly to each patch’s feature tensor to generate the patch-level prediction (see, 830 in FIG. 8), regardless of whether the SABMIL(b) configuration was used during training. Thus, during inference, the patch-level prediction for each patch is generated solely by the AB MIL classifier 970 applied to the feature tensor, and attention or self-attention mechanisms do not contribute to this prediction. While it is possible during inference to compute self-attention or attention and visualize these as attention maps, such attention scores are calculated independently of patch prediction, and do not influence the classification of patches. This is in contrast to the training phase, where both self-attention and attention are used in optimizing theclassifier. As further demonstrated in FIG. 13, visualizations based on attention scores typically provide less distinct separation between positive and negative regions than those based on patch prediction probabilities.

[0217] By relying solely on the ABMIL classifier applied to patch features at inference, the model produces patch-level predictions that can be directly used for clinical interpretation, heatmaps, or binary masks (see FIG. 8). Calculation of attention or self-attention scores at inference is entirely optional, serves only interpretability or visualization purposes, and is not part of the patch classification process. This approach ensures that patch-level predictions benefit from the model’s training on global context while maintaining individual patch specificity and efficient computation. The ability to directly compute and visualize per-patch predictions — separate from attention mechanisms — distinguishes this framework from methods that aggregate only at the bag or region level, and enables accurate, interpretable, and granular outputs for diagnostic and quality assurance applications.VIII. TRAINING A MACHINE LEARNING MODEL TO PREDICT BAG LABELS

[0218] FIG. 10 is a flowchart illustrating process 1000 for training a machine learning model to classify bags of patches with respect to any target class for diagnostic and quality assurance applications. The processing depicted in FIG. 10 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof (e.g., the intelligent selection machine). The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 10 and described below is intended to be illustrative and non-limiting. Although FIG. 10 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different orders, or some steps may also be performed in parallel. In some embodiments, such as the embodiments depicted in FIG. 2, FIGs. 3A and 3B, FIGs. 4A and 4B, FIG. 6, FIG. 7, and FIGs. 9A and 9B, the processing depicted in FIG. 10 may be performed by the components of the machine learning pipeline 200 described with respect to FIG. 2, of the CNN described with respect to FIGs. 3A and 3B, of the attention and / or self-attention architectures described with respect to FIGs. 4A and 4B, of the MIL training technique describedwith respect to FIG. 6, of the AB MIL framework described with respect to FIG. 7, and of the SABMIL or SA-AbMILP framework described with respect to FIG. 9B.

[0219] At box 1005, sets of patches are obtained from a plurality of globally labeled WSIs. Each set of patches corresponds to a WSI, and each WSI is labeled with a target class. The target class may comprise a disease-associated feature (such as tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells) or a non-disease-associated feature (such as artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content). The patches may be generated using a system (e.g., data subsystem 205 described with respect to FIG. 2). In various embodiments, at least 1,000 individual WSIs are used and at least 50 individual patches from a single WSI are generated to create the sets of patches. For example, at least 1,000, at least 1,500, at least 2,000, at least 2,500, at least 3,000, at least 3,5000, at least 4,000, or more WSIs are used, where at least 50, at least 60, at least 70, at least 80, at least 90, at least 100, at least 150, at least 200, at least 250, at least 300, at least 400, at least 500, at least 1,000, at least 1,500, at least 2,000, or more individual patches from a single WSI may be generated. In some embodiments, each set corresponds to a different WSI. The patches may be of any appropriate (x,y) pixel dimensions and resolution quality (e.g., 299 x 299 pixels at 40X resolution). In some instances, the patches may be designed with or without overlap (wherein at least one pixel is overlapped) with neighboring tiles / patches. In various embodiments, the number of WSIs and the number of patches generated from a single WSI is a parameter that is tuned during the training of the machine learning model.

[0220] In various embodiments, each individual set of patches in the sets of patches may also be referred to as a “bag” of patches. In other words, the terms “a set” and “sets” are used interchangeably with the terms “a bag” and “bags.” For example, a single WSI can correspond to a single bag where the set of patches generated from that single WSI (e.g., 50-400 patches) comprise that bag of patches. As another example, a single WSI may be used to generate multiple bags of patches, where each bag contains multiple patches (e.g., 50-400 patches) from the WSI (e.g., 1 WSI generates 2, 3, 4, 5, or more bags). The bags (e.g., sets) may either has an equal number of patches among each bag, or the number of patches in each bag may vary. Patch sampling techniques may also be considered to account for (i) intra bag similarities (e.g., patchesbelonging to the same bag share similarities that patches from other bags do not), (ii) instance co-occurrence (e.g., features in a patch are more likely to occur in some environment over others), and (iii) “instance” based bag structure (e.g., a feature is more likely to be observed in advanced tumors compared to early tumors and vice versa, essentially spatial / temporal / relation occurrence). Each bag is assigned a ground truth label indicating the presence or absence of a target class in the originating WSI, where the target class may comprise a disease-associated feature or a non-disease-associated feature, such as an artifact, background, or normal tissue structure. For example, a bag generated from a WSI labeled for the presence of a particular artifact will be considered positive for that artifact if at least one patch contains the relevant feature, and negative if all patches lack the feature. This structure supports training for both diagnostic and quality assurance tasks, including the detection of tumor, necrosis, or any other target class such as air bubbles, tissue folds, staining irregularities, or debris..

[0221] The specimen corresponding to the WSIs may comprise cells, tissue, or biological fluid. More specifically, the specimen is a tissue section collected from a larger tissue sample from a subject suspected of / diagnosed with / being treated for a disease. In the event the subject is suspected of having the disease, the specimen can be entirely healthy (e.g., no disease) indicating the subject does not have the disease or the specimen can comprise diseased tissue. Moreover, the specimen can show pathological features associated with a different disease other than the disease being tested for. By way of example, the disease being tested for may be cancer, but the specimen may show pathological features associated with an infectious disease as opposed to cancer. In various embodiments, the disease is cancer such as pancreatic, kidney, renal, pelvic, colorectal, stomach, thymic, head and neck, mesothelial, prostate, cervical, thyroid, adrenal, testicular, breast, uterine, bone, esophageal, lung, liver, bile duct, ovarian, bladder, nervous system, or skin cancer. Moreover, the specimen comprises disease negative regions (e.g., nontumor regions), disease positive regions (e.g., tumor regions), or any combination thereof; however, which regions corresponding to the disease negative and disease positive regions are not annotated on the WSIs. In addition to disease-associated regions, the specimen may also include areas that contain non-disease-associated features, such as artifacts from slide preparation, background, or normal tissue. The framework described herein is adaptable to label and classify regions for any target class of interest, enabling use in both pathology detection and quality assurance workflows.

[0222] The globally labeled WSIs are generated using a staining system (e.g., staining system 125 described with respect to FIG. 1). For example, slides of the specimen are obtained, wherein the slides are stained using a histological staining technique to enhance contrast and visualize specific cellular components. Examples of staining methods that may be used include, without limitation, Hematoxylin and Eosin (H&E), Masson’s Trichrome, Periodic Acid-Schiff (PAS), silver stains, such as Gomori’s Methenamine Silver (GMS), Reticulin stains, Prussian Blue stain, immunohistochemical stains, immuno-fluorescent staining, multiplexing fluorescent staining, and the like. In various embodiments, the histological staining technique is H&E. In some instances, the WSIs are digitally stained. Based on the staining pattern, one or more pathologists provide a global label to the WSIs indicating if it is tumorous or non-tumorous (e.g., ground truth labels). In some embodiments, the plurality of WSIs used to generate the sets of patches includes at least 1,000 WSIs, and each set of patches corresponds to a single WSI. Ground truth labels for each bag correspond to the presence or absence of the target class in the originating WSI and may be determined by expert annotation, algorithmic estimation, or other suitable methods.

[0223] Additionally or alternatively, ground truth labels for WSIs may be generated from WSIs that are labeled by an algorithm (e.g., STAPLE). The algorithm may be specifically designed to take as input, segmentations of digital pathology images (e.g., WSIs of tissues) and output a probabilistic estimate of the true segmentation that are used to label regions in the WSI that are non-tumor, tumor, or undefined. Accordingly, the patches generated from these WSIs comprise patch-level labels (e.g., non-tumor or tumor). Patch sampling techniques may also be considered such as sampling positive (e.g., tumor) and negative (non-tumor) patches independently from a positive and a negative distribution. For example, the positive distribution of patches may comprise the patch that display tumor tissue (e.g., 25% or more) indicated by any combination of features such as tumor stroma, non-invasive urothelial carcinoma, invasive urothelial carcinoma, low- grade tumors, high-grade tumors, squamous cell differentiation, glandular differentiation, etc. Patches sampled from a negative pool exclude areas with necrosis, extensive cautery effect, washed off tissue, blurred view or any combination thereof.

[0224] At box 1010, features for each patch within the sets (e.g., bags) of patches are identified using a feature extraction model. The feature extraction model may be a convolution neural network, a transformer model, an autoencoder model, or any other appropriate model. In someembodiments, the feature extractor model is a convolution neural network (like CNN described with respect to FIGs. 3A and 3B) that is previously trained to identify features from sets of patches generated from WSIs. The feature extraction model outputs the identified features from a single patch into a data structure, such as a multi-dimensional array (e.g., a tensor). In other words, a patch tensor is output that comprises all the extracted features (e.g., biological / pathological features and / or edges, comers, and other patterns detected in the patch) identified by the feature extraction model for that specific patch. The extracted features can be represented numerically within the tensor, where the numeric value can correspond to one (or more) extracted feature. The tensors may be of various sizes, such as about 100, about 500, about 1,000 about 2,000, about 2,500, or more in size. Additionally or alternatively, the extracted features may include biomarkers (e.g., cell biomarkers) indicating the cells present in the one or more WSIs / patches. The feature extractor outputs the features extracted from a single patch as a tensor. The feature extraction model may also be a transformer model, an autoencoder model, or any other appropriate model. In some embodiments, the feature extraction model is a convolution neural network, and each set of the extracted features is stored in a tensor having a size of at least 400.

[0225] Importantly, the feature extraction model can identify features from the bags of patches even if augmentations have been applied to the patches. For example, augmentations may include using patches by transforming the patches by rotating, flipping, scaling, or altering the lighting / color conditions, as non-limiting examples.

[0226] At box 1015, the tensors are input into an embedding aggregator comprising a selfattention mechanism and / or an attention mechanism. The self-attention mechanism takes the tensor for each patch in the sets of patches as input and generates a set of attention scores and may also generate transformed features corresponding to each patch. The attention mechanism takes the tensor or the transformed features for each patch as input and generates another set of attention scores for the patch. The embedding aggregator may comprise an attention mechanism, a self-attention mechanism, or an attention-transformer mechanism. In certain embodiments, the aggregator is configured such that any patch in a set of patches is able to attend to all other patches in the set, supporting global context modeling. The embedding aggregator may beimplemented as an embedding aggregator architecture, enabling flexible aggregation of patch features across the entire WSI.

[0227] At box 1020, attention-weighted feature vectors are generated using the embedding aggregator. To generate them, the self-attention mechanism and / or the attention mechanism may be used. The self-attention mechanism generates the set of attention score and / or transformed feature vector corresponding to each patch by inputting the tensor into an embedding layer of the self-attention mechanism. The embedding layer generates embedding vectors (e.g., query vector (q), key vector (k), and value vector (v)) using, e.g., fully connected layers for each tensor. To determine the set of attention scores for each tensor against every other tensor, a dot-product may be calculated between the query of the tensor currently being processed, and the key of every other tensor (including itself). The set of attention scores can be normalized and converted into probabilities using the SoftMax function prior to being multiplied by their corresponding value vector to generate attention weighted value vectors. The attention weighted value vectors are aggregated into a single vector representation for the current tensor being processed to output the transformed features for each patch in the set of patches. This process is repeated until each patch tensor has its own corresponding transformed feature vector. Each aggregated vector represents a summary of how the features in the current tensor being processed interacts with the features in every other tensor.

[0228] The attention mechanism may take the tensor or the transformed features for each patch as input and generates another set of attention scores for the patch. The other set of attention scores can be normalized and converted into probabilities using the SoftMax function. The normalized attention scores may be multiplied to its corresponding tensor (e.g., the original patch tensor output from the extraction model) to generate attention-weighted feature vectors.

[0229] At box 1025, the attention-weighted feature vectors are aggregated into a single vector representation that represents a context-aware combination of all the attention weighted feature vectors. The single vector representation is used to generate a bag feature that is a weighted summary of all the features stored in the tensors.

[0230] At box 1030, the bag features are input into a classification model for predicting bag labels. The classification model may be a support vector machine (SVM), a decision tree, a random forest model, logistic regression model, a neural network, an attention neural network, ora k-nearest neighbor model. In various embodiments, the classification model is a neural network. In various embodiments, the classification model and / or the embedding aggregator may be trained using multiple instance learning (MIL) techniques, which is a type of supervised learning where a set of labeled “bags” containing unlabeled instances are used for training as opposed to a set of individually labeled instances. The bags of patches may be passed through the fully connected layers of the classification model to generate a label prediction for the bag (e.g., tumor or non-tumor). Training the classification model and / or the embedding aggregator involves iteratively calculating an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.) based on the bag-level prediction and the ground truth bag label (e.g., the global label from the WSI the sets of patches are generated from), where the goal is to minimize the difference between the two values. In some instances, backpropagation may be used to update the model parameters, including those of the attention mechanism, to minimize the loss. Because attention-based weighted features are used, the classification model learns which features are most important for classifying a bag as tumor positive or tumor negative. Model performance during the training is evaluated based on performance metrics such as accuracy, precision, recall, and Fl -score at the bag level.

[0231] At box 1035, the trained classification model is output and may be deployed into an environment (e.g., a computational, production, and the like) where it can make inferences on real-world data. For example, the trained model is used to determine whether patches from whole slide images comprise tumor tissue or not based on highly relevant features (e.g., patch-level predictions). The environment may be a clinical or research lab that is part of a medical institute that performs histology. The trained model enhances data analysis, improves efficiency and speed with which pathologists are able to process slides, and reduces the cost associated with maintaining a pathology lab. For example, a trained pathologist can take up to 15 minutes to provide patch-level annotations for a WSI, while the Al assisted method described herein can provide patch-level annotations for a WSI in only 1-2 minutes.

[0232] After training, the trained classification model may be deployed for inference in a production environment. For inference, a set of feature tensors for patches from a new WSI is input into the trained model. Patch-level predictions for the target class are generated allowing model to give granular level prediction. The attention score can also be calculated for each patch,providing insight into which regions the model focus most. Patch-level predictions or attentions can be visualized as heatmaps, binary masks, or overlays, supporting transparent clinical review, audit, and quality control. These outputs enable users to directly interpret model predictions and understand which regions of the WSI contributed most to a given classification. By directly calculating attention for each patch and supporting patch-level outputs, the approach improves granularity, interpretability, and computational efficiency, and is applicable to both disease-associated and non-disease-associated features.

[0233] The trained model, once deployed, is continuously monitored to ensure it performs as expected. This involves tracking the model’s prediction accuracy, response times, and other operational metrics. Additionally, the trained model may require retraining or updates based on new data or changing conditions in the environment in which it is applied. This can be useful because trained models can drift over time due to changes in the underlying data the models are making predictions on. The described training and inference processes provide a flexible and efficient framework for digital pathology analysis, supporting classification of a wide range of target classes, including both disease and artifact categories, with bag-level and patch-level outputs. The ability to generate, store, and visualize patch-level predictions enhances the clinical utility and transparency of the system, while the design of the embedding aggregator and attention mechanisms supports global context modeling and improved computational resource use.IX. DEPLOYMENT OF A MACHINE LEARNING MODEL INTO A PRODUCTION ENVIRONMENT

[0234] FIG. 11 is a flowchart illustrating process 1100 for using a trained machine learning model in a deployed environment to make inference phase predictions. The processing depicted in FIG. 11 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof (e.g., the intelligent selection machine). The software may be stored on a non-transitory storage medium (e.g., on a memory device). The method presented in FIG. 11 and described below is intended to be illustrative and non-limiting. Although FIG. 11 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some differentorders, or some steps may also be performed in parallel. In some embodiments, such as the embodiments depicted in FIG. 2, FIGs. 3A and 3B, FIGs. 4A and 4B, FIG. 6, FIG. 7, and FIGa.9A and 9B, the processing depicted in FIG. 11 may be performed by the components of the machine learning pipeline 200 described with respect to FIG. 2, of the CNN described with respect to FIGs. 3A and 3B, of the attention and / or self-attention architectures described with respect to FIGs. 4A and 4B, of the MIL training technique described with respect to FIG. 6, of the AB MIL framework described with respect to FIG. 7, and of the SABMIL or SA-AbMILP framework described with respect to FIG. 9B.

[0235] At box 1105, one or more sets of patches are obtained from a WSI of a specimen associated with a target class, including both disease-associated features and non-disease-associated features. The target class may comprise, for example, tumor, necrosis, fibrosis, abnormal cell morphology, artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content. For example, a WSI may also be labeled for the presence or absence of a non-disease-associated feature, such as an artifact or background region, depending on the intended application. This flexible approach enables use in both diagnostic and quality assurance workflows.. In some embodiments, one set of patches is obtained from the WSI and the set comprises at least 50 patches. In some embodiments, multiple sets of patches may be obtained from the WSI and each set comprises at least 50 patches. The number of patches obtained from the WSI can range between about 50 patches to about 400 patches. In some embodiments, the patch number in one or more sets is greater than 400.

[0236] The WSI can be from an unstained specimen, or generated using a staining system (e.g., staining system 125 described with respect to FIG. 1). For example, slides of the specimen are obtained, wherein the slides are stained using a histological staining technique to enhance contrast and visualize specific cellular components. Examples of staining methods that may be used include, without limitation, Hematoxylin and Eosin (H&E), Masson’s Trichrome, Periodic Acid-Schiff (PAS), silver stains, such as Gomori’s Methenamine Silver (GMS), Reticulin stains, Prussian Blue stain, immunohistochemical stains, immuno-fluorescent staining, multiplexing fluorescent staining, and the like. In some embodiments, the histological staining technique is H&E. The pathological features associated with the disease may also be examined. For example,the examining the slides may comprise (i) identifying one or more regions of interest within the slides; (ii) segmenting the slides into regions associated with a plurality of tissue types or pathological features; (iii) detecting and quantifying morphological features selected from the group consisting of: cell nuclei, cell membranes, cytoplasm, extracellular matrix, tissue architecture, or vascular structures; (iv) detecting and classifying the pathological features selected from the group consisting of: tumor regions, necrotic regions, fibrotic regions, areas of abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or infiltrating immune cells; (v) detecting and classifying non-disease-associated features selected from the group consisting of: artifacts, air bubbles, tissue folds, staining irregularities, debris, background regions, normal tissue structures, or regions lacking diagnostic content; (vi) quantifying an extent or severity of the pathological features; and / or (vii) generating a statistical report on a distribution, frequency, or area of the pathological features within the slides.

[0237] The specimen in the WS1, may comprise cells, tissue, or biological fluid. More specifically, the specimen is a tissue section collected from a larger tissue sample from a subject suspected of / diagnosed with / being treated for a disease. In the event the subject is suspected of having the disease, the specimen can be entirely healthy (e.g., no disease) indicating the subject does not have the disease or the specimen can comprise diseased tissue. Moreover, the specimen can show pathological features associated with a different disease other than the disease being tested for. By way of example, the disease being tested for may be a cancer, but the specimen may show pathological features associated with an infectious disease as opposed to cancer. In various embodiments, the disease is a pancreatic, kidney, renal, pelvic, colorectal, stomach, thymic, head and neck, mesothelial, prostate, cervical, thyroid, adrenal, testicular, breast, uterine, bone, esophageal, lung, liver, bile duct, ovarian, bladder, nervous system, or skin cancer. In some instances, the specimen comprises disease negative regions (e.g., non-tumor regions), disease positive regions (e.g., tumor regions), or any combination thereof. In some instances, regions corresponding to the disease negative and disease positive regions are not annotated on the WS1.

[0238] At box 1110, features within each patch of the one or more sets of patches are extracted using a feature extraction model, such as a CNN (see FIGs. 3A and 3B) and stored in a multidimensional array (e.g., a tensor). In other words, each patch in the one or more sets of patcheshas its own output tensor that comprises all the identified features from the feature extraction model for that patch. The extracted features stored in the one or more sets of patches are related to biological / pathological features and / or edges, comers, and other patterns detected in the patch. Moreover, the tensor may be of various sizes, such as about 100, about 500, about 1,000 about 2,000, about 2,500, or more in size. In some instances, each tensor has a size of at least 400. The extracted features may include biomarkers (e.g., cell biomarkers) indicating the cells present in the one or more WSIs / patches. The extracted features may also relate to non-disease-associated features, such as artifacts or background, enabling the model to support a broader range of classification tasks.

[0239] At box 1115, each tensor is input into a trained classification model to generate a patchlevel prediction for each patch in the one or more sets of patches. The trained classification model may be a support vector machine, a decision tree, a random forest model, logistic regression model, a neural network, an attention neural network, or a k-nearest neighbor model. In some instances, the classification model was trained on training data of at least 1,000 sets of patches. In some instances, each set has at least 50 patches. The sets of patches may be generated from different WSIs with a ground truth label. In some embodiments, the classification model was trained using multiple instance learning techniques. The trained classification model may output patch-level predictions for any target class.

[0240] The patch-level prediction may be generated based on a predicted probability for the patch from the trained classification model and the predicted probability is generated based on attention weighted features. In other words, the trained classification model was trained on training data comprising attention-weighted features generated from an attention mechanism, a self-attention mechanism, or both. During inference, however, the attention mechanism is not used to compute patch-level predictions; the trained classification model is applied directly to each patch's feature tensor to generate the patch-level prediction, without re-computing attention weights or aggregations.

[0241] At box 1120, the patch-level prediction for each patch in the one or more sets of patches are output. The patch- level prediction is disease positive (e.g., tumor) when the predicted probability of the patch is outside of a predetermined threshold and disease negative (e.g., nontumor) when the predicted probability of the patch is within a threshold. The predictedprobabilities for the set of patches are stored in an array representation of the original WSI. The patch-level predictions may be used to generate various visual images to aid pathologists in analysis and / or diagnosis of the WSI. In some embodiments, the patches classified as either disease positive or disease negative based on whether their predicted probably was within or outside a threshold are used to generate a binary mask of the original WSI. The binary mask may be output as a two-color image where one color corresponds to disease positive regions (e.g., tumor regions) and the other color corresponds to disease negative regions (e.g., non-tumor regions). As such, when pathologists review the binary mask, they can see where in the globally labeled WSI the model predicted tumor tissue versus non-tumor tissue. In some embodiments, the predicted probabilities for each patch in the set of patches are used to generate a heatmap representation of the original WSI. The heatmap can clearly indicate patches / areas that the trained classification model predicted to have high or low tumor probability, and intermediate values where the model predictions are not as clear or the model predicted the patch to be both tumor and non-tumor (e.g., when the predicted probabilities for tumor and non-tumor are about 50% each). Patch-level predictions and corresponding visualizations may also be generated for non-disease-associated target classes, such as artifacts, background, or normal tissue, allowing flexible use of the system in both diagnostic and quality assurance settings.

[0242] The patch-level predictions and / or the various visual images (e.g., binary mask and heatmap of the original WSI) can be used to aid pathologists to generate a report comprising an analysis and / or diagnosis of the WSI. Based on the analysis and / or diagnosis made by the pathologist, a clinician or qualified doctor can decide on how to proceed with patient treatment. For example, in an instance where a subject is being tested for a type of cancer, the report could indicate that all the patch-level predictions are tumor negative and by default the WSI is tumor negative, indicating the subject likely does not have cancer. If the predicted results indicate that a portion of the patch-level predictions are tumor positive, a pathologist may optionally confirm this result by comparing the model predictions to the original WSI and if the pathologist confirms the result the patient is likely to be diagnosed with cancer. In some instances, based on the proportion of patch-level predictions that are tumor positive, pathologist may be able to indicate how aggressive the cancer is, where a high proportion indicates a more aggressive cancer, and a lower proportion indicates a less aggressive or benign cancer. In quality assurance workflows, the report may indicate the detection, location, and extent of artifacts or other non-disease-associated features, which can be used for slide review, process improvement, or exclusion from further diagnostic analysis.

[0243] In some embodiments, the trained machine learning model may output labels indicating the degree of severity of a feature. For example, predicted label from patches may include “low-grade tumors” or “high-grade tumors.” Another example includes labels such as “tumor necrosis grade 0”, “tumor necrosis grade 1,” or “tumor necrosis grade 2,” where grade 0 indicates no dying tissue, grade 1 indicates less than 50% of the cancer is dying tissue, and grade 2 indicates more greater than or equal to 50% of the cancer is dying tissue. As such, the report could further indicate the subject shows tumor positive tissue along with the severity of the tumor. Similarly, for non-disease-associated features, the system may provide quantitative or categorical assessments of artifact severity, frequency, or spatial distribution.

[0244] The report may also indicate whether a subject is responding to a treatment regime and if the treatment should be altered or remain the same. For example, before starting treatment, the trained machine learning model may be used to aid pathologists to generate a baseline or initial histological status report for a subject. After one or more rounds of treatment, new samples may be processed by the trained machine learning and reviewed by pathologists, and the final pathologist diagnosis may indicate a worsening of the disease, no change in the disease, or improvement in the disease. A report indicating a worsening of the disease shows that from the initial test to the subsequent test, more patch-level predictions were indicated as tumor positive and / or a more sever tumor grades was observed and confirmed by pathologists. In this instance, the clinician may alter the treatment regime. A report indicating no change in disease state or progression shows that the number of patch-level predictions as tumor positive confirmed by pathologists did not substantially change from baseline to the subsequent visit. In this instance, the clinician may or may not alter the treatment regime and decide to optionally perform additional testing. A report reviewed by pathologists indicating an improvement in disease state shows that the number of patch-level predictions indicated as tumor positive and / or less sever tumor grades is less than the number of patches previously reported in the initial report. In this instance, the clinician may choose to keep the treatment regime the same. The described approach enables longitudinal monitoring and flexible reporting for both disease progression andquality assurance metrics, leveraging the granularity and interpretability of patch-level predictions for any target class.X. EXAMPLES

[0245] The examples below are intended to further illustrate certain aspects of the methods and compositions described herein and are not intended to limit the scope of the claims.Datasets

[0246] WSIs can be obtained from various sources including publicly available WSI databases, commercial databases, or private (e.g., inhouse) databases. WSIs may cover a single tissue type or multiple tissue types. WSIs may be obtained from normal (healthy) subjects and subjects with known tumor. The tissue of origin of tumor may also vary (e.g., bladder, breast, lung, etc.) The WSIs may also be captured at different magnifications (e.g., 20X or 40X). WSIs may be fully annotated (a label with each pixel or voxel) by a practitioner, or with fewer labels (e.g., one label per WSI).Preprocessing

[0247] Described below were the preprocessing steps applied to WSIs before they were fed into models.

[0248] Patch Tiling-. The size of WSIs, gigapixels usually, cannot be used directly as input to a model for computational limitation. This is why WSI needs to be tiled to patches as shown in FIG. 12. After obtaining a WSI 1210, a patch tiling process can be applied to divide the WSI 1210 into patches (e.g., patch 1215). The size of each patch can be predetermined based on clinical needs or computational limitation. In some instances, a tissue segmentation process can be performed to identify tissue patches (red patches) and non-tissue / background patches (diagonally erased patches, e.g., patch 1220). The non-tissue patches may be removed prior to the training or prediction process.

[0249] Random Patch Selection-. During the training of MIL models, for each epoch and each WSI, a random number of patches (e.g., a bag size B) may be chosen. Unlike typical DNN training where the training data has to have a fixed size, the present techniques allow the input data having various size. For example, a different B may be chosen for each epoch in training. Insome instances, the number of patches used for the same WSI may vary. This random patch selection technique maximizes the patch diversity that MIL models may see.

[0250] Tissue Segmentation'. Removing patches not containing tissue can considerably reduce the training or inference time for model use. An example of this segmentation is shown in FIG.12. A simple gray filter (e.g., with a threshold of 50 (over 256)) may be applied to identify the tissue area. The threshold can be chosen arbitrary and could be changed. A patch is considered as tissue if at least one pixel of the patch is considered as tissue with this threshold. Other tissue segmentation method (e.g., using Otsu threshold) may also be used.

[0251] Image Augmentation'. Images (WSIs or patches) may be augmented before further processing. A different combination of the following image augmentation may be applied to patches during the training: random flip, rotate, brightness, contrast, saturation, and hue augmentation. Exemplary values for image augmentation are shown in Table 1.Table 1Methods ValueRandom brightness [0.75, 1.25]Random contrast [0.25, 1.75]Random saturation [0.75, 1.25]Random hue [-0.04. 0.04]Random horizontal flip probability: 0.5Random vertical flip probability: 0.5Random rotation angle: 0, 90, 180, 270

[0252] Normalization-. Pixel values may be normalized before training or further processing. For example, normalization can be performed using mean pixel values (e.g., [0.485, 0.456, 0.406]) and the standard deviation of the pixel values (e.g., STD = [0.229, 0.224, 0.225]) to each patch (or augmented patch) before being input into different feature extractors. These values fornormalization may be calculated with statistics of ImageNet Dataset. StainNet may also be used to normalize each patch in addition to applying previous ImageNet normalization.Training Data Balancing

[0253] Training data may be adjusted to balance the type of data the model can see. For example, databases may store more tumor WSIs than normal WSIs. In order to minimize the imbalance between training data, a predetermined ratio (e.g., 1:1) may be chosen to select WSIs with a tumor label and WSIs with a non-tumor label. The training data comprising the predetermined ratio of WSIs may be further divided into a training set (e.g., about 60-80% of total training data), a testing set (e.g., about 10-20%), and / or a validation set (e.g., about 10-20%).Fegture Extrgctor Models

[0254] Different Feature extractor models may be used to extract features from the patches. Provided in Table 2 is a list of the different CNN models used as feature extractors and the size of extracted features tensor per patch .Table 2Feature Extractor Model Size of extracted features tensorVGG16 4, 096ResNet34 512ResNet50 2,048EfficientNetV2 1,280Prediction gnd Model Compgrison

[0255] FIG. 13 illustrates a comparative evaluation of the disclosed framework for patch-level classification in digital pathology. The leftmost panel displays the ground truth annotation for a representative whole slide image, with regions positive for the target class (such as tumor) highlighted in solid red. Both the center and rightmost panels are heatmaps using a continuousprobability range [0,1] mapped to the same color bar, where red indicates high probability and blue indicates low probability for the target class. The center panel shows the patch-level prediction (e.g., probabilities) generated by the described model, where more intense red regions corresponds to a higher probability of belonging to the target class, while deeper blue regions indicate a lower probability. Compared to the rightmost panel, the prediction probabilities in the center panel are generally more distinct, with most regions being either strongly positive (red) or strongly negative (blue), and fewer intermediate (green) values. These prediction probabilities are the output of the trained classification model (e.g., after applying a SoftMax activation), and are used directly for patch-level interpretation and visualization. This demonstrates the model’s ability to provide high-resolution, spatially resolved outputs that closely align with expert annotations and support granular interpretation.

[0256] The rightmost panel presents the results from a conventional slide-level prediction approach, where the output is visualized as a continuous probability map across the slide. In this visualization, each region is color-coded according to its predicted probability for the target class, with the color bar ranging from blue (low probability) to red (high probability). When attention scores are used for such a heatmap, as in the right panel, there tends to be a greater proportion of intermediate values (green regions), making it harder to visually distinguish between positive and negative areas. This conventional approach, while offering an overall assessment of the slide, does not provide discrete patch-level interpretability or granularity.

[0257] In addition to prediction probabilities, attention scores may also be computed and visualized as continuous values on a [0,1] scale using a similar color mapping. However, during inference, only the prediction probabilities are used to classify patches and generate clinical interpretations. Attention scores are independent of the patch classification, and are provided optionally for interpretability or visualization. The key difference is that prediction probabilities yield clearer separation between positive and negative regions, resulting in more binary-like (red or blue) outputs, while attention scores often yield more intermediate values (green), reducing visual separability.

[0258] Beneath the panels, a summary table reports mean performance metrics for patch-level prediction and slide-level attention across 52 whole slide images. For the disclosed patch-level prediction, the area under the curve (AUC) is 0.86, Fl score at a threshold of 0.5 is 0.74,precision at 0.5 is 0.77, and sensitivity at 0.5 is 0.76. For the conventional slide-level attention scores, the AUC is 0.73, Fl score is 0.60, precision is 0.68, and sensitivity is 0.59. These results demonstrate that the patch-level prediction probabilities achieve higher accuracy, precision, and interpretability compared to attention scores, enabling more precise localization of relevant regions and supporting transparent clinical review and quality assurance.XI. ADDITIONAL CONSIDERATIONS

[0259] Implementation of the techniques, blocks, steps and means described above can be done in various ways. For example, these techniques, blocks, steps and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro- controllers, microprocessors, other electronic units designed to perform the functions described above, and / or a combination thereof.

[0260] Also, it is noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be rearranged. A process is terminated when its operations are completed, but could have additional steps not included in the figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0261] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting language, and / or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine-readable medium such as a storage medium. A code segment or machineexecutable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions,data structures, and / or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc.

[0262] For a firmware and / or software implementation, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example, software codes can be stored in a memory. Memory can be implemented within the processor or external to the processor. As used herein the term “memory” refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular type of memory or number of memories, or type of media upon which memory is stored.

[0263] Moreover, as disclosed herein, the term “storage medium,” “storage,” or “memory” can represent one or more memories for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other machine -readable mediums for storing information. The term "machine-readable medium" includes, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage mediums capable of storing that contain or carry instruction(s) and / or data.

[0264] While the principles of the disclosure have been described above in connection with specific apparatuses and methods, it is to be clearly understood that this description is made only by way of example and not as limitation on the scope of the disclosure.

Claims

WHAT IS CLAIMED IS:

1. A computer- implemented method comprising:obtaining, from a whole slide image (WSI) of a specimen, a set of patches; extracting, using a feature extraction model, a set of features from each patch in the set of patches;generating, using a classification model, a patch-level prediction for each patch in the set of patches, wherein the classification model outputs a predicted probability for each patch, and wherein the predicted probability is based on attention weighted features used to train the classification model; andoutputting the patch-level prediction for each patch, wherein:the patch-level prediction is positive for a target class when the predicted probability of the patch is outside of a predetermined threshold, andthe patch-level prediction is negative for the target class when the predicted probability of the patch is within the predetermined threshold.

2. The computer- implemented method of claim 1, wherein the target class comprises a disease-associated feature or a non-disease-associated feature.

3. The computer- implemented method of claim 2, wherein the disease-associated feature comprises at least one of: a tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells.

4. The computer-implemented method of claim 2, wherein the non-disease-associated feature comprises at least one of: an artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content.

5. The computer-implemented method of any one of claims 1-4, wherein the classification model is trainable with feature tensors corresponding to patches input to an embedding aggregator, the embedding aggregator comprising a self-attention mechanism and an attention mechanism, wherein attention scores are generated with respect to the feature tensorsand wherein model parameters are adjustable based on a comparison between predicted outputs and reference labels.

6. The computer- implemented method of claim 5, wherein the embedding aggregator is configured such that any patch in the set of patches is able to attend to all other patches in the set to support global context modeling.

7. The computer- implemented method of any one of claims 1-6, further comprising:obtaining slides of the specimen, wherein the slides are stained using a histological staining technique, andexamining the slides for pathological features associated with a disease.

8. The computer-implemented method of claim 7, wherein the histological staining technique is hematoxylin and eosin staining.

9. The computer-implemented method of claim 7, wherein examining the slides comprises:(i) identifying one or more regions of interest within the slides;(ii) segmenting the slides into regions associated with a plurality of tissue types or pathological features;(iii) detecting and quantifying morphological features selected from the group consisting of: cell nuclei, cell membranes, cytoplasm, extracellular matrix, tissue architecture, or vascular structures;(iv) detecting and classifying the pathological features selected from the group consisting of: tumor regions, necrotic regions, fibrotic regions, areas of abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or infiltrating immune cells;(v) detecting and classifying non-disease-associated features selected from the group consisting of: artifacts, air bubbles, tissue folds, staining irregularities, debris, background regions, normal tissue structures, or regions lacking diagnostic content;(vi) quantifying an extent or severity of the pathological features; and / or (vii) generating a statistical report on a distribution, frequency, or area of the pathological features within the slides.

10. The computer- implemented method of any one of claims 1-9, wherein the specimen comprises cells, tissue, or biological fluid.

11. The computer- implemented method of any one of claims 1-9, wherein the specimen is a tissue sample.

12. The computer-implemented method of any one of claims 1-11, wherein the specimen is collected from a tissue sample suspected of disease positive.

13. The computer- implemented method of any one of claims 1-12, where the target class is a cancer-positive class.

14. The computer- implemented method of claim 13, where the cancer is a pancreatic, kidney, renal, pelvic, colorectal, stomach, thymic, head and neck, mesothelial, prostate, cervical, thyroid, adrenal, testicular, breast, uterine, bone, esophageal, lung, liver, bile duct, ovarian, bladder, nervous system, or skin cancer.

15. The computer-implemented method of any of claims 1-14, wherein the set of patches comprises at least 50 patches from the WSI.

16. The computer- implemented method of any of claims 1-15, wherein the feature extraction model is a convolution neural network.

17. The computer- implemented method of any of claims 1-16, wherein the set of features is related to pathological features, edges, comers, patterns detected in the patch, or any combination thereof associated with a disease negative specimen, a disease positive specimen, or both.

18. The computer-implemented method of any of claims 1-17, wherein the set of features is stored in a specific data structure.

19. The computer-implemented method of claim 18, wherein the specific data structure is a tensor having a size of at least 400.

20. The computer- implemented method of any of claims 1-19, wherein the classification model is a support vector machine, a decision tree, a random forest model, logistic regression model, a neural network, an attention-based neural network, or a k-nearest neighbor model.

21. The computer- implemented method of any of claims 1-20, wherein the classification model was trained on training data comprising attention-weighted features generated from an attention mechanism, a self-attention mechanism, or both.

22. The computer- implemented method of claim 21 , wherein the training data comprises at least 1,000 sets of patches, wherein each set of patches is generated from a unique WSI with a global label.

23. The computer- implemented method of any of claims 1-22, wherein the classification model was trained using multiple instance learning (MIL) techniques.

24. The computer- implemented method of any of claims 1-23, wherein the predicted probabilities for the set of patches are stored in an array representation of the WSI.

25. The computer-implemented method of any of claims 1-24, further comprises generating, using the patch-level prediction for each patch in the set of patches, a binary mask of the WSI, wherein the binary mask indicates regions in the WSI that are predicted to be either positive or negative for the target class.

26. The computer- implemented method of any of claims 1-25, further comprises generating, using the predicted probability for each patch in the set of patches, a heatmap of the WSI.

27. The computer- implemented method of any of claims 1-26, further comprising generating a slide-level prediction for the whole slide image based on an aggregation of the patch-level predictions.

28. The computer-implemented method of any of claims 1-27, further comprising displaying, on a graphical user interface, the WSI with the patch-level predictions overlayed on the WSI.

29. A computer-implemented method comprising:obtaining, from a plurality of whole slide images (WSIs), sets of patches, wherein each set of patches corresponds to a WSI of the plurality of WSIs, and wherein each WSI is labeled with a target class;extracting, using a feature extraction model, a set of features for each patch in the sets of patches;inputting the sets of the extracted features for the sets of patches into an aggregator, wherein the aggregator comprises: (i) an attention mechanism that takes a set of extracted features as input and generates a first set of attention scores, (ii) a self-attention mechanism that takes a set of extracted features as input and generates a second set of attentionscores, and / or (iii) an attention-transformer mechanism that takes transformed features of a set of extracted features as input and generates a third set of attention scores;generating, by taking the product of one or more attention scores and corresponding set of extracted features for each patch in the set of patches, attention-weighted feature vectors;aggregating the attention-weighted feature vectors into a single representation to generate bag features;generating, by a classification model using the bag features, a predicted bag label for each bag, wherein each bag corresponds to a set of patches of the sets of patches, and wherein each bag corresponds to the set of patches from a single WSI and a ground truth label for each bag indicates a presence or absence of the target class in the corresponding WSI;performing iterative operations to adjust model parameters of the aggregator and / or the classification model to minimize a loss or error function, wherein the loss or error function is configured to measure a difference between the predicted bag label and a ground truth label for the target class; andoutputting the aggregator and / or the classification model to be deployed into a production environment.

30. The computer- implemented method of claim 29, wherein the target class comprises a disease-associated feature or a non-disease-associated feature.

31. The computer-implemented method of claim 30, wherein the disease-associated feature comprises at least one of: a tumor, necrosis, fibrosis, abnormal cell morphology, abnormal tissue architecture, abnormal biomarker expression, or presence of infiltrating immune cells.

32. The computer-implemented method of claim 30, wherein the non-disease-associated feature comprises at least one of: an artifact, air bubble, tissue fold, staining irregularity, debris, background region, normal tissue structure, or region lacking diagnostic content.

33. The computer-implemented method of any one of claims 29-32, wherein each set of extracted features is stored in a specific data structure.

34. The computer-implemented method of claim 33, wherein the specific data structure is a tensor having a size of at least 400.

35. The computer-implemented method of any one of claims 29-34, wherein the plurality of WSIs comprises at least 1,000 WSIs.

36. The computer-implemented method of any one of claims 29-35, wherein the feature extraction model is a convolution neural network.

37. The computer-implemented method of any one of claims 29-36, wherein each set of the extracted features is stored in a tensor having a size of at least 400.

38. The computer-implemented method of any one of claims 29-37, wherein the self-attention mechanism in (ii) generates the transformed features of each set of extracted features.

39. The computer-implemented method of any one of claims 29-38, wherein each bag includes a different number of patches.

40. The computer-implemented method of any one of claims 29-39, wherein each bag comprises at least 50 patches generated from a single WSI.

41. The computer- implemented method of any one of claims 29-40, wherein the aggregator is an embedding aggregator and / or the aggregator is configured such that any patch in a set of patches is able to attend to all other patches in the same set.

42. A system comprising:one or more processors; andone or more computer-readable media storing instructions which, when executed by the one or more processors, cause the system to perform part or all of the operations disclosed in any one of claims 1-41.

43. One or more non-transitory computer-readable media storing instructions which, when executed by one or more processors, cause a system to part or all of the operations disclosed in any one of claims 1-41.

Images