Systems and methods for immune-guided ai for reproducible regions of interest selection in digital pathology images

Abstract

A system includes one or more processors. The one or more processors can perform automated immune scoring (alS) on a first set of grids of digital pathology images; can extract, using one or more neural networks, features of the first set of grids based on automated immune scores; can extract, using the one or more neural networks, features of a second set of grids of the digital pathology images based on manual immune scores; can perform similarity matching on the features of the first set of grids and the features of the second set of grids; can select a subset of grids for further analysis based on the similarity matching; and can predict one or more regions of interest (ROIs) in the subset of grids.

Description

Atty. Dkt. No.: 642631-0109SYSTEMS AND METHODS FOR IMMUNE-GUIDED Al FOR REPRODUCIBLE REGIONS OF INTEREST SELECTION IN DIGITAL PATHOLOGY IMAGES CROSS-REFERENCE TO RELATED PATENT APPLICATIONS

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application No.: 63 / 750,036 filed January 27, 2025, the entirety of which is incorporated herein by reference.TECHNICAL FIELD

[0002] The present disclosure relates to systems and methods of artificial intelligence (Al) guided selection of reproducible regions of interest (ROI) in digital pathology images. Specifically, it involves the use of digital pathology images and neural networks to accurately estimate immune infiltration and precisely delineate ROIs. This technology is particularly relevant in the context of digital pathology, where accurate and reproducible ROI selection is crucial for diagnostic and research purposes.BACKGROUND

[0003] The digitization of whole slide images (WSI) of histological specimens has revolutionized histological practice enabling downstream analysis such as spatial molecular profiling. However, the selection of regions of interest (ROIs) to perform these molecular assays remains a challenge due to the intrinsic inefficiency and variability of currently in place manual approaches. Manual selection of ROIs is labor-intensive and time-consuming. Lack of consistency in manually selecting ROIs sprawls to studies’ reproducibility, questioning generalization, and hindering scalability for clinical impact.SUMMARY

[0004] The present technology provides systems and methods for Al for reproducible ROI selection in digital pathology images. The technology addresses several unmet needs in digital pathology by eliminating subjective variability in ROI selection, providing a standardized and reproducible method. It reduces the manual, time-consuming process, allowing pathologists to focus more on other tasks. Additionally, it offers precise quantification in complex cases, aiding better treatment decisions. Designed for flexibility, the technology can adapt to various markers and integrate with existing platforms,-1- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109addressing the limitations of current solutions.

[0005] At least one aspect relates to a system. The system can include one or more processors and a memory storing computer instructions. The computer instructions, when executed by the one or more processors can cause the system to perform automated immune scoring (alS) on a first set of girds of digital pathology images; extract features from the first set of grids based on automated immune scores using neural networks; extract features from a second set of grids of the digital pathology images based on manual immune scores (mIS) using neural networks; perform similarity matching on the features of the first set of grids and the features of the second set of grids; select a subset of grids from the first set of grids based on the similarity matching; and predict one or more regions of interest (ROIs) in the subset of grids.

[0006] In some implementations, the digital pathology images comprise brightfield-based images. In some implementations, the one or more processors are to generate virtual multiplex immunofluorescent (vmlF) images from H&E images. In some implementations, the digital pathology images comprise fluorescence-based images. In some implementations, the one or more processors are to calculate a feature similarity score based on the features of the first set of grids and the features of the second set of grids. In some implementations, the one or more processors are to select the subset of grids as a subset of grids with a highest feature similarity score. In some implementations, the feature similarity score is calculated using cosine similarity. In some implementations, the one or more processors can cause the system to perform a sliding window method to predict the one or more ROIs. The sliding window method can include determining a size of a window, sliding the window across each grid of the subset of grids, performing alS at each window position, and selecting the window position with an automated immune score closest to an automated immune score of the entire grid as an ROI.

[0007] Another aspect relates to a system. The system can include one or more processors and a memory storing computer instructions. The computer instructions, when executed by the one or more processors can cause the system to acquire multiplex immunofluorescence (mIF) images or generate virtual mIF (vmlF) images; perform automated immune scoring (alS) on a first set of girds of the mIF images or the vmlF images; extract features from the first set of grids based on automated immune scores using neural networks; extract features from a second set of grids based on manual immune scores (mIS) using neural networks;-2- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109perform similarity matching on the features of the first set of grids and the features of the second set of grids; select a subset of grids from the first set of grids based on the similarity matching; and predict one or more regions of interest (ROIs) in the subset of grids.

[0008] In some implementations, the one or more processors can perform alS based at least on classifying pixels within the first set of grids as immune based on immune attributes and quantifying immune cell infiltration of the pixels within the first set of grids based on a color range determined by cellular features. In some implementations, the one or more processors can preprocess the mIF images using a contour mask to distinguish between glass and tissue regions. In some implementations, the one or more processors can perform similarity matching by comparing the features of the first set of grids to the features of the second set of grids. The one or more processors can calculate a feature similarity score based on the features of the first set of grids and the features of the second set of grids. Spearman’s correlation can be used for comparing the features of the first set of grids to the features of the second set of grids. Cohen’s K score can be used for comparing the features of the first set of grids to the features of the second set of grids. In some implementations, the feature similarity score can be calculated using cosine similarity. In some implementations, the subset of grids selected by the one or more processors can comprise a subset of grids with a highest feature similarity score. In some implementations, the one or more processors can cause the system to perform a sliding window method to predict the one or more ROIs. The sliding window method can include determining a size of a window, sliding the window across each grid of the subset of grids, performing alS at each window position, and selecting the window position with an automated immune score closest to an automated immune score of the entire grid as an ROI. In some implementations, the one or more neural networks is a custom trained convolutional neural network (CNN) model. The CNN model can be trained using a cross-fold validation technique.

[0009] Another aspect relates to a method. The method can include at least one of acquiring multiplex immunofluorescence (mIF) images of tissue samples or generating virtual mIF (vmlF) images of tissue samples; performing automated immune scoring (alS) on a first set of grids of the mIF images or the vmlF images; extracting, using one or more neural networks, features of the first set of grids based on automated immune scores; extracting, using one or more neural networks, features of a second set of grids based on manual immune scores; performing similarity matching on the features of the first set of-3- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109grids and the features of the second set of grids; selecting a subset of grids from the first set of grids further analysis based on the similarity matching; and predicting one or more regions of interest (ROIs).

[0010] In some implementations, performing automated immune scoring (alS) can include classifying pixels within the first set of grids as immune based on immune attributes and quantifying immune cell infiltration of the pixels within the first set of grids based on a color range. In some implementations, the second set of grids can be received as input via a user interface. In some implementations, the method can include preprocessing the mIF images using a contour mask to distinguish between glass and tissue regions. In some implementations, similarity matching can include calculating a feature similarity score between the features of the first set of grids and the features of the second set of grids. In some implementations, the feature similarity score can be calculated using cosine similarity. In some implementations, selecting the subset of grids from the first set of grids can comprise selecting a number of grids with a highest feature similarity score. In some implementations, predicting one or more ROIs can comprise using a sliding window method on the subset of grids. In some implementations, the sliding window method can comprise determining a size of a window based on a target ROI size and, for each grid of the subset of grids, sliding the window across the grid, performing alS at each window position, and selecting the window position with an automated immune score closest to the immune score of an entire grid as an ROI. In some implementations, generating vmlF images of tissue samples can comprise acquiring mIF images of the tissue samples, restaining the mIF images with hematoxylin & eosin (H&E) to obtain H&E images, spatially aligning the H&E images with the mIF images, and providing the H&E images and the mIF images to a trained convolutional neural network (CNN) model.

[0011] Another aspect relates to a method. The method can include acquiring hematoxylin & eosin (H&E) images of tissue samples; dividing the H&E images into a plurality of tiles using one or more neural networks; extracting features from each tile of the plurality of tiles using one or more neural networks; assigning a probability to each tile of the plurality of tiles based on the extracted features; and predicting one or more regions of interest (ROIs) based on the probability assigned to each tile.

[0012] In some implementations, assigning the probability to each tile of the plurality of tiles can include providing a trained neural network with the features extracted from each-4- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109tile and comparing, using the trained neural network, the features extracted from each tile to features extracted from manually identified ROIs. In some implementations, predicting one or more ROIs can include a sliding window method. In some implementations, the sliding window method includes determining a size of a window based on a target ROI size, overlaying the window onto the H&E images, sliding the window across a plurality of window positions, and, for each window position, calculating an average probability and selecting the window position with a highest average probability as an ROI.

[0013] At least one aspect relates to a non-transitory computer-readable medium, which can include machine-readable instructions to cause one or more processors to acquire multiplex immunofluorescence (mIF) images of tissue samples; perform automated immune scoring (alS) on a first set of grids of the mIF images; extract, using one or more neural networks, features of the first set of grids based on automated immune scores; extract, using the one or more neural networks, features of a second set of manually scored grids based on manual immune scores; compare, using a similarity matching, the features of the first set of grids to features of a second set of grids; select grids for further analysis based on the similarity matching; and predict one or more regions of interest (ROIs) based on the selected grids.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The disclosure will be readily understood by the following detailed description in conjunction with the accompanying drawings, wherein like reference numerals designate like elements, and in which:

[0015] FIG. 1 is a block diagram depicting an example system suitable for Al-guided ROI selection in digital pathology images, in accordance with one or more implementations;

[0016] FIG. 2 is a block diagram depicting an example system suitable for Al-guided ROI selection in digital pathology images, in accordance with one or more implementations;

[0017] FIG. 3 shows a flow diagram illustrating a method for selecting grids for analysis from mIF images, in accordance with one or more implementations;

[0018] FIG. 4 shows a flow diagram illustrating a method for quantifying immune cell infiltration using a sliding window, in accordance with one or more implementations;-5- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109

[0019] FIG. 5 shows a flow diagram illustrating a method for selecting ROIs, in accordance with one or more implementations;

[0020] FIG. 6 shows a flow diagram illustrating a method for extracting features of mIF images using a convolutional neural network, in accordance with one or more implementations;

[0021] FIG. 7 shows a flow diagram illustrating a method for training a model to select ROIs from H&E images, in accordance with one or more implementations;

[0022] FIG. 8 shows a flow diagram illustrating a method for selecting ROIs from H&E images, in accordance with one or more implementations;

[0023] FIG. 9 shows a flow diagram illustrating a method for selecting ROIs from H&E images, in accordance with one or more implementations;

[0024] FIG. 10 shows a flow diagram illustrating a method for selecting ROIs from H&E images, in accordance with one or more implementations;

[0025] FIG. 11 shows a flow diagram illustrating a sliding window technique for selecting ROIs, in accordance with one or more implementations;

[0026] FIG. 12 shows a flow diagram illustrating a method for selecting ROIs from vmlF images, in accordance with one or more implementations;

[0027] FIG. 13 shows a block diagram depicting an example computer system, in accordance with one or more implementations;

[0028] FIG. 14 shows a block diagram of an example computing system, in accordance with one or more implementations;

[0029] FIG. 15 shows an example of a study employing the method of FIG.8, in accordance with one or more implementations;

[0030] FIG. 16 shows an example of a chart of an ROI probability map , in accordance with one or more implementations;

[0031] FIG. 17 shows an example of a chart showing performance metrics, in accordance with one or more implementations;-6- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109

[0032] FIG. 18 shows an example of a study employing the method of FIG. 9, in accordance with one or more implementations;

[0033] FIG. 19 shows an example of charts showing performance metrics, in accordance with one or more implementations;

[0034] FIG. 20 shows an example of a study employing the method of FIG. 10, in accordance with one or more implementations;

[0035] FIG. 21 shows an example of a chart showing performance metrics, in accordance with one or more implementations; and

[0036] FIG. 22 shows an example of a study, in accordance with one or more implementations.DETAILED DESCRIPTION

[0037] Following below are more detailed descriptions of various concepts related to, and implementations of, systems and methods for Al-guided and / or immune-guided Al for reproducible ROI selection in digital pathology images. It should be appreciated that various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways, as the disclosed concepts are not limited to any particular manner of implementation. Examples of specific implementations and applications are provided primarily for illustrative purposes.I. Overview

[0038] The selection of regions of interest (ROIs) is a crucial step for accurate and reproducible analysis of tissue samples, particularly in the context of multiplex immunofluorescence (mIF) imaging used for pathological research and diagnostics. Since ROI selection is a critical early step in the analysis process, it is imperative that this selection is performed efficiently to ensure timely evaluations, such as cancer diagnosis and efficacy analysis of tumor treatments. Manual immune scoring by expert pathologists, while precise, is labor-intensive and lacks both efficiency and scalability. The variability in manual methodologies arises from differences in pathologists' experience, interpretation, and subjective judgment. This variability can lead to inconsistencies in research findings and diagnostic outcomes, such as tumor treatment evaluation. Automated methodologies, on-7- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109the other hand, leverage computational algorithms to analyze tissue samples, providing efficient, consistent, and scalable results in histology assessments of a subject. In automated processes, precision is essential in the nuanced understanding of tumor microenvironments, leading to better-targeted therapies and improved patient outcomes. Automated approaches for selecting ROIs can allow for streamlined tissue analysis using biomarkers; however, these approaches have not surpassed the capabilities of manual selection methods in terms of precision and versatility in evolving with the advancing needs of mIF imaging and analysis.

[0039] In the current disclosure, automated immune scoring (alS) is employed to quantify immune cell infiltration and classify pixels based on immune attributes, the results of which are used by one or more neural networks to identify ROIs for further analysis. Automated immune scoring involves segmenting images into sub-regions (e.g., grids) and calculating scores for each grid based on immune attributes. Once the grid scores are determined, implementations described herein involve extracting various image features of the grids using one or more neural networks and assessing features in terms of their correlations with a set of manually scored reference grids. Grids with optimal feature assessment can be selected and used with a sliding window approach to predict ROIs within the selected grids.

[0040] As used herein, the terms “cancer” or “tumor” are used interchangeably and refer to the presence of cells possessing characteristics typical of cancer-causing cells, such as uncontrolled proliferation, treatment resistance and immune-surveillance evasion, metastatic potential, rapid growth and proliferation rate, and certain characteristic morphological features. Cancer cells are often in the form of a tumor, but such cells can exist alone within an animal, or can be a non-tumorigenic cancer cell. As used herein, the term “cancer cells” includes precancerous (e.g., benign), malignant, pre-metastatic, metastatic, and non-metastatic cells. Cancers of virtually every tissue are known to those of skill in the art, including solid tumors such as carcinomas, sarcomas, glioblastomas, melanomas, etc., and circulating cancers such as leukemias. Examples of cancer include, but are not limited to, ovarian cancer, breast cancer, colon cancer, lung cancer, prostate cancer, gastric cancer, pancreatic cancer, cervical cancer, ovarian cancer, liver cancer, bladder cancer, cancer of the urinary tract, thyroid cancer, renal cancer, carcinoma, melanoma, head and neck cancer, and brain cancer. The phrase “cancer burden” or “tumor burden” refers to the quantity of cancer cells or tumor volume in a subject. Reducing cancer burden accordingly may refer to-8- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109reducing the number of cancer cells, or the tumor volume in a subject. The term “cancer cell” refers to a cell that exhibits cancer-like properties, e.g., uncontrollable reproduction, resistance to anti- growth signals, ability to metastasize, and loss of ability to undergo programmed cell death (e.g., apoptosis) or a cell that is derived from a cancer cell, e.g., clone of a cancer cell.II. System for Immune-Guided Al ROI Selection

[0041] FIG. 1 depicts an example of a system 100. The system 100 can be a classification and detection system, such as to perform functions including quantifying immune infiltration of digital pathology images and detecting or identifying regions of interest. The techniques described in the present disclosure may apply to enhancing the accuracy and reproducibility of regions of interest (ROI) selection in digital pathology images for carcinoma research and diagnostics. The techniques described herein can be used to process digital pathology images obtained from the acquisition of whole slide image (WSI) data, such as, brightfield-based images and / or fluorescence-based images. For example, and without limitation, the system 100 may be used to process multiplex immunofluorescence (mIF) images, Hematoxylin & Eosin (H&E) stained images, bioluminescence images, and / or confocal images. The system 100 can be particularly useful in the context of mIF images, virtual mIF images, and / or H&E stained images. Various aspects of the system 100 can be implemented using one or more components of the computing system 1400 described with reference to FIG. 14.

[0042] The system 100 can include or be coupled with data input and / or output devices, such as a user interface 110. The system 100 can receive inputs from the user interface 110 such as text, speech, audio, image, and / or video data indicative of information regarding a subject, such as information provided by one or more pathologists. For example, the system 100 can receive manual immune scoring (mIS) data and / or whole slide image (WSI) data. The system 100 can output results using the user interface 110. For example, upon selecting ROIs from digital pathology images, the selected ROIs can be output to one or more users with the user interface 110.

[0043] The system 100 can include at least one machine learning (ML) model 120, which can include any of various supervised and / or unsupervised machine learning models. The ML model 120 can include any one or more of decision trees, graph networks, random-9- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109forest models, Bayesian models, regressions, support vector machines, gradient-boosted trees, or various combinations thereof. For example, the ML model 120 can include one or more neural networks 115 useful for feature extraction from input data from data sources 105. The one or more neural networks 115, can include, for example and without limitation, any one or more artificial neural networks, deep learning networks, convolutional neural networks, recurrent neural networks; various such neural networks can be useful for realtime processing of data such as periodically detected and / or received data regarding tissue samples. The one or more neural networks 115 can include a plurality of nodes arranged in one or more layers, such as an input layer, an output layer, and / or one or more intermediate layers. The processor 140 can configure the one or more neural networks 115 by modifying or updating one or more parameters, such as weights and / or biases, of various nodes of the neural network responsive to evaluating outputs of the neural network.

[0044] The system 100 can include a similarity matcher 125. In some implementations, the one or more neural networks 115 can extract features from a first set of inputs, to obtain a first set of features, and the one or more neural networks 115 can extract features from a second set of inputs to obtain a second set of features. In some implementations, the set of inputs are sub-regions (e.g., grids, tiles) of the digital pathology images, as in the first set of inputs is a first set of grids and the second set of inputs is a second set of grids. The features can include, without limitation, texture, shape, color intensity, and spatial distribution. The extracted features can be represented as numerical values useful for similarity matching and ROI selection. One or more outputs of the ML model 120 can be provided as one or more inputs to the similarity matcher 125. For example, the first set of features and the second set of features can be input into the similarity matcher 125 to obtain one or more results by analyzing the first set of features and the second set of features. This can involve calculating a feature similarity score between the first set of features and the second set of features using various similarity measurement methods described further herein. These methods can evaluate the similarity between feature vectors, for example, and provide a measure of how closely the results from different scoring approaches align. The one or more results output by the similarity matcher 125 can represent the evaluated similarities between the first set of grids and the second set of grids.

[0045] The system 100 can include a grid selector 130 for identifying and selecting grids from a set of grids acquired from digital pathology data such as mIF images, virtual mIF-10- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109(vmlF) images, H&E images, and / or confocal images. One or more outputs of the similarity matcher 125 can be provided as one or more inputs to the grid selector 130. The grid selector 130 can evaluate the similarity matcher outputs to determine optimal grids for further analysis. The grid selector 130 can use the evaluation to select inputs from the first set of inputs with optimal evaluated similarities to the corresponding inputs from the second set of inputs. For example, the grid selector 130 can analyze the evaluated similarities between the first set of grids and the second set of grids from the similarity matcher 125 and select grids from the first set of grids that are associated with the maximum evaluated similarities between the first set of grids and the second set of grids. By focusing on grids that maximize feature similarity, the system 100 can ensure that the subsequent analysis targets the most relevant and informative regions of the tissue samples, thereby enhancing the accuracy and reliability of the results. The selected grids can then be further processed to identify regions of interest (ROIs) for detailed examination.

[0046] The system 100 can include an ROI selector 135 for identifying and selecting (i.e., predicting) regions of interest in a digital pathology image (e.g., mIF images, vmlF images, H&E images, confocal images, etc.).. The ROI selector 135 can receive, as input, the selected grids identified by the grid selector 130 and process the selected grids to identify ROIs. For example, the ROI selector 135 can identify regions of interest by leveraging a sliding window method. The sliding window method can include overlaying a window of a predetermined size across the grid and calculating the automated immune score (alS) of the area of the grid within the window position. The ROI selector 135 can slide (i.e., move) the window across the entire grid in a systematic manner, calculating alS for each position. The ROI selector 135 can ensure that every position within the grid is analyzed. At each window position, the ROI selector 135 can calculate the alS by analyzing pixels within the window to quantify immune cell infiltration based on immune attributes, such as color intensity, shape, and size. The ROI selector 135 can keep track of the alS for each window position. In some implementations, the alS for each window system is stored in memory 145. The window position with the alS closest to the immune score of the entire grid can be chosen as the ROI, ensuring that the selected ROI is representative of the immune cell infiltration patterns of the entire grid, thereby facilitating enhanced accuracy and reliability in analyses. The identified ROIs can then be used for further diagnostic and research purposes.

[0047] The system 100 can include memory 145, such as a RAM or other dynamic-11- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109storage device, for storing information, and instructions to be executed by the processor 140. Memory 145 can also be used for storing temporary variables, weights, parameters, or other intermediate information during execution of instructions by the processor 140.Memory 145 can be used to maintain datasets for training and inference, cache intermediate results during complex computations, and manage the state of ongoing processes.III. System for Al Guided ROI Selection

[0048] FIG. 2 depicts an example of a system 200. The system 200 can be a classification and detection system, such as to identify regions of interest in bright-field based images, such as H&E stained whole slide images (WSI), based on morphology and structure. The system 200 can incorporate features of the system 100. For example, the system 200 can include the user interface 110, the ML model 120, the data sources 105, the processor 140, and / or the memory 145.

[0049] The system 200 can receive inputs from the user interface 110 such as text, speech, audio, image, and / or video data indicative of information regarding a tissue sample, such as information provided by one or more pathologists. For example, the system 200 can receive whole slide image (WSI) data and / or annotated WSI data (e.g., annotated with manually selected ROIs). In various implementations, the data sources 105 includes the WSI data. The system 200 can output results using the user interface 110. For example, upon selecting ROIs from digital pathology images, the selected ROIs can be output to one or more users with the user interface 110.

[0050] The system 200 can include one or more tissue segmentors 205. The one or more tissue segmentors 205 can segment WSI data to identify different tissue classes. The one or more tissue segmentors 205 can include one or more specialist task models (e.g., a neural network). The one or more specialist task models can generate pixel-level tissue segmentation masks. In some implementations, the one or more tissue segmentors 205 can divide the WSI data into tiles. For example, the one or more tissue segmentors 205 can divide the WSI data into 256x256 pixel tiles.

[0051] The system 200 can include a feature extractor 210. The feature extractor 210 can include one or more neural networks (e.g., the one or more neural networks 115). In some implementations, the feature extractor 210 is a general foundation model. The feature extractor 210 can extract morphological features from the WSI data and represent the -12- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109extracted features as a numerical value (e.g., a numerical embedding vector). The morphological features can include, for example, texture, shape, size, and spatial distribution. The feature extractor 210 can receive an output from the one or more tissue segmentors 205. For example, the feature extractor 210 can receive, as input, a tissue segmentation mask from the one or more tissue segmentors 205 and output the numerical value.

[0052] The system 200 can include the at least one machine learning (ML) model 120, which can include any of various supervised and / or unsupervised machine learning models, as described herein.

[0053] The system 200 can include an ROI selector 215 for identifying and selecting (i.e., predicting) regions of interest in a digital pathology image, such as a region of interest within H&E stained WSI data. In some implementations, the ROI selector 215 receives, as input, the numerical values outputted by the feature extractor 210. In other implementations, the ROI selector 215 receives, as input, the tissue segmentation mask outputted by the one or more tissue segmentors 205. In various implementations, the ROI selector 215 identifies regions of interest by leveraging a sliding window method. Furthermore, in various implementations, the ROI selector 215 includes (e.g., uses, communicates with) the at least one ML model 120.IV. Selecting Grids for Analysis from Digital Pathology Images

[0054] FIG. 3 depicts an example of a method 300 of ROI selection. The method 300 can be performed using various systems and devices described herein, including, for example, one or more components of the system 100 and / or the system 200. The method 300 can be performed as part of any one or more testing, diagnosis, or evaluation processes.

[0055] At step 305, the method 300 can include acquisition of whole slide images (WSIs). In some implementations, the WSIs comprise fluorescent images, such as mIF images. In such implementations, tissue samples can be stained with multiple fluorescent markers, each targeting specific immune cell types, to enable the simultaneous visualization of different immune cell populations within the same tissue sample. In other implementations, the WSIs comprise one or more of brightfield based images, H&E stained images, confocal images, or bioluminescent images.-13- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109

[0056] In fluorescent whole slide images (e.g., mIF images, confocal images, etc.) auto-fluorescent regions can be identified and excluded, such as those produced by hemorrhages and blood vessels, to avoid confounding the analysis. The WSI data can be processed (e.g., preprocessed) using contour masks to distinguish between glass backgrounds and tissue regions of the slides. For example, hole-filling techniques can be employed to construct comprehensive contour masks, ensuring images with clear tissue segmentation. This step can involve excluding pixels to focus on the stroma, wherein the stroma is defined as the region within the tumor devoid of epithelial cells, artifacts, or necrosis, i.e. where immune cells are typically present and where alS can be calculated. The whole slide images can be divided into a set of 3000x3000-pixel non-overlapping grids (e.g., sub-regions) to obtain manageable sections for detailed analysis. In various implementations, the set of grids are filtered to exclude regions with more than 50% of background pixels, indicating a region devoid of cells.

[0057] At step 310, the method 300 can include performing automated immune scoring (alS) on the set of grids by classifying pixels within each grid as immune based on immune attributes, such as color intensity, shape, and size, and quantifying immune cell infiltration based on a color range and / or immune attributes. For example, alS can be determined by classifying pixels within the stroma as immune (yellow, CD45+), excluding those classified as epithelial (green, PanCK+), then selecting specific color ranges to capture various intensities of the indicated colors accurately, considering that both strong and weak expressions of yellow represent CD45+ immune infiltration. Furthermore, alS can include calculating a ratio of immune pixels to total tissue pixels for each grid of the set of grids. Upon alS for each grid of the set of grids, the grids can be classified into two immune classes: alS < 10% indicating a low-immunity score within the grid, and alS between 10% and 50% indicating a mid-immunity score within the grid. In this example, no grids are scored alS > 50%. To define the number of grids per immune class per slide, the global ratio for each immune class can be computed over the total number of grids, keeping the same ratio for the selected grids.

[0058] At step 315, manual immune scoring (mIS) can be performed on the same set of grids to form a second set of grids (e.g., a second set of grids with manual immune scores). Manual immune scoring can include receiving inputs from a user input device, such as from one or more human pathologists. This dual approach can allow for a comparison between-14- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109automated and manual scoring methods. Upon mIS for each grid of the set of grids, the grids can be classified into two immune classes: mIS < 10% indicating a low-immunity score within the grid, and mIS between 10% and 50% indicating a mid-immunity score within the grid. In this example, no grids are scored mIS > 50%.

[0059] At step 320, features can be extracted from both the first set of grids (the grids on which alS was performed) and the second set of grids (the grids on which mIS was performed). Feature extraction can be performed using one or more neural networks, such as a custom-trained convolutional neural network (CNN) model, as described herein. The extracted features of each set of grids in each immune class can be used to identify which grids from the alS grids should undergo further analysis for ROI selection.

[0060] At step 325, similarity matching can be performed to compare the features of the first set of girds (the alS grids) to the features of the second set of grids (the mIS grids). This can involve calculating a feature similarity score between the features of the first set of grids (alS) and the features of the second set of grids (mIS) using methods such as cosine similarity and comparing features using Spearman’s correlation for continuous values. In some implementations, features can be compared by evaluating categorized scores using Cohen's K score for inter-rater agreement, which quantifies a level of agreement between two scoring bodies, taking into account the possibility of agreement occurring by chance. These methods can evaluate the similarity between feature vectors, providing a measure of how closely the automated and manual scoring results align.

[0061] At step 330, grids from the first set of grids associated with the highest similarity scores can be selected for further analysis, such as for performing ROI selection at step 335.

[0062] To provide a robust evaluation of the framework, the predicted ROIs can be compared to manually selected ROIs by two independent pathologists, who are blinded to the model’s predictions. The overlap of the predictions with the manual selections can be evaluated using mean average precision (mAP) and intersection over union (IOU) scores as performance metrics. Overlap values ranging from 0.1 to 0.9 can be tested, with an overlap value of 0.5 yielding the best results. This approach can ensure that the study identifies ROIs with a high degree of accuracy while quantifying the confidence of these predictions.-15- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109V. Quantifying Immune Infiltration with Sliding Window

[0063] Referring to FIG. 4, a method 400 for quantification of immune cell infiltration can be performed, for example, by computing the density of immune-classified pixels within each grid of the selected grids resulting from method 300. This quantification can provide a numerical representation of immune cell presence, which can be used for further analysis.

[0064] The method 400 can involve a sliding window technique to refine the selection process, comprising a first step 405 for determining a window size. The window size can be chosen based on a target ROI size. To predict potential new ROIs, a window size can be determined that corresponds to the average size of manual pathologist-annotated ROIs. For example, to predict new ROIs of size 900x900 pixels, a sliding window of 900x900 pixels can be used.

[0065] At step 410, the window can be overlain onto a grid in the set of grids. At step 415, immune cell infiltration can be quantified by performing alS on the pixels of the grid within the window. At step 420, the window can then iterate to or “slide” to the next grid position. Steps 410, 415, and 420 can repeat until all window positions are iterated through on each grid of the selected grids. The quantified immune cell infiltration can be used as input for ROI selection.VI. Selecting Regions of Interest

[0066] FIG. 5 shows a method 500 for selecting regions of interest. At step 505, automated immune scoring can be performed for every window position in a grid, using the sliding window method described in FIG. 4. At step 510, the window position with the alS closest to the immune score of the entire grid can be chosen as the potential ROI. This approach can ensure that the selected ROIs accurately represent the immune cell infiltration patterns. Steps 505 and 510 can be repeated to determine an ROI for every grid in a set of grids. For example, the system 100 can determine 9 ROIs from a selected grid set of size 9.VII. Extracting features with Neural Networks

[0067] Referring to FIG. 6, a method 600 for extracting features from digital pathology images (e.g., mIF images, H&E images, confocal images, etc.) using one or more neural-16- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109networks is shown. Feature extraction can be a critical component of the system, enabling the identification of complex patterns and relationships within the digital pathology images. The one or more neural networks can be a custom-trained convolutional neural network (CNN) model to extract features from the grids based on immune scores. The method 600 can begin with step 605, the acquisition of high-resolution whole slide image (WSI) data of tissue samples. In some implementations, the WSI data comprises fluorescence-based images (e.g., mIF images). In other implementations, the WSI data comprises brightfield-based images (e.g., H&E stained images).

[0068] A CNN architecture is effective in capturing intricate patterns in image data, especially trained on H&E-stained WSI of the same cancer type, its architecture still enables robust feature extraction from other digital pathology images, such as fluorescent images (e.g., mIF images, confocal images, etc.). The CNN model can comprise multiple layers, including convolutional layers, pooling layers, and fully connected layers, each performing specific operations to extract features from the input images. Each layer performs specific operations to extract features from the input images. The convolutional layers can apply filters to the input images to detect patterns, while the pooling layers can reduce the dimensionality of the data. The fully connected layers can combine the extracted features to generate a feature vector.

[0069] At step 610, the CNN model can be trained. The CNN model can be trained using a large dataset of annotated WSI, optimizing the model's parameters to minimize the difference between predicted and actual annotations. The training dataset can include images with known immune cell annotations, allowing the model to learn the patterns and features associated with immune cell presence. The CNN can be trained using a cross-fold validation technique, which involves dividing the dataset into multiple subsets or "folds." Each fold is used as a validation set while the remaining folds are used for training, ensuring that the model is evaluated on different subsets of data. This technique helps in assessing the model's performance and generalizability, reducing the risk of overfitting. At step 615, the trained CNN model can analyze the grids and extract features that are indicative of immune cell infiltration. These features can include texture, shape, color intensity, and spatial distribution. The extracted features can be represented as numerical values for similarity matching and ROI selection. The extracted features can be used for similarity matching and ROI selection, enhancing the system's ability to identify representative-17- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109regions within the tissue samples.VIII. Training a Model to Select ROIs from Digital Pathology Images

[0070] FIG. 7 depicts an example of a method 700 for training a model (e.g., a ML model, the ML model 120, a logistic regression model, a generalist foundation model, a neural network, a CNN, etc.) to assign a probability to features extracted from digital pathology images (e.g., brightfield images, fluorescent images, H&E images, mIF images, bioluminescent images, confocal images, etc.). The extracted features can include, for example, color, color intensity, texture, shape, spatial distribution, and size. In a fluorescent image (e.g., mIF images, confocal images, etc.) and / or a luminescent image (e.g., bioluminescent image) color and color intensity may be used for detailed analysis. In a microscopic digital pathology image (e.g., H&E images) color, shape, size, spatial distribution, and texture may be used for detailed analysis.

[0071] The method 700 can be performed using various systems and devices described herein, including, for example, one or more components of the system 200 and / or the system 100. The method 700 can begin at step 705 with the acquisition of WSI data. In some implementations, the WSI data comprises H&E stained WSI data. In such implementations, histopathology slides can be prepared and stained with H&E and digitized (e.g., digitally scanned) to form the WSIs.

[0072] At step 710, the pathologist can manually select ROIs on the WSIs, to form labeled WSIs. The manually selected ROIs can represent tissue area that is representative of tumor regions with viable tissue. The manually selected ROIs can be used by the model to learn the morphological features commonly present in a ROI.

[0073] At step 715, the labeled WSIs can be processed (e.g., preprocessed). The labeled WSIs can be segmented and can be divided into 256x256 pixel tiles to obtain manageable sections for detailed analysis (e.g., by the one or more tissue segmentors 205). Features from each tile can then be extracted and outputted as a numerical value (e.g., by the feature extractor 210). The numerical value can encode visual and morphological features (e.g., cell density, architecture, texture, etc.) of each tile. Each tile can then be assigned a binary label (e.g., positive or negative, 1 or 0, etc.). A positive label can indicate that a center of the tile lies inside the manually selected ROI (e.g., selected at step 710). A negative label can indicate that the center of the tile lies outside of the manually selected ROI (e.g., selected at -18- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109step 710). The numerical values can be normalized (e.g., standardized) and the binary labels can be randomly downsampled so the number of negative tiles matches the number of positive tiles.

[0074] At step 720, the model can be trained as an ROI classifier. The model can be trained as a tile-level ROI classifier. The model can receive, as input, the numerical values representing extracted features from each tile. The model can learn the numerical values that maximizes the likelihood of correctly distinguishing tiles within the ROI from tiles outside of the ROI. The model can then output a probability representing the likelihood that the tile exhibits morphological characteristics consistent with the pathologist selected ROIs. The WSI data containing manually selected ROIs can be split into training and validation datasets. For example, 80% of the WSI data can be used for training and 20% of the WSI data can be used for validation. Performance during training can be measured using an area under a receiver operator curve (AUC). The model can achieve an AUC of 0.963, indicating an excellent ability to distinguish between tiles within the ROI and tiles outside of the ROI.IX. ROI Selection from Digital Pathology Images - Scenario 1

[0075] FIG. 8 depicts an example of a method 800 of ROI selection from digital pathology images (e.g., brightfield images, H&E images). The method 800 can be performed using various systems and devices described herein, including, for example, one or more components of the system 200 and / or the system 100. The method 800 can be performed as part of any one or more testing, diagnosis, or evaluation processes. In various implementations, one or more steps of the method 800 is performed by one or more models (e.g., a ML model, the ML model 120, an ROI classifier, etc.).

[0076] At step 805, the method 800 can include acquisition of digital pathology WSI data. In some implementations, the WSI data comprises H&E stained WSI data. In such implementations, histopathology slides can be prepared and stained with H&E and digitized (e.g., digitally scanned) to form the WSIs, thereby obtaining high-resolution H&E images of tissue samples. The H&E imaging technique involves staining tissue samples with H&E to visualize different cell populations within the tissue sample based on morphological features (e.g., shape, density, texture, size, etc.). The WSI data can then be segmented (e.g., by the one or more tissue segmentors 205) and the WSI can be divided into 256x256 pixel tiles. Morphological features from each tile can then be extracted and represented as a numerical-19- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109value (e.g., by the feature extractor 210).

[0077] At step 810, the method 800 can include assigning a probability to each tile. The numerical values from each tile can be input to a trained model (e.g., as described with respect to FIG. 7) and used by the model to assign the probability. The probability assigned to each tile can represent a likelihood of the tile having a numerical value (e.g., morphological features) commonly present in a ROI. For example, the model can compare the numerical values of each tile to the numerical values commonly present in an ROI to assign the probability.

[0078] At step 815, the method 800 can include generating a probability map. The probability map can be overlaid on the WSI, representing the probability assigned to each tile at step 810. The probability map can indicate the spatial distribution of “ROI-likeness” across the WSI, providing a spatial overview of where ROI-like morphology is concentrated.

[0079] At step 820, the method 800 can include predicting one or more ROIs using a sliding window technique (e.g., a sliding window algorithm, a sliding window method). To predict potential ROIs, a window size can be determined that corresponds to the average size of pathologist selected ROIs (e.g., selected at step 710, described with respect to FIG.7). For example, to predict ROIs of size 900x900 pixels, a sliding window of 900x900 pixels can be used.

[0080] The window can be overlain onto the WSI (e.g., the probability map generated at step 815) and can iterate or slide across the WSI to different window positions. An average probability can be determined across the tiles captured by the window at each window position. A predicted ROI can be selected as the window position with the highest average probability. The predicted ROIs can then be used for further analysis (e.g., spatial transcriptomics assays).

[0081] To provide a robust evaluation of the framework, the predicted ROIs (e.g., predicted at step 820) can be compared to manually selected ROIs (e.g., selected at step 710, described with respect to FIG. 7). An overlap between ROIs selected by pathologists and the ROIs selected via the method 800 can be evaluated using the intersection over union (IOU) scores and a Dice score. The IOU scores and the Dice scores can range from 0.0 to 1.0, with scores closer to 1.0 indicating increased overlap between the predicted ROIs and -20- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109the manually selected ROIs. This approach can ensure that the method 800 identifies ROIs with a high degree of accuracy while quantifying the confidence of these predictions.X. ROI Selection from Digital Pathology Images - Scenario 2

[0082] FIG. 9 depicts an example of a method 900 of ROI selection from digital pathology images, such as H&E images. The method 800 can be performed using various systems and devices described herein, including, for example, one or more components of the system 200 and / or the system 100. The method 900 can be performed as part of any one or more testing, diagnosis, or evaluation processes. In various implementations, one or more steps of the method 900 is performed by one or more models (e.g., a ML model, the ML model 120, a CNN, etc.).

[0083] At step 905, the method 900 can include acquisition of WSI data. The WSI data can be acquired by digitizing (e.g., digitally scanning) one or more digital pathology images to generate the WSI data. In some implementations, the WSI data comprises H&E stained WSI data. In such implementations, histopathology slides can be prepared and stained with H&E and digitized to form the WSIs, thereby obtaining high-resolution H&E images of tissue samples.

[0084] At step 910, the WSI data be segmented into tissue classes (e.g., by the one or more tissue segmentors 205). Each pixel of the WSI data can be segmented into tissue classes to form a predicted segmentation mask. The predicted segmentation mask can be saved (e.g., in the memory 145) as a dense mask. In some implementations, the WSI data is segmented into 10 tissue classes. For example, on a lung tissue sample, the WSI data can be segmented into tumor, stroma, immune (e.g., macrophages, inflammatory cells, etc.), necrosis, alveoli, bronchi epithelium, vessels, adipose tissue, and muscle.

[0085] At step 915, the method 900 can include a sliding window technique (e.g., a sliding window algorithm) to predict one or more ROIs. To predict potential ROIs, a window size can be determined that corresponds to an average size of pathologist selected ROIs. For example, to predict ROIs of size 900x900 pixels, a sliding window of 900x900 pixels can be used.

[0086] The window can be overlain onto the predicted segmentation mask and can iterate or slide across the predicted segmentation mask. A proportion of pixels belonging to each-21- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109tissue class can be computed at each window position. For example, a percentage of pixels belonging to tumor or epithelium and a percentage of pixels belonging to necrosis can be assigned at each window position. A score can be assigned to each window position, where higher scores represent a window position that maximizes tumor and minimizes necrosis. A predicted ROI can be selected as the window position with the highest score. The predicted ROIs can then be used for further analysis (e.g., spatial transcriptomics assays).

[0087] To provide a robust evaluation of the framework, the predicted ROIs (e.g., predicted at step 915) can be compared to ROIs selected manually by pathologists on the same WSI data. An overlap between ROIs selected by pathologists and the ROIs selected via the method 900 can be evaluated using the intersection over union (IOU) scores. A correlation computation (e.g., Pearson’s correlation) can be used to assess a degree to which tissue composition metrics (e.g., the percentage of pixels assigned tumor or epithelium and the percentage of pixels assigned necrosis) is preserved between manually selected ROIs and ROIs selected via the method 900. A high correlation value can indicate that the method 900 identifies ROIs with similar tissue composition to the ROIs selected by pathologists.XI. ROI Selection from Digital Pathology Images - Scenario 3

[0088] FIG. 10 depicts an example of a method 1000 of ROI selection from digital pathology images, such as H&E images. The method 1000 can be performed using various systems and devices described herein, including, for example, one or more components of the system 200 and / or the system 100. The method 1000 can be performed as part of any one or more of testing, diagnosis, or evaluation processes. In various implementations, one or more steps of the method 1000 is performed by one or more models (e.g., a ML model, the ML model 120, a CNN, etc.).

[0089] At step 1005, the method 1000 can include acquisition of WSI data. In some implementations, the WSI data is acquired from H&E stained images, as described herein.

[0090] At step 1010, the method 1000 can include annotation of a tumor mask on the WSI data. For example, one or more pathologists can manually annotate the WSI data to form a tumor mask. The tumor mask can identify a region that a predicted ROI should come from. For example, one or more models executing the method 1000 can constrain the analysis to the tumor mask region.-22- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109

[0091] At step 1015, the method 1000 can include segmentation of the tumor mask. The tumor mask can be segmented as described with respect to FIG. 9 (e.g., at step 910), to identify tissue classes at the pixel level. For example, the tumor mask can be segmented into necrosis, tumor, immune, and stromal tissue classes.

[0092] At step 1020, the segmented tumor mask can be filtered. The segmented tumor mask can be filtered to exclude regions with excess necrosis and emphasize regions with high tumor and stromal tissue classes.

[0093] At step 1025, the method 1000 can include a sliding window technique (e.g., a sliding window algorithm) to predict one or more ROIs. To predict potential ROIs, a window size can be determined that corresponds to an average size of pathologist selected ROIs. For example, to predict ROIs of size 900x900 pixels, a sliding window of 900x900 pixels can be used.

[0094] The window can be overlain onto the segmented tumor mask and can iterate or slide across the segmented tumor mask. Morphological features can be extracted at each window position and represented as a numerical value.

[0095] The numerical values of each window position can be compared to a numerical value extracted from the tumor mask. For example, morphological features from the entire tumor mask region may be extracted and represented as a numerical value. The numerical value of the entire tumor mask region can be compared to the numerical values extracted from each window position to identify a window position that is representative of the entire tissue region. The numerical value at each window position can be compared to the numerical value of the entire tumor mask region using cosine similarity. A predicted ROI can be selected as the window position with the highest cosine similarity.

[0096] To provide a robust evaluation of the framework, the predicted ROIs (e.g., predicted at step 1025) can be compared to ROIs selected manually by pathologists on the same tumor mask regions. An overlap between ROIs selected by pathologists and the ROIs selected via the method 1000 can be evaluated using the intersection over union (IOU) scores and Dice scores as described herein.XII. Sliding Window Technique for ROI Selection-23- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109

[0097] FIG. 11 shows an example of a method 1100 for performing the sliding window technique described herein. The method 1100 can be used, for example, in the method 800, the method 900, and the method 1000. The sliding window technique can be performed by one or more components of the system 100 and / or the system 200.

[0098] The method 1100 can begin at step 1105 with determining a size of a window. As described herein, the size of the window can be selected such that the size of the window corresponds to the average size of pathologist selected ROIs.

[0099] At step 1110, the window can be overlain onto an image. In some implementations, the image is an H&E stained WSI that includes multiple tiles (e.g., as described with respect to FIG. 8). In other implementations, the image is a segmentation mask generated from H&E stained WSIs (e.g., the predicted segmentation mask described with respect to FIG. 9, the segmented tumor mask described with respect to FIG. 10, etc.).

[0100] At step 1115, the method 1100 can include calculating a parameter at a window position. In some implementations, and described with respect to FIG. 8, the parameter is an average probability of the window position. In other implementations, and described with respect to FIG. 9, the parameter is a score representing a percentage of pixels within the window position that belong to a particular tissue class. In yet other implementations, and described with respect to FIG. 10, the parameter is a numerical value representing morphological features within the window position.

[0101] At step 1120, the window can be slid to a next window position on the image. Steps 1115 and 1120 can be repeated until the entire image has been captured by the window.

[0102] Once the entire image has been captured by the window, the method 1100 can proceed to step 1125 with predicting one or more ROIs. In some implementations, and described with respect to FIG. 8, the ROI is selected as the window position with the highest average probability. In other implementations, and described with respect to FIG. 9, the ROI is selected as the window position with the highest score. In yet other implementations, and described with respect to FIG. 10, the ROI is selected as the window position with the highest cosine similarity.XIII. Virtual mIF (vmlF) Images from Digital Pathology Images-24- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109

[0103] FIG. 12 depicts an example of a method 1200 for generating virtual mIF (vmlF) images from digital pathology images, such as H&E images. A vmlF image can be defined as a computationally generated approximation of an mIF image, produced by an algorithm (e.g., a trained neural network) from another imaging modality, such as H&E, without performing physical mIF staining. The method 1200 can be performed using various systems and devices described herein, including, for example, one or more components of the system 200 and / or the system 100. The vmlF images can be used in one or more of the method 300, the method 400, the method 500, or the method 600 for ROI prediction and selection. The method 1200 can be performed as part of any one or more of testing, diagnosis, or evaluation processes. In various implementations, one or more steps of the method 1200 is performed by one or more models (e.g., a ML model, the ML model 120, a CNN, etc.).

[0104] The method 1200 can begin at step 1205 with restaining mIF images (e.g., acquired at step 305, discussed with respect to FIG. 3) with H&E. This can result in paired mIF and H&E images on the same WSI dataset.

[0105] At step 1210, the H&E images can be spatially aligned to their paired mIF images. This can involve image co-regi strati on (e.g., rigid and affine registration) to achieve cellular-level (e.g., pixel level) alignment between the mIF images and the H&E images. An extent of alignment between the mIF images and the H&E images can be evaluated using nuclear segmentation on both the mIF images and the H&E images to form nuclear masks for the mIF images and nuclear masks for the H&E images. Two complimentary scores can be calculated to evaluate the extent of alignment, such as, a multi-scale structural similarity (MS-SSIM) score and the Dice score. MS-SSIM can be used to compare a broad set of features in the nuclear masks of the mIF images and the nuclear masks of the H&E images. The broad set of features can include, for example, luminance, contrast, and structure. Dice scores can be used to measure an overlap between segmented nuclei in the nuclear masks of the mIF images and segmented nuclei in the nuclear masks of the H&E images. The paired mIF images and H&E images can be broken down into tiles of size 256x256 pixels and each tile from the H&E images can be paired to a corresponding tile in the mIF images to form a real H&E / mlF pair.

[0106] At step 1215, the method 1200 can include configuring (e.g., training, updating, etc.) an ML model (e.g., a generative adversarial network (GAN), a CNN-based model, the-25- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109ML model 120, etc.) to generate virtual mIF images (vmlF) from H&E images. During training, the model can use the paired and spatially aligned H&E and mIF tiles, which can be preprocessed by resizing (e.g., to 256x256 pixels) and intensity-normalization. The ML model can include a generator and a discriminator, which can each be implemented with a plurality of convolutional layers. In various implementations, an output from the generator can be used as input to the discriminator. For example, the generator can take an H&E tile as input and produce a vmlF tile, and this vmlF tile can then be concatenated or otherwise combined with the corresponding H&E tile and provided to the discriminator as a candidate H&E / vmlF pair.

[0107] During training, each H&E tile can be provided to the generator, which can output a vmlF tile. For example, each H&E tile can be processed by the generator to produce a vmlF tile of the same spatial dimensions (e.g., 256x256 pixels) as the input H&E tile. Each vmlF tile can be paired with the corresponding H&E tile to form H&E / vmlF pairs.Corresponding mIF tiles can be used as ground truth for a reconstruction loss applied to the generator. For example, real mIF tiles that correspond spatially to the vmlF tiles can be used to compute a pixel -wise LI reconstruction loss. The corresponding mIF tiles and the H&E tiles can also be provided to the discriminator so that the discriminator learns to distinguish real H&E / mlF pairs from H&E / vmlF pairs. For example, the discriminator can receive as input either (i) a real H&E / mlF tile pair or (ii) an H&E / vmlF tile pair generated by the generator, and can be trained to output a label indicating whether the pair is real or virtual. The generator can be optimized to minimize a weighted sum of adversarial loss and reconstruction loss. For example, the adversarial loss can encourage the generator to produce vmlF tiles that are indistinguishable in appearance from real mIF tiles, and the LI reconstruction loss can encourage structural similarity between vmlF tiles and corresponding real mIF tiles. The optimization can be performed using an optimizer with a controlled learning-rate. For example, an adaptive moment estimation (Adam) optimizer can be used with a fixed learning rate initially and a linear decay of the learning rate thereafter. The generator thus learns a mapping from H&E tiles as inputs to virtual mIF tiles as outputs. For example, after training, the generator can take as input an H&E tile from a previously unseen sample and output a vmlF tile.

[0108] At step 1220, the method 1200 can include predicting one or more ROIs on the vmlF images. The ROIs can be selected using one or more of the method 300 or the method-26- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109500, as described with respect to FIGS. 3 and 5.

[0109] To provide a robust evaluation of the framework, the ROIs predicted on the vmlF images (e.g., predicted at step 1220) can be compared to ROIs predicted on corresponding real mIF images. An overlap between ROIs selected on the vmlF images and ROIs selected on the corresponding real mIF images can be evaluated using pairwise IOU scores.Furthermore, alS can be performed on the vmlF images and compared to alS performed on the real mIF images (e.g., as described with respect to FIG. 4).XIV. Computing and Network Environment

[0110] It may be helpful to describe aspects of the operating environment as well as associated system components (e.g., hardware elements) in connection with the methods and systems described herein. The methods described herein, such as methods 300, 400, 500, 600, 700, 800, 900, 1000, 1100, and 1200 can be implemented or executed by any of the devices, or any combination thereof, described in relation to FIG. 13.[OHl] Various operations described herein can be implemented on computer systems. FIG. 13 shows a simplified block diagram of a representative server system 1300, client computing system 1314, and network 1326 usable to implement certain implementations of the present disclosure. In various implementations, server system 1300 or similar systems can execute services or servers described herein or portions thereof. Client computing system 1314 or similar systems can implement clients described herein. Server system 1300 can have a modular design that incorporates a number of modules 1302 (e.g., blades in a blade server implementation); while two modules 1302 are shown, any number can be provided. Each module 1302 can include processing unit(s) 1304 and local storage 1306.

[0112] Processing unit(s) 1304 can include a single processor, which can have one or more cores, or multiple processors. In some implementations, processing unit(s) 1304 can include a general-purpose primary processor as well as one or more special-purpose coprocessors such as graphics processors, digital signal processors, or the like. In some implementations, some or all processing units 1304 can be implemented using customized circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some implementations, such integrated circuits execute instructions that are stored on the circuit itself. In other implementations, processing unit(s) 1304 can execute instructions stored in local storage 1306. Any type of processors in any combination -27- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109can be included in processing unit(s) 1304.

[0113] Local storage 1306 can include volatile storage media (e.g., DRAM, SRAM, SDRAM, or the like) and / or non-volatile storage media (e.g., magnetic or optical disk, flash memory, or the like). Storage media incorporated in local storage 1306 can be fixed, removable or upgradeable as desired. Local storage 1306 can be physically or logically divided into various subunits such as a system memory, a read-only memory (ROM), and a permanent storage device. The system memory can be a read-and-write memory device or a volatile read-and-write memory, such as dynamic random-access memory. The system memory can store some or all of the instructions and data that processing unit(s) 1304 need at runtime. The ROM can store static data and instructions that are needed by processing unit(s) 1304. The permanent storage device can be a non-volatile read-and-write memory device that can store instructions and data even when module 1302 is powered down. The term “storage medium” as used herein includes any medium in which data can be stored indefinitely (subject to overwriting, electrical disturbance, power loss, or the like) and does not include carrier waves and transitory electronic signals propagating wirelessly or over wired connections.

[0114] In some implementations, local storage 1306 can store one or more software programs to be executed by processing unit(s) 1304, such as an operating system and / or programs implementing various methods or steps thereof such as methods 400 and 600 of FIGS. 4 and 6 or any other methods or processes described herein.

[0115] “ Software” refers generally to sequences of instructions that, when executed by processing unit(s) 1304 cause server system 1300 (or portions thereof) to perform various operations, thus defining one or more specific machine implementations that execute and perform the operations of the software programs. The instructions can be stored as firmware residing in read-only memory and / or program code stored in non-volatile storage media that can be read into volatile working memory for execution by processing unit(s) 1304.Software can be implemented as a single program or a collection of separate programs or program modules that interact as desired. From local storage 1306 (or non-local storage described below), processing unit(s) 1304 can retrieve program instructions to execute and data to process in order to execute various operations described above.

[0116] In some server systems 1300, multiple modules 1302 can be interconnected via a-28- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109bus or other interconnect 1308, forming a local area network that supports communication between modules 1302 and other components of server system 1300. Interconnect 1308 can be implemented using various technologies including server racks, hubs, routers, etc.

[0117] A wide area network (WAN) interface 1310 can provide data communication capability between the local area network (interconnect 1308) and the network 1326, such as the Internet. Technologies can be used, including wired (e.g., Ethernet, IEEE 6002.3 standards) and / or wireless technologies (e.g., Wi-Fi, IEEE 6002.11 standards).

[0118] In some implementations, local storage 1306 is intended to provide working memory for processing unit(s) 1304, providing fast access to programs and / or data to be processed while reducing traffic on interconnect 1308. Storage for larger quantities of data can be provided on the local area network by one or more mass storage subsystems 1312 that can be connected to interconnect 1308. Mass storage subsystem 1312 can be based on magnetic, optical, semiconductor, or other data storage media. Direct attached storage, storage area networks, network-attached storage, and the like can be used. Any data stores or other collections of data described herein as being produced, consumed, or maintained by a service or server can be stored in mass storage subsystem 1312. In some implementations, additional data storage resources may be accessible via WAN interface 1310 (potentially with increased latency).

[0119] Server system 1300 can operate in response to requests received via WAN interface 1310. For example, one of modules 1302 can implement a supervisory function and assign discrete tasks to other modules 1302 in response to received requests. Work allocation techniques can be used. As requests are processed, results can be returned to the requester via WAN interface 1310. Such operation can generally be automated. Further, in some implementations, WAN interface 1310 can connect multiple server systems 1300 to each other, providing scalable systems capable of managing high volumes of activity. Other techniques for managing server systems and server farms (collections of server systems that cooperate) can be used, including dynamic resource allocation and reallocation.

[0120] Server system 1300 can interact with various user-owned or user-operated devices via a wide-area network such as the Internet. An example of a user-operated device is shown in FIG. 13 as client computing system 1314. Client computing system 1314 can be implemented, for example, as a consumer device such as a smartphone, other mobile phone,-29- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109tablet computer, wearable computing device (e.g., smartwatch, eyeglasses), desktop computer, laptop computer, and so on.

[0121] For example, client computing system 1314 can communicate via WAN interface 1310. Client computing system 1314 can include computer components such as processing unit(s) 1316, storage device 1318, network interface 1320, user input device 1322, and user output device 1324. Client computing system 1314 can be a computing device implemented in a variety of form factors, such as a desktop computer, laptop computer, tablet computer, smartphone, other mobile computing device, wearable computing device, or the like.

[0122] Processing unit(s) 1316 and storage device 1318 can be similar to processing unit(s) 1304 and local storage 1306 described above. Suitable devices can be selected based on the demands to be placed on client computing system 1314; for example, client computing system 1314 can be implemented as a “thin” client with limited processing capability or as a high-powered computing device. Client computing system 1314 can be provisioned with program code executable by processing unit(s) 1316 to enable various interactions with server system 1300.

[0123] Network interface 1320 can provide a connection to the network 1326, such as a wide area network (e.g., the Internet) to which WAN interface 1310 of server system 1300 is also connected. In various implementations, network interface 1320 can include a wired interface (e.g., Ethernet) and / or a wireless interface implementing various RF data communication standards such as Wi-Fi, Bluetooth, or cellular data network standards (e.g., 3G, 4G, LTE, etc ).

[0124] User input device 1322 can include any device (or devices) via which a user can provide signals to client computing system 1314; client computing system 1314 can interpret the signals as indicative of particular user requests or information. In various implementations, user input device 1322 can include any or all of a keyboard, touch pad, touch screen, mouse or other pointing device, scroll wheel, click wheel, dial, button, switch, keypad, microphone, and so on.

[0125] User output device 1324 can include any device via which client computing system 1314 can provide information to a user. For example, user output device 1324 can include a display to display images generated by or delivered to client computing system 1314. The display can incorporate various image generation technologies, e.g., a liquid -30- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109crystal display (LCD), light-emitting diode (LED) including organic light-emitting diodes (OLED), projection system, cathode ray tube (CRT), or the like, together with supporting electronics (e.g., digital-to-analog or analog-to-digital converters, signal processors, or the like). Some implementations can include a device such as a touchscreen that function as both input and output device. In some implementations, other user output devices 1324 can be provided in addition to or instead of a display. Examples include indicator lights, speakers, tactile “display” devices, printers, and so on.

[0126] Some implementations include electronic components, such as microprocessors, storage and memory that store computer program instructions in a computer-readable storage medium. Many of the features described in this specification can be implemented as processes that are specified as a set of program instructions encoded on a computer-readable storage medium. When these program instructions are executed by one or more processing units, they cause the processing unit(s) to perform various operation indicated in the program instructions. Examples of program instructions or computer code include machine code, such as is produced by a compiler, and files including higher-level code that are executed by a computer, an electronic component, or a microprocessor using an interpreter. Through suitable programming, processing unit(s) 1304 and 1316 can provide various functionality for server system 1300 and client computing system 1314, including any of the functionality described herein as being performed by a server or client, or other functionality.

[0127] It will be appreciated that server system 1300 and client computing system 1314 are illustrative and that variations and modifications are possible. Computer systems used in connection with implementations of the present disclosure can have other capabilities not specifically described here. Further, while server system 1300 and client computing system 1314 are described with reference to particular blocks, it is to be understood that these blocks are defined for convenience of description and are not intended to imply a particular physical arrangement of component parts. For instance, different blocks can be but need not be located in the same facility, in the same server rack, or on the same motherboard.Further, the blocks need not correspond to physically distinct components. Blocks can be configured to perform various operations, e.g., by programming a processor or providing appropriate control circuitry, and various blocks might or might not be reconfigurable depending on how the initial configuration is obtained. Implementations of the present-31- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109disclosure can be realized in a variety of apparatus including electronic devices implemented using any combination of circuitry and software.

[0128] While the disclosure has been described with respect to specific implementations, one skilled in the art will recognize that numerous modifications are possible.Implementations of the disclosure can be realized using a variety of computer systems and communication technologies including but not limited to the specific examples described herein. Implementations of the present disclosure can be realized using any combination of dedicated components and / or programmable processors and / or other programmable devices. The various processes described herein can be implemented on the same processor or different processors in any combination. Where components are described as being configured to perform certain operations, such configuration can be accomplished, e.g., by designing electronic circuits to perform the operation, by programming programmable electronic circuits (such as microprocessors) to perform the operation, or any combination thereof. Further, while the implementations described above may make reference to specific hardware and software components, those skilled in the art will appreciate that different combinations of hardware and / or software components may also be used and that particular operations described as being implemented in hardware might also be implemented in software or vice versa.

[0129] Computer programs incorporating various features of the present disclosure may be encoded and stored on various computer-readable storage media; suitable media include magnetic disk or tape, optical storage media such as compact disk (CD) or DVD (digital versatile disk), flash memory, and other non-transitory media. Computer-readable media encoded with the program code may be packaged with a compatible electronic device, or the program code may be provided separately from electronic devices (e.g., via Internet download or as a separately packaged computer-readable storage medium).

[0130] Thus, although the disclosure has been described with respect to specific implementations, it will be appreciated that the disclosure is intended to cover all modifications and equivalents within the scope of the following claims.XV. Example Computing System

[0131] FIG. 14 is a block diagram of an example computing system 1400 suitable for use in the various arrangements described herein. In a non-limiting example, the computing -32- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109system 1400 may implement the system 100 of FIG. 1 and / or the system 200 of FIG. 2, or various other example systems and devices described in the present disclosure.

[0132] The computing system 1400 includes a bus 1402 or other communication component for communicating information and a processor 1404 coupled to the bus 1402 for processing information. The computing system 1400 also includes main memory 1406, such as a RAM or other dynamic storage device, coupled to the bus 1402 for storing information, and instructions to be executed by the processor 1404. Main memory 1406 may also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor 1404. The computing system 1400 may further include a ROM 1408 or other static storage device coupled to the bus 1402 for storing static information and instructions for the processor 1404. A storage device 1410, such as a solid-state device, magnetic disk, or optical disk, is coupled to the bus 1402 for persistently storing information and instructions.

[0133] The computing system 1400 may be coupled via the bus 1402 to a display 1414, such as a liquid crystal display, or active-matrix display, for displaying information to a user. An input device 1412, such as a keyboard including alphanumeric and other keys, may be coupled to the bus 1402 for communicating information, and command selections to the processor 1404. In another implementation, the input device 1412 has a touch screen display. The input device 1412 may include any type of biometric sensor, or a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor 1404 and for controlling cursor movement on the display 1414.

[0134] In some implementations, the computing system 1400 may include a communications adapter 1416, such as a networking adapter. Communications adapter 1416 may be coupled to bus 1402 and may be configured to enable communications with a computing or communications network or other computing systems. In various illustrative implementations, any type of networking configuration may be achieved using communications adapter 1416, such as wired (e.g., via Ethernet), wireless (e.g., via Wi-Fi, Bluetooth), satellite (e.g., via GPS) pre-configured, ad-hoc, LAN, WAN, and the like.

[0135] According to various implementations, the processes of the illustrative implementations that are described herein may be achieved by the computing system 1400-33- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109in response to the processor 1404 executing an implementation of instructions contained in main memory 1406. Such instructions may be read into main memory 1406 from another computer-readable medium, such as the storage device 1410. Execution of the implementation of instructions contained in main memory 1406 causes the computing system 1400 to perform the illustrative processes described herein. One or more processors in a multi-processing implementation may also be employed to execute the instructions contained in main memory 1406. In alternative implementations, hard-wired circuitry may be used in place of or in combination with software instructions to implement illustrative implementations. Thus, implementations are not limited to any specific combination of hardware circuitry and software.

[0136] Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations or implementations.XVI. Examples

[0137] The systems and methods described herein can identify and / or predict ROIs within a digital pathology image (e.g., an mIF image, a vmlF image, an H&E image). The ROIs can be identified and / or predicted using one or more ML models (e.g., custom trained CNN, linear regression models, general foundation models (GFM), specialist task-oriented models (STM), etc.). Inputs to the one or more ML models can include features extracted from new (e.g., unseen by the model) whole slide images. In various implementations, the one or more ML models compares the features extracted from the new whole slide images to features extracted from manually selected ROIs to identify and / or predict an ROI. In various implementations, a sliding window approach with a window size of 6.5mm x 6.5mm and a step size (e.g., a stride) of 12 pixels was used. The identified and / or predicted ROIs can be used for further analysis, for example, for validation of the performance of the one or more ML models and / or spatial transcriptomics assays.

[0138] FIGS. 15-17 show an example of a study following the method 800. In the study,-34- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109H&E WSIs from four tumor types can be used, including, glioblastoma (GBM), cholangiocarcinoma (CCA), lung adenocarcinoma (LUNG), and upper-tract urothelial carcinoma (UTUC). The study can be used to assess a need to select ROls that capture representative tumor areas while maximizing viable tissue coverage within the capture region (FIG. 15). A GFM-based approach that encodes tissue features for ROI selection, such as viable tissue with low hemorrhages and necrosis, can be used. Each WSI can be tiled and passed through a GFM for embedding extraction (e.g., for representing features as a numerical value). With the embeddings, an ML model (e.g., an ROI classifier, a GFM) can be trained using pathologist annotated ROIs on a training dataset 1502 and tested on a test dataset 1504. As shown in FIG. 15, the training dataset 1502 can include tissue samples from LUNG (N=9), CCA (N=9), and UTUC (N=2), and the test dataset 1504 can include tissue samples from GBM (N=27). A logistic regression model can be fitted to generate a probability of each tile being within the pathologist annotated ROI. A sliding window technique 1506 can be used to determine an average probability of each tile captured by the window. FIG. 16 shows an example of a probability map 1602 generated using the sliding window technique 1506. FIG. 17 shows a plot 1702. The plot 1702 shows the median Dice and IOU scores of the predicted ROIs in GBM tissue samples as compared to the manually selected ROIs. As shown in FIGS. 16 and 17, ROI selection using the ML model generated accurate results for GBM tissue samples, visualized by the probability map 1602 (FIG. 16), and quantitatively supported by the plot 1702, showing a high overlap with pathologist annotated ROIs (FIG. 17, N = 27, median Dice ± SD= 0.83 ± 0.32, median IOU ± SD= 0.71 ± 0.33). The ML model can leverage contextual information and provide a fast and generalizable ROI selection approach for spatial transcriptomics assays.

[0139] FIGS. 18 and 19 show an example of a study following the method 900. FIG. 18 shows an example workflow 1800 of the study. The study can be used to addresses requirements for including or excluding certain tissue components, such as necrosis, in an ROI. In the workflow 1800, an ML model (e.g., an STM) can be trained using breast and lung cancer tissue samples and can be used for ROI selection in LUNG cases (N = 9). The ML model can segment the tissue samples into ten categories, for example, epithelium / tumor, stroma, immune, necrosis, alveoli, bronchi epithelium, vessels, adipose tissue, and muscle, to generate a WSLlevel segmentation mask. After the generation of WSl-level segmentation mask, a region with the highest tumor / epithelial-to-necrosis ratio can be selected as an ROI using a sliding window technique. The selected ROIs can be-35- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109compared to pathologist's manual ROIs. FIG. 19, shows an example of a plot 1902 showing the overlap between the selected ROI and the manual ROI. The plot 1902 indicates a moderate to-high overlap with the manual ROI as represented by the IOU score (median IOU± SD= 0.45 ± 0.21). FIG. 19 also shows an example of a plot 1904 comparing the tumor / epithelium composition in the selected ROI and the manual ROI and a plot 1906 comparing the necrosis composition in the selected ROI and the manual ROI. As shown in plot 1904, there is a significant correlation in the tumor / epithelium composition (Pearson's correlation r = 0.98, p = 6.21x10-6) and, as shown in plot 1906, a significant correlation in the necrosis composition (Pearson's r = 0.86, p= 0.003).

[0140] FIGS. 20 and 21 show an example of a study following the method 1000. The study can be used to assess human-Al collaboration in ROI selection. The previous two studies (as described with respect to FIGS. 15-19) can be combined into a hybrid specialistgeneralist approach guided by coarse pathologists' manual segmentation (FIG. 20). This study can be particularly useful when only a region is of interest and there is no specialist model for automated segmentation, thus requiring expert domain annotation. FIG. 20 shows an example workflow 2000 of the study. As shown in FIG. 20, a pathologist can annotate the tumor bed as the macroscopic area of interest. The macroscopic area of interest can be segmented using an ML model (e.g., an STM) to distinguish tumor / epithelium and stromal compartments. An ML model (e.g., a GFM) can be used to generate morphological representation of the pathologist annotated region. Candidate ROIs can then be obtained using a sliding window technique. The candidate ROIs with highest cosine similarity with the tumor / epithelium and stromal compartments in pathologist annotated ROIs can be selected as the ROI. FIG. 21 shows an example of a plot 2102. The plot 2102 compares an overlap between the selected ROI and the pathologist annotated ROI. In an evaluation set (CCA, N = 19), and shown in plot 2102, variable performance compared with the pathologist annotated ROI was observed (median Dice ± SD = 0.46 ± 0.31). While some cases demonstrated effective ROI selection (Dice score > 0.5), others with more homogenous tissue types remained challenging.

[0141] Further, as a proof of concept, the studies of FIGS. 15-17 and FIGS. 20 and 21 can be applied to prospectively select ROIs in CCA samples (N = 4). FIG. 22 shows an example workflow 2200 of the study. In the study, a pathologist can evaluate ROIs selected by the one or more ML models and decide between keeping the ROIs selected by the one or more-36- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109ML models or manually annotating a new ROI. In three cases, the pathologist approved the ROIs predicted by the one or more ML models employed in the study of FIGS. 20 and 21 (manual tumor bed annotation + STM + GFM). In only one case, the predicted ROI following the study of FIGS. 15-17 (GFM approach) was too similar to the predicted ROI following the study of FIGS. 20 and 21 (manual tumor bed annotation + STM + GFM), and the predicted ROI following the study of FIGS. 20 and 21 was kept for consistency. For these four cases, the predicted ROIs were used as input for a spatial transcriptomics assay. In this way, it is possible to combine generalist foundation models (GFMs), specialist task-oriented models (STMs), and domain-expert input for ROI selection through a reproducible and scalable workflow that can be adapted to different hypothesis-driven scenarios.

[0142] The phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of “including” “comprising” “having” “containing” “involving” “characterized by” “characterized in that” and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, acts, or components.

[0143] Implementations of the subject matter and the operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them. Moreover, while a computer storage medium is not a propagated-37- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).

[0144] The operations described in this specification can be performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources. The term “data processing apparatus” or “computing device” encompasses various apparatuses, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of them. The apparatus and execution environment can realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0145] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a circuit, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more circuits, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0146] Processors suitable for the execution of a computer program include, by way of example, microprocessors, and any one or more processors of a digital computer. A-38- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109processor can receive instructions and data from a read only memory or a random-access memory or both. The elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. A computer can include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. A computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a personal digital assistant (PDA), a Global Positioning System (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few. Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0147] To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0148] The implementations described herein can be implemented in any of numerous ways including, for example, using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers.

[0149] Also, a computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible-39- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109presentation of output. Examples of input devices that can be used for a user interface include keyboards, and pointing devices, such as mice, touch pads, and digitizing tablets. As another example, a computer may receive input information through speech recognition or in other audible format.

[0150] Such computers may be interconnected by one or more networks in any suitable form, including a local area network or a wide area network, such as an enterprise network, and intelligent network (IN) or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.

[0151] A computer employed to implement at least a portion of the functionality described herein may comprise a memory, one or more processing units (also referred to herein simply as “processors”), one or more communication interfaces, one or more display units, and one or more user input devices. The memory may comprise any computer-readable media, and may store computer instructions (also referred to herein as “processorexecutable instructions”) for implementing the various functionalities described herein. The processing unit(s) may be used to execute the instructions. The communication interface(s) may be coupled to a wired or wireless network, bus, or other communication means and may therefore allow the computer to transmit communications to or receive communications from other devices. The display unit(s) may be provided, for example, to allow a user to view various information in connection with execution of the instructions. The user input device(s) may be provided, for example, to allow the user to make manual adjustments, make selections, enter data or various other information, or interact in any of a variety of manners with the processor during execution of the instructions.

[0152] The various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0153] In this respect, various inventive concepts may be embodied as a computer readable storage medium (or multiple computer readable storage media) (e.g., a computer-40- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109memory, one or more floppy discs, compact discs, optical discs, magnetic tapes, flash memories, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other non-transitory medium or tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various implementations of the solution discussed above. The computer readable medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computers or other processors to implement various aspects of the present solution as discussed above.

[0154] The terms “program” or “software” are used herein to refer to any type of computer code or set of computer-executable instructions that can be employed to program a computer or other processor to implement various aspects of implementations as discussed above. One or more computer programs that when executed perform methods of the present solution need not reside on a single computer or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present solution.

[0155] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Program modules can include routines, programs, objects, components, data structures, or other components that perform particular tasks or implement particular abstract data types. The functionality of the program modules can be combined or distributed as desired in various implementations.

[0156] Also, data structures may be stored in computer-readable media in any suitable form. For simplicity of illustration, data structures may be shown to have fields that are related through location in the data structure. Such relationships may likewise be achieved by assigning storage for the fields with locations in a computer-readable medium that convey relationship between the fields. However, any suitable mechanism may be used to establish a relationship between information in fields of a data structure, including through the use of pointers, tags or other mechanisms that establish relationship between data elements.

[0157] Any references to implementations or elements or acts of the systems and methods herein referred to in the singular can include implementations including a plurality of these-41- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109elements, and any references in plural to any implementation or element or act herein can include implementations including only a single element. References in the singular or plural form are not intended to limit the presently disclosed systems or methods, their components, acts, or elements to single or plural configurations. References to any act or element being based on any information, act or element may include implementations where the act or element is based at least in part on any information, act, or element.

[0158] Any implementation disclosed herein may be combined with any other implementation, and references to “an implementation,” “some implementations,” “an alternate implementation,” “various implementations,” “one implementation” or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

[0159] References to “or” may be construed as inclusive so that any terms described using “or” may indicate any of a single, more than one, and all of the described terms. References to at least one of a conjunctive list of terms may be construed as an inclusive OR to indicate any of a single, more than one, and all of the described terms. For example, a reference to “at least one of ‘A’ and ‘B’” can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Elements other than ‘A’ and ‘B’ can also be included.

[0160] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods.

[0161] Although the figures and description may illustrate a specific order of method steps, the order of such steps may differ from what is depicted and described, unless specified differently above. Also, two or more steps may be performed concurrently or with partial concurrence, unless specified differently above.

[0162] Where technical features in the drawings, detailed description or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims. Accordingly, neither the -42- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109reference signs nor their absence have any limiting effect on the scope of any claim elements.

[0163] The systems and methods described herein may be embodied in other specific forms without departing from the characteristics thereof. The foregoing implementations are illustrative rather than limiting of the described systems and methods. Scope of the systems and methods described herein is thus indicated by the appended claims, rather than the foregoing description, and changes that come within the meaning and range of equivalency of the claims are embraced therein.-43- 4936-8535-4889.1

Claims

Atty. Dkt. No.: 642631-0109WHAT IS CLAIMED IS:

1. A system comprising:one or more processors; anda memory storing computer instructions, the computer instructions when executed by the one or more processors cause the system to:perform automated immune scoring (alS) on a first set of grids of digital pathology images;extract, using one or more neural networks, features of the first set of grids based on automated immune scores;extract, using the one or more neural networks, features of a second set of grids of the digital pathology images based on manual immune scores;perform similarity matching on the features of the first set of grids and the features of the second set of grids;select a subset of grids from the first set of grids for further analysis based on the similarity matching; andpredict one or more regions of interest (ROIs) in the subset of grids.

2. The system of claim 1, wherein the digital pathology images comprise brightfieldbased images.

3. The system of claim 1, wherein the one or more processors are to generate virtual multiplex immunofluorescent (vmlF) images from hematoxylin & eosin (H&E) stained images.

4. The system of claim 1, wherein the digital pathology images comprise fluorescence-based images.

5. The system of claim 1, wherein the one or more processors are to calculate a feature similarity score based on the features of the first set of grids and the features of the second set of grids.

6. The system of claim 5, wherein the one or more processors are to select the subset of grids as a subset of grids with a highest feature similarity score.-44- 4936-8535-4889.1Atty. Dkt. No.: 642631-01097. The system of claim 5, wherein the feature similarity score is calculated using cosine similarity.

8. The system of claim 1, wherein the one or more processors cause the system to perform a sliding window method to predict the one or more ROIs.

9. The system of claim 8, wherein the sliding window method comprises:determining a size of a window based on a target ROI size; andfor each grid of the subset of grids:sliding the window across the grid,performing alS for each window position; andselecting the window position with an automated immune score closest to an automated immune score of an entire grid as an ROI.

10. A system comprising:one or more processors; anda memory storing computer instructions, the computer instructions when executed by the one or more processors cause the system to:at least one of (i) generate virtual multiplex immunofluorescence (vmlF) images or (ii) acquire multiplex immunofluorescence (mIF) images of tissue samples;perform automated immune scoring (alS) on a first set of grids of at least one of the vmlF images or the mIF images;extract, using one or more neural networks, features of the first set of grids based on automated immune scores;extract, using the one or more neural networks, features of a second set of grids based on manual immune scores;perform similarity matching on the features of the first set of grids and the features of the second set of grids;select a subset of grids from the first set of grids for further analysis based on the similarity matching; andpredict one or more regions of interest (ROIs) in the subset of grids.

11. The system of claim 10, wherein the one or more processors are to perform alS based at least on:-45- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109classifying pixels within the first set of grids as immune based on immune attributes; andquantifying immune cell infiltration of the pixels within the first set of grids based on a color range determined by cellular features.

12. The system of claim 10, wherein the one or more processors are to preprocess the mIF images using a contour mask to distinguish between glass and tissue regions.

13. The system of claim 10, wherein the one or more processors are to perform similarity matching by comparing the features of the first set of grids to the features of the second set of grids.

14. The system of claim 13, wherein the one or more processors are to calculate a feature similarity score based on the features of the first set of grids and the features of the second set of grids.

15. The system of claim 13, wherein Spearman’s correlation is used for comparing the features of the first set of grids to the features of the second set of grids.

16. The system of claim 13, wherein Cohen’s K score is used for comparing the features of the first set of grids to the features of the second set of grids.

17. The system of claim 14, wherein the feature similarity score is calculated using cosine similarity.

18. The system of claim 14, wherein the one or more processors are to select the subset of grids as a subset of grids with a highest feature similarity score.

19. The system of claim 10, wherein the one or more processors cause the system to perform a sliding window method to predict the one or more ROIs.

20. The system of claim 19, wherein the sliding window method comprises:determining a size of a window based on a target ROI size; andfor each grid of the subset of grids:-46- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109sliding the window across the grid,performing alS for each window position; andselecting the window position with an automated immune score closest to an automated immune score of an entire grid as an ROI.

21. The system of claim 10, wherein the one or more neural networks is a custom trained convolutional neural network (CNN) model.

22. The system of claim 21, wherein the CNN is trained using a cross-fold validation technique.

23. A method comprising:at least one of (i) acquiring multiplex immunofluorescence (mIF) images of tissue samples or (ii) generating virtual mIF (vmlF) images of tissue samples;performing automated immune scoring (alS) on a first set of grids of at least one of the mIF images or the vmlF images;extracting, using one or more neural networks, features of the first set of grids based on automated immune scores;extracting, using one or more neural networks, features of a second set of grids based on manual immune scores;performing similarity matching on the features of the first set of grids and the features of the second set of grids;selecting a subset of grids from the first set of grids for further analysis based on the similarity matching; andpredicting one or more regions of interest (ROIs).

24. The method of claim 23, wherein performing alS comprises:classifying pixels within the first set of grids as immune based on immune attributes; andquantifying immune cell infiltration of the pixels within the first set of grids based on a color range.

25. The method of claim 23, wherein the second set of grids are received as input via a user interface.-47- 4936-8535-4889.1Atty. Dkt. No.: 642631-010926. The method of claim 23, further comprising preprocessing the mIF images using a contour mask to distinguish between glass and tissue regions.

27. The method of claim 23, wherein similarity matching comprises calculating a feature similarity score between the features of the first set of grids and the features of the second set of grids.

28. The method of claim 27, wherein the feature similarity score is calculated using cosine similarity.

29. The method of claim 27, wherein selecting the subset of grids from the first set of grids comprises selecting a number of grids with a highest feature similarity score.

30. The method of claim 23, wherein predicting one or more ROIs comprises using a sliding window method on the subset of grids.

31. The method of claim 30, wherein the sliding window method comprises:determining a size of a window based on a target ROI size; andfor each grid of the subset of grids:sliding the window across the grid,performing alS at each window position, andselecting the window position with an automated immune score closest to an automated immune score of an entire grid as an ROI.

32. The method of claim 23, wherein generating vmlF images of tissue samples comprises:acquiring mIF images of the tissue samples;restaining the mIF images with hematoxylin & eosin (H&E) to obtain H&E images; spatially aligning the H&E images with the mIF images; andproviding the H&E images and the mIF images to a trained convolutional neural network (CNN) model.

33. A method compri sing :acquiring hematoxylin & eosin (H&E) images of tissue samples;-48- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109dividing the H&E images into a plurality of tiles using one or more neural networks; extracting features from each tile of the plurality of tiles using one or more neural networks;assigning a probability to each tile of the plurality of tiles based on the extracted features; andpredicting one or more regions of interest (ROIs) based on the probability assigned to each tile.

34. The method of claim 33 wherein assigning the probability to each tile of the plurality of tiles comprises:providing a trained neural network with the features extracted from each tile; and comparing, using the trained neural network, the features extracted from each tile to features extracted from manually identified ROIs.

35. The method of claim 33, wherein predicting one or more ROIs comprises a sliding window method.

36. The method of claim 35, wherein the sliding window method comprises:determining a size of a window based on a target ROI size;overlaying the window onto the H&E images;sliding the window across a plurality of window positions;for each window position:calculating an average probability; andselecting the window position with a highest average probability as an ROI.

37. A non-transitory computer-readable medium storing computer instructions, the computer instructions when executed by one or more processors cause the one or more processors to:acquire multiplex immunofluorescence (mIF) images of tissue samples; perform automated immune scoring (alS) on a first set of grids of the mIF images; acquire a second set of grids of the mIF images with manual immune scores (mIS); extract features of the first set of grids based on automated immune scores; extract features of the second set of grids based on manual immune scores;-49- 4936-8535-4889.1Atty. Dkt. No.: 642631-0109compare features of the first set of grids to features of the second set of grids using a similarity matching method;select grids for further analysis based on the similarity matching method; and predict one or more regions of interest (ROIs) based on the selected grids.-50- 4936-8535-4889.1

Images