Digital image processing for generating multi-channel images

Abstract

Systems, apparatuses, and methods disclosed herein relate to a digital image processing system. In one aspect, the digital image processing system includes one or more processors and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the digital image processing system to access a digital image depicting a portion of a biological sample obtained from a subject. A projecting space and a staining category for an image conversion is defined by the digital image processing system, causing the digital image to be projected to the projecting space using a neural network model. The projected image is output by the digital image processing system.

Description

Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT DIGITAL IMAGE PROCESSING FOR GENERATING MULTI-CHANNEL IMAGES FIELD

[0001] The present disclosure relates to digital image processing techniques, and in particular to techniques for converting unstained or single-marker-stained slides into images with artificially added stains using generative adversarial networks.BACKGROUND

[0002] The integration of artificial intelligence (Al) and digital imaging into the pathology workflow has significantly transformed the traditional role of pathology in the clinical setting. Advancements in Al algorithms, computer-aided diagnostic techniques, and the development of whole slide imaging (WSI) have expanded the tools available to clinicians and researchers for deciphering disease pathology and advancing drug discover}', translation, and clinical applications. This technological advancement is known as digital pathology, which involves the management and interpretation of pathology information in a digital environment. High-resolution images of biological samples are generated using scanning devices, which may be stained to produce digital images such as WSIs. Digital pathology software facilitates faster processing, cheaper storage, and easy management of these images, enhancing the ability to share slide images across the clinical community.

[0003] Initially, Al-aided processing of digital images was primarily used for image-based diagnosis in fields such as radiology and cardiology. However, the advent of WSI has significantly enhanced the application and benefits of Al processing. Unlike X-ray and magnetic resonance imaging, WSI provides extensive information due to its large size (e.g., 100k x 100k pixels) and increased image complexity. This complexity arises from features often being in color due to various staining techniques (e.g., Hematoxylin and Eosin (H&E), or immunohistochemical techniques), the availability' of information at multiple scales (e.g., 4x, 20x, etc.), and multiple z-stack levels that offer different view s of the image depending on the plane of focus. WSI also benefits from not being restricted by image orientation (e.g., anatomical orientation) as encountered in radiology. Consequently, Al-assisted processing offers several technological advancements, including the ability7to analyze digital images within a three-dimensional spatial context, process large volumes of data rapidly, generate labels at a lower cost, and provide consistent outputs free from human fatigue or subjective variability.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0004] Standard histological and immunohistochemical staining procedures in pathology laboratories entail complex and time-intensive sample preparation and staining processes. For instance, H&E staining typically requires about 45 minutes to an hour, while IHC staining can take between 4 to 8 hours or more. Additionally, these chemical-based staining methods are inherently destructive, rendering the tissue sections unsuitable for additional staining and further molecular analysis. Thes limitations highlight the urgent need for innovative approaches that can streamline the workflow and maintain the integrity of the samples for subsequent analysis.BRIEF SUMMARY

[0005] Disclosed are methods, systems, and computer readable storage media for converting unstained or single-marker-stained slides into images with artificially added stains using generative adversarial networks. The methods, systems, and computer readable storage media may be embodied in a variety of ways.

[0006] In various embodiments, a digital image processing system is provided comprising one or more processors and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the digital image processing system to perform operations comprising: accessing a digital image depicting a portion of a biological sample obtained from a subject; defining a projecting space and a staining category for an image conversion, wherein the projecting space is a multi-dimensional space with at least one dimension depicting a luminance of an image; projecting, using a neural network model, the digital image to the projecting space to generate a projected image, wherein each voxel of the digital image is projected to a voxel of the projected image with a same location of the digital image, and wherein the neural network model is trained to generate a virtually stained image of an input image based on the staining category ; and outputting the projected image.

[0007] In some embodiments, the operations further comprise training the neural network model, wherein the training comprises: accessing a set of digital images and their corresponding ground truth images, wherein each digital image depicts a portion of a biological sample obtained from a subject, and wherein each corresponding ground truth image depicts the portion of the biological sample after the biological sample being stained based on the staining category’; aligning each pair of the digital image and its corresponding ground truth image using a registration algorithm; extracting a set of ground truth patches from each ground truth image, wherein each ground truth patch covers a portion of the stained biological sample;Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT generating a set of input patches from each corresponding aligned digital image, wherein each input patch is corresponding to a ground truth patch in the set of ground truth patches; accessing a set of hyperparameters for the neural network model, wherein the set of hyperparameters comprises a depth of a neural network used by the neural network model and a number of filters used by each layer of the neural network; accessing a loss function for the training; iteratively training the neural network model for a number of epochs using training data based on the set of hyperparameters and the loss function, wherein the training data comprises the set of input patches from each digital image and the set of ground truth patches from each corresponding aligned ground truth image; and providing the neural network model.

[0008] In some embodiments, the training further comprises: randomly selecting at least 1000 locations on each ground truth image; and segmenting the set of ground truth patches from each ground truth image on the at least 1000 locations, wherein each ground truth patch of the set of ground truth patches is of a predetermined size, and wherein the set of input patches is segmented from the at least 1000 locations on the corresponding aligned digital image.

[0009] In some embodiments, the predetermined size is 50 x 50, 100 x 100, 256 x 256, or 512 x 512.

[0010] In some embodiments, the training further comprises: performing tissue detection on the ground truth image before selecting the at least 1000 locations, wherein the at least 1000 locations are selected from the detected tissue portion on the ground truth image; examining, manually or using an artifact detection algorithm, whether each ground truth patch of the set of ground truth patches comprises an artifact; and removing the ground truth patch and its corresponding input patch from the training when the ground truth patch is examined to comprise the artifact.

[0011] In some embodiments, the artifact is a blur, an out-of-focus (OOF), a noise, a vignetting, or a chromatic aberration.

[0012] In some embodiments, the registration algorithm is an affine registration, a B-spline registration, an elastic registration, a mutual information registration, a Demons algorithm, or a deep learning-based registration.

[0013] In some embodiments, the set of ground truth patches cover at least a predetermined portion of the biological sample.

[0014] In some embodiments, the predetermined portion is about 10%.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0015] In some embodiments, the neural network is a Generative Adversarial Network (GAN) comprising a generator and a discriminator, wherein the neural network model is provided with the generator.

[0016] In some embodiments, the generator is a U-net and the discriminator is a convolutional neural network (CNN).

[0017] In some embodiments, the loss function is determined based on a quadratic generator loss and a quadratic discriminator loss, and wherein parameters of the loss function is included in the set of hyperparameters.

[0018] In some embodiments, the staining category is selected from a group consisting of: Hematoxylin and Eosin (H&E) staining, Masson’s Trichrome staining, Periodic Acid-Methenamine Silver (PAMS) staining, Periodic Acid-Schiff (PAS) staining, Gram staining, Periodic Acid-Schiff (PAS) staining. Jones' silver staining, Immunohistochemistry (IHC) staining, a Giemsa staining, a Congo red staining, and a Ziehl-Neelsen staining.

[0019] In some embodiments, the digital image processing system further comprises an image scanning device, wherein the image scanning device is configured to capture images of biological sample using a plurality7of scopes, wherein a slide containing the biological sample to be imaged is loaded to the image scanning device, and wherein the operations further comprise: selecting a region of interest in a scannable portion of the slide; calibrating the image scanning device using an autofocus algorithm; and capturing the digital image of the biological sample using the calibrated image scanning device.

[0020] In some embodiments, the biological sample is pre-stained with DAPI (4',6-diamidino-2-phenylindole), Opal 480, Opal 520, Opal 570, Opal 620. Opal 690, or Opal 780.

[0021] In various embodiments, a method is provided, comprising: accessing a set of digital images and their corresponding ground truth images, wherein each digital image depicts a portion of a biological sample obtained from a subject, and wherein each corresponding ground truth image depicts the portion of the biological sample after the biological sample being stained based on a staining category; aligning each pair of the digital image and its corresponding ground truth image using a registration algorithm; extracting a set of ground truth patches from each ground truth image, wherein each ground truth patch covers a portion of the stained biological sample; generating a set of input patches from each corresponding aligned digital image, wherein each input patch is corresponding to a ground truth patch in theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT set of ground truth patches; accessing a set of hyperparameters for a neural network model, wherein the set of hyperparameters comprises a depth of a neural network used by the neural network model and a number of filters used by each layer of the neural network; accessing a loss function for training the neural network model; iteratively training the neural network model for a number of epochs using training data based on the set of hyperparameters and the loss function, wherein the training data comprises the set of input patches from each digital image and the set of ground truth patches from each corresponding aligned ground truth image; and providing the neural network model.

[0022] In some embodiments, the method further comprises: randomly selecting at least 1000 locations on each ground truth image; and segmenting the set of ground truth patches from each ground truth image on the at least 1000 locations, wherein each ground truth patch of the set of ground truth patches is of a predetermined size, and wherein the set of input patches is segmented from the at least 1000 locations on the corresponding aligned digital image.

[0023] In some embodiments, the predetermined size is 50 x 50, 100 x 100, 256 x 256, or 512 x 512.

[0024] In some embodiments, the method further comprises: performing tissue detection on the ground truth image before selecting the at least 1000 locations, wherein the at least 1000 locations are selected from the detected tissue portion on the ground truth image; examining, manually or using an artifact detection algorithm, whether each ground truth patch of the set of ground truth patches comprises an artifact; and removing the ground truth patch and its corresponding input patch from the training when the ground truth patch is examined to comprise the artifact.

[0025] In some embodiments, the artifact is a blur, an out-of-focus (OOF), a noise, a vignetting, or a chromatic aberration.

[0026] In some embodiments, the registration algorithm is an affine registration, a B-spline registration, an elastic registration, a mutual information registration, a Demons algorithm, or a deep learning-based registration.

[0027] In some embodiments, the set of ground truth patches cover at least a predetermined portion of the biological sample.

[0028] In some embodiments, the predetermined portion is about 10%.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0029] In some embodiments, the neural network is a Generative Adversarial Network (GAN) comprising a generator and a discriminator, wherein the neural network model is provided with the generator.

[0030] In some embodiments, the generator is a U-net and the discriminator is a convolutional neural network (CNN).

[0031] In some embodiments, the loss function is determined based on a quadratic generator loss and a quadratic discriminator loss, and wherein parameters of the loss function is included in the set of hyperparameters.

[0032] In some embodiments, a computer implemented method is provided to perform the actions or operations disclosed herein.

[0033] In some embodiments, a computing system is provided, comprising: one or more processors; and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the computing system to perform actions or operations in any one of the methods disclosed herein.

[0034] In some embodiments, a non-transitory computer readable storage medium is provided comprising computer program instruction that, when executed by a computer processor, cause the computer processor to perform actions or operations of part or all of one or more methods disclosed herein.

[0035] In some embodiments, a computer-program product is provided that is tangibly embodied in a non-transitory computer-readable memory that includes instructions which, when executed by the one or more processors, cause the one or more processors to perform any of the methods disclosed herein.

[0036] The terms and expressions which have been employed are used as terms of description and not of limitation, and there is no intention in the use of such terms and expressions of excluding any equivalents of the features shown and described or portions thereof, but it is recognized that various modifications are possible within the scope of the disclosure. Thus, it should be understood that although the claimed techniques have been specifically disclosed by embodiments and optional features, modification and variation of the concepts herein disclosed may be resorted to by those skilled in the art, and that such modifications and variations are considered to be within the scope of this application as defined by the appended claims.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings illustrate certain embodiments of the technology7and are not limiting. For clarity and ease of illustration, the drawings are not made to scale and, in some instances, various aspects may be shown exaggerated or enlarged to facilitate an understanding of particular embodiments.

[0038] FIG. 1 shows an exemplary computing environment for processing, storing, transmitting, and displaying digital pathology7images in accordance with various embodiments.

[0039] FIG. 2 is a flowchart illustrating a process for training a machine learning model that can be implemented in a digital image processing environment to perform a digital staining process in accordance with various embodiments.

[0040] FIG. 3 shows an exemplary image processing in accordance with various embodiments.

[0041] FIG. 4 shows an exemplary patch extraction from a brightfield image in accordance with various embodiments.

[0042] FIG. 5 shows an exemplary framework of a generative adversarial network (GAN) to perform the virtual staining of a digital image in accordance with various embodiments.

[0043] FIG. 6 illustrates an exemplary generator with a U-net architecture used in a GAN to generate synthesized image in accordance with various embodiments.

[0044] FIG. 7 shows an exemplary7CNN with a fully connected output layer used as a discriminator of a GAN in accordance with various embodiments.

[0045] FIG. 8 is a flowchart illustrating a process for digital image processing using a trained machine learning model in accordance with various embodiments.

[0046] FIG. 9 illustrates the training and implementation of a machine learning model that processes digital pathology' images in accordance with various embodiments.TERMS

[0047] As used herein, the articles "a" and "an" are used herein to refer to one or to more than one (i.e. , at least one) of the grammatical object of the article. By way of example, an element means at least one element and can include more than one element.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0048] As used herein, the terms "about." “similarly,'’ “substantially,” and “approximately” are defined as being largely but not necessarily wholly what is specified (and include wholly what is specified) as understood by one of ordinary skill in the art. In any disclosed embodiment, the term “about,” “similarly,” “substantially,” or “approximately” may be substituted with “within [a percentage] of what is specified, where the percentage includes 0.1 percent, 1 percent, 5 percent, and 10 percent, etc. Moreover, the term terms “about,” “similarly,” “substantially,” and “approximately” are used to provide flexibility to a numerical range endpoint by providing that a given value may be slightly above or slightly below the endpoint without affecting the desired result.

[0049] As used herein, when an action is “based on” something, this means the action can be based at least in part on at least a part of the something.

[0050] As used herein, the term “brightfield image” refers to a microscopy image of a sample captured using brightfield illumination, where the sample is illuminated with white light from below, producing a contrast based on the sample’s light absorption or scattering properties. The term “brightfield image” can be used interchangeably as “ground truth” for validating machine learning models or virtual staining techniques.

[0051] As used herein, the term “or” encompass both the inclusive sense, where either one or both of the conditions or elements can be present, and the exclusive sense, where only one of the conditions or elements can be present. In some instances, the term “or” is used interchangeably with “and / or.”

[0052] As used herein, the term “digital image,” “pathological image,” or “pathology image,” includes but is not limited to a histopathology image, a cytopathology image, a neuropathology image, or a molecular pathology image. It should be understood that the term can represent any visual representation of tissues, cells, or organs that may have been affected by various medical conditions, including cancers, infections, inflammatory' diseases, and degenerative disorders. These visual representation or images are typically obtained through medical imaging techniques and / or staining processes, allowing for detailed examination and analysis by pathologists or computers.

[0053] As described herein, the terms “patient” and “subject” are used interchangeably and refer to any single animal, more preferably a mammal (including humans and non-human animals such as dogs, cats, horses, rabbits, rats, cows, pigs, sheep, and non-human primates) that may have or may not have been diagnosed of a genetic or medical condition. In someAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT embodiments, subjects are “patients” e.g., living humans that are receiving medical care for a disease or condition. This includes persons with no defined illness who are being investigated for signs of pathology. In some embodiments, the patient or subject may be at risk of / diagnosed with / being treated for cancer.

[0054] As used herein, the term “pixel” or “voxel” refers to the smallest individual unit of a digital image, representing a specific intensity' or color value at a given spatial location. The term “pixel” can be used interchangeably with the term “voxel” in data representations.

[0055] As used herein, the term “staining category” refers to classifications of histological staining techniques used to visualize tissue or cellular components. Examples include Hematoxylin and Eosin (H&E) staining, Masson’s Trichrome staining, Periodic Acid-Methenamine Silver (PAMS) staining, Periodic Acid-Schiff (PAS) staining, Gram staining, Periodic Acid-Schiff (PAS) staining. Jones’ silver staining, Immunohistochemistry (IHC) staining, a Giemsa staining, a Congo red staining, and a Ziehl-Neelsen staining.

[0056] As used herein, the term “virtual staining images” or “virtually stained images” refers to digitally generated representations of biological samples that simulate the appearance of physically stained biological samples without requiring the actual physical staining. These images are created through computational techniques, such as machine learning or neural networks, applied to raw, label-free imaging data (e.g., autofluorescence or hyperspectral data), mimicking the effects of stains like H&E or PAS. Virtual staining reduces or eliminates the need for physical reagents, enabling nondestructive analysis, reducing costs, and streamlining workflows in diagnostic and research.

[0057] The use herein of the terms including, comprising, or having, and variations thereof, is meant to encompass the elements listed thereafter and equivalents thereof as well as additional elements. Embodiments recited as including, comprising, or having certain elements are also contemplated as consisting essentially of and consisting of those certain elements. As used herein, the term “and / or” refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0058] Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein, and each separate value is incorporated into the specification as if it were individually recited herein. For example, if a concentration range is stated as 1% to 50%. it is intended that values such as 2% to 40%, 10% to 30%, or 1% to 3%, etc., are expresslyAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT enumerated in this specification. A range can include the edge point value of the range. In some instances, the range excludes the edge point values. These are only examples of what is specifically intended, and all possible combinations of numerical values (e.g., integer, whole number, decimal, fraction, and the like) between and including the lowest value and the highest value enumerated are to be considered to be expressly stated in this disclosure.DETAILED DESCRIPTION

[0059] The ensuing description provides preferred exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the ensuing description of the preferred exemplary' embodiments will provide those skilled in the art with an enabling description for implementing various embodiments. It is understood that various changes may be made in the function and arrangement of elements without departing from the spirit and scope as set forth in the appended claims.

[0060] Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood that the embodiments may¬ be practiced without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary- detail in order to avoid obscuring the embodiments.

[0061] Also, it is noted that individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart or diagram may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in a figure. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0062] Publications cited herein and the material for which they are cited are hereby specifically incorporated by reference in their entireties.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT I. INTRODUCTION

[0063] Large medical centers typically have an average caseload of approximately 80,000 to over 100,000 different pathology samples per year, with daily averages ranging from 500 to 750 tissue blocks and more than 1,500 slides. As samples are acquired, histopathology' labs follow a series of standardized steps (e.g., tissue fixation, embedding, sectioning, and staining) to process the samples for examination and diagnosis. The processing of pathology samples is a crucial part of diagnostic workflows, with staining being a pivotal step in this process. Staining is essential for visualizing specific components within tissue samples, such as cell structures, proteins, or nucleic acids, which are critical for identifying pathological changes and making accurate diagnoses. Techniques like Hematoxylin and Eosin (H&E) staining, immunohistochemistry, and special stains are commonly employed to highlight various cellular elements and disease markers. The precision and reliability of these staining procedures are vital, as they directly impact the pathologist's ability to detect and diagnose conditions such as cancer, infections, and other diseases accurately.

[0064] Just the staining process alone can take hours to days to complete, especially when multiple stains for the same sample are required. Moreover, common histological staining techniques, such as H&E staining, require specialized reagents, equipment, and skilled technicians, making them very costly to run. The labor costs associated with staining pathology samples are substantial, driven by the need for highly trained personnel and the intricate nature of the procedures. Histotechnologists and laboratory technicians must have specialized training to perform these tasks with the required accuracy, as any errors in staining can lead to misdiagnosis or the need for repeat testing. Consequently, clinical centers must invest significantly in skilled labor and resources to ensure that the staining of pathology samples is conducted efficiently and effectively, balancing the essential need for diagnostic precision with the associated labor and material costs. In addition to time and cost, many staining methods involve chemical treatments that alter or destroy the sample. Thus, multiple biopsies, which are highly invasive procedures, may be necessary' to acquire a sufficient amount of sample, increasing the burden on the patient.

[0065] While manual staining is the current standard, integration of computational approaches, such as machine learning (ML) and Al, into pathology offer tremendous potential to provide assistance to pathologists as well as enhance data analysis, efficiency, speed, and cost. The development of Al introduces opportunities to transform unstained images intoAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT virtually stained images, reducing the need for physical staining equipment and reagents and speeding up the diagnostic process. However, the implementation of Al in digital pathology is not without challenges, particularly concerning the availability of high-quality training data (e.g., training data having accurate ground truths) for supervised and semi-supervised learning. Another challenge is ensuring the accuracy and reliability7of the virtual staining process, as any errors could lead to misdiagnosis or incorrect scientific conclusions. Additionally, there are technical hurdles related to the integration of Al systems with existing laboratory equipment and workflows. The interpretation of Al-generated results by medical and research professionals can also be complex, requiring specialized training. Furthermore, regulatory and ethical considerations, including data privacy and the validation of Al tools for clinical use, add another layer of complexity to the adoption of Al for virtual staining.

[0066] To address the problems and limitations in virtual staining approaches, disclosed herein are systems, methods, and apparatuses for improved virtual staining using well-trained Al-based models. One illustrative embodiment of the present disclosure is directed to a digital image processing system that includes one or more processors and one or more computer-readable media storing instructions which, when executed by the one or more processors, cause the digital image processing system to perform virtual staining operations. These virtual staining operations include: (i) accessing a digital image depicting a portion of a biological sample obtained from a subject; (ii) defining a projecting space and a staining category7for an image conversion, wherein the projecting space is a multi-dimensional space with at least one dimension depicting a luminance of an image; (iii) projecting, using a neural network model, the digital image to the projecting space to generate a projected image, wherein each voxel of the digital image is projected to a voxel of the projected image with a same location of the digital image, and wherein the neural network model is trained to generate a virtually stained image of an input image based on the staining category; and (iv) outputting the projected image.

[0067] The neural network is typically pretrained and integrated into the digital image processing system prior to the system’s implementation. To train the neural network to perform virtual staining, training paired images are acquired. The training paired images can be a set of digital images and their corresponding ground truth images, wherein each digital image depicts a portion of a biological sample obtained from a subject, and each corresponding ground truth image depicts substantially the same portion of the biological sample after the biological sample being stained based on a staining category7. Instead of using the WSI to train the neural network, a patch selection technique can be adopted to reduce computational cost and memoryAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT requirements, making the training process more efficient. Additionally, focusing on smaller regions allows the model to leam localized patterns and fine-grained details more effectively, improving its ability to generalize across diverse image features.

[0068] The training patches are aligned or registered before inputting into the neural network. Aligning the input patches with the ground truth patches before training ensures that the model leams more accurately from the data by reducing discrepancies and misalignments, which can lead to improved performance and better prediction accuracy. This alignment process also helps in minimizing noise and errors, thus enhancing the overall quality and reliability of the trained model. In some embodiments, the alignment is performed using a registration algorithm (e.g., rigid registration, affine registration, non-rigid (deformable, elastic) registration, featurebased registration, intensity-based registration, or the like). In some embodiments, the alignment process is performed before patch generation. Hyperparameters (including loss functions) for the neural network model may be preset or default values can be used to iteratively train the neural network model. Training stops based on one or more criteria (convergence, epoch number, time, or the like) to ensure efficiency and good generalization to new data. The trained neural network model is then deployed and can be updated periodically based on clinical or research needs.II. COMPUTING ENVIRONMENT

[0069] FIG. 1 shows an exemplary' computing environment 100 for processing, storing, transmitting, and displaying digital pathology images. The computing environment 100 includes a digital image processing platform 11 that enables the processing and analyzing of digital image (e.g., pathological images such as microscopic images of tissues). The digital image processing platform 110 can be implemented using software only (e.g., each module of the platform is a digital entity implemented using programs, code, or instructions executable by one or more processors), using hardware (e.g., a medical tool to perform sample collection, a digital microscope to generate digital image, a GPU, a CPU, or the like), or using a combination of hardware and software. The computing environment 100 also includes one or more storage(s) 115, a network 120, and one or more client devices 130A-130N. Although FIG. 1 illustrates a particular number of the components, it should be understood that any suitable number of components (e.g., client devices 130A-130N) may be included in the computing environment 100. It should also be understood that the computing environment 100Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT may include additional components such as a camera system, an optical system, a whole slide scanner, an image management platform, or the like.A. Digital Image Processing Platform

[0070] The digital image processing platform 110 may include multiple components or modules. For example, as shown in FIG. 1, the digital image processing platform 110 includes a sample collector 101. an imaging module 102, a preprocessing module 104, and an analyzing module 106. Each of the components or modules may be a solely software-based component or module, a solely hardware-based component or module, or a combination of software and hardware. Each component or module can be physically connected or integrated, or interact with each other through computing programming codes, scripts, or software. In different embodiments, the design of the digital image processing platform can be specifically tailored based on research, clinical, or commercial needs.

[0071] The sample collector 101 can be a physical device that collects and prepares samples for imaging. For example, the sample collector 101 can include automated slide loaders or robotic arms for precise handling and positioning of samples, sample trays or carousels for organizing multiple samples, and sample preparation tools such as tissue processors, microtomes, and staining apparatus. The sample collector 101 can also incorporate sample labeling and tracking mechanisms like barcode scanners and RFID tags to maintain sample identification and metadata management. The sample collector 101 may also include controlling chips that enable environmental controls, such as temperature and humidity regulation to ensure sample integrity. The sample collector 101 can also include software or systems that facilitate the collection, preparation, tracking, and management of samples. In some embodiments, the sample collector 101 can also be controlled by a user through a user interface and control software. By ensuring consistent and high-quality sample preparation, the sample collector 101 supports seamless integration with subsequent imaging, preprocessing, and analysis, facilitating reliable and efficient digital pathology workflows.

[0072] In some embodiments, the sample collector 101 can also be configured to perform physical staining of the collected samples. The staining functionality can be achieved by using the automated slide loaders and robotic arms to handle and position the collected samples with high precision, ensuring consistent and accurate application of staining reagents. The sample collector can be equipped with various staining preparation tools, such as staining reagents and software, which collectively ensure that samples are properly fixed, sliced, stained, andAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT prepared. The control software allows operators to set specific staining parameters, monitor progress, and ensure reproducibility of staining protocols. The configuration of the sample collector 101 ensures that the physical staining of samples is performed efficiently and to a high standard, preparing them for detailed imaging and training purposes.

[0073] Samples collected and prepared by the sample collector 101 can be sent to the imaging module 102 of the digital image processing platform 110 for converting the samples into high-resolution digital images. In some embodiments, the imaging module 102 is an integrated part of the sample collector 101. The imaging module 102 may comprise an optical unit 103 (e.g., advanced optical components such as high-quality objective lenses and precise light sources), along with a camera unit 105 (e.g., high-resolution cameras or image sensors) to capture detailed images of the samples. In some embodiments, the imaging module includes a motorized stage for precise movement and positioning of the slides, coupled with an autofocus system to ensure sharpness across the entire sample. Additionally, the imaging module integrates powerful control and processing software to manage image capture, real-time viewing, and initial enhancement of the images. By delivering high-quality digital representations of the samples, the imaging module 102 provides the foundation for accurate and detailed analysis in subsequent processing and analyzing stages.

[0074] The imaging module 102 can interact with the sample collector 101 in real time, facilitating a seamless workflow for sample processing and imaging. For instance, once a tissue sample is collected and prepared by the sample collector 101, it can be immediately imaged by the imaging module 102 to capture its initial state. Following this, the sample collector 101 performs the physical staining of the tissue sample, applying specific staining protocols to enhance the visibility of various structures within the sample. After the staining process is complete, the imaging module 102 captures a second set of images, now highlighting the stained features. This real-time interaction between the sample collector 101 and the imaging module 102 ensures that each step of the sample preparation and imaging process is precisely coordinated, resulting in high-quality, detailed images for accurate training, analysis, and diagnosis.

[0075] The digital images captured by the imaging module 102 can be sent to the preprocessing module 104. The preprocessing module 104 can include preprocessing software that performs tasks such as noise reduction, contrast enhancement, image normalization, and artifact removal to ensure that the images are of the highest quality before analysis. It may alsoAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT incorporate hardware accelerators (e.g., GPUs, TPUs, FPGAs, or the like) to speed up these preprocessing tasks, ensuring efficient handling of large image datasets.

[0076] In some embodiments, the preprocessing module 104 also align multiple images or sections of a slide to create a cohesive and accurate representation of the sample. The alignment may involve correcting any rotational, translational, or scaling discrepancies to create unified images. Alignment algorithms used may include: (i) feature-based methods, such as Scale-Invariant Feature Transform (SIFT) and Speeded-Up Robust Features (SURF), detect and match distinctive key points across image patches to achieve alignment; (ii) intensity -based methods, including cross-correlation and mutual information, align patches by comparing pixel intensity7distributions and maximizing similarity; (iii) optimization-based methods, such as Lucas-Kanade algorithm, are used to iteratively refine the alignment by minimizing differences between patches; and (iv) deep learning-based methods, leverage neural networks to learn the transformation that aligns two images. These algorithms can be combined with geometric transformations like affine, rigid, or perspective, help to correct any misalignments and distortions in the images and ensure precise alignment.

[0077] In some embodiments, the preprocessing module 104 prepares images or patches for machine learning applications (e.g., training and validating a machine learning model). The preprocessing module may chop digital images into patches and may randomly select a predetermined number of patches to be used for training purposes. Patches may also be selected based on algorithms to ensure that different features of the digital images are represented by the selected patches. In some embodiments, stratified sampling methods are used to ensure that all features and regions are represented (e.g., dividing the image into different strata or regions based on criteria (e.g., intensity, texture) and then randomly sampling patches from each stratum). In some embodiments, quality7checks are implemented to ensure that the selected patches are representative and of high quality7(e.g., verifying that patches are not too noisy, blurred, or contain artifacts that could skew the training or analysis). The quality check may also be performed via human intervention.

[0078] Proper registration and sampling can enhance both the accuracy and efficiency of training neural networks (e.g., in virtual staining). By ensuring precise alignment of input and target images, these steps minimize computational redundancies and misaligned features, enabling the model to leam consistent patterns more effectively and complete training faster. Furthermore, Al-powered tools can be used to normalize color variations in histopathologicalAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT images caused by differences in staining protocols, reducing experimental variability between sample batches, enhancing consistency and accuracy in diagnosis. The integration of Al in digital pathology also supports pathologists by automating routine and time-consuming tasks, reducing the workload for pathologists, allowing pathologists to focus on more complex cases and decision-making processes, and ultimately leading to cost savings for healthcare institutions. Overall, the application of Al in digital pathology holds great promise for transforming diagnostic practices and enhancing the overall quality and cost of healthcare.

[0079] In some embodiments, the preprocessing module 104 also performs training functions. For example, the prepared patches may be input into a selected machine learning model to train or fine-tune the machine learning model. The trained machine learning model may be used by the analyzing module 106 to perform virtual staining or segmentation, or by the diagnostic module 108 to perform disease prediction, diagnosis, or design a personalized treatment. The training performed at the preprocessing module 104 may be a supervised training, an unsupervised learning, a semi-supervised learning, or a reinforcement learning. The machine learning model may be selected from linear regression, logistic regression, decision trees, random forests, support vector machines (SVMs). k-nearest neighbors (KNN), artificial neural network architectures like convolutional neural networks (CNNs), recurrent neural networks (RNNs), or generative adversarial network (GANs), k-means clustering, hierarchical clustering, Gaussian mixture models, principal component analysis (PCA), Q-leaming, deep Q-networks (DQNs), and policy gradient methods. Additionally, ensemble methods like boosting (e.g., AdaBoost, Gradient Boosting Machines), bagging (e.g., Bootstrap Aggregating), and stacking can be used to combine multiple models to enhance learning and performance.

[0080] Images or patches are further processed and analyzed using the analyzing module 106. The analyzing module 106 may include a virtual staining unit 107, which applies computational techniques to simulate traditional staining methods used in pathology, and a segmentation unit 109, which can partition the stained images into distinct regions or segments corresponding to different staining parts. The analyzing module 106 may also perform other functions such as quantitative analysis (e.g., cell counting, biomarker identification and quantification), pattern recognition (e.g.. feature extraction and classification of tissues), spatial analysis (e.g., analyzing spatial relationships by reconstructing three-dimensional images), anomaly detection, and other machine learning or artificial intelligence integration.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0081] The virtual staining unit 107 leverages machine learning and image processing algorithms to digitally stain images and / or enhance the colors of already stained images. The virtual staining unit 107 provides the efficiency and flexibility to apply multiple staining protocols to the same sample digitally, offering a comprehensive view of the specimen without altering or destroying delicate structures within the samples. Virtual staining offers nondestructive techniques that utilizes advanced image processing algorithms to simulate the appearance of stained samples without any physical manipulation, thereby preserving the integrity of the original specimen. The advanced image processing algorithms also ensure that the desired features of the sample are highlighted, facilitating easier and more accurate analysis by pathologists and researchers. By eliminating the variability and potential artifacts introduced by manual staining processes, virtual staining contributes to more consistent and reproducible results.

[0082] In some embodiments, the virtual staining unit 107 employs one or more trained machine learning model to perform the virtual staining of samples. The machine learning model may be provided by the preprocessing module 104 or may be a model stored in the storage 115. The virtual staining unit 107 may take a digital image from the imaging module 102 or from the preprocessing module 104 to input in the trained machine learning model to generate virtually stained image. The input image can be a grayscale image, or a fluorescence image with enhanced contrast. The output image is a digitally stained image that highlights different structures and components of the sample, enabling detailed visualization and analysis.

[0083] The output image can be further processed and used in downstream applications. For example, the segmentation unit 109 may employ segmentation algorithms, such as deep learning-based methods, to segment the staining structures or components from the image. In some embodiments, the structures or components are cells, tissues, nuclei, cytoplasm, cell membranes, proteins, enzymes, lipids, carbohydrates, glycogen, connective tissue, mitochondria, bacteria, fungi, nen e cells, fibers, and other anatomical features. Segmentation allows for the precise measurement and quantification of different structures and components within the sample, including statistical analysis, partem recognition, and anomaly detection. Segmentation also facilitates diagnostic and research purposes.

[0084] The diagnostic module 108 is configured to provide comprehensive and personalized diagnostic insights from the analyzed images. The diagnostic module 108 may integrate algorithms and machine learning models to interpret the data provided by the analyzing moduleAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT 106, offering precise and reliable diagnostic information. It can utilize patern recognition, anomaly detection, and predictive analytics to identify and classify various pathological conditions, aiding in the accurate diagnosis of diseases. The diagnostic module 108 can also generate detailed reports and visualizations, summarizing the findings and highlighting critical areas of concern. By leveraging the power of artificial intelligence and advanced data analysis, the diagnostic module 108 enhances the efficiency and accuracy of the diagnostic process, supporting medical professionals in making informed decisions and improving patient outcomes.B. Data and Model Storage

[0085] The storage 115 can be served as a central repository’ for vast amounts of high-resolution image data and associated metadata. It can also save other medical data and models or algorithms. The storage 115 can be a local storage that is connected to the network 120, facilitating seamless and high-speed data transfer between the storage 115 and components of the digital image processing platform 110, ensuring efficient access and retrieval of data. In some embodiments, the storage 115 is a part of the digital image processing platform 110. The digital image processing platform 1 10 can utilize the storage 115 to archive and access raw images, processed images, and analytical results, maintaining data integrity and availability for various stages of image analysis and diagnostics. The digital image processing platform 110 can also utilize the storage 115 to access or call a machine learning model or algorithm for data processing and analysis. Additionally, the storage 115 interfaces with the client devices BOATS!, allowing medical professionals, researchers, and remote users to access and interact with the stored data through secure, real-time connections. The storage 115 can also be configured to enable authorized use or access by authorized users. In some embodiments, the storage 115 is a cloud-based storage, and may be an integral part of the network 120.C. Network

[0086] The netw ork 120 supports data communications using a variety' of available protocols including yvithout limitation TCP / IP (transmission control protocol / Intemet protocol), SNA (systems network architecture), IPX (Internet packet exchange), AppleTalk®, and the like. Merely by way of example, the network 120 may be a local area network (LAN), networks based on Ethernet, Token-Ring, a wide-area network (WAN), the Internet, a virtual netyvork, a virtual private network (VPN), an intranet, an extranet, a public syvitched telephone network (PSTN), an infra-red network, a wireless network (e.g.. a network operating under any of theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT Institute of Electrical and Electronics (IEEE) 1002.11 suite of protocols, Bluetooth®, and / or any other wireless protocol), and / or any combination of these and / or other networks.

[0087] To connect the digital image processing platform 110 (or any of its modules), the storage 115, and the client devices 130A-N to the network 120 (or to each other), various types of links can be utilized. These links can include wireline connections such as Digital Subscriber Line (DSL) or Data Over Cable Senice Interface Specification (DOCSIS), which provide reliable and high-speed data transfer over copper or coaxial cables. Wireless connections, such as Wi-Fi or Worldwide Interoperability for Microwave Access (WiMAX), offer the flexibility of mobility and ease of installation without the need for physical cables. Additionally, optical links like Synchronous Optical Network (SONET) or Sy nchronous Digital Hierarchy (SDH) provide high bandwidth and long-distance data transmission capabilities, making them suitable for connecting multiple components across different locations. It is important to note that the type of link used to connect these components can vary based on specific network requirements, infrastructure availability, and performance needs. The integration of these diverse connection methods ensures a robust and efficient data communication framework, enabling seamless interaction and real-time data access across the entire computing environment 100. This versatile connectivity supports the complex workflows and data-intensive operations performed by the digital image processing platform 110.D. Client Device

[0088] The client devices 130A-N are electronic devices including hardware, software, or embedded logic components or a combination of two or more such components and capable of interacting with the digital image processing platform 110 and the storage 115, optionally via the network 120. The client devices 130A-N may include various types of computing systems such as portable handheld devices, general purpose computers such as personal computers and laptops, workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computing devices may run various types and versions of software applications and operating systems (e.g., Microsoft Windows®, Apple Macintosh®, UNIX® or UNIX-like operating systems. Linux or Linux-like operating systems such as Google Chrome™ OS) including various mobile operating systems (e g., Microsoft Windows Mobile®, iOS®, Windows Phone®, Android™, BlackBerry®, Palm OS®). Portable handheld devices may include cellular phones, smartphones, (e.g., an iPhone), tablets (e.g., iPad®), personal digital assistants (PDAs), and theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT like. Wearable devices may include Google Glass® head mounted display, and other devices. The client devices 130A-N may be capable of executing various different applications such as various Internet-related apps, communication applications (e.g., E-mail applications, short message service (SMS) applications) and may use various communication protocols. Any suitable client devices 130A-N can be configured to generate and output product target discover}’ content to a user. For example, users may use one client device (e g., BOA) to execute one or more applications, which may generate one or more discovery or storage requests that may then be serviced in accordance with the teachings of this disclosure. Each of the client devices 130A-N provides an interface (interfaces 132A-N) (e.g., a graphical user interface) that enables a user of the client device to interact with the client device. The client devices 130A-N may also output information to the user via the interfaces 132A-N (e.g., displaying a report).

[0089] The client devices 130A-N are capable of inputting data, generating data, and receiving data. For example, a user of the client device BOA may send out a request to perform virtual staining of a designated sample using the interface 132A. The request may be sent out through the network 120 to the digital image processing platform 110. The sample collector 101 can access the designated sample and the imaging module 102 may capture a digital image of the designated sample. After the digital image being preprocessed by the preprocessing module 104, the virtual staining unit 107 of the analyzing module 106 performs the virtual staining based on the preprocessed image and output the stained image to the client device BOA via the interface 132A. The stained image and related data can be saved to the storage 115 for future analysis and diagnosis. Additional information such as patient profiles may also be retrieved from the storage 115 to further facilitate staining or result interpretation. For example, the stained image and the additional information may be used together to predict if a subject where the sample is obtained has developed a genetic condition (e.g., a disorder, a disease, or a cancer). The analyzed data may be transmitted to the client device BOA via the network 120.III. FLOWCHARTS FOR DIGITAL IMAGE PROCESSINGA . 1 raining A Machine Learning Model

[0090] FIG. 2 is a flowchart illustrating a process 200 for training a machine learning model that can be implemented in a digital image processing environment (e.g., the digital image processing platform 110 described with respect to FIG. 1 ) to perform a digital staining process.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT The process 200 depicted in FIG. 2 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof (e.g., the intelligent selection machine). For example, the process 200 may be trained and implemented in the digital image processing platform 110 depicted in FIG. 1. The process 200 presented in FIG. 2 and described below is intended to be illustrative and non-limiting. Although FIG. 2 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different orders, or some steps may also be performed in parallel.Al-Whole Slide Image Preparation

[0091] Process 200 may begin with acquiring the biological samples (not depicted in FIG.2). Each biological sample (specimen) may be acquired from a same or different patient. The specimen can be a cell-containing liquid or a tissue. The specimen can comprise, but is not limited to, amniotic fluid, tissue biopsies, blood, blood cells, bone marrow, fine needle biopsy samples, peritoneal fluid, amniotic fluid, plasma, pleural fluid, saliva, semen, serum, tissue or tissue homogenates, frozen or paraffin sections of tissue. Methods of acquiring the specimen include but are not limited to biofilms, aspirations, tissue sections, swabs, drawing blood or other fluids, surgical or needle biopsies, and the like. The specimen can be acquired from a healthy subject or a subject at risk of / having been diagnosed with / receiving treatment for a disease (e.g., cancer). The sample acquisition may be performed, for example, using the sample collector 101 and the imaging module 102 described with respect to FIG. 1.

[0092] After samples are acquired, a whole slide image (WSI) is prepared for each sample (not depicted in FIG. 2). The WSI preparation may be performed, for example, using the sample collector 101 and the imaging module 102 described with respect to FIG. 1. WSI preparation is a process that transforms physical pathology slides into high-resolution digital images for analysis and diagnosis. In some embodiments, the sample is a tissue sample. The preparation may begin with tissue fixation, where biological tissue samples are collected and fixed using chemicals such as formalin to preserve the tissue structure and prevent decomposition. The processed tissue samples may be further processed by a series of steps involving dehydration, clearing, and paraffin embedding. For example, the fixed tissue can be dehydrated through a series of alcohol baths, cleared with a solvent like xylene, and infiltrated with paraffin wax to solidify the tissue. Then each processed tissue sample is embedded in a block of paraffin wax,Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT which provides support for thin slicing. Using a microtome, the paraffin-embedded tissue block is sliced into ultra-thin sections, e.g., 3-5 micrometers thick, and placed on glass slides. The prepared slides are loaded into a whole slide scanner (e.g., the imaging module 102), and high-resolution digital images of the entire tissue section are captured using advanced optical components with an autofocus system to ensure sharpness across the entire slide. It should be understood that other biological samples can also be used and prepared. The following processes are described using tissue samples as an example, and do not mean to limit the scope of the application.

[0093] The fixation and embedding may be performed using a fixation / embedding system. Fixation is a chemical process that preserves a sample’s chemical composition and cellular structure while preventing degradation. Fixatives, such as formaldehyde, neutral buffered formalin (NBF), paraformaldehyde (PF A), methanol, or Bouin’s solution, can be used to harden tissues, inactivate degradative enzymes, and eliminate bacteria. The fixation process can involve perfusion or immersion of the tissue sample for a predetermined time. For liquid samples like blood, the sample may be smeared onto a slide and dried before fixation, potentially omitting the embedding and sectioning steps.

[0094] Embedding involves infiltrating the fixed tissue sample with a suitable histological wax, such as paraffin, or resins like styrene or polyethylene. Since histological wax is insoluble in water, the tissue must first be dehydrated using graded alcohols, followed by clearing with a solvent like xylene. The tissue is then embedded in warm paraffin wax, which fills the spaces previously occupied by xylene and water. Once cooled, the wax forms a hardened block that can be clamped into a microtome for precise sectioning. Deviations from this procedure can result in improper infiltration, hindering subsequent analyses.

[0095] A tissue sectioning system may be used to slice the fixed and / or embedded tissue sample into thin sections using instruments such as a cryostat, microtome, vibratome, or compresstome. Sectioning produces a series of tissue slices, typically 4-5 microns thick, suitable for microscopy such as light or fluorescent microscopy. In some cases, tissues can be rapidly frozen using dry ice or Isopentane and sectioned in a cryostat with a cold knife. Alternatively, other cooling agents like liquid nitrogen can be used. Sections intended for microscopy are generally between 4-10 micrometers thick, while those embedded in epoxy or acrylic resin can be thinner, less than 2 micrometers. The sections are then mounted on glass slides for further processing.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT A2-Imaging and Digitization

[0096] Once the stained tissue sections are prepared, the next step involves digitizing these slides to create WSIs. The tissue sections or samples may be mounted on glass slides, which are then scanned by a whole slide scanner (e.g., Vectra Polaris) equipped with high-resolution camera automated slide handling system, and advanced imaging software to create digital images. For example, the imaging module 102 described with respect to FIG. 1 may be used to scan the slides. The slides can be loaded into an automated slide loader of the imaging module 102. The camera unit 105 of the imaging module 102, equipped with various objective lenses, can be used to capture detailed images of the entire tissue sample by moving across the slide in a grid pattern. In some embodiments, the imaging module 102 can handle multiple slides simultaneously, allowing for batch processing for improving time efficiency. In some embodiments, the scanned image is stored in the storage 115 and can be retrieved or accessed by the digital image processing platform 110. In some embodiments, a tissue section may first be processed by the imaging module 102 to generate unstained bright field digital images of the section prior to staining.

[0097] In some embodiments, the whole slide scanner is equipped with a scanning mechanism, a light source, optical system, and a camera. The scanning mechanism moves the slide or the camera in a systematic pattern, capturing overlapping fields of view (FOVs) across the entire tissue sample. This process is similar to stitching together multiple photographs to create a panoramic image. The scanner must ensure accurate alignment and focus for each FOV to avoid any blurring or distortion. The light source is a consistent and bright light source, such as an LED, which is used to illuminate the tissue sample. The light generated by the light source passes through the tissue as it is moved by the scanning mechanism and is captured by the optical system. The light source can be used to excite the fluorophores. A high-resolution digital camera such as CCD or CMOS sensors is used to capture the illuminated tissue image and / or fluorescence from the tissue sample. These cameras are often capable of capturing images at a resolution of several gigapixels, ensuring that even the smallest cellular details are accurately recorded. The individual FOVs are stitched together to form a continuous, high-resolution WSIs for each color and / or fluorescence channel.

[0098] The imaging module 102 can also scan the slide at different magnifications (e g., no magnification, 4x magnification, lOx magnification, 20x magnification, and / or 40x magnification, etc.). The choice of magnification depends on the level of detail required for theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT analysis. Higher magnification allows for detailed cellular-level information to be captured, while lower magnification captures less detailed but broader structural features covering a larger area. Accordingly, the scanning process is precise enough to ensure that even the smallest details are accurately captured without introducing artifacts or distortions. The resulting digital images are typically large (e.g., 10 billion plus pixels for a 40x WSI that is several GB in size). In some embodiments, the high-resolution files are stored in specialized formats, such as Tagged Image File Format (TIFF) or proprietary formats designed for pathology images. In some instances, the WSI may be stored in unsigned integer numbers (unit) in 8-bit RBG format. In other instances, the WSI may be stored as a pyramidal image (e.g., Scanned Virtual Slide (SVS) images), where the same image can be stored in different magnifications. These digital files can be managed by the digital image processing platform 110 or by the storage 115 described with respect to FIG. 1. In some embodiments, the digital images may be stored in association with metadata such as an identifier of a subject and a date (e.g., a date when the tissue sample w as collected and / or a date when the image w as captured).A3-Staining

[0099] The tissue samples can be stained using a staining system and digital images of the stained tissue may be captured by the same imaging module 102. The staining can be performed using the preprocessing module 104 described with respect to FIG. 1. Various staining methods can be implemented to enhance contrast and visualize specific biological structures or cellular components. The staining can be performed manually or automatically. Staining can include exposing an individual section of the tissue to one or more different stains (e.g., consecutively or concurrently) to express different characteristics of the tissue. For example, each section may be exposed to a predefined volume of a staining agent for a predefined period of time. In other instances, the staining is performed semi-automatically or automatically by Stainers. The Stainers can be programmed with specific protocols for various types of staining, such as Hematoxylin and Eosin (H&E), immunohistochemistry (IHC), and special stains. The Stainers control parameters such as reagent volumes, incubation times, and temperatures, ensuring reproducibility and reducing error.

[0100] Many staining solutions are aqueous. Thus, to stain tissue sections, the embedding agent (e.g., wax) may need to be dissolved and replaced with water (rehydration) before a staining solution is applied to a section (e.g., deparaffinization). For example, the tissue section may be sequentially passed through xylene, decreasing concentration of ethyl alcohol (fromAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT about 100% to 0%), and water. Once stained, the tissue sections may be dehydrated again and placed in xylene. The tissue section may then be mounted on microscope slides in a mounting medium dissolved in xylene. A coverslip may be placed on top to protect the sample section. The evaporation of xylene around the edges of the coverslip may dry the mounting medium and bond the coverslip firmly to the slide.

[0101] Various types of staining protocols may be used to perform the staining. One exemplary type of tissue staining is histochemical staining, which uses one or more chemical dyes (e.g., acidic dyes, basic dyes) to stain tissue structures. Histochemical staining may be used to indicate general aspects of tissue morphology and / or cell microanatomy (e.g., to distinguish cell nuclei from cytoplasm, to indicate lipid droplets, etc.). One example of a histochemical stain is H&E, which differentiates cellular structures by staining nuclei blue and cytoplasm pink. Additional special stains or immunohistochemical markers may be used to highlight specific proteins or cellular components. Other examples of histochemical stains include trichrome stains (e.g., Masson’s Trichrome), which are used to differentiate between collagen and muscle fibers in tissue sections, Periodic Acid-Schiff (PAS), which is used to detect polysaccharides, muco-substances. and glycoproteins in tissues, silver stains, such as Gomori’s Methenamine Silver (GMS) and Reticulin stains, which are used to highlight certain types of tissue fibers and microorganisms, and iron stains, such as the Prussian Blue stain, which are used to detect ferric iron (Fe3+) in tissues.

[0102] Another type of tissue staining is immunohistochemistry (IHC), also called ‘■immunostaining'’), which is used to visualize specific proteins or antigens within tissue sections by leveraging the principles of antibody-antigen interactions. The process begins with the fixation of tissue samples to preserve cellular structures, followed by embedding in paraffin and sectioning into thin slices placed on glass slides. These sections are then exposed to primary antibodies that specifically bind to the target antigens. A secondary’ antibody, conjugated to a detectable marker such as an enzyme or a fluorescent dye, is applied to bind the primary antibody. In chromogenic IHC, the enzyme catalyzes a reaction that produces a colored precipitate at the antigen site, making it visible under a light microscope. In fluorescent IHC, the fluorescent dye emits light when excited by a specific wavelength, allowing visualization under a fluorescence microscope. IHC staining is widely used in pathology to diagnose diseases, identify specific cell types, and study the distribution and localization of proteins within tissues, providing critical insights into cellular functions and disease mechanisms. In embodiments directed to fluorescent stains, a scanner (e.g., the imaging module 102) can beAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT equipped with high-resolution optics and filters specific to the excitation and emission wavelengths of the fluorophores used to capture stained images.

[0103] Multiplex fluorescence imaging techniques can also be used to allow simultaneous detection of multiple markers on a single tissue section. For example, at least 2, at least 3, at least 4, at least 5, at least 6 or more biomarkers (e.g., proteins) may be marked or detected on a single tissue section using multiplexing. Multiplexing can either be done using fluorescent or chromogenic staining. In fluorescent detection, antibodies are conjugated with a fluorochrome and emit light when stimulated with a light of a shorter wavelength. Alternatively, the antibodies can be conjugated with enzy mes that react with a substrate and leaves a chromogenic deposition at the site of reactivity. Moreover, staining may be performed sequentially or simultaneously. In sequential staining, staining procedures are performed one after the other. For example, a first primary antibody with its detection system is applied to the sample and after a predetermined amount of time, excess antibody is removed. Then a second primary antibody with its detection system is applied to the sample, and so on. Conversely, in simultaneous staining, all the primary' antibodies are applied at the same time. Either sequential or simultaneous staining may also apply the direct or indirect methods described with respect to IHC.

[0104] The digital images of the unstained biological sample or the stained sample may7have various image channels (e.g., different perspectives of the same slide). For example, a single sample slide may be used to generate (i) a bright field (BF) image, (ii) an unstained autofluorescence (AF) image or a slightly stained image (e.g., a DAPI image), (iii) fluorescence images generated with filters, and (iv) a stained image (e.g., H&E, Masson’s tri chrome, Jones’ silver, etc.). The bright field image can be generated by illuminating the sample from underneath with white light. To generate the unstained autofluorescence image, the sample slide can be examined under a fluorescence microscope to detect cellular features that naturally exhibit fluorescence. Fluorescence images can be generated without performing immunohistochemistry' staining methods but using filters. The filters may be of different wavelengths that can cause organic structures within the sample to auto-fluoresce. For example, some cellular components within tissue samples exhibit autofluorescence due to the intrinsic fluorescent properties of certain biomolecules (e.g.. NADH. FAD, lipofuscin, collagen, elastin, riboflavin, porphyrins, amino acids, and chlorophyll). Cellular components that may be identified using fluorescence filters can include mitochondria, lysosomes, cytoplasm, extracellular matrix, red blood cells, nucleus, chloroplasts (in plant cells), and theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT like. In some embodiments, the various image channels may be generated by a specialized imaging machine (e.g., Vectra Polaris (Akoya Bio)) rather than manually. The imaging machine may be equipped with 2, 3, 4, 5, 6, 7, 8, 9, 10, or more channels that can generate BF images, AF images, and numerous fluorescence images at different wavelengths. In some embodiments, the numerous fluorescence channels can include DAPI, Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, Opal 780. or any combination thereof. Finally, after all other image channels have been generated, the sample slide can be stained using any one of the histochemical stains described herein to generate the stained image.A4-Image Acquisition and Alignment

[0105] At block 205, a first set of digital images are acquired. Each digital image can be a digitized WSI depicting a portion of a biological sample. The first set of digital images may be acquired from the digital image processing platform where the digital image of each biological sample is captured. This digital image processing platform could be a microscope connected to a computer system that scans the biological samples and converts them into digital images. Alternatively, the first set of digital images is acquired from a database, such as the storage 115 described with respect to FIG. 1. This database could be a centralized data repository where previously captured digital images are stored and cataloged for easy retrieval.

[0106] The biological samples may be obtained from the same subject, for instance, different tissue sections from a biopsy of a single patient. Alternatively, the biological samples may come from a group of subjects. In some embodiments, the biological sample is obtained from a mass from each subject. In some embodiments, the group of subjects has the same characteristic. For example, all the subjects may have been diagnosed with a particular disease such as cancer. Other common characteristics among the subjects could include, but are not limited to: having the same biological gender, coming from the same geographic region, or belonging to the same age group (e.g., between 18 to 60 years old). In some embodiments, the group of subjects has different characteristics. For example, half of the subjects may have been diagnosed with the particular disease, and the other half have not. The selection of subjects and biological sample may be determined based on a medical or clinical need and different inclusion (e.g., specific medical conditions, age ranges, or other relevant factors) and exclusion criteria (e.g., exclude participants or samples that might confound the results, such as those with comorbid conditions or who are on conflicting medications) may be designed.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0107] The number of subjects or biological samples may be determined based on the training need or study objectives. Power analysis can be used for considering the expected effect size, variability, and desired confidence level to select the desired size of samples or subjects. In some embodiments, at least 100, 200, 300, 400, 500, 600, 700, 800, 900, or 1000 samples are selected.

[0108] At block 210, a second set of digital images are acquired. Each digital image can be a digitized WSI depicting a portion of the biological sample at block 205 but after staining. The second set of digital images may be acquired from the same digital image processing platform where the digital image of each unstained biological sample is captured. Alternatively, the second set of digital images is acquired from the same or a different database where the digital image of each unstained biological sample is obtained.

[0109] The first set of digital images and the second set of digital images are paired images. For each image of a biological sample in the first set, there is a corresponding image in the second set that corresponds to the same biological sample after staining. In some embodiments, the image of the biological sample in the first set is an image of a stained biological sample, and the image in the second set corresponds to the biological sample after a second staining process. The first staining process may use fluorescent dyes such as DAPI, Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, and Opal 780 to improve the contrast and clarity' of the images, and the second staining process may use staining techniques such as H&E, Jones’ Silver stain, PAS stain, Masson’s Trichrome Stain, and IHC to provide detailed information about tissue structure and composition. For example, FIG. 3 shows an exemplary digital image 310 in the first set (a DAPI image), and an exemplarity’ digital image 330 in the second set (an H&E stained image). The first set of digital images may be used input data to train a machine learning model to perform digital staining, and the second set of digital images can be used as ground truth to evaluate performance of the machine learning model. In some embodiments, the first set and the second of digital images are acquired by accessing different channels of images. For example, a DAPI image may be stored as a first channel of an accessed image, and the H&E stained image may be stored as the next three channels of the same accessed image. As used herein, the first set of images may be referred to as “unstained images’’ and the second set of images may be referred to as “stained images.”

[0110] At block 215, the paired images in the first set and in the second set are aligned. The alignment can be feature-based alignment or pixel-based alignment. The alignment can involveAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT in several steps: image preprocessing, feature extraction, coarse transformation, and / or fine alignment. In some embodiments, only the fine alignment is performed. The alignment can be performed using the preprocessing module 104 described with respect to FIG. 1.

[0111] The paired digital images may be slightly different in size, and the preprocessing may use segmentation techniques, padding techniques, or other resizing techniques to resize the paired digital images. In some embodiments, the slight difference is rectified during alignment. When resizing is performed, an image may be used as a base image and the other one is resized to match the size of the base image. In some embodiments, the stained image is used as the base image. The resizing may be performed using interpolation methods (e.g., bilinear or bicubic interpolation) to infer the intensity' or pixel values of the resized image. The resizing can be also performed by padding the smaller image to match the size of the larger image (e.g., by adding extra background pixels around the edges of the smaller image to increase its dimensions). In some embodiments, the resizing is performed by identifying a common region of interest (ROI) to remove differences in image boundaries. The identification can be performed by applying a ROI mask on each of the paired digital images using a segmentation algorithm. Segmentation algorithms used for the identification includes but are not limited to: (i) algorithms to identify and locate specific structures such as cells, tissues, or pathological features within the images, (ii) algorithms to delineate tumor boundaries from non-tumor tissue, and (iii) algorithms to detect and segment objects into predefined classes (e.g., non-cancerous and cancerous cells, different types of tumors, or different cell structures). These algorithms may be supported by machine learning models that are pretrained to perform each specific task.

[0112] The preprocessing may also include noise reduction and normalization. For example, each set of images may be normalized to ensure consistent brightness and contrast levels (e.g., normalized the intensity level). Noise reduction techniques (e.g., Gaussian blur) can also be applied to both images to minimize any background noise that could interfere with the alignment process.

[0113] In some embodiments, features are extracted from paired images before alignment. Features may include edges, comers, textures, and the like. Features may be extracted using feature detection algorithms such as Scale-Invariant Feature Transform (SIFT), Speeded-Up Robust Features (SURF), Binary Robust Independent Elementary Features (BRIEF), Features from Accelerated Segment Test (FAST), Oriented FAST and Rotated BRIEF (ORB), or theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT like. The features in each image may be matched up to find corresponding points in both images. Feature matching algorithms like FLANN (Fast Library for Approximate Nearest Neighbors) or the Brute-Force matcher may be used for the matching.

[0114] The alignment may be also referred to as “registration.” A first coarse alignment may be performed first, followed by a fine alignment. The coarse alignment may estimate a basic transformation matrix (e.g., affine or rigid transformation) that maps the coordinates of the matched features from the pre-staining image to the post-staining image (or vice versa) and apply this coarse transformation to align the paired images roughly. A local transformation matrix (e.g., elastic deformation or non-rigid registration) may be estimated and applied to correct residual discrepancies around key structural features. Other fine alignment techniques (e.g., homography transformation) may also be applied to align the paired images. In some embodiments, the paired images are aligned tile by tile (based on the scanning techniques) and then using the alignment algorithms to stitch tiles into a single aligned image.

[0115] In some embodiments, the alignment is a pixel-based or intensity-based alignment. For example, a global metric (e.g., cross-correlation, mutual information, and sum of squared differences (SSD)) that measures the similarity among pixels between the paired images may be optimized and the optimized metric is used to align the images.

[0116] In certain embodiments, a manual registration (e.g., rigid alignment) is performed prior to the image alignment. The manual registration is triggered when the initial positions of the paired images are not roughly the same (e.g., leading to misalignment). If the images are not approximately aligned initially, automated alignment algorithms may struggle to find the correct correspondences, resulting in poor or failed alignment. In some embodiments, the manual registration is performed when a quality check metric of the alignment process fails. In some embodiments, the manual registration is performed when the metadata of the paired images does not match (e.g., stage coordinates are different). In some embodiments, a user visually inspects the images and manually adjusts the stained image to align it with the base image. After the manual registration, automated alignment process can be performed to refine the manual registration. In some embodiments, a manual refined alignment is performed after the automated alignment.

[0117] In some embodiments, each of the images is resampled to a lower resolution before the automated alignment. WSIs can be very large, making direct processing computationally expensive and time-consuming. Resampling images to a lower resolution (e.g., 5, 10, or 20Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT micrometers per pixel) can substantially reduce the image size while retaining essential structural information, making the registration process more efficient. In some embodiments, each image is segmented into smaller patches before the automated alignment.

[0118] In some embodiments, software package, library, computer script or combination thereof known in the art may be used to perform the alignment, including but not limited to Elastix, SimplelTK, Advanced Normalization Tools (ANTs), FMRIB Software Library (FSL), NiftyReg, Insight Segmentation and Registration Toolkit (ITK). MATLAB, Open-Source Computer Vision Library (OpenCV), ImageJ / Fiji, 3D slicer, and the like. The software / library packages may perform one or more types of image registration tasks including rigid, affine, non-rigid (deformable), and spline registration. By way of example, an image registration software (e.g., Imaged) may initially apply an affine registration (e.g.. linear alignment) on the image followed by a more precise registration such as spline registration (e.g., using spline functions to model the transformation field that maps points from one image to corresponding points in another image). Exemplary' spline registration includes B-Spline registration, elastic spline registration, thin plate spline, free-form deformation (FFD) with splines, and multilevel B-Spline registration.

[0119] In some embodiments, a configuration file is accessed that specifies details for the image alignment. The configuration file may include the registration parameters, including the type of transformation (rigid, affine, non-rigid, spline), the similarity metric (e.g., mutual information, mean squared error), the optimizer, the multi-resolution strategy, and the like. The configuration file helps apply the automated image alignment. The transformation parameters may be pre-optimized to minimize the similarity’ metric. In some embodiments, the pixel-level alignment is achieved using grid matrixes, control points, key features in the images themselves, or any combination thereof.A5-Patch Extraction

[0120] At block 220, a set of paired patches are extracted from each pair of images. That is, for each patch in the base image, there is a corresponding patch (stained patch) in the stained image. The patch extraction may be performed using the preprocessing module 104 described with respect to FIG. 1. Given the size of WSIs. using all the whole WSIs or all patches of WSIs to train a machine learning model can be impractical or time consuming. Therefore, using a set of patches extracted from the WSI significantly reduces computing time while maintaining prediction accuracy. The patch size and the number of patches in each set may beAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT predetermined. In some embodiments, the patch size is about or exactly 50x50, lOOx 100, 128x128. 224x224, 256x256, or 512x512 pixels. In some embodiments, the number of patches extracted from each patch is about or exactly 50, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000, 1500, 2000. The size and the number may be determined based on training needs. For example, the extracted patches may cover at least 10%, 20%, 30%, 40%, or 50% of the WSI or the tissue portion shown on the WSI.

[0121] The extraction can be performed using a feature extraction algorithm or a random selection algorithm. For example, after determining the size of each patch, a random selection is performed to choose a predetermined number of patches from the WSI. The random selection can be performed by griding the WSI to generate patches, numbering the patches, and randomly selecting the patches based on the numbering. The random selection can also be performed by randomly generating coordinates for the top-left comer of each patch. FIG. 4 shows extracted patches (in squares) in a bright field image. As shown in FIG. 4, the patches can be either overlapping or non-overlapping. A specific percentage of each patch contains the tissue sample. For instance, if less than 20% of the patch displays the stained tissue, the patch may be excluded from the patch set. The patches can be saved in various image formats, such as PNG, JPEG, or TIFF, which can also include metadata (e.g., patch coordinates, relevant annotations) associated with each patch.

[0122] In some embodiments, the patches are extracted from the stained images, and the corresponding patches are selected based on the patch coordinates of the patches from the stained images. In some embodiments, the patch selection process is refined by stacking the unstained image and the stained image to generate a multi-channel image. In some embodiments, the set of paired patches are manually verified to be sure they do not overlap with artifacts in either image (e.g., out of focus, debris, and the like). The verification can be firstly done on the unstained image (or DAPI image), and the stained image is verified after that. In some embodiments, the artifact regions on each image are removed before patch extraction.A6-Model Training

[0123] At block 225, a machine learning model is trained to generate stained patches based on unstained patches. The training may be performed on an instance of a cloud service or may be performed on a local computer. In some embodiments, the training can be performed using the preprocessing module 104 described with respect to FIG. 1. A model architecture orAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT algorithm may be predetermined before the training. For example, the machine learning model may be a linear regression model, a logistic regression model, a decision tree model, a k-means model, a k-nearest neighbor (k-NN) model, a neural network, a support vector machine, a random forest model, or a graph model. The neural network includes but is not limited to Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Long Short-Term Memory Networks (LSTMs), and Generative Adversarial Networks (GANs). In some embodiments, the machine learning model is a GAN. In some embodiments, an attention model is used to fine-tune a trained machine learning model.

[0124] The patches extracted from the unstained images are used as input data to train the machine learning model, and the corresponding patches extracted from the stained images are used as the ground truth (or ‘label") to evaluate the performance of the model. The patches may be assigned to different groups before training. For example, the patches may be split into a training set and a test set based on a predefined ratio (e.g., a 50 / 50%, 90 / 10%, or 70 / 30%). The training set may be further split into training subsets and validation subsets. In some embodiments, the training data is split into three subsets (training, validation, and test) for each round of training. Around 70-80% of the training data may be allocated for training, about 10-15% for validation (for tuning hyperparameters and making decisions about model architecture), and about 10-15% for testing (for assessing the model’s generalization ability to new, unseen data). In some embodiments, different validation technique such as holdout validation, k-fold cross-validation, leave-one-out cross-validation, leave-one-group-out cross-validation, nested cross-validation, bootstrap validation, or the like may be performed to minimize sampling bias and overfitting.

[0125] In some embodiments, the training is supervised learning. The machine learning model, such as a GAN, is trained on paired patches extracted from unstained and corresponding stained images. During training, the model learns the spatial and color transformations required to replicate the staining process digitally. Loss functions, such as mean squared error (MSE), perceptual loss or specifically defined loss functions, can be used to minimize the difference between the generated and real stained images to improve the model performance. During training, the model iteratively adjusts its parameters by minimizing the loss function, which quantifies the difference between the model’s predictions (output patch) and the true labels (stained patch). This optimization process can be performed using techniques like stochastic gradient descent (SGD) or Adam optimizer. The validation set is used to monitor the model’s performance and tune hyperparameters, such as learning rate or regularization strength, toAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT prevent overfitting. Once the training converges, the model is evaluated on the test set to assess its generalization ability. This process may be repeated with different architectures or configurations to find the best-performing model.Generative Adversarial Network (GAN)

[0126] FIG. 5 shows an exemplary framework of a generative adversarial network (GAN) 500 to perform the virtual staining of a digital image. The GAN 500 may comprise a generator 505, and a discriminator 510, each of which is a neural network (e.g., a CNN or DNN). The generator 505 can be trained to generate virtually stained images that are substantially the same as the real stained images. The discriminator 510 acts as an adversary' to the generator 505 by¬ distinguishing between real stained images and virtually stained images produced by the generator 505. The discriminator 510 also provides feedback to the generator 505, helping it improve the quality of the virtually stained images over time.

[0127] The generator 505 receives input data 515 (the unstained patches) in batch for the training purpose. The input data 515 may comprise n channels (n represents any natural number) corresponding to different colors. In some embodiments, the input data is singlechannel DAPI images. The generator network leams to perform image transformation on the input data 515 to generate output data 520 (virtually stained patches). The output data 520 may comprise m channels (m represents any natural number) corresponding to different colors. In some embodiments, the output data 520 has a higher dimension than the input data 515 (i.e., m > n). In some embodiments, the output data 520 is stored as YCrCb images. For example, as shown in FIG. 3, an input channel 340 can be projected to three output channels 350. The generator can be trained to transform a pixel Vi (labeled 345) in the input channel 340 to a pixel V’I (labeled 355) in the three output channels 350.

[0128] The output data 520 and the brightfield data 525 are input into the discriminator 510 for training the discriminator 510 to distinguish between “real” data (the brightfield data 525) and “fake” data (the output data 520). An objective function (e.g., a loss function, a cost function, a contrastive loss function, adversarial loss, etc.) may be used to evaluate the performance of the GAN based on the differences between the output data 520 (virtually stained patches) and the brightfield data 525 (the ground truth real stained patches). The objective function can be configured to minimize the difference between the output data 520 and the brightfield data 525. In some embodiments, the objective function is a loss functionAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT comprising a generator loss and a discriminator loss. Equations (1) shows a discriminator loss function, and Equation (2) shows a generator loss function.

[0129] “GT” in Equation (1) represents the expected output (the brightfield data) and “D” in both equations represents the output from the discriminator. The factors a, ft, and y in Equation (2) are hyperparameters that are adjusted by the validation, Genimageis the generated image (output data) from the generator, and “TV” represents total variation of the image.

[0130] In some embodiments, a specific GAN is selected to perform the virtual staining. The specific GAN can be selected from the following: Vanilla GAN, Deep Convolutional GAN; Conditional GAN, CycleGAN, Pix2Pix, StyleGAN. StyleGAN2. Wasserstein GAN, WGAN-GP, BigGAN, Progressive GAN, Super-Resolution GAN, StarGAN, InfoGAN, Auxiliary Classifier GAN, TwinGAN, Text-to-Image GAN, 3DGAN. and Spatio-Temporal GAN. In some embodiments, the generator 505 has a U-Net architecture or its variants (e.g., encoderdecoder or up-sampling structure) and the discriminator 515 has a down-sampling structure.Generator

[0131] FIG. 6 illustrates an exemplary generator 600 with a U-net architecture used in a GAN to generate synthesized images (e g., virtually stained images). The U-net architecture includes an encoder network 605 (down-blocks) that captures features of input patch 602, a bottleneck layer 610 that generates a feature map, and a decoder network 615 (up-blocks) that reconstructs images based on the features and / or the feature map. The U-net architecture may further include skip connections 620 that transfer of spatial details and low-level features from encoder layers to decoder layers. In some embodiments, the bottleneck layer 610 is considered a portion of the encoder network 605 or the decoder network 615. Although FIG. 6 shows the U-net architecture with a specific depth and a specific number of components, it should be understood that the depth and the number of architectural components are considered hyperparameters that can be predefined or adjusted during the training. For example, the depth of the U-net architecture can be 4, 5, 6, 7, or 8, and the convolutional filter size can be 3x3 or 5x5. These hyperparameters depend on specific GANs to be used and / or training or design needs.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0132] The input data for the generator can be image data such as WSIs or patches from WSIs. An input patch 602 may be obtained based on the patch extraction described above. The input patch 602 can be an unstained patch or a fluorescence patch (e.g., DAPI, Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, and / or Opal 780). When training the generator, a plurality of input patches is input into the generation in batch, and the plurality' of input patches may include all patches or selected patches generated from an unstained WSI or a fluorescence WSI. To improve the training speed without sacrificing the accuracy, a randomly selected portion (e.g., about 10% - 50%) of the total patches can be selected.

[0133] The input patch 602 may undergo preprocessing steps prior to being fed into the U-net architecture of the generator. For example, a multi-channel input patch may be transformed into a grayscale image to reduce training time. Preprocessing may also include converting the input patch 602 of a specific format (a tiff file) into a tensor (e.g., a multidimensional array), with each element in the tensor corresponding to a pixel intensity value or a normalized pixel intensity value. In some embodiments, the training is performed using a specific tensor core of a GPU, and the input patch 602 is converted to a specific precision format (e.g., FP16, BF16, TF32, INT8, or FP8) to further accelerate the training process. In some embodiments, the input patch 602 is further processed into specific size (e.g., 4x4 blocks for Volta architecture, FP16 and 8x8 blocks for Turing, Ampere, and Hopper architectures, mixed-precision formats like FP16 and INT8).

[0134] The encoder network 605 captures features of the input patch 602 by progressively reducing the spatial dimensions while increasing the depth (number of feature channels / feature maps). In other words, the encoder network 605 creates a compressed representation of the input patch 602 that retains information needed for the target task (e.g., staining). The encoder network 605 may include multiple encoder blocks 625a-d with each down-block including a number of convolutional layers that perform convolution functions followed by a down sampling operation (e.g., pooling). The convolutional layers can be followed by an activation function and optionally batch normalization layers. As illustrated in FIG. 6, the encoder network 605 includes four encoder blocks 625a-d, each comprising 3 convolutional layers. Exemplary activation functions include Rectified Linear Unit (ReLu), Leaky' ReLU, Hyperbolic Tangent (tanh), Sigmoid, Gaussian Error Linear Unit (GELU) or Swish. In some embodiments, the activation function is a Leaky ReLU or a derivative of Leaky ReLU to ensures small gradient flows for negative inputs, preventing neurons from being inactive during training. The output of the encoder blocks 625a-d is input into a pooling layer 630a-d. TheAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT pooling can be a max pooling, a min pooling, an average pooling, a stochastic pooling, a spatial pyramid pooling, and a pyramid pooling. The pooling can also be within a specific pooling window (e.g., 2x2 or 3x3) to further reduce the training time.

[0135] The convolution and pooling layers work in tandem to progressively extract high-level features from the input patch 602. The convolutional layers are designed to apply a set of learnable filters or kernels to the input data. Each filter slides over the entire image, performing an element-wise multiplication and summation to produce a feature map. This process, known as a convolution operation, helps in detecting local patterns such as edges, textures, and shapes within the image. The filters in the initial convolutional layers (e.g., in encoder blocks 625a-b) typically capture basic features like edges and comers and the resulting feature maps represent the original image, but highlight certain features (e.g., edges and comers). On the other hand, filters in deeper layers (e.g., encoder blocks 625c-d) capture more complex patterns and structures, which produce more abstract representations. The convolution operation also helps in presenting the spatial relationships between pixels, which is cmcial for understanding the content of the image.

[0136] After the convolution operation is performed, resulting in feature maps, an activation function can be applied to introduce non-linearity into the GAN. This non-linearity is useful because it enables the netw ork to learn and represent complex patterns and relationships in the data that linear operations alone cannot capture. Examples of activation functions that may be used include the linear activation function, the rectifier linear unit (ReLU), the leaky ReLU, the sigmoid activation function, the SoftMax activation function, the hyperbolic tangent activation function, and the exponential activation function. In certain instances, the activation function used for each convolutional layer is the ReLU, which applies the function (f(x) = max(0, x)) to each element of the feature map. This means that any negative values in the feature map are set to zero, while positive values remain unchanged. In some instances, the activation function is the same across all convolution layers. In other instances, the activation function is varied across the convolution layers (e.g., one or more convolutional layers use ReLU and another one or more convolutional layers use leaky7ReLU).

[0137] The application of the activation function occurs elementwise across the entire feature map produced by the convolution operation. By introducing non-linearity at each convolutional layer, the network can model more complex functions and is better equipped to handle diverse and intricate data. This step is useful because, without it, the model would essentially7beAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT equivalent to a single linear transformation, regardless of the number of layers, severely limiting its capacity to solve complex tasks. Therefore, by applying activation functions after each convolutional operation, the encoder network 605 can develop a more nuanced and powerful representation of the input data, leading to more accurate and robust performance in tasks such as image recognition and classification.

[0138] The convolution process generates a large amount of data that needs to be compressed by its corresponding pooling layer (e.g., convolution output 625a is compressed to generate pooled layer 630a) to reduce the spatial dimensions of the feature maps, which serves several purposes. Pooling operations, such as max pooling or average pooling, down sample the feature maps by summarizing the presence of features in patches of the feature map. For instance, max pooling takes the maximum value from a patch of the feature map / numerical tensor, effectively reducing the size while retaining the most significant information. This not only reduces the computational load and memory requirements but also adds a degree of translational invariance, meaning the network becomes more robust to slight translations of the input image. By alternating between convolutional and pooling layers, the encoder network 605 hierarchically builds a rich and abstract representation of the input image. The repeated application of convolution and pooling operations allows the network to leam and recognize increasingly complex features at different levels of abstraction, which allows for the effective processing and understanding of image data.

[0139] In between the encoder network 605 and the decoder network 615, is the the bottleneck layer 610. At this point, the most compressed and abstract representation of the input data is captured, meaning that the spatial dimensions have been significantly reduced, and the depth (number of feature channels) has been increased, resulting in a dense representation of high-level features. Moreover, the bottleneck layer 610 captures the context of the input image, which is essential for understanding the overall structure and relationships within the image to ensure accurate reconstruction of the output during the decoding process. As illustrated, the bottleneck layer 610 includes several convolutional layers with activation functions (e.g., ReLu). Unlike the encoder, the bottleneck does not perform any down sampling operations (such as pooling). Instead, it focuses on further processing and refining the features extracted by the encoder to output the final feature map representation. The final feature map is fed into the decoder network 615, where up sampling operations are performed to reconstruct the original image dimensions.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0140] The decoder network 615 (also referred to as the up-blocks or the expansive network) is responsible for reconstructing the image from the compressed representation created by the down-blocks by progressively increasing the spatial dimensions (e.g., height and width) while reducing the depth (number of feature channels). To achieve this, each decoder block 635a-d starts with a transpose convolution 640a-d (also referred to as deconvolutions or up sampling convolutions) that use a learned filter (e g., kernel) to perform the up sampling. The transpose convolution 640a-d applies a skip connection function (e.g., identity skip connection, projection skip connection, or concatenation skip connection) between the feature maps from the corresponding layers of the encoder with the up sampled feature maps from the decoder. For example, between encoder block 625d and decoder block 635d, providing direct paths for information and gradients to flow. In so doing, skip connections 620 alleviate issues like vanishing gradients, improving feature propagation, and enabling the training of deeper models. In various instances, the skip connection function is a concatenation skip connection where the feature maps from the corresponding layers of the encoder are concatenated (e.g., multiplied) by the up sampled feature maps from the decoder. This provides the subsequent layer with access to both the high-level features from the deeper layers and the low-level features from the earlier layers. After concatenation, each decoder block processes the skip connection output with several convolutional layers (e.g., 2) with an appropriate kernel (e.g., kernel size of 3x3), followed by an activation function (e.g., ReLu).

[0141] The purpose of the output layer 645 is to generate the final predictions from the processed features. It transforms the high-dimensional feature maps produced by the preceding layers into the desired output format, typically matching the dimensions of the input image. To achieve this, the output layer 645 performs dimensional reduction on the number of feature channels to match the number of output channels required for the task. For instance, in a binary task, the output layer 645 will reduce the feature channels to a single channel representing the probability of each pixel belonging to the foreground class. On the other hand, for multi-class tasks, the output layer 645 reduces the number of feature channels to the appropriate number based on the number of output classes. As such, the structure and function of the output layer 645 can vary slightly depending on whether the task is binary or multi-class. The output layer 645 comprises a convolution layer that applies a convolution smaller than those applied in the encoder network 605 and the decoder network 615. In so doing, the final convolution layer reduces number of feature channels to the number of desired output classes. This operation ensures that each pixel in the output map has a vector of length equal to the number of classes,Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT representing the class scores for that pixel. Finally, an activation function (e.g., SoftMax for multi-class tasks or sigmoid for binary tasks) is applied to produce the final predicted image (a synthesized patch 650). The activation function serves to normalize the final tensors making up the synthesized patch 650.Discriminator

[0142] The discriminator 510 in FIG. 5 is used to distinguish between “real” data (the brightfield data 525) and "fake" data (the output data 520). The architecture of the discriminator is tailored to the specific type of data and application. In some embodiments, the architecture of the discriminator 510 is a fully connected neural network (FCN), a convolutional neural network (CNN), a recurrent neural network (RNN). a transformer-based network, a graph neural network (GNN), or a combination thereof. In some embodiments, the architecture of the discriminator is a CNN with a fully connected output layer.

[0143] FIG. 7 shows an exemplary CNN 700 with a fully connected output layer used as a discriminator of a GAN. The CNN 700 can be a feedforward neural network where data moves from the input layer through a set of hidden layers in one direction (e.g., having no cycles contained within it). However, it should be understood that other types of neural networks could be implemented or combined with the feedforward CNN. The CNN 700 can take a patch (a “real” image or a “fake” image) as input, extract features from the input using convolutional layers, assigns importance (learnable weights and biases) to the features in the patch, and differentiates one from another to perform the image discrimination (e.g., perform a binary classification).

[0144] As shown in FIG. 7, the CNN 700 comprises an input layer 705, convolution-pooling layers 710, and classification-output layers 715. The input layer 705 of the CNN 700 is the initial stage where input data (e.g., the “real” patch or the “fake” patch) is fed into the network. The input data may be in a particular data structure such as a tensor (e.g.. multi-dimensional array) so that the training of the discriminator can be performed efficiently by specialized computer hardware (e.g., a GPU with specific tensor cores). In some embodiments, the input data is a patch with multiple channels (e.g., 3-channel YCrCb image or a 3 -channel RGB image).

[0145] While the input layer 705 itself may not perform complex computations; it ensures that the input data is formatted correctly for subsequent layers. The convolution-pooling layersAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT 710, rely on this structured data to perform operations like convolution, which involves sliding a filter or kernel over the input data to detect features such as edges, textures, and patterns. The convolution-pooling layers 710 include one or more convolution layers 720a-n (solid boxes) that perform convolution functions followed by pooling layers 725a-n (dashed boxes) that perform pooling functions. For both the convolution layers 720a-n and the pooling layers 725a-n, 'n' represents any natural number. It should be understood that the number of convolution layers does not necessarily equal to the number of pooling layers. Similar to the description of convolution and pooling provided with respect to the generator (505 in FIG. 5 and 600 in FIG.6), the convolution and pooling layers within the discriminator work in tandem to progressively extract high-level features from the input data received from the input layer. Kernels slide over the input data with a size (e.g., kernel size of 3x3x3 or 5x5x3) performing element-wise multiplication and summation to produce feature maps where the early convolution layers capture basic features (e.g., edges, comers, textures, and shapes) and the later convolution layers capture more complex patterns and structures. The number of filters (e.g., kernel size) may be considered a hyperparameter that is fine-tuned during training / validation. In some embodiments, the discriminator has five convolutional layers with increasing number of filters (e.g., 64 filters for the first layer, and 2x filters for each subsequent layer (128, 256, 512, and 1024 respectively)). The convolution-pooling operations help to preserve the spatial relationships between pixels.

[0146] After each convolution operation, an activation function is applied across the feature map to introduce non-linearity into the model so that it leams complex patterns and relationships in the data that linear operations alone cannot capture. Examples of activation functions that may be used include the linear activation function, the rectifier linear unit (ReLU), the leaky ReLU, the sigmoid activation function, the SoftMax activation function, the hyperbolic tangent activation function, and the exponential activation function. In certain instances, the activation function used for each convolutional layer is the leaky ReLU, which introduces a small, non-zero gradient (e.g., 0.01 or 0.02) for negative input values. This means that negative values in the feature map do not get converted to zeros (as in ReLU) and instead keep a small positive value. This helps keep the neurons active and learning, even if they receive negative input values. In some instances, the activation function is the same across all convolution layers. In other instances, the activation function is varied across the convolution layers (e.g., one or more convolutional layers use ReLU and another one or more convolutional layers use leaky ReLU).Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0147] Following convolution and activation, the data is compressed by its corresponding pooling layer (e.g.. convolution output of 720a is compressed to generate pooled layer 725a) to reduce the spatial dimensions of the feature maps. A variety of pooling methods (e.g., max pooling, average pooling, or the like) can be performed. The pooling operation may be performed at an appropriate size (e.g., pooling window of size 2x2).

[0148] After the convolution-pooling layers 710, the data flows into the classification-output layers 715. which are configured to make final classification (e.g., how distinguishable the ■‘fake” image is from the “real” image) based on the extracted features or the feature map. The classification-output layers 715 may comprise a flattened layer 730, one or more fully connected layers 735, and an output layer 740A-B that provides the predicted classifications. The feature map obtained from the final pooling layer 725n is multi-dimensional arrays. To transition from the multi-dimensional feature map to a format suitable for the fully connected layers 735, the feature map can be flattened into a one-dimensional vector (e g., the flattened layer 730). This process involves taking the multi-dimensional array of features and converting it into a long vector that maintains the same numerical values (e.g., by concatenation or using a flatten operation).

[0149] Once the feature maps / tensors are flattened, the resulting vectors are passed into the fully connected layers 735. Each neuron of the fully connected layers 735 is connected to every neuron in the previous layer and / or the next layer. These layers perform linear transformations followed by the application of activation functions (such as ReLU or sigmoid), which introduce non-linearity and help in learning complex patterns. The fully connected layers 735 serve to combine the features extracted by the convolution-pooling layers 710 in various ways to produce higher-level representations. These representations are useful as they consolidate all the information gathered from the earlier stages of the network and are tailored towards the specific task of classification. In some embodiments, the activation function(s) used in the fully connected layers 735 are the same or different than the activation function(s) used in the convolution layers 720a-n of the convolution-pooling layers 710. In some embodiments, the CNN includes only one fully connected layer.

[0150] The final layer of the classification-output layers 715 is the output layer 740A-B, which provides the predicted binary classifications. In many classification tasks, this is typically a SoftMax layer, especially when dealing with multi-class classification problems. The SoftMax function converts the output scores from the final fully connected layer intoAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT probabilities that sum to one, providing a probabilistic interpretation of the predictions. Each output node in this layer corresponds to a different class label, and the node with the highest probability is often taken as the network's prediction. In other scenarios, such as binary classification, a sigmoid function might be used instead. The output layer 740A-B thus translates the learned features and patterns into a final prediction, completing the process of image recognition or classification within the CNN. As shown in FIG. 7A, the CNN 700 is trained to provide two different predictions or classifications 740A or 740B; however, this example is not meant to be limiting and other prediction problems where more or fewer predictions are made may also be considered.Training the Generator and the Discriminator

[0151] Training the GAN involves in training both the generator 505 and the discriminator 510 as shown in FIG. 5. A configuration file may be acquired before the training. The configuration file may be acquired from a client device 130A described with respect to FIG. 1, from a storage 115, or from the preprocessing module 104. The configuration file may include hyperparameters of the generator 505 and the discriminator 510 (e.g.. depth of encoder and decoder of a U-net, kernel size, stride numbers, activation functions, loss functions, optimizers, default weights and biases, and the like).

[0152] Training data is also obtained. The training data can be the set of paired patches extracted at block 220 of FIG. 2. After initializing the generator and the discriminator, the training of the generator can be run first on a batch of training patches for one or tw o epochs to produce the batch of generated patches, and the batch of real stained images and generated stained images (from the generator) are fed into the discriminator to compute the discriminator loss (e.g., using Equation (1)) and update the weights of the discriminator. The training of the generator can be run for another one or two epochs to generate new stained images and the adversarial loss is computed based on the discriminator’s feedback. A pixel-level generator loss is calculated and combined with the discriminator loss to update the generator’s weights. In some embodiments, the training of the GAN proceeds in alternating periods: the discriminator trains for one or more epochs, the generator trains for one or more epochs, and repeat to continue to train the generator and discriminator. In some embodiments, the discriminator training ignores the generator loss (e g., generator kept constant during the discriminator training phase) and just uses the discriminator loss. In some embodiments, theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT discriminator updates its weights through backpropagation. In some embodiments, both the discriminator and the generator update their weights through backpropagation.

[0153] In some embodiments, a vector of random noise is used as the initial input for the generator. The generator then transforms this noise into an output (a “fake” image). The random noise can be Gaussian noise or generated using a uniform distribution. The output is then processed by the discriminator to generate “fake” or “real” classification. A discriminator loss is determined from the classification. Since the discriminator processes the generated data, the gradients of the discriminator’s output with respect to its input (the “fake” image) are computed first. These gradients are propagated back through the discriminator layers to the point where the generated data enters the discriminator. The gradients from the discriminator (with respect to the “fake” image) are then propagated back through the generator to compute how the generator’s parameters influenced the “fake” image that the discriminator evaluated. At the end of this process, gradients are calculated to show how changes to the generator’s parameters affect the discriminator’s output. In some embodiments, the gradients are used to change only the generator’s weights, for example, using an optimization algorithm, such as Stochastic Gradient Descent (SGD) or Adam. That is, while backpropagation passes through the discriminator to compute the gradients, the discriminator’s weights are not updated during the backpropagation. The discriminator acts as a fixed evaluator when training the generator.

[0154] The training of the GAN may be performed using software tools such as Python (Python 3.8), TensorFlow (e.g., TensorFlow 2), and / or Pytorch (e.g., Pytorch 2). In some instances, the training is performed for up to 1000 epochs (e.g., 100-300 epochs). In some embodiments, mini-batches are used during training (e.g., batch size of 16 or 32) to improve training stability. Performance of the generator is periodically evaluated on the validation set to monitor overfitting and adjust hyperparameters. The generator to be implemented in the inference phase (e.g., used by a digital image staining system 320 shown in FIG. 3) can be the one with the lowest loss on the validation set. In some embodiments, the trained generator can be fine-tuned using a new training set comprising images of a specific task. For example, the generator may be initially trained to produce stained images of any kinds (e.g., H&E and IHC), and the new training set includes H&E-stained images only. The fine-tuning may be performed using techniques including transfer learning, attention models, progressive growing, perceptual loss, domain adaptation, feature matching, and few-shot fine-tuning.

[0155] Below is an example process of training the GAN.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0156] Training Loop: Iterate for a specified number of epochs or until convergence:- For each batch of real data samples:1. Train the Discriminator:- Sample a batch of real data samples from the dataset.- Sample a batch of random noise vectors from the noise distribution.- Generate fake data samples.- Compute the discriminator loss using both real and fake data samples.- Perform backpropagation and update the discriminator’s weights using the discriminator loss.2. Train the Generator:- Sample a batch of random noise vectors.- Generate fake data samples.- Compute the generator loss using the discriminator’s output on the fake samples. - Perform backpropagation and update the generator’s weights using the generator loss. Loss functions:a. Equation (1);b. Equation (2);c. Binary Cross-Entropy (BCE) Lossd. Minimax loss (the generator tries to minimize the function while the discriminator tries to maximize it);e. Mean Squared Error (MSE) Loss;f. Least Squares GAN (LSGAN);g. Total Variation Loss;h. Non-saturating loss;i. Perceptual Loss;j . Kullback-Leibler (KL) Divergence;k. Feature Matching Lossl. Relativistic Average GAN (RaGAN)m. Wasserstein GAN (WGAN);n. Gradient Penalty;o. Wasserstein GAN with Gradient Penalty (WGAN-GP);p. Adversarial Loss; andAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT q. Hinge Loss.

[0157] Table 1 shows a use-case summary’ of some of the loss functions in training GANs.Table 1Validation and Verification of the Training

[0158] Validation of the trained GAN focuses on assessing the GAN's performance during training using quantitative metrics such as the Inception Score (IS) and Frechet Inception Distance (FID), which measure the quality and diversity’ of generated samples and their alignment with the real data distribution. Metrics like precision and recall evaluate the realism and diversity of generated data, while reconstruction error is used in conditional GANs to measure pixel or feature-level similanty to ground truth. Monitoring adversarial loss curves forAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT both the generator and discriminator is also crucial to identify potential issues like mode collapse, overfitting, or unstable training. Qualitative validation, such as visual inspection and comparison with real data, complements quantitative metrics by providing insights into the perceptual quality of the outputs. The training result may be reviewed or verified by a pathologist or a trained expert. Pathologists review the annotated images, ensuring that the automated detections align with histopathological criteria and clinical relevance.

[0159] Verification ensures that the GAN meets task-specific goals and produces outputs suitable for the intended application. For example, in conditional tasks like image-to-image translation or virtual staining, verification involves ensuring that generated outputs align accurately with the input conditions and ground truth. Metrics such as structural similarity index (SSIM) and peak signal-to-noise ratio (PSNR) can be used to evaluate the qualify of outputs in tasks requiring precise reconstruction. Diversify’ of generated samples is another key aspect of verification, ensuring the generator captures the full range of the real data distribution without overfitting. Employing a validation set, saving model checkpoints, and inspecting intermediate outputs throughout training are practical techniques to monitor and verify GAN performance. By combining quantitative metrics, qualitative evaluation, and applicationspecific validation, GANs can be trained and verified to produce high-quality, diverse, and realistic outputs tailored to their intended use.

[0160] It should be understood that, although the model training is described with respect to using GANs to perform staining tasks (projecting ID spectral signals representing light absorbance or reflectance across different wavelengths into a multi-dimensional data space, such as a 2D spatially resolved stained image (H&E-stained)), the scheme and processes are suitable for any image conversion or image projection tasks. For example, the techniques can be used for converting Electrocardiograms (ECGs) into a 2D heatmap representation for automated disease detection (e.g., arrhythmia)A 7-Output

[0161] At block 230, the trained machine learning model or a portion thereof (e.g., the generator 505 of the trained GAN 500 described with respect to FIG. 5) is output or provided. The output model may be implemented in the digital image processing platform 110 (described with respect to FIG. 1) to perform image preprocessing (e.g., using by the preprocessing module 104), virtual staining (e.g., using by the virtual staining unit 107), image segmentation (e.g., using by the segmentation unit 109), and / or disease diagnosis (e.g., using by theAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT diagnostic module 108). The output model may also be used in the digital image staining system 320 (described with respect to FIG. 3) to perform digital H&E staining of a sample. In some embodiments, the output model is used to perform patch extraction as shown in FIG. 4. The output model may be stored in the storage 115 described with respect to FIG. 1 for later use (e.g., inference, fine-tuning, or deployment). In some embodiments, the output model is stored in the cloud (the same cloud where the training is performed, while in the same or different instances).B. Processing Digital Images Using the Trained Machine Learning Model

[0162] FIG. 8 is a flowchart illustrating a process 800 for using a trained machine learning model to perform digital image processing (e.g., to perform a digital staining process). The process 800 depicted in FIG. 8 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the respective systems, hardware, or combinations thereof (e.g., the intelligent selection machine). For example, the process 800 may be implemented in the computing environment 100 depicted in FIG. 1 for processing, storing, transmitting, and displaying digital pathology images. The process 800 presented in FIG. 8 and described below is intended to be illustrative and nonlimiting. Although FIG. 8 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the steps may be performed in some different orders, or some steps may also be performed in parallel.

[0163] At block 805, a digital image depicting a portion of a biological sample obtained from a subject is accessed. In some embodiments, the digital image is a whole slide image or a portion thereof (e.g., a patch). The biological sample can be collected or obtained using the sample collector 101 described with respect to FIG. 1 and the digital image of the biological sample may be captured using the imaging module 102. Similar techniques used in Step Al and Step A2 described above may be used to capture and access the digital image. In some embodiments, the digital image is a grayscale image or a single-channel fluorescence image (e.g., a DAPI image). In some embodiments, the digital image has the same dimension as the input image used to train the machine learning model at Step A6.

[0164] For example, an image scanning device is configured and used to capture images of biological sample using a plurality of scopes. A slide containing the biological sample to be imaged is loaded to the image scanning device. A region of interest is selected in a scannableAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT portion of the slide, and the image scanning device is calibrated using an autofocus algorithm to accurately capture the digital image of the biological sample. In some embodiments, the image is a brightfield image of the biological sample. Alternatively, or additionally, the image can be a DAPI, Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, or Opal 780 image. The image can be captured by pre-staining the biological sample using DAPI, Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, or Opal 780. In some embodiments, the image is obtained by generating the DAPI, Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, or Opal 780 stained using autofluorescence-based techniques (e.g., converting a brightfield image to a DAPI image).

[0165] At block 810, a projecting space and a staining category for an image conversion are defined. For example, the projecting space may be a three-dimensional space (e.g., a YCrCb space) and the staining category is H&E staining. In some embodiments, the staining category is selected from Hematoxylin and Eosin (H&E) staining, Masson’s Trichrome staining, Periodic Acid-Methenamine Silver (PAMS) staining, Periodic Acid-Schiff (PAS) staining, Gram staining, Periodic Acid-Schiff (PAS) staining, Jones’ silver staining, Immunohistochemistry (IHC) staining, a Giemsa staining, a Congo red staining, and a Ziehl-Neelsen staining. The projecting space and staining category information may be considered as metadata or hyperparameters of the trained machine learning model. In some embodiments, the defining is performed by acquiring the metadata or hyperparameters of the training machine learning model. The information may be stored in an instance or a service of a cloud, or in the storage 115 described with respect to FIG. 1 In some embodiments, the projecting space has a higher dimension (e.g., a multi-dimensional space) than the initial space of the digital image accessed at block 805. In some embodiments, at least one of the dimensions of the projecting space depicts a luminance of an image. In some embodiments, the projecting space is a YCrCb space or an RGB space. In some embodiments, the staining category can be a segmentation category (e.g., segment boundaries or boxes of a target object).

[0166] At block 815, the digital image is projected to the projecting space using the trained machine learning model (e.g., a neural network) based on the staining category. Each pixel of the original digital image is projected to a pixel of the projected image at a same location of the digital image (see e.g., a pixel Vi of the input channel 340 is projected to a voxel V’I of the output channels 350). The trained machine learning model may be a generator of a GAN trained to perform the image projection based on the projecting space and staining category information. The training process can be the same or similar process in Step A6 describedAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT above. In some embodiments, the generator generates a virtually stained image of the digital image based on the staining category (e.g., H&E staining). The projection may be performed using the virtual staining unit 107 described with respect to FIG. 1.

[0167] In some embodiments, the training process begins with accessing a set of digital images and their corresponding ground truth images. Each digital image depicts a portion of a biological sample obtained from a subject, and a corresponding ground truth image depicts the portion of the biological sample after the biological sample being stained based on the staining category. Both images capture a substantially similar portion of the biological sample. As used herein, the term “substantially similar” refers to a condition in which two images capture the same or nearly identical portion of a biological sample, with any differences being minor and not materially altering the content or structure of the sample. For example, two images of a tissue section might be considered substantially similar if they overlap by at least 80%, 90%, 95%, or 99%, or if corresponding features (e.g., nuclei or cellular structures) are aligned with a positional difference of less than 1, 2, 3, 4, or 5 micrometers. In some embodiments, the digital images are unstained images or fluorescence images, and the fluorescence images may be generated by staining the sample or using autofluorescence-based techniques. In some embodiments, the corresponding ground truth images are real stained images. The ground truth images may be stained using the same staining category (e.g., H&E staining) or different staining categories. The biological samples may be obtained from the same subjects or different subjects. The biological samples may be obtained from the same origin (e.g., the lung) or different origins. The biological samples may be of the same type (tissue) or different.

[0168] Each digital image is aligned with its corresponding ground truth image using a registration algorithm (see Step A4 described above). In some embodiments, the registration algorithm is an affine registration, a B-spline registration, an elastic registration, a mutual information registration, a Demons algorithm, or a deep learning-based registration. The alignment may involve identifying and matching key features in both images, followed by applying transformations (e.g., affine, rigid, or non-rigid) to align the digital image with the ground truth. To improve efficiency and accessibility for training purposes, the aligned images and their transformations (e.g., transformation matrices or deformation fields) are stored in a structured format with metadata (e.g., alignment parameters, image identifiers, and preprocessing details). The metadata can be used for quick retrieval and ensures consistency across training batches, enabling efficient loading and processing during model training while maintaining alignment accuracy across the dataset.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0169] To further improve training efficiency, a set of patches is extracted from each of the aligned images. The extraction can be performed by following techniques in Step A5 described above. In some embodiments, about 10% to 50% of the total patches are extracted from each image, and each patch covers a portion of the biological sample (removing background patches). In some embodiments, at least 10% of each patch depicts the biological sample (at most 90% of the area in the patch is the background). In some embodiments, a certain number of locations is selected (e.g., about 1000), and patches are extracted from the selected locations with filtering (e.g., removing background patches). For example, tissue detection may be performed using the ground truth image before selecting the 1000 locations, and a manual examination is performed to remove patches with an artifact (blurring, folding, color variation, debris, out of focus, pen marking, noise, vignetting, chromatic aberration, or the like) or using an artifact detection algorithm. The ground truth patches with artifacts and their corresponding input patches are removed from the training.

[0170] The extraction criteria can be predesigned and stored in a configuration file. In some embodiments, each patch is of the same size (e.g., 20 x 20 pixels, 50 x 50 pixels, 100 x 100 pixels, 128 x 128 pixels, 256 x 256 pixels, or 512 x 512 pixels). In some embodiments, the patches are extracted using a random generator. In some embodiments, the patches are first extracted from the ground truth images, and corresponding patches at the same locations are then generated from the digital images.

[0171] A set of hyperparameters for the machine learning model to be trained to perform the digital staining task is accessed. The hyperparameter information may be stored in a configuration file in a database (e.g., storage 115 in FIG. 1), in the cloud, or accessed from a client device based on a user’s instruction. For example, if the machine learning model is a neural network, the set of hyperparameters may include a depth of the neural network and a number of filters used by each layer of the neural network. One or more loss functions can also be accessed for the training. In some embodiments, the machine learning model is a Generative Adversarial Network (GAN) comprising a generator and a discriminator. In some embodiments, the generator is a U-net and the discriminator is a convolutional neural network (CNN). In some embodiments, the loss function is determined based on a quadratic generator loss and a quadratic discriminator loss, and wherein parameters of the loss function are included in the set of hyperparameters. In some embodiments, the loss function is a combination of Equation (1) and Equation (2).Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0172] The machine learning model can be iteratively trained for a number of epochs using a training dataset based on the set of hyperparameters and the loss function(s) or until a condition is met. The training dataset includes the paired patches (input patches and ground truth patches) or a portion of them. For example, 70% of the total paired patches is assigned to the training dataset, 20% to the validation dataset, and 10% to the test dataset. The trained machine learning model (or a portion, e.g., a generator portion of a GAN) is output for implementation, fine-tuning, or deployment.

[0173] At block 820, the projected image is output. The projected image may be further used for segmentation (e.g., using the segmentation unit 109 in FIG. 1) or for disease diagnosis (using the diagnostic module 108 in FIG. 1). The projected image may be further displayed on a graphical user interface (e.g., the interface 132A) of a client device (e.g., the client device 130A). In some embodiments, both the digital image accessed at block 805 and the projected image generated at block 815 are displayed on the interface. In some embodiments, the digital image and the projected image are paired up (e.g., using the same identifier) and stored in a database (e.g., the storage 115) or in the cloud.IV. TRAINING AND USING A MACHINE LEARNING MODEL WITH DIGITAL IMAGES

[0174] FIG. 9 shows a block diagram of a machine learning pipeline 900 comprising several subsystems that work together to train, validate, and implement one or more machine learning models in accordance with various embodiments. The machine learning pipeline 900 may be executed as part of the preprocessing module 104, the analyzing module 106, or the diagnostic module 108 of the computing environment 100 described with respect to FIG. 1, or the digital image staining system 320 in FIG. 3. The machine learning pipeline 900 comprises a data subsystem 905 for collecting, generating, preprocessing, and labeling of training and validation datasets 910, training and validation subsystem 915 that facilitates the training and validation of one or more machine learning algorithms 920, and inference subsystem 925 for deploying and implementing one or more trained machine learning models 930 independently or in combination with one or more other systems or services 935 for downstream processes.

[0175] As used herein, machine learning algorithms (also described herein as simply algorithm or algorithms) are procedures that are run on datasets (e.g., training and validation datasets) and perform pattern recognition on datasets, learn from the datasets, and / or are fit on the datasets. Examples of machine learning algorithms include linear and logistic regression,Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT decision trees, artificial neural networks, k-means, and k-nearest neighbor. In contrast, machine learning models (also described herein as simply model or models) are the output of the machine learning algorithms and are comprised of model data and a prediction algorithm. In other words, the machine learning model is the program that is saved after running a machine learning algorithm on training data and represents the rules, numbers, and any other algorithmspecific data structures required to make inferences. For example, a linear regression algorithm may result in a model comprised of a vector of coefficients with specific values, a decision tree algorithm may result in a model comprised of a tree of if-then statements with specific values, or neural network, backpropagation, and gradient descent algorithms together result in a model comprised of a graph structure with vectors or matrices of weights with specific values.A Data Subsystem

[0176] Data subsystem 905 is used to collect, generate, preprocess, and label data to be used to train and validate one or more machine learning algorithms 920. The data collection can include exploring various data sources such as public datasets, private data collections, or realtime data streams, depending on a project’s needs. In some instances, a data source is a public or online repository of information or examples pertinent to a general or target domain space. Many domains have publicly available datasets provided by governments, universities, or organizations. For example, many government and private entities offer datasets on healthcare, environmental data, and more through various portals. For proprietary needs, data might be available through partnerships or purchases from private companies that specialize in data aggregation. In other instances, a data source is a private repository of information or examples pertinent to a general or target domain space. For example, a data source can be the storage 115 that stores digital WSIs generated by the digital image processing platform 110 described with respect to FIG. 1. Once a data source is identified, data subsystem 905 can be used to collect data through appropriate methods such as downloading from online repositories, web scraping, using APIs for real-time data, creating datasets through surveys and experiments, or by running assays. The acquired raw data may be further preprocessed to generate the training and validation datasets 910.

[0177] In some instances, raw data (e.g., images or patches) may be generated as opposed to being collected or acquired. Data generating may comprise data synthesis and / or data augmentation. Different data synthesis and / or data augmentation techniques may be implemented by the data subsystem 905 to generate data to be used for the training andAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT validation subsystem 915. Data synthesizing involves creating entirely new data points from scratch. This technique may be used when real data is insufficient, too sensitive to use, or when the cost and logistical barriers to obtaining more real data are too high. The synthesized data should be realistic enough to effectively train a machine learning model, but distinct enough to comply with regulations (e.g., copyright and data privacy), if necessary. Techniques such as Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs) may be used to generate new data examples. These models leam the distribution of real data and attempt to produce new data examples that are statistically similar but not identical. Data augmentation, on the other hand, refers to techniques used to artificially expand the size of a dataset by creating modified versions of existing data examples. The primary goal of data augmentation is to increase variation in the data in order to make the model more robust to variations it might encounter in the real world, thereby improving its ability to generalize from the training data to unseen data. This is especially common in image and speech recognition tasks but is applicable to other data types as well. For images, data augmentation may include rotations, flipping, scaling, or altering the color / lighting conditions. For text, data augmentation may include synonyms replacement, back translation, or sentence shuffling. For audio, data augmentation may include changes made to pitch, speed, or background noise.

[0178] Preprocessing may be implemented using data subsystem 905, serving as a bridge between ra \' data acquisition and effective model training. The primary objective of preprocessing is to transform raw data into a format that is more suitable and efficient for analysis, ensuring that the data fed into machine learning algorithms is clean, consistent, and relevant. This step can be useful because raw data often comes with a variety of issues such as missing values, noise, irrelevant information, and inconsistencies that can significantly hinder the performance of a model. By standardizing and cleaning the data beforehand, preprocessing helps in enhancing the accuracy and efficiency of the subsequent analysis, making the data more representative of the underlying problem the model aims to solve.

[0179] Preprocessing may be performed using a processor (e.g., a CPU, GPU, TPU, FPGA, the like, or any combination thereof), memory', and storage that operates software or computer program instructions (e.g.. TensorFlow, PyTorch, Keras, and the like) to execute arithmetic, logic, input and output commands for processing acquired data. The processor measures weights, coefficients, and importance values for all quality features, a subset of quality features, or both by scaling acquired datasets (images or patches for training, validation, and test). Data scaling comprises adjusting the features in a machine learning model so that all features are onAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT a relatively similar scale close to normal distribution. Further, data scaling also helps to identify quality features that have the highest impact on the performance of the machine learning models being assessed (e.g., false positive capture rates and true positive flag rates) and remove redundant quality features. Methods for data scaling include: MinMaxScaler, RobustScaler, Standards cal er, Normalizer, and any other methods known to one of skill in the art.

[0180] An example of preprocessing, particularly for scanned WSI, is inter-WSI variability. Sources that can cause inter-WSI variability include the use of different scanners, different staining processes, tissue thickness, magnification, etc. These differences in experimental and digitization of WSIs can result in artifacts like blurring, folding, and color variation between WSIs having the same sample tissue. To overcome this, data subsystem 905 can apply techniques that correct for color variations, differences in image sizes / magnification. and / or signal normalization. To address stain variation or color variation, data subsystem 905 may apply grayscale conversion, color normalization, and / or color augmentation (application of random hue, saturation, brightness, and contrast) to the WSIs. Data subsystem 905 may apply a quality control filter that removes low quality WSIs (e.g., blurry, folded tissue) from the acquired images.

[0181] Raw data preprocessing may comprise generating small ‘"patches” of an original WSI. Digitization of pathology slides produces WSIs reaching gigapixel sizes. For example, an image of a 20 mm x 20 mm sample of tissue scanned at 40X objective power may have 80,000 x 80,000 pixels. Further, if this image is in standard 24-bit color, the flattened version of the image would be about 19.2 GB in size. Digital images of such size require very lengthy computational run times and multiple rounds of processing. Accordingly, data subsystem 905 may preprocess the WSI into patches that are small regions corresponding to a portion of the WSI or the corresponding biological sample. The patches may be of any size appropriate to train a machine learning model. For example, a patch can be a region of a whole slide scan or a region of interest having appropriate pixel dimension (e.g., 50 x 50 pixels) and resolution quality (e.g., no magnification, 4x magnification, lOx magnification, 20x magnification, and / or 40x magnification, etc.). Moreover, the degree of overlap allowed between patches may also be controlled for. In some embodiments, the size of the patches, resolution quality, and overlap may be adjustable parameters or hyperparameters of the machine learning model and dependent on the type of sample (e g., cell, tissue, fluid) present on the slide. In addition to patch generation, manual or computational tools (e.g., segmentation tools) may also be applied to the WSI to distinguish between tissue patches and non-tissue / background patches. ManualAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT removal of background can include outlining the tissue or applying a box around the tissue that minimizes the amount of background. More advanced tools, like segmentations, may apply filters and thresholds to the WSI, where any patch below the threshold (chosen arbitrarily) is removed. In so doing, the amount of computational power needed during training and / or inference is reduced as only patches with sample on them are used for analysis. In some instances, data subsystem 905 may save the patches as individual image files or as coordinates of each individual patch with respect to the original image.

[0182] In various embodiments, data preprocessing involves generating patches from pixelwise aligned WSIs generated by the preprocessing module 104 described with respect to FIG.1. The data subsystem 905 may deploy a software platform (e.g., QuPath) to generate patches from aligned WSIs. Initially, each image channel that is included in the aligned WSI can be imported into the software. Of the imported image channels, one image channel is used to define regions of interest within the WSI. The selected channel may be a fluorescence channel (e.g., DAPI) as these channels typically include the most artifacts (e.g., debris, out of focus, etc.) that need to be removed or avoided during patch generation. The regions of interest may be selected manually using tools like freehand drawing, rectangular or circular selection tools, or they may be selected automatically using built in algorithms that can define the regions of interest based on a set of parameters (e.g., color or texture). Once regions of interest are indicated, the software platform adds square annotations, representing patches, within the regions based on user specified size parameters.

[0183] Other raw data preprocessing techniques include data cleaning, normalization, feature extraction, dimensionality reduction, and the like. Data cleaning may involve removing duplicates, filling in missing values, or filtering out outliers to improve data quality. Normalization involves scaling numeric values to a common scale without distorting differences in the ranges of values, which helps prevent biases in the model due to the inherent scale of features. Feature extraction involves transforming the input data into a set of useable features, possibly reducing the dimensionality of the data in the process. For instance, in image analysis, feature reduction techniques such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA). t-Distributed Stochastic Neighbor Embedding (t-SNE), autoencoders, and feature selection can be used for simplifying images, improving model performance, and gaining insights into the underlying structure of the images. These techniques not only help in reducing the computational load on the model but also in mitigating issues like overfitting by simplifying the data without losing critical information.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0184] In the instance that machine learning pipeline 900 is used for supervised or semisupervised learning of machine learning models, labeling techniques can be implemented as part of the data collection. The quality and accuracy of data labeling directly influence the model’s performance, as labels serve as the definitive guide that the model uses to leam the relationships between the input features and the desired output. Particularly in complex domains such as image recognition, natural language processing, or medical diagnosis, precise and consistent labeling is important because it provides the ground truth or target outcomes against which the model’s predictions are compared and adjusted during training. Effective labeling ensures that the model is trained on correct and clear examples, thus enhancing its ability to generalize from the training data to real-world scenarios.

[0185] Labeling techniques can vary significantly depending on the type of data and the specific requirements of the project. Manual labeling, where human annotators label the data, is one method that can be used. This approach is useful when a detailed understanding and judgment are required, such as in labeling medical images or categorizing text data where context and subtlety are important. However, manual labeling is time-consuming and prone to inconsistency, especially with many annotators. To mitigate this, semi-automated labeling tools may be used as part of data subsystem 905 to pre-label data using algorithms, which human annotators may then review and correct as needed. Another approach is active learning, a technique where the model being developed is used to label new data iteratively. The model suggests labels for new data points, and human annotators may review and adjust certain predictions such as the most uncertain predictions. This technique optimizes the labeling effort by focusing human resources on a subset of the data, e.g., the most ambiguous cases, improving efficiency and label quality through continuous refinement.

[0186] Once collected, generated, preprocessed, and / or labeled, the data may then be split into the training and validation datasets 910. The training and validation datasets 910 may comprise the raw data and / or the preprocessed data. The training and validation datasets 910 are typically split into at least three subsets of data: training, validation, and testing. The training set is used to fit the model, where the machine learning model learns to make inferences based on the training data. The validation set, on the other hand, is utilized to tune hyperparameters and prevent overfitting by providing a sandbox for model selection. Finally, the test set serves as a new and unseen dataset for the model, used to simulate real-world application and evaluate the final model’s performance. The process of splitting ensures thatAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT the model can perform well not just on the data it was trained on, but also on new. unseen data, thereby validating and testing its ability to generalize.

[0187] Various techniques can be employed to split the data effectively, with each method aiming to maintain a good representation of the overall dataset in each subset. A simple random split (e.g., a 70 / 20 / 10%, 80 / 10 / 10%, or 60 / 25 / 15%) is the most straightforward approach, where examples from the data are randomly assigned to each of the three sets. However, more sophisticated methods may be necessary to preserve the underlying distribution of data. For instance, stratified sampling may be used to ensure that each split reflects the overall distribution of a specific variable, particularly useful in cases where certain categories or outcomes are underrepresented. Another technique, k-fold cross-validation, involves rotating the validation set across different subsets of the data, maximizing the use of available data for training while still holding out portions for validation. These methods help in achieving more robust and reliable model evaluation and are useful in the development of predictive models that perform consistently across varied datasets.

[0188] Data subsystem 905 is also used for collecting, generating, setting, or implementing model hyperparameters 940 for the training and validation subsystem 915. The hyperparameters control the overall behavior of the models. Unlike model parameters 945 that are learned automatically during training, hyperparameters 940 are set before training begins and have a significant impact on the performance of the model. For example, in a neural network, hyperparameters include the learning rate, number of layers, number of neurons / nodes per layer, activation functions, convolution kernel width, the number of kernels for a model, the number of graph connections to make during a lookback period, and the maximum depth of a tree in a random forest among others. These settings can determine how quickly a model leams, its capacity to generalize from training data to unseen data, and its overall complexity’. Correctly setting hyperparameters is important because inappropriate values can lead to models that underfit or overfit the data. Underfitting occurs when a model is too simple to leam the underlying pattern of the data, and overfitting happens when a model is too complex, learning the noise in the training data as if it were signal.(B) Training, Validation, and Testing

[0189] The training and validation subsystem 915 is comprised of a combination of specialized hardware and software to efficiently handle the computational demands required for training, validating, and testing a machine learning model. On the hardware side, high-Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT performance GPUs (Graphics Processing Units) may be used for their ability to perform parallel processing, drastically speeding up the training of complex models, especially deep learning networks. CPUs (Central Processing Units), while generally slower for this task, may also be used for less complex model training or when parallel processing is less critical. TPUs (Tensor Processing Units), designed specifically for tensor calculations, provide another level of optimization for machine learning tasks. On the software side, a variety of frameworks and libraries are utilized, including TensorFlow, PyTorch, Keras, and scikit-leam. These tools offer comprehensive libraries and functions that facilitate the design, training, validation, and testing of a wide range of machine learning models across different computing platforms, whether local machines, cloud-based systems, or hybrid setups, enabling developers to focus more on model architecture and less on underlying computational details.

[0190] Training is the initial phase of developing machine learning models 930 where the model leams to make predictions or decisions based on data training data provided from the training and validation datasets 910. During this phase, the model iteratively adjusts its internal model parameters 945 to achieve a preset optimization condition. In a supervised machine learning training process, the preset optimization condition can be achieved by minimizing the difference between the model output (e.g., predictions, classifications, or decisions) and the ground truth labels in the training data. In some instances, the preset optimization condition can be achieved when the preset fixed number of iterations or epochs (full passes through the training dataset) is reached. In some instances, the preset optimization condition is achieved when the performance on the validation dataset stops improving or starts to degrade. In some instances, the preset optimization condition is achieved when a convergence criterion is met, such as when the change in the model parameters falls below a certain threshold between iterations. This process, known as fitting, is fundamental because it directly influences the accuracy and effectiveness of the model.

[0191] In an exemplary training phase performed by the training and validation subsystem 915, the training subset of data is input into the machine learning algorithms 920 to find a set of model parameters 945 (e.g., weights, coefficients, trees, feature importance, and / or biases) that minimizes or maximizes an objective function (e.g., a loss function, a cost function, a contrastive loss function, a cross-entropy loss function, an Out-of-Bag (OOB) score, etc.). To train the machine learning algorithms 920 to achieve accurate predictions, “errors” (e.g., a difference between a predicted label and the ground truth label) need to be minimized. In order to minimize the errors, the model parameters can be configured to be incrementally updated byAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT minimizing the objective function over the training phase ("optimization”). Various different techniques may be used to perform the optimization. For example, to train machine learning algorithms such as a neural network, optimization can be done using back propagation. The current error is typically propagated backwards to a previous layer, where it is used to modify the weights and bias in such a way that the error is minimized. The weights are modified using the optimization function. Other techniques such as random feedback, Direct Feedback Alignment (DFA), Indirect Feedback Alignment (IF A), Hebbian learning, and the like can also be used to update the model parameters 945 in a manner as to minimize or maximize an objective function. This cycle is repeated until a desired state (e.g., a predetermined minimum value of the objective function) is reached.

[0192] The training phase is driven by three primary components: the model architecture (which defines the structure of the algorithm(s) 920), the training data (which provides the examples from which to leam), and the learning algorithm (which dictates how the model adjusts its model parameters). The goal is for the model to capture the underlying patterns of the data without memorizing specific examples, thus enabling it to perform well on new, unseen data.

[0193] The model architecture is the specific arrangement and structure of the various components and / or layers that make up a model. In the context of a neural network, the model architecture may include the configuration of layers in the neural network, such as the number of layers, the type of layers (e.g., convolutional, recurrent, fully connected), the number of neurons in each layer, and the connections between these layers. In the context of a random forest consisting of a collection of decision trees, the model architecture may include the configuration of features used by the decision trees, the voting scheme, and hyperparameters such as the number of trees in the forest, the maximum depth of each tree, the minimum number of samples required to split a node, and the maximum number of features to consider when looking for the best split. In some instances, the model architecture is configured to perform multiple tasks. For example, a first component of the model architecture may be configured to perform a feature selection function, and a second component of the model architecture may be configured to perform a feature scoring function. The different components may correspond to different algorithms or models, and the model architecture may be an ensemble of multiple components.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0194] Model architecture also encompasses the choice and arrangement of features and algorithms used in various models, such as decision trees or linear regression. The architecture determines how input data is processed and transformed through various computational steps to produce the output. The model architecture directly influences the model's ability to leam from the data effectively and efficiently, and it impacts how well the model performs tasks such as classification, regression, or prediction, adapting to the specific complexities and nuances of the data it is designed to handle.

[0195] The model architecture can encompass a wide range of algorithms 920 suitable for different kinds of tasks and datatypes. Examples of algorithms 920 include, without limitation, linear regression, logistic regression, decision tree, Support Vector Machines, Naives Bayes algorithm, Bayesian classifier, linear classifier, K-Nearest Neighbors, K-Means. random forest, dimensionality reduction algorithms, grid search algorithm, genetic algorithm, AdaBoosting algorithm, Gradient Boosting Machines, and Artificial Neural Networks such as convolutional neural network (CNN), an inception neural network, a U-Net, a V-Net, a residual neural network (ResNet), a transform neural network, a recurrent neural network, a Generative adversarial network (GAN), or other variants of Deep Neural Networks (DNN) (e.g., a multilabel n-binary DNN classifier or multi-class DNN classifier). These algorithms can be implemented using various machine learning libraries and frameworks such as TensorFlow, PyTorch, Keras, and scikit-leam, which provide extensive tools and features to facilitate model building, training, validation, and testing.

[0196] The learning algorithm is the overall method or procedure used to adjust the model parameters 945 to fit the data. It dictates how the model learns from the data provided during training. This includes the steps or rules that the algorithm follows to process input data and make adjustments to the model's internal parameters (e.g., weights in neural networks) based on the output of the objective function. Examples of learning algorithms include gradient descent, backpropagation for neural networks, and splitting criteria in decision trees.

[0197] Various techniques may be employed by training and validation subsystem 915 to train machine learning models 930 using the learning algorithm, depending on the type of model and the specific task. For supervised learning models, where the training data includes both inputs and expected outputs (e.g.. ground truth labels), gradient descent is a possible method. This technique iteratively adjusts the model parameters 945 to minimize or maximize an objective function (e.g., a loss function, a cost function, a contrastive loss function, etc.).Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT The objective function is a method to measure how well the model’s predictions match the actual labels or outcomes in the training data. It quantifies the error between predicted values and true values and presents this error as a single real number. The goal of training is to minimize this error, indicating that the model's predictions are, on average, close to the true data. Common examples of loss functions include mean squared error for regression tasks and cross-entropy loss for classification tasks.

[0198] The adjustment of the model parameters 945 is performed by the optimization function or algorithm, which refers to the specific method used to minimize (or maximize) the objective function. The optimization function is the engine behind the learning algorithm, guiding how the model parameters 945 are adjusted during training. It determines the strategy to use when searching for the best weights that minimize (or maximize) the objective function. Gradient descent is a primary example of an optimization algorithm, including its variants like stochastic gradient descent (SGD), mini-batch gradient descent, and advanced versions like Adam or RMSprop, which provide different ways to adjust learning rates or take advantage of the momentum of changes. For example, in training a neural network, backpropagation may be used with gradient descent to update the weights of the network based on the error rate obtained in the previous epoch (cycle through the full training dataset). Another technique in supervised learning is the use of decision trees, where a tree-like model of decisions is built by splitting the training dataset into subsets based on an attribute value test. This process is repeated on each derived subset in a recursive manner called recursive partitioning.

[0199] In unsupervised learning, where training data does not include labels, different techniques are used. Clustering is one method where data is grouped into clusters that maximize the similarities of data within the same cluster and maximize the differences with data in other clusters. The K-Means algorithm, for example, assigns each data point to the nearest cluster by minimizing the sum of distances between data points and their respective cluster centroids. Another technique. Principal Component Analysis (PCA), involves reducing the dimensionality of data by transforming it into a new set of variables, the principal components, which are uncorrelated and ordered so that the first few retain most of the variation present in all of the original variables. These techniques help uncover hidden structures or patterns in the data, which can be essential for feature reduction, anomaly detection, or preparing data for further supervised learning tasks.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0200] Validating is another phase of developing machine learning models 930 where the model is checked for deficiencies in performance and the hyperparameters 940 are optimized based on validation data provided from the training and validation datasets 910. The validation data helps to evaluate the model's performance, such as accuracy, precision, recall, or Fl -score, to gauge how well the model is likely to perform in real-world scenarios. Hyperparameter optimization, on the other hand, involves adjusting the settings that govern the model's learning process (e.g., learning rate, number of layers, size of the layers in neural networks) to find the combination that yields the best performance on the validation data. One optimization technique is grid search, where a set of predefined hyperparameter values are systematically evaluated. The model is trained with each combination of these values, and the combination that produces the best performance on the validation set is chosen. Although thorough, grid search can be computationally expensive and impractical when the hyperparameter space is large. A more efficient alternative optimization technique is random search, which samples hyperparameter combinations from a defined distribution randomly. This approach can in some instances find a good combination of hyperparameter values faster than grid search. Advanced methods like Bayesian optimization, genetic algorithms, and gradient-based optimization may also be used to find optimal hyperparameters more effectively. These techniques model the hyperparameter space and use statistical methods to intelligently explore the space, seeking hyperparameters that yield improvements in model performance.

[0201] An exemplary validation process includes iterative operations of inputting the validation subset of data into the trained algorithm(s) using a validation technique such as K-Fold Cross-Validation, Leave-one-out Cross-Validation, Leave-one-group-out Cross-Validation, Nested Cross-Validation, or the like, to fine-tune the hyperparameters and ultimately find the optimal set of hyperparameters. In some instances, a 5-fold cross-validation technique may be used to avoid overfitting the trained algorithm and / or to limit the number of selected features per split to the square-root of the total number of input features. In some instances, training dataset is split into 5 equal-size cohorts (or about equal-size), and every four of the cohorts are used to train an algorithm to generate five models (e.g., cohorts #1, 2, 3, and 4 are used to train and generate model 1, cohorts #1, 2, 3. and 5 are used to train and generate model 2, cohorts #1, 2, 4, and 5 are used to train and generate model 3, cohorts #1, 3, 4, and 5 are used to train and generate model 4, and cohorts #2, 3, 4 and 5 are used to train and generate model 5). Each model is evaluated (or validated) using the unused cohort in the training (e.g., for model 5, cohort #1 is used for validation). The overall performance of the training can beAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT evaluated by an average performance of the five models. K-fold cross-validation provides a more robust estimate of a model's performance compared to a single training / validation split because it utilizes the entire dataset for both training and evaluation and reduces the variance in the performance estimate.

[0202] Once a machine learning model has been trained and validated, it undergoes a final evaluation using test data provided from the training and validation datasets 910, which is a separate subset of the data that has not been used during the training or validation phases. This step is crucial as it provides an unbiased assessment of the model's performance in simulating real-world operation. The test dataset serves as new, unseen data for the model, mimicking how the model would perform when deployed in actual use. During testing, the model’s predictions are compared against the true values in the test dataset using various performance metrics such as accuracy, precision, recall, and mean squared error, depending on the nature of the problem (classification or regression). This process helps to verify the generalizability of the model — its ability to perform well across different data samples and environments — highlighting potential issues like overfitting or underfitting and ensuring that the model is robust and reliable for practical applications. The machine learning models 930 are fully validated and tested once the output predictions have been deemed acceptable by user defined acceptance parameters. Acceptance parameters may be determined using correlation techniques such as Bland- Altman method and the Spearman's rank correlation coefficients and calculating performance metrics such as the error, accuracy, precision, recall, receiver operating characteristic curve (ROC), etc.Training, Testing, and Validation of CNNs

[0203] With regards to training a CNN, many of the techniques described for machine learning pipeline 900 with respect to FIG. 9 apply. For example, a data subsystem (like data subsystem 905) is used to collect, generate, preprocess, and label data to be used to train, test, and validate a CNN (e.g., CNN 700 in FIG. 7). Data may be generated by the digital image processing platform 110 described with respect to FIG. 1 that generates and stores digital WSIs. The WSIs generated by the imaging module 102 of the digital image processing platform 110 can be preprocessed by the preprocessing module 104 to generate paired patches corresponding to portions of the WSIs. In some instances, the WSIs may include a global / slide level annotation (e.g., as metadata) indicating if the specimen in the WSI is non-tumor or diseased (e.g., cancerous). The global annotation may be provided by one or more pathologistsAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT specifically trained to detect the presence or absence of tumor tissue on a slide or using an automated detection algorithm.

[0204] The paired patches are used by the training and validation subsystem 915 to train the CNN 700 to identify features within the patches. In this scenario, the algorithm 920 is the CNN architecture with specific hyperparameters obtained by the hyperparameters 940 of the data subsystem 905. Model parameters 945 of the CNN 700 can initially be set using default values. The CNN 700 learns features specifically associated with corresponding labels (e.g.. the stained patches) from the input data (e.g., the unstained patches). Features specifically associated with stained pattern (or other biomarkers relevant to the staining) may be assigned higher weights during training and the model parameters 945 will be automatically updated accordingly.

[0205] In some instances, the hyperparameters 940 for CNN 700 include a learning rate, a momentum (e.g.. 0.9, 0.99), number of convolutional layers (e.g., 2, 3, 4, 5, 6, 7, 8), number of filters per layer (e.g., 32 filters in the first layer, increasing by 2x per layer), size of filters (e.g., 3x3 or 5x5), batch size (e.g., 64), number of epochs (e.g., 500 or 1000), and the like. These settings can determine how quickly a model leams, its capacity' to generalize from training data to unseen data, and its overall complexity. In some instances when a CNN is initially trained, the hyperparameters 940 may be assigned at random. In other instances, the hyperparameters may be explicitly defined to control different aspects of the CNN such as memory or cost of execution. Hyperparameters may be tuned by a practitioner or during validation.

[0206] The adjustment of the CNN’s model parameters 945 is performed iteratively during training by backpropagation and optimization. The training begins with an initial forward pass, where input data is passed through the network, and predictions are generated by applying convolutional operations, activation functions, and pooling layers. The predictions are compared with the ground truth labels using a loss function (e.g., cross-entropy or mean squared error) to calculate the error. This error is then propagated backward through the network using backpropagation, which computes the gradients of the loss with respect to each parameter using the chain rule of calculus. These gradients indicate the direction and magnitude of change needed to minimize the loss. An optimization algorithm, such as Stochastic Gradient Descent (SGD) or Adam, updates the parameters by subtracting the gradients scaled by the learning rate, ensuring that the parameters gradually converge toward values that minimize the loss function. This process is repeated iteratively for multiple epochs over the training dataset, progressively improving the CNN’s ability to make accurate predictions. RegularizationAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT techniques like dropout or weight decay may also be applied during training to prevent overfitting.

[0207] Once the model parameters 945 of the CNN 700 have been trained by optimizing the loss or error function, the CNN 700 should be able to predict one or more features (e.g., generate an output image) from the input data. In addition to tracking the training loss, the testing loss can additionally be tracked to implement early stopping, which halts training of a neural network when a clear divergence between training and testing loss is noted. The early stopping process combats overfitting, preventing scenarios in which the neural network performs extremely well on the training data at the expense of generalizing to validation data.

[0208] Validation of CNN 700 is the process of checking for deficiencies in performance and the hyperparameters can be updated using the validation dataset. The validation data helps to evaluate the model's performance during the training phase and ensures that it is general. Once training is complete the model performance can be evaluated using the test dataset, using measurements such as accuracy, precision, recall, or Fl -score, to gauge how well the model is likely to perform in real-world scenarios.

[0209] Once CNN 700 has been trained and validated, it undergoes a final evaluation using test data provided from the training and validation datasets, which is a separate subset of the data that has not been used during the training or validation phases. This step is crucial as it provides an unbiased assessment of the model’s performance in simulating real-world operation. The test dataset sen es as new, unseen data for the CNN 700, mimicking how the model would perform when deployed in actual use. During testing, the model’s predictions are compared against the true values in the test dataset using various performance metrics such as accuracy, precision, recall, and mean squared error, depending on the nature of the problem (classification or regression). This process helps to verify the generalizability of CNN 700 — its ability- to perform well across different data samples and environments — highlighting potential issues like overfitting or underfitting and ensuring that the model is robust and reliable for practical applications. CNN 700 is fully validated and tested once the output predictions have been deemed acceptable by user defined acceptance parameters and will be used as the models 930 in the inference subsystem 925.

[0210] As should be understood, other training-validation mechanisms are contemplated and may be implemented. For example. CNN 700 may be trained and hyperparameters may be tuned on data from a first subset of data and the data from a second subset of data may only beAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT used for testing and evaluating performance of the model. Moreover, although the trainingvalidation mechanisms described herein focus on training a new CNN 700, these trainingvalidation mechanisms can also be utilized to fine-tune existing CNNs trained from other datasets. For example, in some instances, a CNN might have been pre-trained using digital pathology' images for one type of cancer to predict the detection of another ty pe of cancer. In those cases, CNN 700 can be used for transfer learning and retrained / validated using new sets of digital pathology images for a second type of cancer. Further, the learned neural network parameters may be frozen and only the last layer (e.g., the fully connected layer) of CNN 700 is retrained / validated to learn feature predictions associated w ith a second type of cancer. The above example also works for the U-net architecture of the generator 600 in FIG. 6.(C) Inference Phase for Machine Learning Models

[0211] The inference subsystem 925 is comprised of various components for deploying the machine learning models 930 in a production environment (e.g., use in the digital image processing platform 110 described with respect to FIG. 1). Deploying the machine learning models 930 includes moving the models from a development environment (e.g., the training and validation subsystem 915, where it has been trained, validated, and tested), into a production environment where it can make inferences on real-world data (e.g., input data 950). This step starts with the model being saved after training, including its parameters and configuration such as final architecture and hyperparameters. It is then converted, if necessary, into a format that is suitable for deployment, depending on the deployment environment. For instance, a model trained in a scientific computing environment such as Python might be converted into a Java-friendly format for integration into a larger enterprise application. Deployment can be conducted on various platforms, including on-premises servers, cloud environments like AWS, Azure, Google.

[0212] Once deployed, the model is ready to receive input data 950 and return outputs (e.g., inferences 955). In some instances, the model resides as a component of a larger system or service (e.g., including additional downstream applications provided by the systems or services 935). In some instances, the models 930 and / or the inferences 955 can be used by the downstream systems or services 935 to provide further information. For example, the inferences 955 can be used to aid qualified personnel (e.g., oncologists) to determine whether treatment should be administered to a patient. In some instances, the inferences subsystem 925 can be used to aid qualified personnel to determine a specific type of treatment to administerAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT to a patient based on the inference results. The downstream applications can be configured to generate an output 960. In some instances, the output 960 comprises a report including inferences 955 and information generated by the downstream systems or sen-ices 935.

[0213] In an exemplary inference subsystem 925, the input data 950 includes images generated from one or more biological samples from a patient having been diagnosed with a disease (e.g., cancer). The images may be generated by performing WSI processing (independently or as part of a digital pathology assay) on the one or more biological samples collected from the patient as described with respect to FIG. 1. The one or more biological samples may be a single or multiple tissue samples (e.g., a bladder tissue, lung tissue, tumor section, etc.) obtained from the patient.

[0214] To manage and maintain its performance, a deployed model may be continuously monitored to ensure it performs as expected over time. This involves tracking the model's prediction accuracy, response times, and other operational metrics. Additionally, the model may require retraining or updates based on new data or changing conditions in the environment it is applied in. This can be useful because machine learning models can drift over time due to changes in the underlying data they are making predictions on — a phenomenon known as model drift. Therefore, maintaining a machine learning model in a production environment often involves setting up mechanisms for performance monitoring, regular evaluations against new test data, and potentially periodic updates and retraining of the model to ensure it remains effective and accurate in making predictions.V. EXPERIMENTS AND EXAMPLES

[0215] The examples below are intended to further illustrate certain aspects of the methods and compositions described herein and are not intended to limit the scope of the claims.Example 1: Analyzing Multiplex Fluorescence Assay

[0216] Step 1: Singleplex DAB Optimization

[0217] The first step involves generating multiplex fluorescence imaging data by staining samples to create multiplex tissue microarrays (TMAs). As part of this process, singleplex DAB (3,3’-diaminobenzidine) slides are generated using brightfield microscopy. Whole Slide Images (WSIs) are captured for both unstained and stained slides. The stained slides may include singleplex DAB-optimized stains for various biomarkers, such as cancer markers.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT These markers include CD4, CD45, CD20, Ki-67, CD56, PK-L1, CD28, CD38, PD-1, CD137, CD8. CD68. FoxP3, Perforin, ICOS, CD3, CD16, and PanCK. Table 2 below shows 14 multiplex panels.Table 2

[0218] Step 2: Multiplex Fluorescent Optimization

[0219] In this step, the goal is to optimize multiplex fluorescent imaging by ensuring pixel-accurate alignment of the stained images. Image registration is performed using affine transformations to align the staining positions. A plugin in ImageJ, such as the BDV Warpy extension can be used for this purpose. This plugin combines multiple steps, starting with affine registration using Elastix, followed by a more precise spline registration.

[0220] Fluorochrome selection can be a part of this optimization. Images are captured, e.g., using the Vectra Polaris system, which supports multiple fluorescence channels, including DAPI, Opal 480. Opal 520, Opal 570, Opal 620, Opal 690, and Opal 780. For multiplex panel creation, alignment of multiple autofluorescence channels is performed. The system generates an autofluorescence (AF) channel that is added to the image stack, and various combinations of channels are used for different models. A focus is placed on blue-shifted channels like DAPI,Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT Opal 480, and Opal 520, as well as the AF channel, although some models utilize all available channels.

[0221] Step 3: Image Analysis

[0222] The step of analyzing the images is performed through a series of computational processes. First, the images are unmixed to separate the signals from different fluorophores. The resulting unmixed images are then processed using specialized algorithms (e.g., a trained GAN to perform virtual staining) implemented in software platforms such as QuPath and Visiopharm. The results of the algorithmic processing are verified by trained pathologists to ensure accuracy and reliability. This step can extract meaningful insights and validate the output of the multiplex fluorescence imaging workflow.

[0223] To quality7control the multiplex process, TMAs can be stained and scanned. Consistency across different runs is ensured by comparing the fluorescence signals from all detected cells to a reference standard. This comparison includes analyzing the brightness of the 1000 most fluorescent cells in each channel across multiple TMAs. To generate a diverse range of reference data, a bootstrapping technique is applied to the TMAs, which involves swapping cores between the reference slides to create new virtual slide configurations. The newly created slides are then compared to the reference to determine whether their signals fall within the acceptable range.VI. ADDITIONAL CONSIDERATIONS

[0224] Implementation of the techniques, blocks, steps and means described above can be done in various ways. For example, these techniques, blocks, steps and means can be implemented in hardware, software, or a combination thereof. For a hardware implementation, the processing units can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described above, and / or a combination thereof.

[0225] Also, it is noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operationsAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT can be re-arranged. A process is terminated when its operations are completed but could have additional steps not included in the figure. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0226] Furthermore, embodiments can be implemented by hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting language, and / or microcode, the program code or code segments to perform the necessary tasks can be stored in a machine-readable medium such as a storage medium. A code segment or machine-executable instruction can represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a script, a class, or any combination of instructions, data structures, and / or program statements. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, ticket passing, network transmission, etc.

[0227] For a firmware and / or software implementation, the methodologies can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. Any machine-readable medium tangibly embodying instructions can be used in implementing the methodologies described herein. For example, software codes can be stored in a memory. Memory can be implemented within the processor or external to the processor. As used herein the term “memory” refers to any type of long term, short term, volatile, nonvolatile, or other storage medium and is not to be limited to any particular ty pe of memory7or number of memories, or type of media upon which memory is stored.

[0228] Moreover, as disclosed herein, the term "storage medium", “storage” or “memory ” can represent one or more memories for storing data, including read only memory (ROM), random access memory7(RAM), magnetic RAM, core memory, magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other machine-readable mediums for storing information. The term "machine-readable medium" includes but is not limited to portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage mediums capable of storing that contain or carry instruct! on(s) and / or data.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT

[0229] While the principles of the disclosure have been described above in connection with specific apparatuses and methods, it is to be clearly understood that this description is made only by way of example and not as limitation on the scope of the disclosure.

Claims

1. Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT WHAT IS CLAIMED IS;1. A digital image processing system, comprising:one or more processors; andone or more computer-readable media storing instructions which, when executed by the one or more processors, cause the digital image processing system to perform operations comprising:accessing a digital image depicting a portion of a biological sample obtained from a subject;defining a projecting space and a staining category for an image conversion, wherein the projecting space is a multi-dimensional space with at least one dimension depicting a luminance of an image;projecting, using a neural network model, the digital image to the projecting space to generate a projected image, wherein each voxel of the digital image is projected to a voxel of the projected image with a same location of the digital image, and wherein the neural network model is trained to generate a virtually stained image of an input image based on the staining category; andoutputting the projected image.

2. The digital image processing system of claim 1, wherein the operations further comprise training the neural network model, wherein the training comprises:accessing a set of digital images and their corresponding ground truth images, wherein each digital image depicts a portion of a biological sample obtained from a subject, and wherein each corresponding ground truth image depicts the portion of the biological sample after the biological sample being stained based on the staining category;aligning each pair of the digital image and its corresponding ground truth image using a registration algorithm;extracting a set of ground truth patches from each ground truth image, wherein each ground truth patch covers a portion of the stained biological sample;generating a set of input patches from each corresponding aligned digital image, wherein each input patch is corresponding to a ground truth patch in the set of ground truth patches;Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT accessing a set of hyperparameters for the neural network model, wherein the set of hyperparameters comprises a depth of aneural network used by the neural network model and a number of filters used by each layer of the neural network;accessing a loss function for the training;iteratively training the neural network model for a number of epochs using training data based on the set of hyperparameters and the loss function, wherein the training data comprises the set of input patches from each digital image and the set of ground truth patches from each corresponding aligned ground truth image; andproviding the neural network model.

3. The digital image processing system of claim 2, wherein the training further comprises:randomly selecting at least 1000 locations on each ground truth image; and segmenting the set of ground truth patches from each ground truth image on the at least 1000 locations, wherein each ground truth patch of the set of ground truth patches is of a predetermined size, and wherein the set of input patches is segmented from the at least 1000 locations on the corresponding aligned digital image.

4. The digital image processing system of claim 3. wherein the predetermined size is 50 x 50, 100 x 100, 256 x 256, or 512 x 512.

5. The digital image processing system of claim 3, wherein the training further comprises:performing tissue detection on the ground truth image before selecting the at least 1000 locations, wherein the at least 1000 locations are selected from the detected tissue portion on the ground truth image;examining, manually or using an artifact detection algorithm, whether each ground truth patch of the set of ground truth patches comprises an artifact; andremoving the ground truth patch and its corresponding input patch from the training when the ground truth patch is examined to comprise the artifact.

6. The digital image processing system of claim 5, wherein the artifact is a blur, an out-of-focus (OOF), a noise, a vignetting, or a chromatic aberration.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT 7. The digital image processing system of claim 2, wherein the registration algorithm is an affine registration, a B-spline registration, an elastic registration, a mutual information registration, a Demons algonthm, or a deep learning-based registration.

8. The digital image processing system of claim 2, wherein the set of ground truth patches cover at least a predetermined portion of the biological sample.

9. The digital image processing system of claim 8, wherein the predetermined portion is about 10%.

10. The digital image processing system of claim 2, wherein the neural network is a Generative Adversarial Network (GAN) comprising a generator and a discriminator, wherein the neural network model is provided with the generator.

11. The digital image processing system of claim 10, wherein the generator is a U-net and the discriminator is a convolutional neural network (CNN).

12. The digital image processing system of claim 10, wherein the loss function is determined based on a quadratic generator loss and a quadratic discriminator loss, and wherein parameters of the loss function is included in the set of hyperparameters.

13. The digital image processing system of claim 1, wherein the staining category is selected from a group consisting of: Hematoxylin and Eosin (H&E) staining, Masson’s Trichrome staining, Periodic Acid-Methenamine Silver (PAMS) staining, Periodic Acid-Schiff (PAS) staining, Gram staining, Periodic Acid-Schiff (PAS) staining, Jones’ silver staining, Immunohistochemistry (IHC) staining, a Giemsa staining, a Congo red staining, and a Ziehl-Neelsen staining.

14. The digital image processing system of claim 1, further comprising an image scanning device, wherein the image scanning device is configured to capture images of biological sample using a plurality of scopes, wherein a slide containing the biological sample to be imaged is loaded to the image scanning device, and wherein the operations further comprise:selecting a region of interest in a scannable portion of the slide; calibrating the image scanning device using an autofocus algorithm; andAttorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT capturing the digital image of the biological sample using the calibrated image scanning device.

15. The digital image processing system of claim 1, wherein the biological sample is pre-stained with DAPI (4',6-diamidino-2-phenylindole), Opal 480, Opal 520, Opal 570, Opal 620, Opal 690, or Opal 780.

16. A method, comprising:accessing a set of digital images and their corresponding ground truth images, wherein each digital image depicts a portion of a biological sample obtained from a subject, and wherein each corresponding ground truth image depicts the portion of the biological sample after the biological sample being stained based on a staining category;aligning each pair of the digital image and its corresponding ground truth image using a registration algorithm;extracting a set of ground truth patches from each ground truth image, wherein each ground truth patch covers a portion of the stained biological sample;generating a set of input patches from each corresponding aligned digital image, wherein each input patch is corresponding to a ground truth patch in the set of ground truth patches;accessing a set of hyperparameters for a neural network model, wherein the set of hyperparameters comprises a depth of a neural network used by the neural network model and a number of fdters used by each layer of the neural network;accessing a loss function for training the neural network model; iteratively training the neural network model for a number of epochs using training data based on the set of hyperparameters and the loss function, wherein the training data comprises the set of input patches from each digital image and the set of ground truth patches from each corresponding aligned ground truth image; andproviding the neural network model.

17. The method of claim 16, further comprising:randomly selecting at least 1000 locations on each ground truth image; and segmenting the set of ground truth patches from each ground truth image on the at least 1000 locations, wherein each ground truth patch of the set of ground truth patches is of a predetermined size, and wherein the set of input patches is segmented from the at least 1000 locations on the corresponding aligned digital image.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT 18. The method of claim 17, wherein the predetermined size is 50 x 50, 100 x lOO, 256 x 256, or 512 x 512.

19. The method of claim 17, further comprising:performing tissue detection on the ground truth image before selecting the at least 1000 locations, wherein the at least 1000 locations are selected from the detected tissue portion on the ground truth image;examining, manually or using an artifact detection algorithm, whether each ground truth patch of the set of ground truth patches comprises an artifact; andremoving the ground truth patch and its corresponding input patch from the training when the ground truth patch is examined to comprise the artifact.

20. The method of claim 19, wherein the artifact is a blur, an out-of-focus (OOF), a noise, a vignetting, or a chromatic aberration.

21. The method of claim 16, wherein the registration algorithm is an affine registration, a B-spline registration, an elastic registration, a mutual information registration, a Demons algorithm, or a deep learning-based registration.

22. The method of claim 1 , wherein the set of ground truth patches cover at least a predetermined portion of the biological sample.

23. The method of claim 22, wherein the predetermined portion is about 10%.

24. The method of claim 16, wherein the neural network is a Generative Adversarial Network (GAN) comprising a generator and a discriminator, wherein the neural network model is provided with the generator.

25. The method of claim 24, wherein the generator is a U-net and the discriminator is a convolutional neural network (CNN).

26. The method of claim 24, wherein the loss function is determined based on a quadratic generator loss and a quadratic discriminator loss, and wherein parameters of the loss function is included in the set of hyperparameters.Attorney Docket No.: 057618-1544088Client Reference No. : LC-2024-05-WO-PCT 27. A computer implemented method comprising the operations in any one of claims 1-15.

28. A computing system, comprising:one or more processors; andone or more computer-readable media storing instructions which, when executed by the one or more processors, cause the computing system to perform the operations in any one of claims 16-26.

29. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform the operations in any one of claims 1-26.

Images