Spatial image registration to ground truth

Abstract

Systems and methods for registering a microscope image of a tissue sample to a ground truth coordinate system are described herein. The method may include: (i) obtaining a plurality of fields of view (FOVs) of a microscope image of a tissue sample; (ii) stitching the plurality of FOVs together to map the plurality of FOVs to global image locations; (iii) determining locations of fiducials in the stitched plurality of FOVs; (iv) comparing the determined locations of the fiducials to known locations for the fiducials in the stitched plurality of FOVs to compute a transform; and (v) applying the transform to the stitched plurality of FOVs to register the microscope image to a ground troth coordinate system. Additional systems and methods for registering data associated with a spatial transcript image of a substrate to a ground truth coordinate system may similarly be described herein.

Description

33080 / IP-2900 / PCSPATIAL IMAGE REGISTRATION TO GROUND TRUTH CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 735,535, filed December 18, 2024, entitled “Spatial Image Registration to Ground Truth,” the entire disclosure of which is hereby expressly incorporated by reference herein.FIELD OF THE INVENTION

[0002] The present disclosure generally relates to techniques for registering microscope images and spatial transcriptomics data for tissue samples and / or substrates to a ground truth, such as applying transforms based on detected locations of fiducials and performing transforms (e.g., shifts, rotations, scaling, skew adjustment, etc.) based on overlapping fiducials and / or nucleotide data.BACKGROUND

[0003] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

[0004] In spatial transcriptomics, transcripts are obtained from tissue samples placed on a substrate after tissue permeabilization and subsequent capture using target capture sites on the surface. The captured transcripts are then assigned locations to generate a spatial transcriptomics map based on a ground truth coordinate system. Prior to tissue permeabilization, the tissue can be stained and imaged using a microscope. The microscope image may then depict cells, nuclei, and / or other features of the tissue sample. However, the microscope image may not be aligned with the spatial transcriptomics map such that the transcripts cannot be accurately overlaid on the microscope image depicting the tissue.SUMMARY

[0005] To register the spatial transcript image to the ground truth coordinate system, an image alignment system uses fiducials present in the plurality of tiles that make up the overall spatial33080 / IP-2900 / PC transcript image captured by the sequencer. In some implementations, the image alignment system determines corresponding coordinates for the detected fiducials in the ground truth coordinate system. The image alignment system then detects shifts in the images (e.g., due to various imperfections) using spatial barcodes (SBCs) representative of nucleotide pairs present in pairs of the tiles and shifts the fiducial coordinates accordingly. By detecting and shifting the fiducials and / or SBCs within the tiles, the image alignment system can correct for various imperfections, such as rotation, translation, skew, distortion, etc. when stitching the tiles together. Then the image alignment system can register the image tiles to a common ground truth coordinate system using a transform calculated based on the shifts for the fiducials.

[0006] The microscope may also be part of the image alignment system and may be used to capture a microscope image which the image alignment system registers to the ground truth coordinate system. In some such implementations, the microscope may initially be retrieved from the microscope. In some such implementations, the image alignment system may initially obtain a plurality of fields of view (FOVs) of a microscope image of a tissue sample. The image stitching system may then stitch the FOVs (e.g. using common features or fiducials in pairs of the FOVs that make up the overall image) and may subsequently determine shifts in the FOVs. The image stitching system may then apply a transform to the stitched FOVs based on the determined locations of fiducials to register the image to the ground truth coordinate system.

[0007] Depending on the implementation, the image alignment system may use fiducial-only alignment techniques (e.g., an initial alignment based on detected fiducial coordinates and a shift using overlapping fiducials amongst tiles), hybrid SBC-fiducial alignment techniques (e.g., an initial alignment based on detected fiducial coordinates and a shift using the nucleotide pairs), microscope-based fiducial alignment techniques (e.g., using the stitched FOVs and shifting the fiducials in the FOVs), and / or a combination of such. For example, the image alignment system may perform the alignment for human review to determine which aligned image is preferable.

[0008] In this manner, the image alignment system may register the microscope image to the ground truth coordinate system while mitigating or removing the effects of traditional complications. For example, the image alignment system may align the microscope image without performing stitching at the image level to obtain global fiducial locations, which would otherwise potentially lead to the erroneous stitching between tiles or image manipulation at33080 / IP-2900 / PC overlap regions (e.g., blending) that may affect the fiducial integrity and detection accuracy. Similarly, the image alignment system may register the sequencing tile images to the ground truth coordinate system while mitigating the effects of defects in the substrate (e.g., by incorporating fiducial locations) without requiring accurate base-calling at a single cycle rather than at each of the cycles as required for SBC. Moreover, the microscope may minimize stitching errors by incorporating fiducial locations, such as using the shape and location of the fiducial to fine tune feature detection algorithms by introducing additional image processing and filtering.

[0009] Then the image alignment system may overlay the spatial transcript image registered to the ground truth coordinate system over the microscope image registered to the ground truth coordinate system to generate an aligned image which is then presented for display to a user. The aligned image can be used to identify the transcript density in a particular cell, elevated expression of a certain gene in a cell, cell-to-cell interactions, etc.

[0010] In some aspects, the techniques described herein relate to a method for registering a microscope image of a tissue sample to a ground truth coordinate system, the method including: obtaining, by one or more processors, a plurality of fields of view (FOVs) of a microscope image of a tissue sample; stitching, by the one or more processors, the plurality of FOVs together to map the plurality of FOVs to global image locations; determining, by the one or more processors, locations of fiducials in the stitched plurality of FOVs; comparing, by the one or more processors, the determined locations of the fiducials to known locations for the fiducials in the stitched plurality of FOVs to compute a transform; and applying, by the one or more processors, the transform to the stitched plurality of FOVs to register the microscope image to a ground truth coordinate system.

[0011] In some aspects, the techniques described herein relate to a method, wherein the stitching is based on common image features of the plurality of FOVs.

[0012] In some aspects, the techniques described herein relate to a method, further including: shifting, by the one or more processors, the stitched plurality of FOVs based on a difference between the known locations for the fiducials and the determined locations of the fiducials.33080 / IP-2900 / PC

[0013] In some aspects, the techniques described herein relate to a method, further including: mapping, by the one or more processors, the shifted plurality of FOVs to the global image locations.

[0014] In some aspects, the techniques described herein relate to a method, wherein the plurality of FOVs are organized in a series of rows and columns, a respective FOV of the plurality of FOVs at least partially overlaps with an adjacent FOV, and the determined locations of the fiducials include an overlapping fiducial location present in the respective FOV and the adjacent FOV.

[0015] In some aspects, the techniques described herein relate to a method, further including: registering, by the one or more processors, data associated with a plurality of sequencing tile images related to the microscope image to the ground truth coordinate system.

[0016] In some aspects, the techniques described herein relate to a method, wherein the ground truth coordinate system is a universal ground truth coordinate system to which a plurality of devices map data including at least a microscope capturing the microscope image and a sequencer analyzing a plurality of sequencing tile images related to the microscope image.

[0017] In some aspects, the techniques described herein relate to a method, wherein the ground truth coordinate system is a coordinate system common to a sequencing tile global image coordinate system used by a sequencer analyzing a plurality of sequencing tile images related to the microscope image.

[0018] In some aspects, the techniques described herein relate to a method, wherein the fiducials are common fiducials present in the plurality of sequencing tile images and in the microscope image.

[0019] In some aspects, the techniques described herein relate to a method for registering data associated with a spatial transcript image of a substrate to a ground truth coordinate system, the method including: obtaining, by one or more processors, data associated with a plurality of sequencing tile images of a spatial transcript image of a substrate; determining, by the one or more processors, locations of fiducials in the plurality of sequencing tile images; shifting, by the one or more processors, respective locations of respective fiducials of respective sequencing tile images of the plurality of sequencing tile images based on overlap data for the respective33080 / IP-2900 / PC sequencing tile images; comparing, by the one or more processors, the shifted respective locations of the respective fiducials to known locations for the respective fiducials to compute a transform; and applying, by the one or more processors, the transform to the data associated with the plurality of sequencing tile images to register the data associated with the plurality of sequencing tile images to a ground truth coordinate system.

[0020] In some aspects, the techniques described herein relate to a method, wherein the overlap data includes matching nucleotide data associated with the substrate in adjacent sequencing tile images.

[0021] In some aspects, the techniques described herein relate to a method, wherein shifting the locations of respective fiducials includes: receiving, by the one or more processors, spatial barcodes (SBCs) representative of the nucleotide data in the adjacent sequencing tile images; and shifting, by the one or more processors, the locations of the respective fiducials by shifting locations of the adjacent sequencing tile images to align matching SBCs in the adjacent sequencing tile images.

[0022] In some aspects, the techniques described herein relate to a method, wherein applying the transform includes; applying, by the one or more processors, the transform to at least one of (i) one or more locations associated with the SBCs or (ii) the locations of the fiducials.

[0023] In some aspects, the techniques described herein relate to a method, wherein applying the transform includes: applying, by the one or more processors, the transform to the plurality of sequencing tile images.

[0024] In some aspects, the techniques described herein relate to a method, wherein the overlap data includes the determined locations of the fiducials.

[0025] In some aspects, the techniques described herein relate to a method, wherein the plurality of sequencing tile images are organized in a series of swaths, a respective sequencing tile image at least partially overlaps with an adjacent sequencing tile image, and the locations of the fiducials includes an overlapping fiducial location present in the respective sequencing tile image and the adjacent sequencing tile image.33080 / IP-2900 / PC

[0026] In some aspects, the techniques described herein relate to a method, further including: registering, by the one or more processors, data associated with a microscope image related to the plurality of sequencing tile images to the ground truth coordinate system.

[0027] In some aspects, the techniques described herein relate to a method, wherein the ground truth coordinate system is a universal ground truth coordinate system to which a plurality of devices map data including at least a sequencer analyzing the plurality of sequencing tile images and a microscope capturing a microscope image related to the plurality of sequencing tile images.

[0028] In some aspects, the techniques described herein relate to a method, wherein the ground truth coordinate system is a coordinate system common to a microscope global image coordinate system used by a microscope analyzing a microscope image related to the plurality of sequencing tile images.

[0029] In some aspects, the techniques described herein relate to a method, wherein the fiducials are common fiducials present in the plurality of sequencing tile images and in the microscope image.

[0030] In some aspects, the techniques described herein relate to a method, wherein the fiducials include fiducials in an active area of the sequencing tile images.

[0031] In some aspects, the techniques described herein relate to a method, wherein the fiducials include fiducials outside the active area of the sequencing tile images.

[0032] Advantages will become more apparent to those of ordinary skill in the art from the following description of the preferred embodiments which have been shown and described by way of illustration. As will be realized, the present embodiments may be capable of other and different embodiments, and their details are capable of modification in various respects. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The figures described below depict various aspects of the system and methods disclosed herein. It should be understood that each figure depicts an embodiment of a particular33080 / IP-2900 / PC aspect of the disclosed system and methods, and that each of the figures is intended to accord with a possible embodiment thereof.

[0034] There are shown in the drawings arrangements which are presently discussed, it being understood, however, that the present embodiments are not limited to the precise arrangements and instrumentalities shown, wherein:

[0035] Fig. 1 depicts a block diagram of an example computing environment for performing ground truth coordinate system registration techniques to register microscope images of tissue samples and / or spatial transcriptomics maps to a ground truth coordinate system, according to some aspects.

[0036] Fig. 2 depicts an example microscope image of a tissue sample with fiducials side-by- side with an example spatial transcriptomics map for the sample tissue with the same fiducials, demonstrating the alignment of physical structures with genomic data, according to some aspects.

[0037] Fig. 3A depicts an example field of view of a microscope image registered to a global coordinate system, according to some aspects.

[0038] Fig. 3B depicts another example field of view of a microscope image registered to a global coordinate system, according to some aspects.

[0039] Fig. 4A depicts a diagram of an example method for performing sequencing tile stitching without calculating fiducial and / or SBC shifts, according to some aspects.

[0040] Fig. 4B depicts various exemplary stitching artifacts due to errors in the sequencing tile stitching process and / or microscope image stitching process, according to some aspects.

[0041] Fig. 4C depicts a diagram of an example method for performing sequencing tile stitching by calculating fiducial and / or SBC shifts, according to some aspects.

[0042] Fig. 4D depicts stitched sequencing tiles according to the example method depicted in Fig. 4C which reduces artifacts, according to some aspects.

[0043] Fig. 5A depicts a block diagram of a system for registering a spatial transcriptomics image to a ground truth coordinate system, according to some aspects.33080 / IP-2900 / PC

[0044] Fig. 5B depicts a block diagram of a system for registering a microscope image to a ground truth coordinate system, according to some aspects.

[0045] Fig. 6 depicts a flow diagram of an example method for registering a microscope image of a tissue sample to a ground truth coordinate system, which may be implemented by a server device, according to some aspects.

[0046] Fig. 7 depicts a flow diagram of an example method for registering a spatial transcriptomics image of a substrate to a ground truth coordinate system, which may be implemented by a server device, according to some aspects.

[0047] The Figures depict preferred implementations for purposes of illustration only. Alternative implementations of the systems and methods illustrated herein may be employed without departing from the principles of the invention described herein.DETAILED DESCRIPTION

[0048] Although the following text discloses a detailed description of implementations of methods, apparatuses and / or articles of manufacture, it should be understood that the legal scope of the property right is defined by the words of the claims set forth at the end of this document. Accordingly, the following detailed description is to be construed as examples only and does not describe every possible implementation, as describing every possible implementation would be impractical, if not impossible. Numerous alternative implementations could be implemented, using either current technology or technology developed after the filing date of this patent. It is envisioned that such alternative implementations would still fall within the scope of the claims.EXEMPLARY COMPUTING ENVIRONMENT

[0049] Fig. 1 is a block diagram of an example computing environment 100 for performing the error correction and fiducial or spatial barcode (SBC) stitching techniques described herein. This computing environment 100 is designed to detect fiducial and / or SBC locations within a tissue sample (e.g., tissue sample 102a, 102b), calculate shifts in the locations, stitch corresponding image data based on the shifts, and register the stitched data to a ground truth coordinate system. The example computing environment 100 may include a sequencing device 110, a server device 120, a client device 130, a microscope 140, a tissue sample 102a, 102b, and a network 150.33080 / IP-2900 / PC

[0050] The tissue sample 102a may be sectioned and placed in proximity to a substrate (e.g., a flow cell) having fiducials 104a, 106a or markers at known, physical locations. For example, the fiducials 104a, 106a may be placed in a specific pattern on the flow cell (e.g., a grid) where the spacing between the fiducials 104a, 106a is known and can be used to locate each fiducial. In other implementations, each fiducial 104a, 106a has associated physical coordinates (e.g., (5 pm, 10 pm)). The tissue sample 102a may be stained, for example with hematoxylin and eosin (H&E), and imaged via the microscope 140. In some implementations, the microscope 140 images different subsections or fields of view (FOVs) of the tissue sample 102a. Then each of the FOVs can be combined to generate a microscope image of the tissue sample 102a.

[0051] The microscope 140 may transmit the microscope image of the tissue sample 102a with fiducials 104a, 106a or FOVs of subsections of the tissue sample 102a with fiducials 104a, 106a to the server device 120 via the network 150. The microscope image may depict cells, nuclei, and / or other cellular and subcellular structures within the tissue sample.

[0052] Additionally, the tissue sample 102b may be sectioned and placed in proximity to a substrate (e.g., a flow cell) with a very large set of (e.g. up to hundreds of millions) of barcoded locations, each containing a sequence of capture oligonucleotides constituting a spatial barcode unique to that location. As mentioned above, the flow cell may also include fiducials 104b, 106b or markers at known physical locations. Prior to permeabilization, the sequencing device 110 may decode the spatial barcodes for creating a map of barcode sequences and coordinate locations. During permeabilization, the tissue sample 102b may release mRNA which binds to capture oligonucleotides from a proximal location on the tissue sample 102b. A reverse transcription reaction may occur while the tissue is still in place, generating a library that incorporates the spatial barcodes and preserves spatial information. For example, each library may include a combination of mRNA, a barcode, an index, and / or other known sequences. The barcoded sequences are mapped back to a specific location within the flow cell.

[0053] In further implementations, capture oligonucleotides and spatially barcoded oligonucleotides having an SBC may be separated but proximal to one another on the substrate. In such implementations, a mechanism may be used after molecule capture, where the captured molecules may bridge to a nearby spatially barcoded oligonucleotide, for example as described in U.S. Provisional Patent Application No. 63 / 615,558, which is incorporated by reference herein33080 / IP-2900 / PC in its entirety. A reverse transcription reaction may occur resulting in a library having both mRNA and an SBC, which the sequencing device 110 (and / or another sequencing device (not shown)) may sequence.

[0054] The tissue sample 102a and fiducials 104a, 106a imaged by the microscope 140 may be the same tissue sample 102b and fiducials 104b, 106b imaged by the sequencing device 110.

[0055] Then the sequencing device 110 may image the flow cell having barcoded nucleotide fragments from the tissue sample 102b and having fiducials 104b. 106b. The sequencing device 110 may include a computing device, image sensors, and a sequencing application 112 for sequencing a genomic sample or other nucleic-acid polymer. In some versions, by executing the sequencing application 112 using a processor, the sequencing device 110 may analyze nucleotide fragments or oligonucleotides extracted from genomic samples to generate nucleotide reads or other data utilizing computer implemented methods and systems either directly or indirectly on the sequencing device 110.

[0056] More particularly, the sequencing device 110 may receive the flow cell with barcoded nucleotide fragments extracted from the tissue sample 102b and fiducials 104b, 106b, and the sequencing device may determine the nucleobase sequence of such extracted nucleotide fragments. The sequencing device 110 may be the sequencing system described in U.S. Patent Application No. 18 / 340,795, titled “Split- Read Alignment by Intelligently Identifying and Scoring Candidate Split Groups” and filed on July 23, 2023, which is hereby incorporated by reference in its entirety.

[0057] In some versions, the sequencing device 110 may utilize sequencing-by-synthesis (SBS) to sequence nucleotide fragments into nucleotide reads and determine nucleobase calls for the nucleotide reads. By executing the sequencing application 112, the sequencing device 110 may further store the nucleobase calls as part of base-call data that is formatted as a binary base call (BCL) file and send the BCL file to the server device 120. The sequencing device 110 may communicate the BCL file and / or other data to the server device 120 via one or more network(s) 150 or directly (e.g., bypassing the one or more network(s) 150).

[0058] The sequencing device 110 may generate a map of stitched nucleotide cluster locations (e.g., SBCs as discussed herein) within a flow cell in swaths which are regions of the flow cell. The swaths may overlap with each other. For example, adjacent swaths may have a 1% overlap,33080 / IP-2900 / PC a 3% overlap, a 5% overlap, etc. In some implementations, the swaths may be disposed such that the horizontal overlap is configurable by a user and / or designer.

[0059] As used herein, the term “cluster of oligonucleotides” (or simply “cluster(s)” or “DNA cluster(s)”) refers to a localized group or collection of DNA or RNA molecules on a nucleotide- sample slide, such as a flow cell, or other solid surface. In particular, a cluster includes tens, hundreds, thousands, or more copies of a cloned DNA or RNA segment, or the same DNA or RNA segment. For example, in one or more embodiments, a cluster includes a grouping of oligonucleotides immobilized in a section of a flow cell or other nucleotide- sample slide. In some embodiments, clusters are evenly spaced or organized in a systematic structure within a patterned flow cell. By contrast, in some cases, clusters are randomly organized within a nonpatterned flow cell. A cluster of oligonucleotides can be imaged utilizing one or more light signals. For instance, an oligonucleotide-cluster image may be captured by a camera during a sequencing cycle of light emitted by irradiated fluorescent tags incorporated into oligonucleotides from one or more clusters on a flow cell.

[0060] The server device 120 receives the FOVs of the microscope image of the tissue sample 102a and receives the swaths from the sequencing device 110. The server device 120 may then divide the swaths into sequencing tiles. In some implementations, each tile is generated by selecting one sequencing cycle and using the selected cycle to define the cluster locations and the local coordinate system for the tile. Because the real time analysis (RTA) may select a different cycle for each tile, some of the tiles may be misaligned in relation to the other tiles. In such instances, the tile and swath stitching system may select a desired cycle (e.g., associated with a ground truth coordinate system) to use to align each of the local coordinate systems from the different cycles. Then the tile and swath stitching system may transform the coordinate systems for each of the tiles to the desired cycle, so that the cluster locations in each of the tiles are defined with respect to the same coordinate system. In some implementations, the clusters are associated with a plurality of cycles based on SBC base size. For example, an SBC with 30 bases may have 30 cycles. As such, each cycle of the plurality of cycles may be analyzed separately for correcting errors in the sequencing tiles. In further implementations, only a first cycle is analyzed for correcting errors in the sequencing tiles. In still further implementations, a combination of cycles (and / or channels of the sequencing device 110 used to perform sequencing) may be used to improve the accuracy of fiducial detection.33080 / IP-2900 / PC

[0061] In some implementations, a particular cycle may be designated as a reference cycle, and corresponding tiles in another cycle may be registered to the reference cycle. For example, if a cycle 0 is the reference cycle, then a particular tile X in another cycle (e.g., cycle 1) may be aligned using the cycle 1 image for tile X and the cycle 0 image for tile X. In some such implementations, a transformation (e.g., shift, rotation, skew, etc.) may be applied (e.g., based on camera movement, extent of repeatability, etc.) to the cycle 1 image to ensure that the cycle 1 image is in the same coordinate system as the cycle 0 image.

[0062] The server device 120 may include a memory and one or more processors (CPUs). The memory can be a non-transitory memory and can include one or several suitable memory modules, such as random access memory (RAM), read-only memory (ROM), flash memory, other types of persistent memory, etc.

[0063] The memory may store an operating system (OS), which can be any type of suitable mobile or general-purpose operating system. The memory also stores a ground truth calculation engine 122 that receives the FOVs of the microscope image and the sequencing tiles and registers them to a ground truth coordinate system based on the locations of detected fiducials and / or SBCs in the microscope image and / or the sequencing tiles as well as calculated shifts in location for the detected fiducials and / or SBCs. The ground truth calculation engine 122 may stitch the FOVs to together to generate the microscope image and may stitch the sequencing tiles to generate the spatial transcriptomics map. Similarly, in further implementations, the ground truth calculation engine 122 may stitch shifted locations (e.g., determined using the calculated shifts in locations for the detected fiducials and / or SBCs) for the fiducials and / or SBCs to generate stitched locations (e.g., an output list of the sequences / fiducials and / or locations). In further implementations, the ground truth calculation engine 122 also may align portions of the spatial transcript map with corresponding portions of the microscope image within the ground truth coordinate system.

[0064] To register the microscope image and the spatial transcriptomics map to a common (e.g., ground truth) coordinate system, the server device 120 first detects fiducials within both the microscope image and the spatial transcriptomics map. Fiducials are markers or features at known physical locations to be used as reference points for stitching and / or alignment. The ground truth calculation engine 122 further calculates shifts for the locations of the detected33080 / IP-2900 / PC fiducials based on unique fiducials and / or SBCs present in overlapping areas of adjacent sequencing tiles. The ground truth calculation engine 122 may further transform the locations of the detected and / or shifted fiducials or SBCs to their known physical locations, effectively correcting for any discrepancies due to rotation, translation, scaling, skew, or distortion. Additionally, the ground truth calculation engine 122 may align the spatial transcriptomics map to the microscope image by identifying similar fiducials present in both images, further refining the alignment process.

[0065] To transform the locations of the shifted fiducials, the ground truth calculation engine 122 generates affine transformation matrices for the microscope image and the spatial transcriptomics map. To calculate a transform and generate a first affine transformation matrix for the microscope image, the ground truth calculation engine 122 establishes correspondences between a first set of detected fiducial locations in the microscope image and known physical locations for such. Using the correspondences, the ground truth calculation engine 122 solves for the parameters of the affine transformation, which includes translation, rotation, scaling, and shearing factors. The resulting first affine transformation matrix encapsulates the parameters, enabling the conversion of locations of fiducials within the microscope image to corresponding known physical locations. In some implementations, the affine transformation matrices are calculated using unconstrained least squares based on minimizing the sum of the squares of the correspondences between the known locations and the shifted locations. In other implementations, the affine transformation matrices are calculated using constrained least squares (e.g., if the transformation is within a known range). In still other implementations, the affine transformation matrices are calculated using weighted least squares, where the weights are proportional to confidence levels or correlation peaks reflecting the accuracy of a corresponding fiducial detection.

[0066] To calculate a transform and generate a second affine transformation matrix for the spatial transcriptomics map, the ground truth calculation engine 122 first establishes correspondences between a second set of shifted fiducial locations in the spatial transcriptomics map and corresponding known physical locations. Using the correspondences, the ground truth calculation engine 122 solves for the parameters of the affine transformation, which includes translation, rotation, scaling, and shearing factors. The resulting second affine transformation33080 / IP-2900 / PC matrix encapsulates the parameters, enabling the conversion of locations of fiducials within the spatial transcriptomics map to the corresponding known physical locations.

[0067] Then the ground truth calculation engine 122 applies the first affine transformation matrix to the microscope image. Applying the first affine transformation matrix to the microscope image involves manipulating the microscope image according to the parameters defined in the first affine transformation matrix. In some implementations, the ground truth calculation engine 122 adjusts the position, orientation, and / or scale of the microscope image to align it with the common coordinate system. For example, the microscope image may span from locations (0, 0) to (1000, 1000) whereas the common coordinate system may include physical locations (0 pm, 0 pm) to (500 pm, 450 pm). The transformation ensures that the fiducials in the microscope image are positioned at the correct corresponding physical locations, facilitating accurate alignment with the spatial transcriptomics map.

[0068] The ground truth calculation engine 122 also applies the second affine transformation matrix to the spatial transcriptomics map. Applying the second affine transformation matrix to the spatial transcriptomics map involves manipulating the spatial transcriptomics map according to the parameters defined in the second affine transformation matrix. This process adjusts the position, orientation, and scale of the spatial transcriptomics map to align it with the common coordinate system. For example, the spatial transcriptomics map may span from locations (0. 0) to (800, 900) whereas the common coordinate system may include physical locations (0 pm, 0 pm) to (500 pm, 450 pm). The transformation ensures that the fiducials and / or SBCs in the spatial transcriptomics map are positioned at their correct physical locations, facilitating accurate alignment with the microscope image. In some implementations, the resolution of each image may be different. For example, the spatial transcriptomics map may have a pixel resolution of x pixels per micrometer, and the microscope image may have a pixel resolution of y pixels per micrometer. In further implementations, the ground truth may have another different resolution (e.g., 1 pixel per micrometer). In some such implementations, a computing device may scale the images and / or fine-tune the affine transform based on the differences.

[0069] The server device 120 may then transmit the spatial transcript map and / or microscope image registered within the ground truth coordinate system to the client device 130. The client device 130 may include a memory and one or more processors (CPUs). The memory can be a33080 / IP-2900 / PC non-transitory memory and can include one or several suitable memory modules, such as random access memory (RAM), read-only memory (ROM), flash memory, other types of persistent memory, etc.

[0070] The memory may store an operating system (OS), which can be any type of suitable mobile or general-purpose operating system. The memory also stores a spatial transcriptomics application 132 which may present the spatial transcriptomics map registered to the ground truth coordinate system, the spatial transcriptomics map overlaid on the microscope image where both the spatial transcriptomics map and the microscope image are registered to the ground truth coordinate system, the microscope image registered to the ground truth coordinate system, and / or any other such combination thereof via a user interface. For example, the spatial transcriptomics application 132 may present a heatmap indicating the number of transcripts at various locations overlaid on the microscope image of the tissue. Additionally or alternatively, the spatial transcriptomics application 132 may indicate the cell type (e.g., by color coding the cell types) of cells within the microscope image along with the spatial transcripts heatmap.

[0071] In other implementations, the spatial transcriptomics application 132 may present a user control, such as a slider bar which allows a user to select a transparency level indicating the extent to which they can see the spatial transcripts heatmap overlaid on the microscope image. The spatial transcripts heatmap may cover various portions of the microscope image and by adjusting the transparency, the user can see the entire microscope image when the transparency level is set to a maximum, can see the spatial transcripts heatmap entirely when the transparency level is set to a minimum, and can see both spatial transcripts heatmap features and the microscope image features they cover when the transparency level is in between the minimum and maximum. In yet other implementations, the spatial transcriptomics application 132 may present estimated cell contours of the tissue from the microscope image on the spatial transcriptomics heatmap. For example, the spatial transcriptomics application 132 may estimate cell borders by identifying cell nuclei and expanding the contours of the cell nuclei by a threshold distance to estimate the cell borders. In another example, the spatial transcriptomics application 132 may identify the cell borders directly from the microscope image.

[0072] In this manner, users can identify which transcript corresponds to which cell, determine cell types based on corresponding transcripts, assess transcript density within cells, identify33080 / IP-2900 / PC elevated gene expression, and analyze cell-to-cell interactions. The client device 130 may also perform this analysis and provide an indication of the transcript density within cells, elevated gene expression in particular cells, cell-to-cell interactions, etc. for display to the user.

[0073] Fig. 2 depicts an example microscope image 200 of a tissue sample 202 with fiducials 204 side-by-side with an example spatial transcriptomics map 250 for the tissue sample 252 with the same fiducials 254. The microscope image 200 is a 2D image of a 3D tissue structure from a top-down perspective. This snapshot flattens the 3D structure of the tissue into a 2D plane for visualization and analysis. The depth and complexity of the tissue's structure may be inferred from the arrangement and appearance of cells and other components within the plane of the microscope image 200.

[0074] The spatial transcriptomics map 250 may include a heat map of the transcripts - the clusters of RNA molecules produced from DNA within the cells - at various locations within the tissue sample 252. The spatial transcriptomics map 250 is generated by analyzing the tissue sample 252 for the presence of RNA sequences and then displaying the density of these sequences across the tissue sample 252. The spatial transcriptomics map 250 may use colors to indicate the concentration of transcripts, with warmer colors typically representing higher concentrations.

[0075] In some implementations, the microscope image 200 and spatial transcriptomics map 250 are received by the server device 120 from the microscope 140 and the sequencing device 110, respectively. The server device 120 may also obtain indications of the known, physical locations of the fiducials 204, 254. For example, the server device 120 may receive an indication of the spacing between the fiducials (e.g., 5 pm horizontally and vertically). In other implementations, the server device 120 may receive physical coordinate locations of each fiducial e.g.. 10 pm, 35 pm). In other implementations, the server device 120 may receive FOVs of the microscope image 200 and / or sequencing tiles for the spatial transcriptomics map 250.

[0076] Fig. 3A depicts an example FOV 302 of the microscope image 200 registered to a global coordinate system 300. The global coordinate system 300 is depicted as a grid with the fiducials 204 from the microscope image 200 at their known, physical locations.33080 / IP-2900 / PC

[0077] In some implementations, the ground truth calculation engine 122 registers the FOV 302 to the global coordinate system 300 by transforming the locations in the FOV 302 into physical locations and stitching the FOV 302 to adjacent FOVs 302 within the microscope image using detected fiducials 204 which overlap in adjacent FOVs 302. As mentioned above, the ground truth calculation engine 122 may transform the locations in the FOV 302 into physical locations by calculating a transform and generating an affine transformation matrix for the FOV 302 using correspondences between the shifted fiducial locations in the FOV 302 and corresponding known physical locations. Using the correspondences, the ground truth calculation engine 122 solves for the parameters of the affine transformation, which includes translation, rotation, scaling, and shearing factors. Then the ground truth calculation engine 122 applies the affine transformation matrix to the FOV 302 to adjust the position, orientation, and / or scale of the FOV 302. By correlating the pixel locations of these fiducials to their known physical locations, the ground truth calculation engine 122 can accurately map the FOV 302 from its pixel-based coordinate system to a physical coordinate system that aligns with the global coordinate system 300.

[0078] In some implementations, each fiducial has a known, global location within the microscope image 200. In this manner, the FOV 302 can be mapped to a particular cell 304 (e.g., a row and column) within a grid in the global coordinate system 300 based on the known, global locations of the fiducials within the FOV 302. While the fiducials are shown in Fig. 3A as having a uniform spacing and a uniform geometry (dots), this is merely one example for ease of illustration only.

[0079] The fiducials may have different spacings from each other, such that a fiducial or group of fiducials may be uniquely identified within the flow cell based on the spacing from adjacent fiducials. In this manner, the FOV 302 can be mapped to a particular cell 304 (e.g., a row and column) within a grid in the global coordinate system 300 based on the spacing of the fiducials in the FOV 302. Still further, the fiducials may have different geometries from each other. Some fiducials may have a dot geometry while others have a cross, line, or bullseye ring geometry. A fiducial or group of fiducials may be uniquely identified within the flow cell based on the geometries of the fiducials. In this manner, the FOV 302 can be mapped to a particular cell 304 (e.g., a row and column) within a grid in the global coordinate system 300 based on the33080 / IP-2900 / PC geometries of the fiducials in the FOV 302. In other implementations, an FOV 302 can be mapped to a particular cell 304 (e.g., a row and column) within a grid in the global coordinate system 300 based on any suitable combination of the spacing and geometries of the fiducials in the FOV 302.

[0080] For example, as shown in Fig. 3B, some fiducials include dot geometries while other fiducials include cross geometries. The ground truth calculation engine 122 may determine the global locations of the fiducials within an FOV based on the arrangement of geometries of the fiducials. For example, an FOV with three dot fiducials above three cross fiducials may be in the lower right comer of the global coordinate system.

[0081] In other implementations, the fiducials have known local locations within the FOV 302. In yet other implementations, the ground truth calculation engine 122 obtains a known spacing between the fiducials which can be used to derive local locations of the fiducials within the FOV 302. The FOV 302 may also have an identifier such as an FOV ID which can be used to obtain information regarding the spatial relationship of the FOV 302 within the microscope image 200. For example, the FOV ID may indicate whether the FOV 302 is situated in the top left cell of the global coordinate system 300, directly below the top left cell, directly to the right of the top left cell, or in another specific location.

[0082] The ground truth calculation engine 122 may then stitch the FOV 302 to adjacent FOVs to the cell 304 where the FOV 302 is located. As mentioned above, the FOV 302 includes at least some overlap with adjacent FOVs. As such, the ground truth calculation engine 122 may identify common features within the adjacent FOVs and assign the common features the same or a similar physical location in the global coordinate system 300. For example, the ground truth calculation engine 122 may identify the same fiducials on the right side of one FOV and on the left side of an adjacent FOV. The ground truth calculation engine 122 may assign the fiducials the same physical location in the global coordinate system 300. More specifically, if the FOV 302 includes fiducials having known, local locations, and a first fiducial has local location (2 pm, 50 pm) which matches with or is similar to a second fiducial in an adjacent FOV having global location (300 pm, 320 pm) after stitching, the FOV 302 may be stitched to the adjacent FOV such that the first fiducial has global location (300 pm, 320 pm) or a similar location in the global coordinate system 300. In another example, the ground truth calculation engine 122 may33080 / IP-2900 / PC stitch the FOVs based on common image features, such as cells, nuclei, etc. within the adjacent FOVs.

[0083] In any event, the ground truth calculation engine 122 may stitch each of the FOVs 302 to each other to register the microscope image 200 to the global coordinate system 300. The ground truth calculation engine 122 may perform a similar process for registering the spatial transcript map 250 to the global coordinate system 300. For example, the ground truth calculation engine 122 may transform the locations of fiducials and / or SBCs in each sequencing tile into physical locations and stitch the sequencing tiles to adjacent sequencing tiles within the spatial transcriptomics map using shifted fiducials which overlap in adjacent sequencing tiles and / or SBCs which overlap in adjacent sequencing tiles.

[0084] The fiducials and / or SBCs may have known local locations within the sequencing tile. In yet other implementations, the ground truth calculation engine 122 obtains a known spacing between the fiducials which can be used to derive local locations of the fiducials within the sequencing tile. The sequencing tile may also have an identifier such as a tile ID which can be used to obtain information regarding the spatial relationship of the sequencing tile within the spatial transcriptomics map 250 (e.g., a swath number of a lane, an order of the tile within a swath or lane, etc.). For example, the tile ID may indicate whether the sequencing tile is the top left sequencing tile in the global coordinate system 300, directly below the top left sequencing tile, directly to the right of the top left sequencing tile, or in another specific location.

[0085] The ground truth calculation engine 122 may additionally or alternatively calculate one or more values by which to shift the local fiducial locations to fit the global coordinate system 300. For example, the ground truth calculation engine 122 may use known and / or detected SBC locations or overlap fiducial locations to calculate a shift by which to shift local fiducial location values to fit the global coordinate system. Depending on the implementation, the ground truth calculation engine 122 performs the shift before performing stitching for the sequencing tiles, after performing the stitching, while performing the stitching, etc.

[0086] The ground truth calculation engine 122 may stitch the sequencing tile to adjacent sequencing tiles. As mentioned above, the sequencing tile includes at least some overlap with adjacent sequencing tiles. In this manner, the ground truth calculation engine 122 may identify common SBCs or fiducials within the adjacent sequencing tiles and assign the common SBCs or33080 / IP-2900 / PC fiducials the same physical location in the global coordinate system 300. For example, the ground truth calculation engine 122 may identify the same SBC(s) on the right side of one sequencing tile and on the left side of an adjacent sequencing tile. The ground truth calculation engine 122 may assign the SBC(s) the same or a similar physical location in the global coordinate system 300.

[0087] Fig. 4A is a diagram of an example method 400A for performing sequencing tile stitching without calculating fiducial and / or SBC shifts. In particular, a computing device such as a sequencing device (e.g., sequencing device 110), server device (e.g., server device 120), and / or client device (e.g., client device 130) may perform the steps described with regard to Fig. 4A.

[0088] The computing device may receive one or more sequencing tile images 412 from a sequencing device (e.g., sequencing device 110) and / or from an imaging and / or sequencing component of the device, as described herein. The computing device may subsequently stitch together the sequencing tile images 412 based on one or more common features into a stitched sequencing image 414. In particular, the common features may include one or more overlapping elements that the computing device uses to stitch together the adjacent sequencing tiles. However, some of the overlapping elements may be offset (e.g., due to imaging errors, movement, etc.), causing at least one of the imaging tiles to be offset, as can be seen in Fig. 4.

[0089] In some implementations, the computing device calculates confidences scores for fiducials and / or stitching. As such, the computing device may detect errors (e.g., stitching artifacts) based on the confidence score meeting a predetermined threshold value, through (normalized) cross-correlation peaks of expected locations and calculated locations, etc.

[0090] The computing device may then analyze the stitched sequencing image 414 to detect one or more fiducials 416 and, using the detected fiducials, map the fiducial locations to a ground truth coordinate system and compute an affine transformation matrix to apply to one or more element locations of the stitched sequencing image 414 to generate a ground troth coordinate listing 418, as described herein. However, generating the ground truth coordinate listing 418 using an offset tile may lead to rendering and / or generation errors, as can be seen in Fig. 4B. Similarly, it will be understood that, although Fig. 4A depicts a sequencing side tile33080 / IP-2900 / PC stitching process with errors, similar errors may occur on the microscope side for image stitching.

[0091] Fig. 4B depicts various exemplary stitching artifacts 400B due to errors in the sequencing side tile stitching process and / or microscope side image stitching process. In particular, stitching error 450 depicts a number of fiducial stitching errors 455. In particular, the fiducials stitching errors 455 are not in a grid because of stitching errors during image stitching. Similarly, stitching error 460 depicts a number of tile stitching artifacts 465. In some implementations, the tile stitching artifacts 465 may occur due to an insufficient number of SBCs and / or fiducials in the artifact regions (e.g., due to mismatched tiles), causing the tile stitching artifacts 465 to occur when stitching the tiles.

[0092] Fig. 4C depicts a diagram of an example method 400C for performing sequencing tile stitching including calculations of the fiducial and / or SBC shifts, in contrast to the techniques of Fig. 4A described above. However, similar to Fig. 4A, a computing device such as a sequencing device (e.g., sequencing device 110), server device (e.g., server device 120), and / or client device (e.g., client device 130) may perform the steps described with regard to Fig. 4C.

[0093] Similar to Fig. 4A, the computing device may receive sequencing tile images 422.Unlike Fig. 4A, however, the computing device may detect fiducials 424 present in the sequencing tile images 422 without stitching the sequencing tile images together. The computing device may calculate shifts for the fiducials 424 based on overlap data between the sequencing tile images. For example, the computing device may calculate shifts for the fiducials 424 using SBCs and / or fiducials present in adjacent sequencing tile images to determine a shift value to apply to the fiducials 424. The computing device may then shift the fiducials 424 and stitch the fiducial locations to generate shifted fiducials 426. The computing device may map the shifted fiducial locations to ground truth and / or compute an affine transform based on the shifted fiducials 426 and known locations for the fiducials in the sequencing tile images. The computing device may then apply the affine transform to data associated with the sequencing tile images (e.g., SBC locations, common fiducial locations, the sequencing tile images, etc.) to generate a ground truth coordinate listing 428. Because the fiducial locations are shifted to correct for any errors before stitching and / or mapping, the system is able to reduce or eliminate stitching errors otherwise present in the exemplary method of Fig. 4A. This may result in stitched sequencing33080 / IP-2900 / PC tiles without artifacts, as shown in Fig. 4D, or with reduced artifacts compared to the stitched sequencing tiles 460 as shown in Fig. 4B.

[0094] Fig. 5A depicts an exemplary block diagram of a system 500 for registering a spatial transcriptomics image to a ground truth coordinate system. Depending on the implementation, the system 500 may include components of a device in a larger computing system (e.g., computing environment 100 of Fig. 1). For example, the system 500 may include components of a computing device such as a sequencing device (e.g., sequencing device 110), a computing device communicatively coupled to the sequencing device (e.g., a server device 120 and / or a client device 130), and / or any other such device as described herein. It will be understood that the system 500 may include additional, fewer, and / or alternate components.

[0095] In some implementations, the computing device generates and / or receives sequencing tile images 502 representative of a substrate, as described herein. In some such implementations, the computing device receives the sequencing tile images 502 at a fiducial detection module 504. In some implementations, the fiducial detection module 504 receives and / or is previously programmed with a fiducial template 506 for detecting fiducials in the sequencing tile images. In some implementations, the fiducial detection module 504 detects non-unique fiducials (e.g., fiducials introduced broadly into the substrate for identification). In some such implementations, the fiducials are introduced in line with the fiducial template 506 that is then provided to the fiducial detection module 504. In further implementation, the fiducial template 506 is based on a substrate used and / or a grid of fiducials introduced to a substrate as described with regard to Fig. 1 above.

[0096] In some implementations, the fiducial detection module 504 may assign confidence scores to detected fiducials. For example, if the detected fiducial is blurry, not easily visible, etc., then the fiducial detection module 504 may assign the detected fiducial a low confidence score. The fiducial detection module 504 in some implementations only considers fiducials with a confidence score meeting a predetermined threshold value and filters out other fiducials.

[0097] The fiducial detection module 504 may then generate local fiducial locations 508 present in the sequencing tile images 502. hr some implementations, the local fiducial locations may be determined relative to a local coordinate system for individual sequencing tile images captured by the sequencing device 110. In some implementations, the fiducial detection module33080 / IP-2900 / PC504 generates the local fiducial locations 508 by using the fiducial template 506 and searching within a predetermined range of the template fiducial locations. In further implementations, the fiducial detection module detects fiducials present in the sequencing tile images 502 using additional or alternate techniques (e.g., as discussed herein with regard to Fig. 1).

[0098] In some implementations, the system 500 may calculate fiducial shifts 512 for the local fiducial locations 508. Depending on the implementation, the mapping module 510. a dedicated calculation module (not shown), and / or another module of system 500 performing functionalities as described herein may calculate the fiducial shifts 512. In some implementations, the system 500 may calculate the fiducial shifts 512 based on overlap data present in adjacent sequencing tile images 502. In some such implementations, the overlap data may include one or more spatial barcodes (SBCs) indicative of nucleotide data present in adjacent sequencing tile images 502 (e.g., as described with regard to Fig. 1 above). As such, the system 500 may calculate a shift in the SBCs between adjacent sequencing tile images 502 and subsequently calculate a corresponding shift for the local fiducial locations 508 to generate the calculated fiducial shifts 512.

[0099] In further implementations, the overlap data may include one or more fiducials common to adjacent sequencing tile images 502. Similarly to the SBCs, the system 500 may calculate the shift in the overlapping fiducials and subsequently calculate a corresponding shift for the local fiducial locations 508. The system 500 may then generate the calculated fiducial shifts 512.

[0100] In some implementations, the mapping module 510 receives the local fiducial locations 508 and / or the calculated fiducial shifts 512. The mapping module 510 may use the calculated fiducial shifts 512 to transform the local fiducial locations 508 to global fiducial locations 514 (e.g., fiducial locations for the entire set of sequencing tile images 502 rather than on a per-tile basis (e.g., for the local fiducial locations 508)). The mapping module 510 may map the sequencing tile images 502 in accordance with the global fiducial locations 514, generating a global map of the sequencing tile images 502 based on the global fiducial locations 514. In some implementations, the system 500 may map the global fiducial locations 514 to a ground truth coordinate system as described herein.33080 / IP-2900 / PC

[0101] The registration module 516 may then receive the global fiducial locations 514 and expected fiducial locations 518. In some implementations, the expected fiducial locations 518 are based on the fiducial template 506, based on a separate calculation of the fiducial locations without the calculated shifts, based on an organization and / or arrangement of a corresponding flow cell (e.g., as described above with regard to Fig. 1), based on a pattern for introducing fiducials, and / or any other such technique for determining a physical location of fiducials as described herein. The registration module 516 may then generate affine transformation matrices that transform locations of fiducials or SBCs detected within the map of sequencing tile images 502 (e.g., also referred to as a “spatial transcriptomics map”) to known, physical locations. The registration module 516 may then register the transformed spatial transcriptomics map to generate a registered map 520.

[0102] In some implementations, the system 500 may determine the global fiducial locations 514 and map the global fiducial locations 514 to the ground truth coordinate system (e.g., using the SBCs and / or other overlap data) before a tissue sample is placed in a flow cell and permeabilization has been performed. After which, the system 500 may generate the spatial transcript image using locations for one or more SBCs (e.g., which have been permeabilized). Depending on the implementation, the registered map 520 may be used (e.g.. by another sequencing device) to generate the spatial transcript image.

[0103] Fig. 5B depicts an exemplary block diagram of a system 550 for registering a microscope image to a ground truth coordinate system, similar to that of system 500, but from a microscope-focused perspective rather than a sequencing device-focused perspective. Depending on the implementation, the system 550 may include components of a device in a larger computing system (e.g., computing environment 100 of Fig. 1). For example, the system 500 may include components of a computing device such as a microscope (e.g., microscope 140), a computing device communicatively coupled to the microscope (e.g., a client device 130 and / or a server device 120). and / or any other such device as described herein. It will be understood that the system 550 may include additional, fewer, and / or alternate components.

[0104] In some implementations, the computing device generates and / or receives various FOVs for a microscope image captured by the microscope 140. Depending on the implementations, the computing device may stitch the image together (e.g., using feature-based33080 / IP-2900 / PC image stitching, fiducial-based image stitching, etc.) to generate a stitched image 552 representative of a tissue sample, as described herein. The system 550 may then proceed with detecting fiducials in the stitched image 552 similar to the system 500 of Fig. 5A. For example, in some such implementations, the computing device receives the stitched image 552 at a fiducial detection module 554A. In some implementations, the fiducial detection module 554 receives and / or is previously programmed with a fiducial template 556A for detecting fiducials in the sequencing tile images. In some implementations, the fiducial detection module 554A detects non-unique fiducials (e.g., fiducials introduced broadly into the tissue sample for identification). In some such implementations, the fiducials are introduced in line with the fiducial template 556A that is then provided to the fiducial detection module 554A. In some implementations, the fiducial template 556A matches the fiducial template 506 because variation in a substrate design may vary consistently for both the sequencing side and the microscope imaging side.

[0105] The fiducial detection module 554A may then generate fiducial image data 558. including the stitched image 552 and the detected fiducials. In some implementations, the overlap data and / or the fiducial image data 558 may be or include fiducials in an active area of the image (e.g., an area indicative of the tissue) and / or an inactive area of the image (e.g., an area not indicative of the tissue). In some implementations, the system fiducial image data 558 may include an increased density of fiducials to ensure stitching in an area surrounding the tissue sample (e.g., tissue sample 102a, 102b). In further implementations the image data may include holes in the tissues, and the local fiducial locations may be determined based on edges containing fiducials to prevent errors (e.g.. from detecting edges of the tissues that do not exist due to blank space of a hole in the tissue).

[0106] In further implementations, a registration module 560 receives the fiducial image data 558 and one or more extracted templates 562. In some implementations, the extracted templates 562 are associated with a microscope image pitch (e.g.. a physical size of the pixels in an image sensor for the microscope), and the template may therefore be used to calculate the object field of the microscope. In particular, the registration module 560 may use the extracted templates 562 to calculate the object field of the microscope and apply such to the fiducial image data 558 to register and / or correct elements of a locally registered image 564.33080 / IP-2900 / PC

[0107] In some implementations, the system 550 then analyzes the locally registered image 564 along with a set of expected fiducial locations 566 and a second set of one or more fiducial templates 556B using a fiducial detection module 554B. Depending on the implementation, the fiducial detection module 554B may be the same module as the fiducial detection module 554A but running an algorithm to and / or configured to detect unique fiducials in the locally registered image 564 rather than non-unique fiducials as described above.

[0108] Similar to the expected fiducial locations 518 of Fig. 5 A, the expected fiducial locations 566 may be based on the fiducial template(s) 556A and / or 556B, based on a separate calculation of the fiducial locations, based on an organization and / or arrangement of a corresponding flow cell (e.g., as described above with regard to Fig. 1), based on a pattern for introducing fiducials, and / or any other such technique for determining a physical location of fiducials as described herein.

[0109] The fiducial detection module 554B and / or another component of the system 550 may then calculate a global translation estimate 568 based on the expected fiducial locations 566, the fiducial template 556B, and the locally registered image 564. In some implementations, the system 550 calculates one or more shifts and / or corrections to shifts as part of calculating the global translation estimate 568. For example, the system 550 may calculate the shifts and corrections to shifts based on the unique fiducials (e.g.. included in overlap data between adjacent FOVs of the stitched image 552). In some implementations, the system 550 calculates large errors and / or shifts in stitching using non-overlap region fiducials between FOVs (e.g., where an FOV is off by more than a predetermined margin) and calculates fine errors (e.g., with blending artifacts and / or below a predetermined margin) using the overlap region fiducials. The system 550 may then map the local fiducial locations of the locally registered image 564 to the global translation estimate 568 and map the global translation estimate 568 (and / or a final global image associated with the global translation estimate 568 locations) to a ground truth coordinate system. Depending on the implementation, the ground truth coordinate system may be or include the ground truth coordinate system as registered in system 500.

[0110] The system 550 may then calculate an affine transformation (e.g., as described above) between detected fiducial locations (e.g., by the fiducial detection module 554B) and expected fiducial locations 566. The system 550 may apply the affine transformation to the image and / or33080 / IP-2900 / PC the global translation estimate 568 locations to register the image and generate a globally registered image 570.

[0111] In some implementations, the affine transformation performed by either the system 500 or the system 550 is a linear distortion correction. In further implementations, the affine transformation is preceded by a non-linear distortion correction for a stitched image, such as to be constant across FOVs and / or tiles before stitching. In some implementations, the non-linear distortion correction is instead applied based on the microscope (e.g., microscope 140) to correct distortions inherently present in the microscope. For example, the microscope and / or a computing device communicatively coupled to the microscope may store a look-up table or polynomial with pre-measured and pre-calculated transformations to apply to each FOV and / or tile to normalize each respective FOV and / or tile. In some implementations, the non-linear distortion correction is applied using an optical alignment tool, such as that described in U.S. Patent Application No. 15 / 382,684, titled “Optical alignment tool” and filed on December 18, 2016, which is hereby incorporated by reference in its entirety.

[0112] Fig. 6 is a flow diagram of an example method 600 for registering a microscope image of a tissue sample to a ground truth coordinate system. The method 600 may be implemented by a computing device such as a microscope 140, a computing device communicatively coupled to the microscope 140 (e.g., server device 120 and / or client device 130). and / or any other such device as described herein. It will be understood that additional, fewer, and / or alternate components may be used to implement the example method 600, and / or that the method 600 may include more or fewer blocks than shown (and / or in a different order than shown).

[0113] At block 602, the computing device may obtain a plurality of fields of view (FOVs) of a microscope image of a tissue sample. In some implementations, the plurality of FOVs are organized in a series of rows and columns. In further implementations, a respective FOV at least partially overlaps with an adjacent FOV, and the determined locations of the fiducials include an overlapping fiducial location present in the respective FOV and the adjacent FOV.

[0114] At block 604, the computing device may stitch the plurality of FOVs together to map the plurality of FOVs to global image locations. In some implementations, the stitching is based on common image features of the plurality of FOVs. For example, the stitching may be based on detecting that a feature is at least partially present in two FOVs (e.g., via image analysis33080 / IP-2900 / PC techniques as described herein), and the computing device may stitch the corresponding FOVs together at a corresponding detected point of overlap.

[0115] At block 606, the computing device may determine locations of fiducials in the stitched plurality of FOVs. In some implementations, the computing device shifts the stitched plurality of FOVs based on a difference between the known locations for the fiducials and the determined locations of the fiducials. For example, the computing device may detect one or more erroneous shifts and / or stitching errors. As such, the computing device may use fiducial locations in the plurality of FOVs to determine corrections to apply to the shifts (e.g., using overlap fiducials for fine errors and non-overlapping fiducials for large stitching errors). The computing device may then map the shifted plurality of FOVs to the global image locations. At block 608, the computing device may compare the determined locations of the fiducials to known locations for the fiducials in the stitched plurality of FOVs to compute a transform.

[0116] At block 610, the computing device may apply the transform to the stitched plurality of FOVs to register the microscope image to a ground truth coordinate system. In some implementations, the computing device additionally registers data associated with a plurality of sequencing tile images related to the microscope image to the ground truth coordinate system. In some such implementations, the data associated with a plurality of sequencing tile images is the data described below with regard to Fig. 7. Depending on the implementation, the computing device receives the data already registered and / or mapped to the ground truth coordinate system (e.g., by a sequencing device 110, server device 120, client device 130, etc.). In further implementations, the computing device additionally registers the data such that both the data associated with the plurality of sequencing tile images and the microscope image are coregistered to the ground truth coordinate system and / or each other.

[0117] Depending on the implementation, the ground truth coordinate system may refer to one of multiple coordinate systems. For example, the ground truth coordinate system may be or include a universal ground truth coordinate system used by a plurality of devices (e.g.. the sequencing device 110, the server device 120. the client device 130. the microscope 140, and / or any other such device as described herein). As such, the ground truth coordinate system may be agnostic to any particular device coordinate system. In further implementations, the ground truth coordinate system may be or include a ground truth coordinate system based on a particular33080 / IP-2900 / PC device coordinate system. For example, the ground truth coordinate system may be a coordinate system common to a sequencing tile global image coordinate system used by a sequencing device (e.g., sequencing device 110) for analyzing a plurality of sequencing tile images related to the microscope image. As such, the computing device may register the microscope image to an uncorrected sequencing tile image coordinate system (e.g., as described below). In some such implementations, the detected fiducials are common fiducials present in both the plurality of sequencing tile images and in the microscope image. As such, the computing device may detect a wider range of fiducials (e.g.. common fiducials rather than unique or overlapping fiducials), but may in turn lead to less accuracy in the mapping for faster or more direct mapping.

[0118] Fig. 7 is a flow diagram of an example method 700 for registering a spatial transcriptomics image to a ground truth coordinate system. The method 700 may be implemented by a computing device such as a sequencing device 110, a computing device communicatively coupled to the sequencing device 110 (e.g., server device 120 and / or client device 130), and / or any other such device as described herein. It will be understood that additional, fewer, and / or alternate components may be used to implement the example method 600, and / or that the method 600 may include more or fewer blocks than shown (and / or in a different order than shown).

[0119] At block 702, the computing device may obtain data associated with a plurality of sequencing tile images of a spatial transcript image of a substrate. At block 704, the computing device may determine locations of fiducials in the plurality of sequencing tile images.

[0120] At block 706, the computing device may shift locations of respective fiducials of respective sequencing tile images of the plurality of sequencing tile images based on overlap data for the respective sequencing tile images. In some implementations, the overlap data includes matching nucleotide data associated with the substrate in adjacent sequencing tile images. For example, adjacent sequencing tile images may include nucleotide data such as spatial barcodes (SBCs) indicative of a matching nucleotide pair shared between adjacent sequencing tile images that provide points which the computing device may use as a reference for how / where the sequencing tile images should match. As such, shifting the locations of respective fiducials may include receiving the SBCs and shifting the locations of the respective fiducials by shifting33080 / IP-2900 / PC locations of the adjacent sequencing tile images to align matching SBCs in the adjacent sequencing tile images.

[0121] In further implementations, the overlap data may include the determined locations of the fiducials. As such, the computing device may shift the locations of respective fiducials based on detected common fiducials to adjacent sequencing tile images, the common fiducials indicative of locations in the adjacent sequencing tile images that should match and / or be registered to a same coordinate system point. In some such implementations, the plurality of sequencing tile images are organized in a series of swaths, a respective tile image at least partially overlaps with an adjacent sequencing tile image, and the locations of the fiducials includes an overlapping fiducial location present in the respective sequencing tile image and the adjacent sequencing tile image.

[0122] At block 708, the computing device may compare the shifted locations of the respective fiducials to known locations for the respective fiducials to compute a transform. At block 710, the computing device may apply the transform to the data associated with the plurality of sequencing tile images to register the data associated with the plurality of sequencing tile images to a ground truth coordinate system. Similar to block 610 described above with regard to Fig. 6, the computing device additionally registers data associated with a microscope image, related to the plurality of sequencing tile images, to the ground truth coordinate system. In some such implementations, the microscope image is the image described above with regard to Fig. 6. Depending on the implementation, the computing device receives the image already registered and / or mapped to the ground truth coordinate system (e.g., by a microscope 140, server device 120, client device 130, etc.). In further implementations, the computing device additionally registers the data such that both the data associated with the plurality of sequencing tile images and the microscope image are co-registered to the ground truth coordinate system and / or each other.

[0123] Depending on the implementation, the ground truth coordinate system may refer to one of multiple coordinate systems. For example, the ground troth coordinate system may be or include a universal ground truth coordinate system used by a plurality of devices (e.g., the sequencing device 110, the server device 120, the client device 130, the microscope 140, and / or any other such device as described herein). As such, the ground truth coordinate system may be33080 / IP-2900 / PC agnostic to any particular device coordinate system. In further implementations, the ground truth coordinate system may be or include a ground truth coordinate system based on a particular device coordinate system. For example, the ground truth coordinate system may be a coordinate system common to a microscope global image coordinate system used by a microscope (e.g., microscope 140) for analyzing a microscope image related to the plurality of sequencing tile images. As such, the computing device may register the plurality of sequencing tile images to an uncorrected microscope coordinate system (e.g., as described above). In some such implementations, the detected fiducials are common fiducials present in both the plurality of sequencing tile images and in the microscope image. As such, the computing device may detect a wider range of fiducials (e.g., common fiducials rather than unique or overlapping fiducials), but may in turn lead to less accuracy in the mapping for faster or more direct mapping.

[0124] In some implementations, the computing device may apply the transform to different portions of the data associated with the plurality of sequencing tile images. For example, in some implementations in which shifting the locations of respective fiducials includes receiving SBCs representative of nucleotide data in adjacent sequencing tile images, the computing device may apply the transform to one or more locations associated with the SBCs. hi further implementations, the computing device may apply the transformation to the locations of the fiducials detected in the plurality of sequencing tile images. In still further implementations, the computing device may apply the transform to the plurality of sequencing tile images to transform the images as a whole.

[0125] Aspects of the techniques described in the present disclosure may include any of the following aspects, either alone or in combination:

[0126] Aspect 1. A method for registering a microscope image of a tissue sample to a ground truth coordinate system, the method comprising: obtaining, by one or more processors, a plurality of fields of view (FOVs) of a microscope image of a tissue sample; stitching, by the one or more processors, the plurality of FOVs together to map the plurality of FOVs to global image locations: determining, by the one or more processors, locations of fiducials in the stitched plurality of FOVs; comparing, by the one or more processors, the determined locations of the fiducials to known locations for the fiducials in the stitched plurality of FOVs to compute a33080 / IP-2900 / PC transform; and applying, by the one or more processors, the transform to the stitched plurality of FOVs to register the microscope image to a ground truth coordinate system.

[0127] Aspect 2. The method of aspect 1, wherein the stitching is based on common image features of the plurality of FOVs.

[0128] Aspect 3. The method of aspect 1, further comprising: shifting, by the one or more processors, the stitched plurality of FOVs based on a difference between the known locations for the fiducials and the determined locations of the fiducials.

[0129] Aspect 4. The method of aspect 3, further comprising: mapping, by the one or more processors, the shifted plurality of FOVs to the global image locations.

[0130] Aspect 5. The method of aspect 1, wherein the plurality of FOVs are organized in a series of rows and columns, a respective FOV of the plurality of FOVs at least partially overlaps with an adjacent FOV, and the determined locations of the fiducials include an overlapping fiducial location present in the respective FOV and the adjacent FOV.

[0131] Aspect 6. The method of aspect 1, further comprising: registering, by the one or more processors, data associated with a plurality of sequencing tile images related to the microscope image to the ground truth coordinate system.

[0132] Aspect 7. The method of aspect 1, wherein the ground truth coordinate system is a universal ground truth coordinate system to which a plurality of devices map data including at least a microscope capturing the microscope image and a sequencer analyzing a plurality of sequencing tile images related to the microscope image.

[0133] Aspect 8. The method of aspect 1, wherein the ground truth coordinate system is a coordinate system common to a sequencing tile global image coordinate system used by a sequencer analyzing a plurality of sequencing tile images related to the microscope image.

[0134] Aspect 9. The method of aspect 8, wherein the fiducials are common fiducials present in the plurality of sequencing tile images and in the microscope image.

[0135] Aspect 10. A method for registering data associated with a spatial transcript image of a substrate to a ground truth coordinate system, the method comprising: obtaining, by one or more processors, data associated with a plurality of sequencing tile images of a spatial transcript image33080 / IP-2900 / PC of a substrate; determining, by the one or more processors, locations of fiducials in the plurality of sequencing tile images; shifting, by the one or more processors, respective locations of respective fiducials of respective sequencing tile images of the plurality of sequencing tile images based on overlap data for the respective sequencing tile images; comparing, by the one or more processors, the shifted respective locations of the respective fiducials to known locations for the respective fiducials to compute a transform; and applying, by the one or more processors, the transform to the data associated with the plurality of sequencing tile images to register the data associated with the plurality of sequencing tile images to a ground truth coordinate system.

[0136] Aspect 11. The method of aspect 10, wherein the overlap data includes matching nucleotide data associated with the substrate in adjacent sequencing tile images.

[0137] Aspect 12. The method of aspect 11, wherein shifting the locations of respective fiducials includes: receiving, by the one or more processors, spatial barcodes (SBCs) representative of the nucleotide data in the adjacent sequencing tile images; and shifting, by the one or more processors, the locations of the respective fiducials by shifting locations of the adjacent sequencing tile images to align matching SBCs in the adjacent sequencing tile images.

[0138] Aspect 13. The method of aspect 12, wherein applying the transform includes: applying, by the one or more processors, the transform to at least one of (i) one or more locations associated with the SBCs or (ii) the locations of the fiducials.

[0139] Aspect 14. The method of aspect 10, wherein applying the transform includes: applying, by the one or more processors, the transform to the plurality of sequencing tile images.

[0140] Aspect 15. The method of aspect 10, wherein the overlap data includes the determined locations of the fiducials.

[0141] Aspect 16. The method of aspect 15, wherein the plurality of sequencing tile images are organized in a series of swaths, a respective sequencing tile image at least partially overlaps with an adjacent sequencing tile image, and the locations of the fiducials includes an overlapping fiducial location present in the respective sequencing tile image and the adjacent sequencing tile image.33080 / IP-2900 / PC

[0142] Aspect 17. The method of aspect 10, further comprising: registering, by the one or more processors, data associated with a microscope image related to the plurality of sequencing tile images to the ground truth coordinate system.

[0143] Aspect 18. The method of aspect 10, wherein the ground truth coordinate system is a universal ground truth coordinate system to which a plurality of devices map data including at least a sequencer analyzing the plurality of sequencing tile images and a microscope capturing a microscope image related to the plurality of sequencing tile images.

[0144] Aspect 19. The method of aspect 10, wherein the ground truth coordinate system is a coordinate system common to a microscope global image coordinate system used by a microscope analyzing a microscope image related to the plurality of sequencing tile images.

[0145] Aspect 20. The method of aspect 19, wherein the fiducials are common fiducials present in the plurality of sequencing tile images and in the microscope image.

[0146] Aspect 21. The method of aspect 10, wherein the fiducials include fiducials in an active area of the sequencing tile images.

[0147] Aspect 22. The method of aspect 10, wherein the fiducials include fiducials outside the active area of the sequencing tile images.ADDITIONAL CONSIDERATIONS

[0148] Although the disclosure herein sets forth a detailed description of numerous different implementations, it should be understood that the legal scope of the description is defined by the words of the claims set forth at the end of this patent and equivalents. The detailed description is to be construed as exemplary only and does not describe every possible implementation since describing every possible implementation would be impractical. Numerous alternative implementations may be implemented, using either current technology or technology developed after the filing date of this patent, which would still fall within the scope of the claims.

[0149] The following additional considerations apply to the foregoing discussion. Throughout this specification, plural instances may implement components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order33080 / IP-2900 / PC illustrated. Structures and functionality presented as separate components in example configurations may be implemented as a combined structure or component. Similarly, structures and functionality presented as a single component may be implemented as separate components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

[0150] Additionally, certain implementations are described herein as including logic or a number of routines, subroutines, applications, or instractions. These may constitute either software (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware. In hardware, the routines, etc., are tangible units capable of performing certain operations and may be configured or arranged in a certain manner. In example implementations, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

[0151] The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example implementations, comprise processor-implemented modules.

[0152] Similarly, the methods or routines described herein may be at least partially processor implemented. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented hardware modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example implementations, the processor or processors may be located in a single location, while in other implementations the processors may be distributed across a number of locations.

[0153] The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example implementations, the one or more processors or processor-33080 / IP-2900 / PC implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other implementations, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations.

[0154] This detailed description is to be construed as exemplary only and does not describe every possible implementation, as describing every possible implementation would be impractical, if not impossible. A person of ordinary skill in the art may implement numerous alternate implementations, using either current technology or technology developed after the filing date of this application.

[0155] Those of ordinary skill in the art will recognize that a wide variety of modifications, alterations, and combinations may be made with respect to the above described implementations without departing from the scope of the invention, and that such modifications, alterations, and combinations are to be viewed as being within the ambit of the inventive concept.

[0156] The patent claims at the end of this patent application are not intended to be construed under 35 U.S.C. § 112(f) unless traditional means-plus-function language is expressly recited, such as “means for” or “step for” language being explicitly recited in the claim(s). The systems and methods described herein are directed to an improvement to computer functionality and improve the functioning of conventional computers.

Claims

33080 / IP-2900 / PCCLAIMSWhat is claimed is:

1. A method for registering a microscope image of a tissue sample to a ground truth coordinate system, the method comprising: obtaining, by one or more processors, a plurality of fields of view (FOVs) of a microscope image of a tissue sample; stitching, by the one or more processors, the plurality of FOVs together to map the plurality of FOVs to global image locations; determining, by the one or more processors, locations of fiducials in the stitched plurality of FOVs; comparing, by the one or more processors, the determined locations of the fiducials to known locations for the fiducials in the stitched plurality of FOVs to compute a transform; and applying, by the one or more processors, the transform to the stitched plurality of FOVs to register the microscope image to a ground truth coordinate system.

2. The method of claim 1, wherein the stitching is based on common image features of the plurality of FOVs.

3. The method of claim 1, further comprising: shifting, by the one or more processors, the stitched plurality of FOVs based on a difference between the known locations for the fiducials and the determined locations of the fiducials.

4. The method of claim 3, further comprising: mapping, by the one or more processors, the shifted plurality of FOVs to the global image locations.

5. The method of claim 1, wherein the plurality of FOVs are organized in a series of rows and columns, a respective FOV of the plurality of FOVs at least partially overlaps with an33080 / IP-2900 / PC adjacent FOV, and the determined locations of the fiducials include an overlapping fiducial location present in the respective FOV and the adjacent FOV.

6. The method of claim 1, further comprising: registering, by the one or more processors, data associated with a plurality of sequencing tile images related to the microscope image to the ground truth coordinate system.

7. The method of claim 1, wherein the ground truth coordinate system is a universal ground truth coordinate system to which a plurality of devices map data including at least a microscope capturing the microscope image and a sequencer analyzing a plurality of sequencing tile images related to the microscope image.

8. The method of claim 1, wherein the ground truth coordinate system is a coordinate system common to a sequencing tile global image coordinate system used by a sequencer analyzing a plurality of sequencing tile images related to the microscope image.

9. The method of claim 8, wherein the fiducials are common fiducials present in the plurality of sequencing tile images and in the microscope image.

10. A method for registering data associated with a spatial transcript image of a substrate to a ground truth coordinate system, the method comprising: obtaining, by one or more processors, data associated with a plurality of sequencing tile images of a spatial transcript image of a substrate; determining, by the one or more processors, locations of fiducials in the plurality of sequencing tile images; shifting, by the one or more processors, respective locations of respective fiducials of respective sequencing tile images of the plurality of sequencing tile images based on overlap data for the respective sequencing tile images; comparing, by the one or more processors, the shifted respective locations of the respective fiducials to known locations for the respective fiducials to compute a transform; and33080 / IP-2900 / PC applying, by the one or more processors, the transform to the data associated with the plurality of sequencing tile images to register the data associated with the plurality of sequencing tile images to a ground truth coordinate system.

11. The method of claim 10, wherein the overlap data includes matching nucleotide data associated with the substrate in adjacent sequencing tile images.

12. The method of claim 11, wherein shifting the locations of respective fiducials includes: receiving, by the one or more processors, spatial barcodes (SBCs) representative of the nucleotide data in the adjacent sequencing tile images; and shifting, by the one or more processors, the locations of the respective fiducials by shifting locations of the adjacent sequencing tile images to align matching SBCs in the adjacent sequencing tile images.

13. The method of claim 12, wherein applying the transform includes: applying, by the one or more processors, the transform to at least one of (i) one or more locations associated with the SBCs or (ii) the locations of the fiducials.

14. The method of claim 10, wherein applying the transform includes: applying, by the one or more processors, the transform to the plurality of sequencing tile images.

15. The method of claim 10, wherein the overlap data includes the determined locations of the fiducials.

16. The method of claim 15, wherein the plurality of sequencing tile images are organized in a series of swaths, a respective sequencing tile image at least partially overlaps with an adjacent sequencing tile image, and the locations of the fiducials includes an overlapping fiducial location present in the respective sequencing tile image and the adjacent sequencing tile image.33080 / IP-2900 / PC17. The method of claim 10, further comprising: registering, by the one or more processors, data associated with a microscope image related to the plurality of sequencing tile images to the ground truth coordinate system.

18. The method of claim 10, wherein the ground truth coordinate system is a universal ground truth coordinate system to which a plurality of devices map data including at least a sequencer analyzing the plurality of sequencing tile images and a microscope capturing a microscope image related to the plurality of sequencing tile images.

19. The method of claim 10, wherein the ground truth coordinate system is a coordinate system common to a microscope global image coordinate system used by a microscope analyzing a microscope image related to the plurality of sequencing tile images.

20. The method of claim 19, wherein the fiducials are common fiducials present in the plurality of sequencing tile images and in the microscope image.

21. The method of claim 10, wherein the fiducials include fiducials in an active area of the sequencing tile images.

22. The method of claim 10, wherein the fiducials include fiducials outside an active area of the sequencing tile images.

Images