Systems and methods for image registration

Abstract

Methods and systems are provided herein for image registration. In an example, a system includes one or more processors configured to: obtain a first set of bounding box coordinates, the first set of bounding coordinates identifying respective locations of a plurality of first bounding boxes, the plurality of first bounding boxes identifying a plurality of objects in an initial image of a sample; obtain a second set of bounding box coordinates, the second set of bounding coordinates identifying respective locations of a plurality of second bounding boxes, the plurality of second bounding boxes identifying a plurality of objects in a current image of the sample; and register the current image to the initial image based on the first set of bounding box coordinates and the second set of bounding box coordinates.

Description

Docket No. ARL24304PCTSYSTEMS AND METHODS FOR IMAGE REGISTRATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application claims priority to U.S. Provisional Application No. 63 / 746,842, filed January 17, 2025, and entitled "SYSTEMS AND METHODS FOR IMAGE REGISTRATION,” the entire contents of which is hereby incorporated by reference for all purposes.TECHNICAL FIELD

[0002] Embodiments of the subject matter disclosed herein relate generally to image registration, and more specifically to image registration using object bounding boxes.BACKGROUND AND SUMMARY

[0003] Modem image analysis leverages object detection to identify and locate objects within images. This process involves using coordinates, such as polygons and bounding boxes, to mark each object’s position. Bounding boxes are particularly common, defined by their top-left and bottom-right corners, encapsulating the object within a rectangular frame. This method is widely used across various domains, including autonomous vehicles, security, healthcare, and high-content analysis (HCA) of cell therapy screens

[0004] Some image analysis tasks demand that the detected objects be tracked across multiple image frames. Typically, image-based registration techniques are employed to track objects across image frames, such as feature-based image registration. However, these methods are computationally expensive and time-consuming, which can limit their applications. Further, these methods require processing of the full image frames, further limiting their applicability.

[0005] Methods and systems for image registration are provided herein that address the above issues. In one example, a system includes one or more processors configured to: obtain a first set of bormding box coordinates, the first set of bounding coordinates identifying respective locations of a plurality of first bounding boxes, the plurality of first bounding boxes identify ing a plurality of objects in an initial image of a sample; obtain a second set of bormding box coordinates, the second set of bounding coordinates identifying respective locations of a plurality of second bounding boxes, the plurality of second bounding boxes identify ing a plurality of objects in a current image of the sample; and register the current image to the initial image based on the first set of bounding box coordinates and the second set of bounding box coordinates.

[0006] It should be understood that the brief description above is provided to introduce in simplified form a selection of concepts that arc further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is definedDocket No. ARL24304PCTuniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] FIG. 1 is a diagram of a computing device;

[0008] FIG. 2 is a flow chart illustrating a method for tracking objects across multiple image frames via image registration;

[0009] FIG. 3 is a flow chart illustrating an example method for image registration using bounding boxes;

[0010] FIG. 4 is an example field of view (FOV) image of a sample including a plurality’ of cells after processing to place bounding boxes and identify a first tag for each cell;

[0011] FIG. 5 is an example FOV image of the sample after processing to place bounding boxes and identify a second tag for each cell;

[0012] FIG. 6 is a flowchart illustrating an example method for performing template matching using a FOV image and a reference image to detect cells in the FOV image;

[0013] FIG. 7 is an example brightfield FOV image of a sample including a plurality of cells;

[0014] FIG. 8 is an example reference image; and

[0015] FIG. 9 is a magnified version of a portion of the FOV image of FIG. 7 with bounding boxes identifying cells overlaid on the FOV image.DETAILED DESCRIPTION

[0016] The present description relates to image registration using bounding boxes placed over detected objects in order to track object position across images. Image registration may be performed in a variety of image processing tasks, including but not limited to high-content analysis of cells. In high-content analysis, automated image analysis may be used for processing large volumes of bioimagc data. This involves steps like image normalization, segmentation, and feature extraction, often using bounding boxes to identify and track cells. Bounding boxes simplify the process by providing a clear, manageable way to localize objects, which is advantageous for analyzing cellular behaviors and drug responses. Some high-content analysis tasks may demand registration of respective objects' bounding boxes and / or polygons. For applications such as optical pooled screens, where individual cells have respective DNA or RNA barcoded annotations, each tag (e.g., nucleotide) of the barcode annotation is encoded in a “round” of chemical treatments and respective imaging. For a given “round,” a single tag can be read for each cell imaged by detecting colored spots (or the lack of colored spots) within the cell. Thus, each round results in a single tag associated with each bounding box for each cell being encoded and read. In order to read the full sequence / annotation over all rounds, each cell’s respective nucleotide tags need to be aggregated over the rounds, which means that the objects found in an image from oneDocket No. ARL24304PCTround need to be matched with respective objects in the next round. This matching of objects across images can be confounded by the fact that the cells may move from image to image. For example, the cells are typically adhered to the bottom of a multi-well plate that, after a round of imaging, is removed from the imaging system, processed with various liquid agents, then placed back into the imaging system for a next round of imaging, frequently with a new relative position that may differ by 100-150pm in one or more dimensions, enough to make it difficult to match bounding boxes across images. Therefore, image registration is required for successful nucleotide sequence annotations for optical pooled screens.

[0017] As explained previously, image registration typically includes registering an image to a prior image using features within the images themselves, which is a computationally expensive process that demands the availability of the full images. For example, in optical pooled screens where image registration is used to aggregate encoded nucleotides for each cell across multiple rounds of imaging, an image may be registered to a prior image using computer vision, such as by tracking distinctive features in the images (e g., cell edges, points) or pixel intensity' values, or model-based image registration, such as via deep learning models that are trained to identify a transformation between the images. Each of these approaches demands processing of the images themselves, is sensitive to noise, and is computationally expensive.

[0018] Thus, according to embodiments disclosed herein, image registration may be performed using the coordinates of bounding boxes to register objects across images, rather than traditional image registration methods described above. Traditional methods align images based on pixel intensity patterns, for example, which can be computationally intensive and require the original images. In contrast, using bounding box coordinates for registration focuses on the spatial relationships of detected objects, allowing for several advantages, such as reduced computational complexity', reduced reliance on the original images, and increased efficiency in data handling. For example, operating on bounding box coordinates is significantly less computationally demanding than processing entire images. Further, post-processing (e.g., when image registration is performed) can occur without needing the original images, which might be unavailable or slow to load. This is particularly useful in scenarios where images are stored remotely or need to be processed in real-time. Further still, handling singledimensional bounding box data is exponentially faster and less expensive than dealing with highdimensional image data. This efficiency can lead to quicker analysis and decision-making.

[0019] Turning now to the figures, FIG. 1 shows an exemplary computing system 100, according to an embodiment. The computing system 100 includes a computing device 110, which further includes a processor 112 and a memory 114. The processor 112 may comprise one or more computational components usable for executing machine-readable instructions. For example, the processor 112 may comprise a central processing unit (CPU) or may include, for example a graphics processing unit (GPU).Docket No. ARL24304PCTThe processor 112 may be positioned within the computing device 110 or may be communicatively coupled to the computing device 110 via a suitable remote connection.

[0020] The memory 114 may comprise one or more types of computer-readable media, including volatile and / or non-volatile memory. The volatile memory may comprise, for example, random-access memory (RAM), and the non-volatile memory may comprise read-only memory (ROM). The memory 114 may include one or more hard disk drive(s) (HDDs), solid state drives (SSDs), flash memory, and the like. The memory 114 is usable to store machine-readable instructions, which may be executed by the processor 112. The memory 114 is further configured to store images 116, which may comprise digital images captured or created using a variety of techniques, including digital imaging, digital illustration, and more.

[0021] At least a portion of the images 116 may be acquired via an imager 106. The imager 106 may be one or more of a microscope (e.g., a light microscope, a fluorescence microscope), a multi-well plate imager, and another type of bioassay imager, for example. The imager 106 may include one or more light sources, including broad and / or narrow spectrum light sources. Examples of broad spectrum light sources include light sources that emit light over a wide wavelength range, such as lamps (e.g., mercury lamps, halogen lamps) that emit light spanning the ultra-violet (UV) and visible ranges. Examples of narrow spectrum light sources include light sources that emit light from a narrow wavelength range or wavelength band, such as light-emitting diodes (LEDs) and lasers. The imager 106 may further include at least one image sensor, such as a charge-coupled device (CCD), an electron multiplying CCD (EMCCD), an active pixel sensor (e.g., a complementary mctal-oxidc-scmiconductor, or CMOS, sensor), or another type of sensor that detects light in a location-specific maimer, such as in an array-based fashion. Additionally, the imager 106 may include one or more optical coupling devices (e.g., lenses and mirrors), filters, beam splitters, and the like that may be used to direct light of a desired wavelength or wavelength range to a sample being imaged and receive light transmitted by, reflected by, or emitted by (e.g., depending on the imaging modality) the sample at the image sensor(s).

[0022] The memory 114 further includes a detection module 118, which comprises machine-readable instructions that may be executed by the processor 112 to identify objects in the images 116 and place bounding boxes on the objects. The detection module 118 may utilize suitable methods for identifying cells and placing bounding boxes, such as segmentation. In some examples, as will be descried in more detail below, the detection module 118 may utilize template matching to identify cells. In such examples, the processor 112 may utilize the machine-readable instructions contained by the detection module 118 for obtaining or generating a reference image, obtaining a field of view (FOV) image that includes a plurality of objects to be detected (e.g., cells), performing template matching with the reference image on the FOV image to generate a correlation coefficient map. and outputting a total object cormt and / or bounding boxes indicative of the location of each object.Docket No. ARL24304PCT

[0023] In some examples, the memory 114 may further include a registration module 119, which comprises machine-readable instructions that may be executed by the processor 112 to register images. Image registration includes the alignment or transformation of two or more images into a common coordinate system. In some examples, an initial image may define the coordinate system, and one or more images acquired after the initial image may be registered to the initial image so that common features in the images (e.g., cells) are positioned in the same location in each image. The registration module 119 may be configured to determine pixel shifts between two images that may be applied to register the two images onto the common coordinate system. The registration module 119 may be configured to determine the pixel shifts using cross-correlation of ID arrays that represent the presence of objects in the columns and rows of the coordinates of bounding boxes placed based on the cell identification performed by the detection module 118.

[0024] In some examples, the memory 114 may further include a classification module 120, which comprises machine-readable instructions that may be executed by the processor 112 to segment and / or classify objects (e.g., cells) in the images 116. In some examples, the classification module 120 may be configured to read and aggregate barcode tags in order to generate a barcode aimotation for each cell. In some examples, additionally or alternatively, the classification module 120 may include one or more neural networks trained to classify cells based on morphology, such as whether the cells are alive or dead, mitotic phase of the cells, and / or other morphology. The one or more neural networks may be trained to use instance images of individual cells generated based on the output of the detection module as input to determine cell morphology'.

[0025] The computing system 100 further includes a user interface 102, which may comprise one or more peripherals and / or input devices, including, but not limited to, a ke board, a mouse, a touchpad, or virtually any other input device technology that is communicatively coupled to the computing device 110. The user interface 102 may enable a user to interact with the computing device 110, such as to select one or more images to evaluate, to select one or more parameters of the classification model, and so forth.

[0026] The computing system 100 further includes a display device 104, which may be configured to display outputs of the detection module 118, the registration module 119, and / or the classification module 120, display the images themselves, and / or display other features / parameters. In some examples, the display device 104 may be a liquid crystal display (LCD), a light-emitting diode display (LED), or an organic light-emitting diode display (OLED). The user may select or otherwise input parameters (e.g.. imaging protocols) via the user interface 102 based on options displayed via the display device 104.

[0027] In some examples, the computing device 110 may be implemented over a cloud or other computer netw ork. For example, the computing device 110 is shown in FIG. 1 as constituting a single entity , but it is to be understood that the computing device 110 may be distributed across multipleDocket No. ARL24304PCTdevices, such as across multiple servers. Processors of the computing device may be single core or multicore, and the programs executed thereon may be configured for parallel or distributed processing. A logic subsystem of the computing device (e.g., memory) may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud-computing configuration. For example, the images 116 may be stored in the memory 114 of the computing device 110 and accessed by detection module 118 on a remote device. Output from the detection module on the remote device may be sent back to the computing device 110 for storage in the memory 114.

[0028] As used herein, the terms “system” or “module” may include a hardware and / or software system that operates to perform one or more functions. For example, a module or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non-transitory computer readable storage medium, such as a computer memory. Alternatively, a module or system may include a hard-wired device that performs operations based on hard-wired logic of the device. Various modules or systems shown in the attached figures may represent the hardware that operates based on software or hardwired instructions, the software that directs hardware to perform the operations, or a combination thereof. For example, the detection module 118. the registration module 119, and / or the classification module 120 may be implemented using hardware, software, firmware, or a combination thereof. In some aspects, several portions of the subject matter described herein may be implemented via Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs), semiconductor devices based around a matrix of configurable logic blocks (CLBs), connected via programmable interconnects. Some aspects of the embodiments disclosed herein, in w hole or in part, can be equivalently implemented in standard integrated circuits, as one or more computer programs running on one or more computers (e.g., as one or more programs running on one or more computer systems), as one or more programs running on one or more processors (e.g., as one or more programs running on one or more microprocessors), as firmware, or as a combination thereof.

[0029] FIG. 2 is a flowchart that illustrates a method 200 for performing an optical pooled screen on a sample including a plurality of cells. Performing the optical pooled screen may include performing multiple rounds of cell tagging and imaging. In each round, cells are tagged w ith fluorescently-labeled nucleotides or other tags (e.g.. protein-based tags). The cells are then imaged. From the images, the cells are identified and demarcated w ith bounding boxes via a detection module and the cell tags are classified via a classification module. In order to aggregate the tags across the rounds, image registration is performed via a registration module. The detection module may be a non-limiting example of the detection module 118 of FIG. 1. the classification module may be a non-limiting example of the classification module 120 of FIG. 1. and the registration module may be a non-limiting example of the registration module 119 of FIG. 1. Method 200 may be carried out by a computing device, such as theDocket No. ARL24304PCTcomputing device 110 of FIG. 1 (e.g., by executing instructions stored in memory of the computing device).

[0030] Method 200 may be executed in response to a request to perform an optical pooled screen on the sample. In some examples, the request may be a manual request initiated by a user (e.g., a user may select an image and enter a request to detect objects in the image and read the tags in the image by selecting a user interface element or menu option). In other examples, the request may be an automatic request initiated in response to execution of a particular imaging protocol. For example, a user may select an imaging protocol from among a plurality of stored imaging protocols. The selected imaging protocol may dictate that object detection, tag reading, and tag aggregation be performed on all or on selected images obtained while executing the imaging protocol.

[0031] At 202, method 200 includes obtaining an initial image. The initial image may be an image of a sample including a plurality of cells that have undergone chemical treatment to apply a first round of tags to the cells. The tags may be fluorescently -tagged nucleotides that are incorporated into DNA or RNA of the cells, fluorescently -tagged amino acids that are incorporated into proteins of the cells, or another suitable tag. Thus, the initial image may be a fluorescent image taken in a single channel, a merged image of multiple channels, or another suitable type of image. In some examples, multiple versions of the initial image may be obtained, such as a brightfield version, one or more single-channel fluorescent versions, and / or a merged / multi-channel fluorescent version. The initial image may be acquired with an imager, such as imager 106 of FIG. 1.

[0032] At 204, the initial image is processed in order to place bounding boxes on the initial image (e.g., via the detection module 118) and read an initial tag of each cell / bounding box (e.g., via the classification module 120). The bounding boxes may identify each (identifiable) cell in the initial image. Each bounding box may be a rectangle of the same size that is centered on a respective cell. The bounding boxes may be placed using a suitable method, such as via a deep learning-based model that is trained to identify each cell in the initial image or using template matching. Identifying cells and placing bounding boxes using template matching is described in more detail below with respect to FIG.6. It is to be appreciated that the bounding boxes may or may not actually be superimposed on the initial image, and may be saved in memory as a set of bounding box coordinates, with each bounding box defined by the top left and bottom right coordinates (or top right and bottom left coordinates) relative to the initial image.

[0033] Processing the initial image to read the initial tag of each cell / bounding box may include, once the cells have been identified and the bounding boxes placed, analyzing each cell / image patch demarcated by a respective bounding box to determine whether that cell has a spot of a particular color or set of colors, and if so, determining the color of the spot. The identified color may be mapped back to a tag (e.g., a particular nucleotide). The tag (e.g., a particular nucleotide or no tag identified) may beDocket No. ARL24304PCTsaved in memory along with the coordinates of the bounding box (and each bounding box may be assigned a respective initial tag).

[0034] In some examples, different versions of the initial image may be processed to place the bounding boxes and then read the initial tags. For example, the bounding boxes may be placed using a brightfield image or an image acquired with a first channel (e.g., to visualize a cell stain such as DAPI), while the initial tags may be read from a merged / multi-channel image. In such examples, it is to be appreciated that the sample is not moved between acquisitions of the various versions of the initial image and thus image registration is not needed to align the various versions of the initial image.

[0035] At 206, a second image is obtained. The second image may be acquired with the imager, after the initial image is acquired, and may be similar to the initial image, but after the cells have undergone further chemical treatment to apply a second round of tags to the cells. Due to the application of the further chemical treatment to apply the second round of tags, the sample is moved between acquisition of the initial image and acquisition of the second image. When the sample is placed back into the imager to acquire the second image, the sample may be moved in one or more directions (e.g., left, right, up, down) relative to the position of the sample when the initial image was acquired.

[0036] At 208, the second image is processed to place bounding boxes and read a second tag of each cell / bounding box. The processing to place the bounding boxes on the second image and read the second tags may be the same as the processing to place the bounding boxes on the initial image and read the first tags. As such, each second tag may be saved in memory along with the coordinates of the bounding boxes (and each bounding box may be assigned a respective second tag).

[0037] At 210, the second image is registered to the initial image using the coordinates of bounding boxes of the initial image and the coordinates of the bounding boxes of the second image. Additional detail about registering the second image to the initial image using the bounding box coordinates is presented below with respect to FIG. 3. Briefly, ID arrays (of both column and row coordinates) may be generated for each image and cross-correlation on the ID arrays used to calculate pixel shifts that define the difference in position of the second image relative to the initial image. The image registration may use only the bounding box coordinates and does not utilize the initial image or the second image.

[0038] At 212, an annotation is created for each cell by adding the second tag to the corresponding initial tag. The annotation (e.g., aggregation of tags) may be created after the images are registered. As explained above, the initial tags may be saved in memory along with the bounding box coordinates of the initial image, with each bounding box assigned an initial tag. Likewise, the second tags may be saved in memory along with the bounding box coordinates of the second image, with each bounding box assigned a second tag. The coordinates of the bounding boxes of the second image may be shifted based on the pixel shift values determined during the registration of the second image to the initial image and the (shifted) bounding boxes of the second image (after registration to the initial image) mayDocket No. ARL24304PCTbe matched to respective bounding boxes of the initial image based on overlap (which is described in more detail below). In this way, at least some / most of the bounding boxes of the initial image may align with (e.g., have the same coordinates as) a respective bounding box of the second image. A bounding box of the second image that is determined to match a bounding box of the initial image may be assumed to be identifying the same cell as identified by that bounding box in the initial image, and the second tag associated with that bounding box may be aggregated with the initial tag of the corresponding bounding box of the initial image to form the annotation for that cell. Each annotation may be saved in memory with the bounding box coordinates of the initial image.

[0039] In some examples, the sample may be moved between acquisition of the initial image and acquisition of the second image by a large enough amount that one or more cells present in the initial image may not be present in the second image or vice versa. In such cases, each bounding box of the initial image may not have a corresponding bounding box of the second image. When this is detected, a notification may be output to alert an operator so that another second image can be acquired with the sample moved so that all cells in the initial image are imaged during the second round of imaging. Additionally or alternatively, if a cell identified in the initial image is determined to not be present in the second image, that cell’s annotation may be assigned an indicator that indicates the second tag is unknown (such as a dash, e.g., “-”). In some examples, the full annotation / sequence may include redundancies or be sufficiently long that the annotation is tolerant to one or a few missing tags (e.g., an annotation may still be identifiable if the annotation includes one or more “unknown” tags). As explained below, if a cell is present in the second image that is not present in the initial image, the cell may be identified as not matching a cell of the initial image and otherwise ignored (e.g., not assigned an annotation). The initial image may act as the reference image for the purposes of image registration and identification of which cells are to be processed for determining the full annotation.

[0040] At 214. method 200 determines if there are more tags to be read as part of the optical pooled screen protocol. The detennination of whether there are more tags to be read may be based on a predefined protocol (which may be specified by the operator via user input, such as a menu selection) or based on user input. If there are no more tags to be read (e.g., a desired number of tags has been read), method 200 proceeds to 216 to output the annotation for each cell. Outputting the aimotations may include displaying the annotations on a display device and / or saving the annotations in memory. Method 200 then ends.

[0041] However, if at 214 it is determined that there are more tags to be read, method 200 proceeds to 218 to obtain a next image. The next image may be acquired with the imager, after the prior image is acquired (e.g., after the second image), and may be similar to the initial image and second image, but after the cells have undergone further chemical treatment to apply a next round of tags to the cells. Due to the application of the further chemical treatment to apply the next round of tags, the sample is moved between acquisition of the prior image and acquisition of the next image. When the sample is placedDocket No. ARL24304PCTback into the imager to acquire the next image, the sample may be moved in one or more directions (e.g., left, right, up, down) relative to die position of the sample when the prior image was acquired.

[0042] At 220, the next image is processed to place bounding boxes and read a next tag of each cell / bounding box. The processing to place the bounding boxes on the next image and read the next tags may be the same as the processing to place the bounding boxes on the initial image and read the initial tags. As such, each next tag may be saved in memory along with the coordinates of the bounding boxes (and each bounding box may be assigned a respective next tag).

[0043] At 222. the next image is registered to the initial image using the coordinates of bounding boxes of the initial image and the coordinates of the bounding boxes of the next image, using the method of FIG. 3 described below. At 224, each annotation is updated by adding the corresponding next tag to that annotation. The process of obtaining a next image, processing the next image to place bounding boxes and read tags, registering the next image to the initial image, and updating the annotations may continue until it is determined that there are no more tags to be read, at which point the final annotation for each cell is output and the method ends, as explained above.

[0044] FIG. 3 is a flowchart that illustrates a method 300 for registering a current image to an initial image using bounding box coordinates via a registration module. The registration module may be a non-limiting example of the registration module 119 of FIG. 1. Method 300 may be carried out by a computing device, such as the computing device 110 of FIG. 1 (e.g., by executing instructions stored in memory of the computing device). In some examples, method 300 may be carried out as part of method 200. such as at 210 and 222.

[0045] At 302, method 300 includes obtaining a first set of bounding box coordinates of an initial image. The initial image may be the initial image described above with respect to FIG. 2. However, it is to be appreciated that the method 300 described herein for registering images may be applied on any images that have bounding boxes demarcating / identifying objects in the images, and thus the initial image may be any suitable image and the bounding boxes may be placed using any suitable method. As explained previously, the first set of bounding box coordinates may include two coordinates for each bounding box, such as a top left coordinate and a bottom right coordinate. The coordinates may be in the coordinate system of the initial image, e.g., (x, y) pixel coordinates. However, other bounding box coordinates are possible, particularly when the bounding boxes are all of equal size, such as a coordinate of the centroid of each bounding box, a coordinate for only one corner of each bounding box (e.g., the top left comer), etc. In some examples, the first set of bounding box coordinates may be obtained without obtaining the actual initial image. For example, the first set of bounding box coordinates may be retrieved from memory or obtained from a separate computing device.

[0046] At 304, method 300 includes obtaining a second set of bounding box coordinates of a current image. The current image may be the second image or one of the next images described above with respect to FIG. 2. The current image may be acquired after the initial image and may be an imageDocket No. ARL24304PCTof the same sample as the initial image, and may include bounding boxes identifying objects in the current image. Similar to the first set of bounding box coordinates, the second set of bounding box coordinates may include two coordinates for each bounding box, such as a top left coordinate and a bottom right coordinate. The coordinates may be in the coordinate system of the current image, e.g., (x, y) pixel coordinates. In some examples, the second set of bounding box coordinates may be obtained without obtaining the actual current image. For example, the second set of bounding box coordinates may be retrieved from memory or obtained from a separate computing device.

[0047] At 306. method 300 includes separating the column values and the row values for each set of bounding box coordinates. In this way. a first set of row coordinates and a first set of column coordinates are generated from the first set of bounding box coordinates and a second set of row coordinates and a second set of column coordinates are generated from the second set of bounding box coordinates.

[0048] As an example, the first set of bounding box coordinates may include the following coordinates (in x, y format, where x is the pixel column and y is the pixel row): (1,2636); (2,1476); (4,521); (6,3639); (8.1727). The first set of column coordinates may include [1, 2, 4. 6, 8] and the first set of row coordinates may include [2636, 1476, 521. 3639, 1727], The second set of bounding box coordinates may include the following coordinates (in x, y format, where x is the pixel column and y is the pixel row): (1,2581); (4,3583); (6,467); (11,1673); (12,2104). The second set of column coordinates may include [1, 4, 6, 11, 12] and the second set of row coordinates may include [2581, 3583, 467, 1673, 2104], It is to be appreciated that some images, such as images of cells, may include hundreds or even thousands of bounding boxes and thus the example bounding box coordinates presented herein are only a subset of the overall bounding box coordinates that may be obtained and processed as described. In the example presented above, the bounding box coordinates may be listed by column, such that all bounding box coordinates that are in the first column are listed first, followed by all bounding box coordinates that are in the second column, then the third column, and so forth. However, other ways of listing / ordering the bounding box coordinates are possible, such as by row, without departing from the scope of this disclosure, so long as the order of the bounding box coordinates is consistent among all images / sets of bounding box coordinates.

[0049] At 308. a ID array is created for each set of row coordinates, to generate a first ID row array (generated from the first set of row coordinates) and a second ID row array (generated from the second set of row coordinates). At 310, a ID array is created for each set of column coordinates, to generate a first ID column array (generated from the first set of column coordinates) and a second ID column array (generated from the second set of column coordinates).

[0050] The ID arrays may be binary arrays that represent the row locations and column locations, respectively, of the bounding box coordinates. For example, the first ID row array may include [1, 0, 0, 1, 1] and the second ID row array may include [1, 0. 0, 0, 1], The first ID column array may includeDocket No. ARL24304PCT[1, 1, 0, 1, 0] and the second ID column array may include [1, 0, 0, 1, 0], As one non-limiting example, the ID column arrays may be calculated according to the following:

[0051] max column = max(max(initial_columns), max(current columns))

[0052] initial image = np.zeros(max_column + 1, dtype=np.float32)

[0053] current image = np.zeros(max_column + 1, dtype=np.float32)

[0054] In the above example, “initial image” is the first ID column array and “current image” is the second ID column array. In this way. the ID column arrays may be generated by identifying the maximum number of columns in each of the sets of column coordinates, setting each ID column array to have a total number of values that is based on the maximum number, and populating each ID column array based on the respective set of column coordinates such that if a column has a coordinate, that column (in the ID column array) is given a value of one, and if the column does not have a coordinate, that column is given a value of zero. Each ID column array may thus be a “fingerprint” of the spatial distribution of the respective bounding boxes (e.g., of the initial image for the first ID column array and of the current image for the second ID column array) in the column dimension. The ID row arrays may be calculated in a similar maimer, such that the ID row arrays are generated by identifying the maximum number of rows in each of the sets of row coordinates, setting each ID row array to have a total number of values that is based on the maximum number, and populating each ID row array based on the respective set of row coordinates such that if a row has a coordinate, that row (in the ID row array) is given a value of one, and if the row does not have a coordinate, that row is given a value of zero. Each ID row array may thus be a “fingerprint” of the spatial distribution of the respective bounding boxes (e.g., of the initial image for the first ID row array and of the current image for the second ID row array) in the row dimension. However, in some examples, counting may be used instead of a binary representation. In such examples, if a row (or column) has more than one coordinate in it (e.g., two bounding boxes have a coordinate in the same row or column), the value for that row (or column) may be the total number of coordinates (e.g., tw o).

[0055] At 312, a row similarity metric is calculated. The row similarity metric may be a suitable metric that indicates a level of similarity between the ID row arrays as a function of displacement of the arrays relative to each other, that can be used to determine how much the second ID row array is shifted relative to the first ID row array. The similarity metric may be cross-correlation, convolution, cosine similarity, or another suitable metric. At 314, a column similarity metric is calculated. As an example, the similarity metrics may be cross-correlations such that cross-correlation is calculated between the first ID row array and the second ID row array, and cross-correlation is calculated between the first ID column array and the second ID column array. In a non-limiting example, the crosscorrelation may be calculated according to the following (using the column cross-correlation as an example):

[0056] correlation = np.correlate(current_image, initial image, mode=full)Docket No. ARL24304PCT

[0057] The column correlation may include, as an initial subset of the total correlation values, [1, 0. 0, 2, 1], The row correlation may include, as an initial subset of the total correlation values, [1, 1, 0, 0. 1], To calculate the column correlation, for example, the second ID column array may be “slid” across the first ID column array. At each position of the second ID column array relative to the first ID column array (e.g., at each pixel / column displacement), the integral of the product of the ID column arrays is calculated. The integral may be calculated for each position of the second ID column array relative to the first ID row array, to generate a cross-correlation value for each pixel / column shift. In this way, a cross-correlation value may be calculated for each column. To calculate the row correlation, for example, the second ID row array may be “slid” across the first ID row array. At each position of the second ID row array relative to the first ID row array (e.g.. at each pixel / row displacement), the integral of the product of the ID row arrays is calculated. The integral may be calculated for each position of the second ID row array relative to the first ID row array, to generate a cross-correlation value for each pixel / row shift. In this way. a cross-correlation value may be calculated for each row.

[0058] At 316, pixel shifts are calculated (shifts from the current image to the initial image) from the row and column similarity metrics. For example, the column cross-correlation may be used to calculate a column shift and the row cross-correlation may be used to calculate a row shift. As explained above, the column cross-correlation may include a cross-correlation value for each column. The maximum cross-correlation value may be identified, and the index of the maximum cross-correlation value determined. The index may represent the number of pixels / columns the second ID column array is shifted to equal the first ID column array. As a non-limiting example, the column shift may be calculated according to the following:

[0059] col shift = (len(initial image) - 1) - correlation, ar gmax()

[0060] Thus, the column shift may be calculated by determining the number of characters (e.g., Is) in the first ID column array (minus one) and then subtracting the correlation argmax from the number of characters. The correlation argmax is the index of the maximum value in the column correlation array. The col shift indicates how many elements the second ID column array is shifted relative to the first ID column array to achieve the maximum correlation. A positive shift means the second ID column array is shifted to the right, and a negative shift means the second ID column array is shifted to the left. The row shift may be calculated in a similar manner from the row cross-correlation (e.g., the row shift may include how many elements the second ID row array is shifted relative to the first ID row array to achieve the maximum correlation). In a ID signal, "cross-correlation" for object location detection essentially means sliding a known "template" signal along the signal of interest, calculating the similarity between the template and the underlying signal at each position, and identifying the location where the similarity is highest, which indicates the likely position of the object represented by the template; essentially finding the "best match" by shifting the template across the dataDocket No. ARL24304PCTand measuring how well it aligns at each position. Cross-correlation is a commonly used similarity measurement for this purpose.

[0061] At 318, the current image is registered to the initial image based on the pixel shifts. For example, the pixels of the current image may be shifted by the amounts specified by the row and column shifts. Additionally or alternatively, the bounding box coordinates of the current image may be shifted by the amounts specified by the row and column shifts. At 320, each bounding box in the current image is matched to a respective bounding box in the initial image based on a level of overlap. The matching of bounding boxes may be performed after image registration, such that the bounding boxes of the current image are shifted based on the pixel shifts. Matching based on overlap may include, for a first bounding box of the current image, determining an area of overlap between the first bounding box and each bounding box of the initial image and identifying the bounding box of the initial image that provides the highest area of overlap with the first bounding box. The first bounding box may then be determined to match the identified bounding box. If the first bounding box does not overlap any bounding boxes the initial image, the first bounding box may be considered not to match any bounding boxes of the initial image. The process may be repeated for each bounding box of the current image. If a bounding box of the current image does not match any bounding boxes of the initial image, the object (e.g., cell) defined / identified by that bounding box may be determined not to be present in the initial image. Method 300 then ends.

[0062] FIGS. 4 and 5 show example images of a sample including a plurality of cells that may be processed according to methods 200 and 300 in order to encode nucleotide barcode tags for the plurality of cells and aggregate the encoded tags. FIG. 4 shows an example initial image 400 after the initial image has been processed to place bounding boxes identifying the cells in the initial image and after a first round of tags have been read, as explained above with respect to FIG. 2. The initial image 400 may be a non-limiting example of the initial image described above with respect to FIG. 2. In the example shown, the bounding boxes are color-coded to indicate the detected tag for each cell, such as red indicating a first nucleotide was incorporated (e.g., T) in that cell, blue indicating a second nucleotide (e.g., C) was incorporated in that cell. etc. As one example, a first cell 402 is identified via a first (blue) bounding box. and a second cell 404 is identified with a second (green) bounding box.

[0063] FIG. 5 shows an example second image 500 after the second image has been processed to place bounding boxes identifying the cells in the second image and after a second round of tags have been read, as explained above with respect to FIG. 2. The second image 500 may be a non-limiting example of the second image described above with respect to FIG. 2. In the example shown, the bounding boxes are color-coded as explained above with respect to FIG. 4. The first cell 402 is also present in the second image 500 and is identified via a first (green) bounding box. The second image 500 is shifted relative to the initial image 400 by a large enough degree that the second cell 404 is onlyDocket No. ARL24304PCTpartially imaged in the second image 500, and a bounding box has not been placed on the second cell 404.

[0064] To aggregate the tags into annotations for each identified cell, the second image 500 is registered to the initial image 400 using only the coordinates of the bounding boxes of the initial image 400 and the coordinates of the bounding boxes of the second image 500, as explained above with respect to the method 300. Using the calculated pixel shifts, the second image 500 may be registered to the initial image 400 such that the first bounding box associated with the first cell 402 in the second image 500 may be moved and thereby align (and match) with the first bounding box associated with the first cell 402 in the initial image 400. Accordingly, the annotation for the first cell 402 may be generated by aggregating the first nucleotide (e.g.. T) and the second nucleotide (e.g.. C) as identified for the first cell 402 in the initial image 400 and the second image 500, respectively.

[0065] As appreciated from the second image 500 relative to the initial image 400. even when imaging the same sample using the same imager, the actual imaged FOV may shift enough between images that entire cells may be missing in the second image 500 that were imaged in the initial image 400. and cells may be present in the second image 500 that were not imaged in the initial image 400. Thus, it may not be possible to accurately register the second image 500 to the initial image 400 using only one or two cells / bounding boxes, as it may be impossible to determine how much one bounding box has moved between images given the number of bounding boxes and how much the bounding boxes can move betw een images. By performing the image registration method described herein, that utilizes die coordinates of all the bounding boxes, accurate image registration may be performed w ithout having to do feature-based on or pixel-intensity based image registration with the actual images. In some examples, depending on the number of objects (e.g., cells) in the images to be registered, the image registration process may be performed w ith subimages rather than the whole images. For example, each image may be divided into quarters and a selected quadrant (e.g., top left) of each image may be used for calculating the pixel shifts based on the bounding box coordinates of the two subimages (e.g., the top left quadrant of the initial image and the top left quadrant of the current image). Once the pixel shifts are calculated, the pixel shifts may be applied to register the entire current image to the initial image and match all of the bounding boxes of the initial image to bounding boxes of the current image.

[0066] As explained above, any suitable method may be used to place the bounding boxes on the images. However, in some examples, the bounding boxes may be placed using template matching. Template matching may be a computationally -efficient way to identify cells in the images, and may be performed on brightfield images or stained / fluorescent images. In the example of performing an optical pooled screen, identifying the cells and placing the bounding boxes using template matching on a brightfield image may be advantageous in that it may allow all the fluorescent channels to be used to visualize the various tags. In such examples, the bounding boxes may be placed using a brightfield version of the initial image, a brightfield version of the second image, etc.Docket No. ARL24304PCT

[0067] FIG. 6 is a flowchart that illustrates a method 600 for performing template matching on a field of view (FOV) image. The FOV image may be a suitable image of a sample having a plurality of objects, such as a plurality of cells. Method 600 may be carried out by a computing device, such as the computing device 110 of FIG. 1 (e.g., by executing instructions stored in memory of the computing device). In some examples, method 600 may be carried out as part of method 200, such as at 204 and 220 of method 200.

[0068] At 602, method 600 includes validating the FOV image and a reference image. As explained above, the FOV image may be a brightfield image of a plurality of cells. The reference image may be an image of (or representation of) a cellular structure such as an endoplasmic reticulum (ER). Validating the FOV image and the reference image may include confirming both the FOV image and reference image have been obtained and are 2D arrays. Further, in some examples, validating may include confirming that the size / resolution of the reference image matches the size / resolution of the FOV image. The reference image may be created with an expected magnification (and hence size) of the cells in the FOV image, and the reference image may be scaled (e.g., reduced or increased in size) to match the expected size of the ERs in the FOV image, if needed.

[0069] At 604, template matching on the FOV image with the reference image is performed to generate a correlation coefficient map. Template matching may utilize a 2D correlation-based method to determine how well each region of the FOV image matches the reference image, with the degree of matching represented by a correlation coefficient (e.g., Pearson’s linear correlation coefficient) for each region. For example, the reference image may be compared to a first region of the FOV image (that is the same size and shape as the reference image) at the upper-left corner of the FOV image, and a first correlation coefficient may be calculated for that region (and placed in the map at a location corresponding to the centroid pixel of the first region). The reference image may be “slid” one pixel to the right relative to the FOV image to compare the reference image to a second region of the FOV image. A second correlation coefficient may be calculated for the second region (and placed in the map at a location corresponding to the centroid pixel of the second region). The process may continue until the reference image has been slid over (and compared to) each respective region of the top of the FOV image; the reference image may then be slid down one pixel and compared to each respective region across the FOV image back toward the left; and so forth until a correlation coefficient has been calculated for each region of the FOV image.

[0070] The template matching may include Fourier domain-based correlation. As such, the FOV image and reference image are transformed to the frequency domain (e.g., using a Fourier transform) and the comparison described above takes place in the frequency domain. In the context of identifying cells in brightfield images. Fourier domain correlation offers significant advantages over traditional time domain methods. By transforming images into the Fourier domain, the correlation coefficient detection can be leveraged to perform template matching more efficiently. This approach allows forDocket No. ARL24304PCTrapid identification of cell patterns, as it focuses on frequency components rather than pixel intensity¬ variations, which can be influenced by noise and lighting conditions.

[0071] Using Fourier transforms, the correlation process becomes computationally faster, enabling real-time analysis of large datasets. This is particularly beneficial in high-throughput microscopy, where quick and accurate cell detection is crucial. In contrast, time domain methods often demand extensive preprocessing and can be slower due to the need for pixel-wise comparisons, making them less suitable for large-scale applications. Overall, Fourier-based techniques enhance both the speed and robustness of cell identification in brightfield imaging.

[0072] At 606, method 600 may include filtering and / or thresholding the correlation coefficient map to create a filtered binary map. The filtering / thresholding may include filtering by correlation coefficient threshold, local luminance threshold, and / or local contrast threshold, which may act to reduce instances of false positives and false negatives. For example, for correlation coefficient threshold filtering, each correlation coefficient may be compared to a correlation coefficient threshold and the correlation coefficients that are greater than the correlation coefficient threshold may be identified as matching / passing and represented as a value of one on the map. The correlation coefficients that are less than the correlation coefficient threshold may be represented as a value of zero on the map. For local luminance and / or local contrast thresholding, a similar process may be applied. For example, a mean luminance of each region of the FOV image determined to match the reference image (e.g., the regions assigned a value of one based on the correlation coefficient) may be calculated and compared to a luminance threshold. If the mean luminance is above the luminance threshold, the region is confirmed as matching the reference image and the value of one is maintained; if the mean luminance is not above the threshold, the region is not confirmed as matching the reference image and the value is changed to zero. A similar process may be applied for local contrast thresholding using a contrast threshold. Each threshold may be a minimum allowable value. The thresholds may be selected manually via example image audits or selected automatically by comparing benchmarks of bounding box counts and locations with respective bounding boxes found using nuclear seeding (e.g., nuclear stain detected in a fluorescent channel, which provides high contrast for easy detection, location, and counting).

[0073] At 608. method 600 includes performing local maxima detection in the filtered binary map. The local maxima detection may identify pixels of the filtered binary map that have a value higher than all of their immediate neighboring pixels. Each pixel / local maximum identified may be designated as an initial detected object (e.g., ER / nucleus). At 610, initial bounding boxes are calculated around the initial detected objects. The bounding boxes may be centered on the identified pixel and have a shape that matches the reference image and a size based on an expected size of the detected objects (e.g.. an average or expected cell size).

[0074] At 612. the bounding boxes are filtered by a bounding box overlap threshold. The bounding box overlap threshold may be a percentage of overlap betw een neighboring bounding boxes,Docket No. ARL24304PCTsuch as 50% overlap or 75% overlap. If neighboring bounding boxes overlap by more than the bounding box overlap threshold, one of the bounding boxes may be removed or a bounding box may not be placed to begin with. For example, bounding boxes are considered to be added for local maxima points. When a bounding box is in the process of being added, the bounding box is checked relative to the locations of any existing bounding boxes. If the bounding box to be added overlaps an existing bounding box by more than the bound box overlap threshold, the bounding box is not added. The bounding boxes that remain may be the final bounding boxes that may be used to generate the output (e.g., cell / nucleus / ER count, the bounding boxes themselves).

[0075] At 614, method 600 determines if mask processing is to be performed. Mask processing may be performed when it is desired to segment the detected objects in order to do morphology analysis or another desired task on the detected objects. If mask processing is to be performed, method 600 proceeds to 616 to perform mask processing. The mask processing may include normalizing and thresholding a mask for each detected object and optionally removing speckle noise. Each mask may be a binary image where each pixel is classified as part of a detected object or not, with pixels identified as part of the detected object given a value of 1 and pixels that are not part of the detected object given a value of 0. The mask processing may be performed with a suitable model, such as a neural network. The bounding boxes placed as described above may be initial bounding boxes that are centered on the initial detected object (e.g., cellular structure such as the ER). The initial bounding boxes may be used to divide the FOV image into sub-images / image patches and each sub-image / image patch may be processed via the mask processing. In some examples, the mask processing may include using masks (e.g., nuclear or cellular masks) to identify the nucleus or cell in each sub-image / image patch. One or more or all of the initial bounding boxes may be adjusted in position so that the final bounding boxes are re-centered over the nucleus or cell. By using sub-images instead of the entire FOV image or tiles of the FOV image, the generation of the masks may be performed faster and with smaller neural networks that utilize less training data. Further, the template matching described herein is much more efficient than using a neural network to find cell (via ER) instances.

[0076] At 618, the output is generated. As mentioned above, the output may include a total cell / nucleus / ER count, the final bounding boxes, and / or an image that includes the final bounding boxes overlaid on the FOV image. If mask processing is perfonned, the output may include the binary image / mask for each detected object / sub-image. Method 600 then ends.

[0077] FIG. 7 illustrates an example brightfield FOV image 700. FOV image 700 may be an example of an image of images 116 obtained with imager 106. FOV image 700 may include a plurality of cells, such as human epithelial cells. Due to the use of broad spectrum light and lack of cell stain, the cells are relatively transparent other than at the edges of the cell membranes (where the index of refraction causes light to reflect or not based on the critical angle). The FOV image 700 may include hundreds or thousands of cells.Docket No. ARL24304PCT

[0078] In order to detect each cell in the FOV image 700, template matching using a reference image, such as the reference image shown in FIG. 8, may be performed, which may result in bounding boxes being placed over each identified cell in the FOV image 700. A region 702 of FOV image 700 is shown in magnified view in FIG. 9 including bounding boxes placed based on the template matching.

[0079] FIG. 8 shows an example reference image 800 that may be used to perform template matching to identify areas of an input image (e.g., FOV image 700) that match the reference image 800. The reference image 800 may be an image of an ER or may approximate an average or expected ER. The ER may be used as the cell structure for matching via the template matching because ERs are present in most eukaryotic cells, ERs are associated with cell nuclei and so are a proxy for the nuclei, and ERs are visible in brightfield images. Further, ERs consistently modulate the brightfield image across cells. The mean intensify is less and the size and shape of the ER is somewhat low variance compared to the cell shape and size. Said another way, ERs are visible, can be differentiated from other cell structures visible in brightfield images, and have low variance in size and shape. While each ER in each cell may not be symmetrical, the reference image 800 may be symmetrical, which may avoid orientation-specific effects.

[0080] As explained above, after transformation to the frequency domain, the reference image 800 may be “slid” across the FOV image 700 and the similarity between the reference image 800 and the underlying FOV image 700 at the region where the reference image is located may be calculated and represented as a correlation coefficient for each position of the reference image 800. The correlation coefficients may be represented as map of the same size / pixel resolution of the FOV image 700, with each pixel of the map corresponding to a location of the reference image 800 relative to the FOV image 700 and having value that represents the correlation coefficient calculated for that location. After thresholding and / or filtering to reduce false negatives and false positives, a bounding box may be placed at each local maxima of the map. and the bounding boxes may be overlaid on the FOV image 700, as shown in FIG. 9. In some examples, the bounding boxes that are ultimately output (e.g., and overlaid on the FOV image) may be adjusted based on nuclear masks to more accurate identify each cell.

[0081] FIG. 9 shows a bounding box image 900 that includes the region 702 of the FOV image 700 with bounding boxes overlaid thereon, such as bounding box 902. The region 702 in image 900 is magnified relative to the FOV image 700 to better visualize individual cells. When the bounding boxes are overlaid on the FOV image, the bounding boxes may include rectangles that are visible on the FOV image. However, it is to be appreciated that the bounding boxes may be represented as pixel coordinates of the FOV image that identify the location and size of each bounding box. For example, each bounding box may be represented as the coordinates of the FOV image for the top left corner and bottom right comer of the bounding box (or the top right comer and bottom left comer).

[0082] It is to be appreciated that other cellular structures could be used as the basis for the reference image, as long as the cellular structure is present in each cell and has high contrast. OtherDocket No. ARL24304PCTexample cellular structures that could be used include portions of the cell membrane, for example. Depending on the cell type and / or other parameters defining the FOV image, other consistent aspects of the cells may be present that can be used to identify the cells. However, the ER is a readily -apparent organelle of commonly -used cell types in cell-based assays for cell-based therapies such as drug discovery, gene therapy, etc. Further, while the template matching disclosed herein is particularly advantageous for use on brightfield FOV images, the template matching may be performed on FOV images of samples that include stained cells, so long as the reference image represents a cellular structure as it would appear when stained.

[0083] The technical effect of registering a current image to an initial image using bounding box coordinates is that the current image and the initial image may be registered without having to process the current image and the initial image, using less computational resources and providing faster registration results.

[0084] In another representation, a system includes one or more processors configured to: obtain one or more initial images of a sample including a first subset of a plurality of cells; process the one or more initial images to place a plurality of first bounding boxes, the plurality of first bounding boxes identifying the first subset of the plurality of cells; process the one or more initial images to read a respective first tag of each cell of the first subset of the plurality of cells and store each first tag in the memory; obtain one or more current images of the sample including a second subset of the plurality of cells; process the one or more current images to place a plurality of second bounding boxes, the plurality’ of second bounding boxes identifying the second subset of the plurality of cells; process the one or more current images to read a respective second tag of each cell of the second subset of the plurality of cells; register the plurality of second bounding boxes to the plurality of first bounding boxes based on a first set of bounding box coordinates and a second set of bounding box coordinates, the first set of bounding coordinates identifying respective locations of the plurality of first bounding boxes, the second set of bounding coordinates identify ing respective locations of the plurality of second bounding boxes; create an annotation for each cell by aggregating each second tag with a corresponding first tag identified based on the registering; and output each annotation for display and / or save each annotation in the memory. In a first example of the system, the one or more initial images includes an initial brightfield image of the sample including the first subset of a plurality of cells and an initial fluorescence image of the sample including the first subset of a plurality of cells, wherein processing the one or more initial images to place the plurality of first bounding boxes comprises processing the initial brightfield image to place the plurality of first bounding boxes using template matching of a reference image to the initial brightfield image, and wherein processing the one or more initial images to read the respective first tag of each cell of the first subset of the plurality of cells comprises processing the initial fluorescence image to read the respective first tag of each cell of the first subset of the plurality of cells. In a second example of the system, optionally include the first example, the one or more current images includes a current brightfield image of the sample including the second subset of a plurality of cells and a currentDocket No. ARL24304PCTfluorescence image of the sample including the second subset of a plurality of cells, wherein processing the one or more current images to place the plurality of second bounding boxes comprises processing the current brightfield image to place the plurality of second bounding boxes using template matching of a reference image to the current brightfield image, and wherein processing the one or more current images to read the respective second tag of each cell of the second subset of the plurality of cells comprises processing the current fluorescence image to read the respective second tag of each cell of the second subset of the plurality of cells. In a third example of the system, optionally including one or both of the first and second examples, registering the plurality of second bounding boxes to the plurality of first bounding boxes based on the first set of bounding box coordinates and the second set of bounding box coordinates comprises: calculating a row shift and a column shift based on the first set of bounding box coordinates and the second set of bounding box coordinates; shifting the plurality of second bounding boxes to the plurality of first bounding boxes by shifting the plurality of second bounding boxes by the row shift and column shift; and matching each of the plurality of second bounding boxes to a respective one of the plurality of first bounding boxes to determine which respective first tag corresponds to each second tag.

[0085] The following claims particularly point out certain combinations and sub-combinations regarded as novel and non-obvious. These claims may refer to “an” element or “a first” element or the equivalent thereof. Such claims should be understood to include incorporation of one or more such elements, neither requiring nor excluding two or more such elements. Other combinations and subcombinations of the disclosed features, functions, elements, and / or properties may be claimed through amendment of the present claims or through presentation of new claims in this or a related application. Such claims, whether broader, narrower, equal, or different in scope to the original claims, also are regarded as included within the subject matter of the present disclosure.

Claims

Docket No. ARL24304PCTCLAIMS1. A system, comprising:one or more processors configured to:obtain a first set of bounding box coordinates, the first set of bounding coordinates identifying respective locations of a plurality of first bounding boxes, the plurality of first bounding boxes identifying a plurality of objects in an initial image of a sample;obtain a second set of bounding box coordinates, the second set of bounding coordinates identifying respective locations of a plurality of second bounding boxes, the plurality of second bounding boxes identifying a plurality of objects in a current image of the sample; and register the current image to the initial image based on the first set of bounding box coordinates and the second set of bounding box coordinates.

2. The system of claim 1, wherein registering the current image to the initial image comprises shifting the respective locations of the plurality of second bounding boxes to be in a coordinate system of the initial image, the shifting based on similarity metrics of the first set of bounding box coordinates and the second set of bounding box coordinates.

3. The system of claim 1 or 2, wherein the registering comprises:separating column and row values for the first set of bounding box coordinates and the second set of bounding box coordinates to generate a first set of row coordinates and a first set of column coordinates from the first set of bounding box coordinates, and a second set of row coordinates and a second set of column coordinates from the second set of bounding box coordinates;creating a first ID row array from the first set of row coordinates, a second ID row array from the second set of row coordinates, a first ID column array from the first set of column coordinates, and a second ID column array from the first set of column coordinates;determining a row shift based on a first similarity metric calculated between the first ID row array and the second ID row array;determining a column shift based on a second similarity metric calculated between the first ID column array and the second ID column array; andshifting the respective locations of the plurality of second bounding boxes based on the row shift and the column shift.

4. The system of claim 2 or 3. wherein the similarity metric is cross-correlation.

5. The system of any one of claims 1-4, wherein the plurality of first bounding boxes is placed relative to the initial image using template matching of a reference image to the initial image and theDocket No. ARL24304PCTplurality of second bounding boxes is placed relative to the current image using template matching of the reference image to the current image.

6. The system of any one of claims 1-5, wherein the registering is performed only with the first set of bounding box coordinates and the second set of bounding box coordinates and not with the initial image or the current image.

7. The system of any one of claims 1-6, wherein the sample includes a plurality of cells, the plurality' of objects in the initial image includes a first subset of the plurality of cells, the plurality of objects in the current image includes a second subset of the plurality of cells, and a portion of cells in the first subset are also in the second subset.

8. A method, comprising:obtaining a first set of bounding box coordinates, the first set of bounding coordinates identifying respective locations of a plurality of first bounding boxes, the plurality of first bounding boxes identifying a plurality of objects in an initial image of a sample;obtaining a second set of bounding box coordinates, the second set of bounding coordinates identifying respective locations of a plurality of second bounding boxes, the plurality of second bounding boxes identifying a plurality of objects in a current image of the sample; and registering the current image to the initial image based on the first set of bounding box coordinates and the second set of bounding box coordinates.

9. The method of claim 8, wherein registering the current image to the initial image based on the first set of bounding box coordinates and the second set of bounding box coordinates comprises registering the current image to the initial image based on a row pixel shift and a column pixel shift calculated from respective similarity metrics of the second set of bounding box coordinates relative to the first set of bounding box coordinates.

10. The method of claim 9 or 10, wherein the registering comprises creating a first ID row array and a first ID column array from the first set of bounding box coordinates and a second ID row array and a second ID column array from the second set of bounding box coordinates, the first ID row array representing a spatial distribution of the first set of bounding box coordinates in a row dimension, the first ID column array representing a spatial distribution of the first set of bounding box coordinates in a column dimension, the second ID row array representing a spatial distribution of the second set of bounding box coordinates in the row dimension, the second ID column array representing a spatial distribution of the second set of bounding box coordinates in the column dimension.Docket No. ARL24304PCT11. The method of claim 10, wherein the registering comprises calculating row cross-correlation values for each position of the second ID row array relative to the first ID row array and identifying the row pixel shift based on the row cross-correlation values and calculating column cross-correlation values for each position of the second ID column array relative to the first ID column array and identifying the column pixel shift based on the column cross-correlation values.

12. The method of any one of claims 8-11, wherein the registering is performed only with the first set of bounding box coordinates and the second set of bounding box coordinates and not with the initial image or the current image.

13. The method of any one of claims 8-12, wherein the sample includes a plurality of cells, the plurality of objects in the initial image includes a first subset of the plurality of cells, the plurality of objects in the current image includes a second subset of the plurality of cells, and a portion of cells in the first subset are also in the second subset.

14. The method of any one of claims 8-13, wherein the plurality of first bounding boxes is placed relative to the initial image using template matching of a reference image to the initial image and the plurality of second bounding boxes is placed relative to the current image using template matching of the reference image to the current image.

15. The method of claim 14, wherein the reference image is an image of an endoplasmic reticulum.

Images