Three-dimensional cellular segmentation

Abstract

A system implements techniques for three-dimensional (3D) cellular segmentation by providing a 3D input image to a 3D cellular segmentation engine, for identifying cells in 3D images, and to a 2D cellular segmentation engine, for identifying cells in 2D images. The system receives 3D cell labels as an output of the 3D cellular segmentation engine and 2D cell labels from the 2D cellular segmentation engine for each of Z planes of the 3D input image. The system identifies overlaps between the 3D and 2D cell labels, clusters the 3D cell labels into 3D cell groups based on the identified overlaps between the 3D cell labels and the 2D cell labels and maps each of the 3D cell groups with 2D cell labels, based on the identified overlap, before updating the 2D cell labels based on the mapping and generating a labeled 3D image based on the updated labels.

Description

PATENT Docket No. 240613A-008PCTTHREE-DIMENSIONAL CELLULAR SEGMENTATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of the filing date of provisional U.S. Patent Application No. 63 / 760,299, entitled “THREE-DIMENSIONAL CELLULAR SEGMENTATION’' and filed on February 19, 2025, the entire contents of which is hereby expressly incorporated herein by reference.BACKGROUND

[0002] Many biological applications require identification of cells, membranes and / or nuclei from microscopic images. Identifying cells or cellular structures from images is often referred to as cellular segmentation or cell segmentation. The process of cell segmentation commonly involves delineating the boundaries of cells by distinguishing a cell from its background or from other cells. Cell segmentation in densely packed tissue samples such as samples having multiple layers is of utmost importance in many biological applications. For example, cell segmentation is a core component of spatial transcriptomics, which is a technique that combines gene expression profiling with spatial information from tissue samples and allows researchers to map and analyze the location of individual molecules within tissue sections, providing insights into how gene expression varies across different regions of the tissue. Accuracy of cell segmentation in such techniques is very important to the results provided and any subsequent analysis performed via these techniques. However, accurate cell segmentation in tissue samples is challenging and often requires use of Three-dimensional (3D) images. 3D cell segmentation is the segmentation of individual cells with volumetric data in samples with one or more layers of cells, such as various tissue sections. While 3D cell segmentation techniques exist, current mechanisms suffer from many shortcomings. For example, current techniques often require high sampling rates along all 3 axes of 3D images, including the Z axis. This limits the usage of these techniques for systems with limited optical resolution along the Z axis. Furthermore, the use of the currently available mechanisms is often restricted due to practical constraints on scanning and / or imaging speed, data storage limitations, processing capabilities and the like. Moreover, many current mechanisms result in high error rates for cells with elongated shapes or long processes, even with high resolution images. Hence, there is a need for improved systems and methods of 3D cell segmentation.PATENT Docket No. 240613A-008PCTBRIEF DESCRIPTION OF THE DRAWINGS

[0003] The drawing figures depict one or more implementations in accord with the present teachings, by way of example only, not by way of limitation. In the figures, like reference numerals refer to the same or similar elements. Furthermore, it should be understood that the drawings are not necessarily to scale.

[0004] FIGs. 1A-1B depict examples of results generated by prior art cell segmentation techniques.

[0005] FIG. 2 is a diagram of an example system in which the techniques for 3D cell segmentation disclosed herein are implemented.

[0006] FIG. 3A depicts an example heatmap displaying the results of overlap (ratio of intersection area over the smaller area of a pair) between 2D cell labels of one z plane and 3D cell labels identified in the same z stack.

[0007] FIG. 3B depicts an example heatmap displaying the results of overlap in identified cells between 2D and 3D segmentation results across multiple Z planes of a 3D image.

[0008] FIG. 3C depicts an example heatmap displaying the results of correlations between cells identified via 3D segmentation based on their overlapping profiles with respect to the 2D segmentation results of the same z stack.

[0009] FIG. 3D depicts an example mapping result between identified 2D cells and identified 3D groups, where the 3D groups are generated by clustering the 3D cell labels based on their pairwise correlation with each other as shown in FIG. 3C.

[0010] FIG. 4A depicts an example of a comparison between results of prior art cell segmentation techniques (2ndrow for 2D and 3rdrow for 3D cell segmentation) and results of the 3D cell segmentation solution disclosed herein (4throw).

[0011] FIG. 4B depicts an example of a result from the 3D cell segmentation solution disclosed herein.

[0012] FIG. 5 is a flow chart of an example process for 3D cellular segmentation according to the techniques disclosed herein.

[0013] FIG. 6 is a block diagram showing an example software architecture, various portions of which may be used in conjunction with various hardware architectures herein described, which may implement any of the described features.

[0014] FIG. 7 is a block diagram showing components of an example machine configured to read instructions from a machine-readable medium and perform any of the features described herein.PATENT Docket No. 240613A-008PCTDETAILED DESCRIPTION

[0015] Systems and methods for an improved 3D cell segmentation mechanism are disclosed. These techniques provide a technical solution to the technical problems associated with the current 3D segmentation techniques in accurately performing cell segmentation for 3D images including the ones with limited z axis information (e.g., limited resolution and / or sampling rate). The technical solution improves the performance of 3D segmentation for irregular cells. For example, currently available mechanisms do not provide optimal results for images with limited optical resolution, resulting in limited usage of these techniques for images with limited optical resolution along the Z axis. Furthermore, current mechanism may suffer from limited sampling rates which could arise from a requirement of faster turnaround time and / or systems with limited data storage and / or processing capabilities.

[0016] FIGs. 1A-1B depict examples of results generated by prior art cell segmentation techniques. The images in FIGs. 1A-1B are images of a biological sample. The images 102, 104, 106, 108 and 110 of FIG. 1A are two-dimensional (2D) images of the same sample in different Z planes. Thus, the images in the first row (i.e., images 102, 104, 106, 108 and 110) are morphological images taken at different Z positions. For example, image 102 is a 2D image of the sample in a Z plane at the Z position of 1 micrometer, image 104 is the 2D image of the sample in the Z plane at the Z position of 2 micrometer, image 106 is the 2D image of the sample in the Z plane at the Z position of 3 micrometer, and so on. As depicted, each of the images 102, 104. 106, 108 and capture different subsets of the 3D sample at different volumetric planes at different Z positions. The images 112 (i.e. images in the second row) represent 2D segmentation results (e.g., labeled images) for each of the images 102, 104, 106, 108 and 110 using a known 2D segmentation technique. The colors in the images 112 represent the cell identification (ID) based on stitching 2D cell segmentation outcomes, such that labels in adjacent Z planes with significant intersection over union (loU) areas with each other are assigned with the same cell ID (e g., same color). As depicted, the images 112 shows that 2D cell segmentation can identify some cells accurately in the 2D space (XY direction) within each z plane separately. However, identifying the cells correctly in the 3D dimension requires correlating the identified cells in the Z direction. Linking the identified cells across the z planes is error prone, particularly for highly irregular cells. The linking is achieved in some prior art algorithms by stitching the 2D images 112 together in the Z direction based on some heuristic rules such as thresholding on ratio of intersection area over union (loU) or centroid-to-centroid distance of 2D labels in adjacent z planes. The result is the 3D image 120 depicted in FIG. IB. While depicted as a 2D image, the image 120 includesPATENT Docket No. 240613A-008PCTa depth aspect and can be examined in a 3D space by visualizing a cross-section view such as the side bar 122 or botom bar 124. By using the side bar 122, a user can view the depth aspect in the YZ direction. The botom bar 124 can be used to view the depth aspect in the XZ direction. While the stitched image 120 displays some correct cell segmentations in the XY direction, it suffers from a large number of cell spliting errors in the Z direction. One example is shown as the pale-color-highlighted cell at the center of cross-hair in image 120 where the cell’s footprint in the XZ and YZ direction is limited and does not cover the entire cell volume correctly as indicated by the underlying morphological staining in the side bars 122 and 124.

[0017] An alternative mechanism for 3D segmentation of the images 102, 104, 106, 108 and 110 is to use existing 3D segmentation algorithms. The images 114 (i. e. , images in the 3rdrow of FIG. 1 A) display the resulting cell segmentation labels from a 3D implementation of a machine-learning cell segmentation model at the same Z plane as images 102, 104, 106, 108 and 110. The image 130 of FIG. IB displays the corresponding 3D segmentation result in the 3D space. Thus, the images 114 of the 3rdrow depict the 3D cell segmentation of the whole z-stack using prior art mechanisms. The direction arrows in FIG. IB demonstrate the direction of the different views in the image 120 and image 130. Each of the image 120 and 130 contains three parts with top left, top right and botom parts showing the XY, YZ and XZ views of the 3D z-stack, respectively. While depicted as a 2D image, the image 130 includes a depth aspect and can be examined in a 3D space by utilizing a cross-section view such as the side bar 132 or the botom bar 134. The image 130 achieves better continuity of cell identification in the Z direction. However, it results in high cell spliting errors and small object errors in the XY direction. Thus, there exists a technical problem of inaccurate identification of cells in the 3D space in existing 3D cell segmentation technologies.

[0018] To address these technical problems and more, in an example, this description provides technical solutions for improved systems and methods of 3D cell segmentation that achieve more accurate results. The improved systems and methods include performing both a 2D cell segmentation and a 3D cell segmentation on the same sample, matching the two sets of results and then stitching the 2D results along the Z axis based on the best matched 3D cell segmentation labels. This may involve performing a 2D cell segmentation for each Z plane of the 3D image while also performing a 3D cell segmentation on the 3D image stack. The results of the tw o cell segmentations are then compared to calculate a maximum area ratio of intersection over individual cell pairs within each cell label pair between the results from 3D and. 2D cell segmentations. The 3D cell identifications are then grouped based on thePATENT Docket No. 240613A-008PCTmatching results across the 2D cell identifications, using a clustering technique. The group identifications of the 3D cell labels are then mapped to the 2D cell labels generated by the 2D segmentation to correctly identify cells that belong to the same 3D group. Thus, the provided 3D cell segmentation mechanism utilizes 3D implementation of machine-learning models to guide the Z continuity of cells identified via the 2D cell segmentation technique. This results in cell segmentations that are more accurate across the 3D space and more robust against the sampling rate along the Z direction. Moreover, the resulting mechanism produces accurate results for thin tissue samples with limited information in the Z axis and / or high anisotropic ratios (e.g., large Z step sizes).

[0019] The technical solutions improve current cell segmentation systems by increasing the accuracy and efficiency of identifying cells in 3D spaces. These and other technical benefits of the techniques disclosed herein will be evident from the discussion of the example implementations that follow. It should be noted that while the solutions disclosed herein are discussed in relation to anisotropic samples, the solutions can also be applied to 3D segmentation of any samples.

[0020] FIG. 2 is a diagram of an example system 200 in which the techniques for 3D cell segmentation disclosed herein are implemented. The example system 200 includes an imaging instrument 204, a target sample 202, and a 3D cell segmentation system 208. The example system 200 shows one possible configuration of an environment that may be used to implement the techniques disclosed herein. Other implementations may include additional components instead of or in addition to one or more of the components shown in the example implementation of FIG. 2.

[0021] The imaging instrument 204 is a device configured to capture 3D images of objects. In an example, the imaging instrument 204 is an imaging device, such as a 3D digital microscope, used to capture images of various samples such as biomolecular objects. For example, the imaging instrument 204 may be a 3D X-ray microscope. In an example, the imaging instrument 204 captures multiple sets of magnified images from a range of viewing angles which are then collated and digitally constructed into a 3D data set to generate 3D image stacks. Processing of the multiple images to generate a 3D image stack of the target sample 202 may take place within the imaging instrument 204 or it may be done in a different device to generate the 3D image.

[0022] The target sample 202 may be a biological or biomedical sample. For example, the target sample can include a tissue sample, including a tissue sample taken from a subject. A target sample can include fresh-frozen (FF) tissue, such as a fresh-frozen biopsyPATENT Docket No. 240613A-008PCTsample or a surgical sample taken from a subject. A target object can include formalin-fixed paraffin-embedded (FFPE) tissue, such as a biopsy sample or a surgical sample. In some situations, a tissue can be stored for some time (months or years) prior to use. A target sample can include a section taken from a sample and positioned on a microscope slide using standard techniques in histochemistry and pathology laboratories. A target sample can include cultured cells, such as a cell culture or a cell culture pellet. Cultured cells can be FF or FFPE preserved, depending on the embodiment. In some implementations, a target sample includes a fresh cell culture. In an example, the target sample 202 is an anisotropic sample. In some implementation, the target biological sample is stained to visualize morphological features such as nuclei, membrane, cytoplasm, and the like, using techniques such as histochemical staining, immuno-fluorescent staining, or Hematoxylin and Eosin staining.

[0023] Once images of the target sample 202 are taken via the imaging instrument 204 and turned into a 3D image stack, a z-stack of input images 206 is transmitted to the 3D cell segmentation system 208 for processing. In some implementations, the 3D cell segmentation system 208 is part of the imaging instrument 204. In other implementations, the 3D cell segmentation system 208 operates on one or more servers or other computing devices that are communicatively connected to the imaging instrument 204. Moreover, in alternative implementations, the input images 206 may be provided by a user. As such, in some implementations, the 3D cell segmentation system 208 is not connected to the imaging instrument 206. The input images 206 may be transmitted from a user and / or the imaging instrument 204 via a network. In some implementations, a user utilizes a user interface screen to submit the input images.

[0024] In some implementations, the 3D cell segmentation sy stem 208 includes a 3D segmentation engine 210, 2D segmentation engine 212, overlap identification engine 218, correlation engine 220, clustering engine 222, mapping engine 224, and labeling engine 226. It should be noted that while each of the 3D segmentation engine 210, 2D segmentation engine 212, overlap identification engine 218, correlation engine 220, clustering engine 222, mapping engine 224, and labeling engine 226 are shown as being part of the 3D cell segmentation system 208, one or more of these elements may be separate from the 3D segmentation system 208. For example, as discussed further below, the 3D segmentation engine 210 and / or 2D segmentation engine 212 may include standard segmentation models that are used by the 3D cell segmentation system 208, and as such may be included in the system or be separate elements.PATENT Docket No. 240613A-008PCT

[0025] The z-stack of input images 206 is provided to the 3D segmentation engine 210 and the 2D segmentation engine 212. The 3D segmentation engine 210 may include a machine learning model (e.g., a deep learning model) that has been trained to examine 3D microscopic images of biological samples to identify cells in the 3D space of the image. In an example, the 3D segmentation model is the ilastik model. In other implementations, the 3D segmentation model is a locally trained segmentation model which is trained to segment one or more types of images for a specific or a variety of types of target samples. In an example, a locally trained segmentation model uses a pretrained model which is finetuned and improved based on one or more datasets. In other implementations, the 3D segmentation engine 210 may use or include a machine learning model trained to perform 2D cell segmentations. In such an implementation, the 3D segmentation engine 210 may extend the application of the 2D segmentation model to 3D cell segmentation using a volumetric approach where the 2D model is applied to each 2D slice of a 3D image stack along all its three 2D view s (i.e., XY, XZ, and YZ). In such an implementation, the 2D model prediction results are integrated to create a coherent model prediction in the 3D space and then the 3D predictions are converted into segmented cells in the 3D space. In an example, the 2D segmentation model is the Cellpose model, which is a deep learning-based segmentation model that can segment cells from a w ide range of images. Other examples of 2D segmentation models that can be used include but are not limited to StarDist, Mesmer, U-Net3, and Mask R-CNN. In some implementations, the 3D segmentation engine 210 utilizes two or more segmentation models.

[0026] The 3D segmentation engine 210 receives the Z-stack of 3D input images 206 and analyzes the images to identify- cells within the 3D images. The Z-stack of input images 206 are a stack of 2D images and as such include multiple 2D images in the Z direction that together create a volumetric image. The 3D segmentation engine 210 analyzes each of the images in the Z direction and takes the Z direction into account to generate cell identification labels for the objects identified in the input images 206. In some implementations, the 3D input images 206 are downsampled by or before the 3D segmentation engine 210, for example, as part of a preprocessing mechanism. In some examples, the 2D images (XY direction) are downsampled such that the resolution of the image is within a predetermined range of the step size in the Z direction (e.g., in a range of 1 / 3 to 3 of the step size). In an example, the down sampling is such that the resolution of the image is approximately close to the size of the step in the Z direction (e.g., if the Z step is 1 micrometer, the resolution in the XY direction is reduced to 1 micrometer). In one implementation, the sampling rate of the input image is adjusted across one or more of the X, Y and Z axis (e.g., all axes) to generatePATENT Docket No. 240613A-008PCTan anisotropic value in the range of 0.3 to 3 before the image is provided to a segmentation engine. In an example, the parameters used for the 3D segmentation engine 210 are a bin equivalent to 4 and anisotropy of 0.8 um / 0.48 urn. The parameters may be predetermined or adjustable. In an example, a user is able to adjust the parameters before performing cell segmentation. In response to receiving the input images 206, the 3D segmentation engine 210 generates a set of 3D cell identification (ID) labels for the image.

[0027] In addition to providing the input images 206 to the 3D segmentation engine 210, the Z-stack of the input images 206 is also transmitted to 2D segmentation engine 212 for processing. In some implementations, the input images are preprocessed separately for each of the 3D cellular segmentation engine 210 and 2D cellular segmentation engine 212 to provide an optimized version of the input images to each of the 3D cellular segmentation engine and 2D cellular segmentation engine. For example, the sampling rate, resolution, etc. of the images provided to each of the 3D and 2D segmentation engines may be different and optimized for each model. In some examples, preprocessing includes downsampling or upsampling the input image. The 2D segmentation engine 212 may include one or more machine learning image segmentation models such as the 2D segmentation model(s) 214. At least one of the 2D segmentation model(s) 214 may be a model that is trained for cell segmentation of 2D images. In an example, the 2D segmentation model 214 is a locally trained image segmentation machine learning model. For example, a pretrained image segmentation machine learning model may be locally trained and finetuned using a training dataset of specific type of images and cells (e.g., microscopic image of biological cells). Examples of 2D cell segmentation models that can be used include but are not limited to Cellpose, StarDist and Mesmer. The 2D cell segmentation models may include models used for nuclei segmentation, whole cell segmentation or multi-modality segmentation (e.g. multiple input channels). When more than one 2D segmentation model is used, each model may be used on the same sample at given Z planes and the results from the multiple models are transmitted to the combination engine 216, which harmonizes and combines multiple sets of results into one set to get a more robust 2D cell segmentation result. It should be noted that the use of multiple 2D segmentation models and combination engine 216 is optional and only one 2D segmentation model may be required to achieve the required results.

[0028] The 2D segmentation model performs a segmentation for each of the images in the Z plane. This results in 2D cell ID labels for each of the images in the Z plane of the input images 206. In some implementations, labels for out-of-focus objects at each of the Z planes are removed to improve the results. The 2D results can then be stitched in the Z directionPATENT Docket No. 240613A-008PCTbased on some heuristic rules. In some examples, the 2D cell labels in adjacent Z planes with an area ratio of intersection over union that is larger than a given threshold (e.g., larger than 0.7) are assigned with the same cell ID. In some examples, the 2D cell labels in adjacent Z planes with a centroid-to-centroid distance that is smaller than a certain threshold (e.g. smaller than 1 micrometer, or smaller than 0.4 times of smaller diameter among the cell labels under comparison) are assigned with the same cell ID. This may require calculating the intersection of area over union for the labels in the Z direction and use of a stitching algorithm. This may also require calculating the centroid-to-centroid distance for labels in the Z direction and comparing the distance to the local diameter of cell labels. The stitching may be done by a stitching algorithm (not shown).

[0029] The stitching algorithm may be a mechanism that stitches images together along the Z axis and may be referred to as a Z-stitching algorithm. The Z-stitching algorithm stitches the images labeled with the same ID labels together along the Z axis. In an example, the stitching is done based on thresholding the centroid distance of nuclei segmentation results (e.g., a distance of smaller or equivalent to 0.5 micrometer or a distance of 0.4 times smaller than the diameter of a given pair of nuclei).

[0030] In some implementations, the 2D input image is downsampled before or by the 2D segmentation engine 212 to match the optimal diameter range of the cell segmentation model for more accurate results. In some examples, the resolution used for 2D segmentation is higher than the resolution used for 3D segmentation. In an example, the parameters used for 2D segmentation are a bin equivalent to 2 and a resolution of 0.24 um per XY pixel. The parameters may be predetermined or adjustable. In an example, a user is able to adjust the parameters before performing cell segmentation.

[0031] In some implementations, the 2D segmentation model(s) 214 include a nuclei segmentation model and the input images 206 are also provided to the nuclei segmentation model to perform 2D nuclei segmentation on each 2D image in the Z plane of the input images 206. The nuclei segmentation model may be a deep learning machine model designed for cell segmentation with parameters that are specifically set for nucleus identification. In other implementations, the nuclei segmentation model may be a machine learning model that is specifically trained for identification of nuclei in microscopic images. The nuclei segmentation model may be a pretrained or known model or may be trained locally. The nuclei segmentation model analyzes the Z plane images of the input image 206 to identify nucleus in the images and then stitches the Z images (e.g., Z-stitching the labeled images) along the Z direction. To achieve this, the 2D segmentation engine 212 may include or makePATENT Docket No. 240613A-008PCTuse of a nuclei segmentation model as well as a Z-stitching algorithm. The Z-stitching algorithm may operate as discussed above to stitch the images. This results in identification of nucleus in the 3D input images in the form of nucleus ID labels in the Z plane.

[0032] When multiple 2D segmentation models (e.g., a 2D whole cell segmentation model and a nuclei segmentation model or 2 whole cell segmentation models) are used, the results from the multiple models (e.g., 2D segmentation model and the nuclei segmentation model) are provided to the combination engine 216. which maps the nucleus ID labels to the 2D whole-cell ID labels. This is achieved, in one implementation, by calculating the intersection of union and the intersection of area between 2 sets of labels on the same image (e.g., the 2D whole-cell ID labels and the nucleus ID labels in each Z plane). To achieve this, the combination engine 216 may include an intersection of union engine and an intersection of area engine. The intersection of area (loA) engine may be a model or algorithm that calculates the area ratio of the intersected region of a pair of cell IDs over the smaller area within the pair. The intersection of union (loU) may be an algorithm that calculates the area ratio for the intersected region over the union of between a pair of cell IDs from the results from the 2D segmentation model and the nuclei segmentation model. In some implementations, the results of the loA and loU calculations are used to find cell IDs that fall within a desired threshold (e.g., loU of larger than 0.4 or loA of larger than 0.9 indicates the cell IDs should be combined). Other mechanisms of combining the results of whole cell segmentation and nuclei segmentation are used in other implementations.

[0033] The resulting output of the combination engine 216 helps identify areas of the cells that include nuclei and / or areas that contain cytoplasm. This not only increases the robustness of cell segmentation for biological samples with various degrees of staining quality for either nuclei or plasma membranes or cytoplasm, but also provides additional information about the subcellular structure of the identified cells such as which parts of an identified cell contain the nuclei and which parts contain cytoplasm. In some implementations, where subcellular segmentation is not needed or desired, the combination engine 216 is not utilized. Instead, the output of the 2D segmentation engine 212 is generated from one 2D segmentation model and directly provided to the overlap identification engine 218. It should be noted that the specific 2D segmentation model(s) 214 used are not limited to those discussed here. Other 2D segmentation model(s) may be used. Furthermore, the 2D segmentation engine 212 may include additional elements based on the number and / or type of models used. The solution disclosed herein can make use of various 2D and 3D segmentation engines to achieve the desired results.PATENT Docket No. 240613A-008PCT

[0034] Once the 2D and 3D segmentations are complete, the outputs of the 2D segmentation engine 212 and the 3D segmentation engine 210 are provided to the overlap identification engine 218 to identify overlaps between the 2D and 3D results. This involves identifying the intersection regions between each cell pair in each Z plane (e.g., each Z image). The intersection is examined over a smaller area within the cell pair and the ratio of intersection of the cell areas between corresponding Z planes is calculated. In some implementations, the result of this overlap identification is used to generate a table or graph such as a heatmap.

[0035] FIG. 3A depicts an example heatmap displaying the results of overlap between 2D cell labels and 3D cell labels in one Z plane. Each row 302 of the table 300A corresponds to a cell ID label from the 2D segmentation results. Thus, the label Zl_225 corresponds to a label assigned to a cell identified in a plane designated as the Z1 plane in the 2D segmentation output. Similarly, the remaining labels Zl_237, Zl_238, Zl_242 and the like each correspond to one identified cell in plane Z1 by the 2D segmentation engine. Each column 304 in table 300A represents a label assigned to an identified cell in the 3D segmentation output. Thus, the labels 3D_354. 3D 370. 3D_377 and the like are the labels assigned to each of the identified cells in the whole z-stack image in the 3D segmentation output. The values depicted in each row represent the ratio of the intersection over smaller area of the corresponding cells. Thus, the value of 1 depicted in column 2, row 2 of the table 300A describes the intersection between cell Zl_225 identified in the 2D segmentation output and cell 3D_354 identified in the 3D segmentation output. The value of 1 shows that these two cells identified in the two different segmentation mechanisms have one cell completely covered by the other cell, which signals that these two labels are from the same actual cell in 3D. When the value is 0. that demonstrates that there is no overlap between a cell identified in the 2D segmentation output and the cell identified in the 3D segmentation output. For example, cells Zl_225 and 3D_370 do not appear to have any overlap. Thus, the overlap results could be used as surrogate markers to the likelihood of a cell identified in the 2D segmentation being the same as a cell identified in the 3D segmentation.

[0036] The process of overlapping the 2D and 3D segmentation results is repeated for multiple Z planes (e.g., for the entire Z stack) of the 3D input image to determine overlap in each Z plane. As the overlap is calculated, a more complete picture of the overlap between the 2D and 3D segmentation results can be generated, in some implementations.

[0037] FIG. 3B depicts an example heatmap displaying the results of the overlap in the identified cells between the 2D and 3D segmentation results across multiple Z planes of a 3DPATENT Docket No. 240613A-008PCTimage. Similar to the table 300A of FIG. 3A, table 300B of FIG. 3B displays correspondence between cells identified in the 2D segmentation output and those identified in the 3D segmentation out, but extends the results across the multiple Z planes of the 3D image to generate a more comprehensive picture of overlap between the cell labels identified by 2D cell segmentation engine and 3D cell segmentation engine. This results in a more detailed heatmap. Thus, each row 310 of the table 300B corresponds to a cell ID label from the 2D segmentation results for one given Z plane. For example, the label Z2_237 corresponds to a label assigned to a cell identified in a plane designated as the Z2 plane in the 2D segmentation output. Each column 314 in table 300B represents a label assigned to an identified cell in the 3D segmentation output across all the Z planes. In some implementations, identified 3D cells with minimal or no 2D cell intersection across all z planes are identified as noise and removed from the data fed into the downstream correlation engine 220.

[0038] The output of the overlap identification engine 218, which is the calculated overlap ratios, is provided to the correlation engine 220 for further processing. The correlation engine 220 receives the overlap data and uses the data to identify correlations between cells identified via the 3D segmentation engine (i.e., between the 3D ID labels). This is achieved by examining the overlap profiles of 3D cell results (e.g., each identified 3D ID label), determining how each identified 3D cell interacts with the identified 2D cells from the 2D segmentation engine and comparing that interaction to other 3D cells to determine which other 3D cells have similar profiles (e.g., similar interactions with the same set of 2D cells) to identify grouping of 3D cells. This is because if two identified 3D cell labels interact the same way or in a similar manner with a specific set of identified 2D cell labels, then the two 3D cell labels are likely to belong to the same actual cell in the 3D physical space and thus would be grouped together by the downstream clustering engine 222. It should be noted that the terms cell, cell result, cell ID label and cell label are used interchangeably in this disclosure to refer to a cell or cell identification parameter identified via a segmentation technique.

[0039] In some implementations, the correlation engine 220 generates a pairwise correlation matrix among the 3D ID labels. This may be achieved by utilizing a function that calculates pairwise correlations between pairs of data. A pairwise correlation matrix is a statistical tool that displays correlations between every' pair of numeric variables in a dataset. By generating a correlation matrix among the 3D ID labels, the correlation engine 220 provides an overview of correlations between the identified 3D cells, thus enabling morePATENT Docket No. 240613A-008PCTaccurate identification of actual cells. In some implementations, a correlation network is used. The result of the correlation can be displayed in a table or used to generate a heatmap.

[0040] FIG. 3C depicts an example heatmap displaying the results of correlations between cell labels identified via the 3D segmentation engine as indicated by their overlap profiles against 2D segmentation results. Each row 322 and column 324 of the table 300C corresponds to a cell ID label from the 3D segmentation results. Thus, both the rows and columns of the table 300C represent cells identified by the 3D segmentation engine and the values in each cell of the table is the calculated correlation value between corresponding identified 3D cells. For example, examining the second row of the table 300C, the label 3D_354 corresponds to a label assigned to a cell identified in the 3D segmentation output. The same label 3D 354 is also included in the columns along with other identified 3D cell labels, with labels 3D_370, 3D_371, 3D_377 to 3D_446. The values depicted in each field of the second row represent the correlation between corresponding cells, as measured by the overlap profiles between each 3D cell label and all 2D cell labels across all Z planes. Thus, the value of 1 depicted in row 2 and column 2 of the table 300C illustrates that correlation value between cell 3D_354 and 3D 354 is 1 (i.e., 100%). which is accurate as these are comparisons between exactly the same cell label. The value of -0.0294 between cells 3D_354 and 3D_370, however, shows that cell label 3D_354 and 3D_370 have a small correlation with each other and thus have low likelihood to arise from same actual cell.

[0041] The correlation values in table 300C help identity different 3D cell labels that may belong to the same group. In an example, correlation values between different 3D cell labels that are higher than a certain threshold value (e.g., equal to or higher than 0.5) are visually identified by a visual cue (e.g., same color such as red). This helps identify 3D cell labels that should be grouped together. For example, the value of 1 depicted in row 5 and column 6 of table 300C illustrates the correlation value between cells labeled as 3D_377 and 3D_378, which indicates these two 3D cell labels have the same or highly similar overlap profiles with respect to the 2D cell labels and thus likely belong to the same actual cells in 3D. Similarly, the value of 0.9999 depicted in row 5 and column 7 of the table 300C shows a high considerable correlation value between cells 3D 377 and 3D 379, which indicates these two 3D cell labels likely arise from the same actual cells in 3D. The high correlation values help the clustering engine 222 group the cells together in clusters, as described in further detail below. The cluster grouping results are displayed in row 324. As depicted, the three 3D cell labels 3D_377. 3D_378 and 3D_379 which display high pairwise correlation values betweenPATENT Docket No. 240613A-008PCTeach other were assigned with same group ID, grp_4, by the downstream clustering engine 222.

[0042] Referring back to FIG. 2, the correlation matrix generated by the correlation engine 220 is provided to the clustering engine 222, which uses the correlation results to cluster the 3D cell ID labels into a plurality of groups based on the similarity' between their overlap profiles with respect to 2D cell labels across all Z planes. In some implementations, the clustering engine 222 utilizes a known clustering model and / or clustering algorithm such as the Leiden clustering algorithm or Louvian clustering algorithm. In an example, the clustering engine 222 utilizes an unsupervised clustering model to cluster the 3D cell labels. In an implementation, the clustering engine 222 converts the pairwise correlation matrix into an adjacency matrix between 3D cell labels with edge weights defined by the correlation values, and further filters the edges with correlation values below a certain threshold (e.g., correlation values smaller than 0.6) before clustering. This filtering is optional and may be performed prior to the correlations results being provided to the clustering engine 222. In an implementation, the Leiden clustering algorithm with a resolution of 1 is used to group the 3D ID labels based on the correlation values. The type of clustering algorithm, resolution and filtering step is used can vary. The result is a list of group identifications (i.e., group labels) with each group corresponding to a set of 3D ID labels (e.g., cells 3D_377, 3D_378 and 3D_379 belonging to group grp_4).

[0043] Once the 3D cell labels are clustered into multiple groups, the clustering result is provided to the mapping engine 224. The mapping engine 224 maps the group IDs to the 2D cell IDs for each Z plane, based on the intersection of area results generated by the overlap identification engine 218. In an example, this involves, for each 2D cell ID of a given Z plane, ranking the group IDs based on the absolute intersection area between the 2D cell ID and the 3D cell labels within that group. This involves comparing the intersection between the 3D cell labels represented by a given group with each 2D cell ID and ranking the 3D groups based on the sum of intersection area for the 3D group. In some implementations, each 2D cell ID in a given Z plane w ith a maximum intersection of area ratio that is lower than a given threshold (e.g., lower than 0.5) is considered a new cell that is unique to the 2D segmentation and thus is assigned with a new cell ID. In an example, if there are multiple matches, the 3D group with the largest intersection area with the 2D cell is selected as the correct match. In another example, the stitched and combined 2D cells are maintained as they are. if the ratio of the intersection of area is below a given threshold (e.g.. below 0.5). Thus, by utilizing the mapping engine 224, the 2D cell IDs are mapped to the 3D group IDs.PATENT Docket No. 240613A-008PCT

[0044] FIG. 3D depicts an example mapping result betw een identified 2D cells and identified 3D groups. The mapping results include a list of 2D cell ID labels for each Z plane in column 332 of the 300D table, with each 2D cell ID label being mapped to a corresponding group ID as displayed in column 334 of the 300D table. For example, 2D ID labels Z1 243, Z2_243 and Z_3_243 are all mapped to grp_7 of the 3D cell groups. This illustrates that the identified 2D cells Zl_243, Z2_243 and Z_3_243 likely belong to the same actual cell in the 3D physical space, which is the same cell as the 3D cell labels within grp_7 identified by the 3D segmentation.

[0045] The mapping information along with the 2D ID labels and 3D ID labels are provided to the labeling engine 226 for generating the final results. The labeling engine 226 updates the per Z plane 2D ID labels with the identified 3D group IDs. Furthermore, the labeling engine 226 may utilize the 2D cell ID labels (e.g., cell and nuclei labels) to determine the foreground and cell borders in the XY direction per Z plane. The 3D group IDs are used in this process to determine cellular continuity7across the Z planes. This provides the technical advantage of using the more accurate cell identification information from the 2D segmentation, while at the same time, accounting for cellular continuity across the Z planes.

[0046] In some implementations, the result is examined to address 2D ID labels that do not correspond to any 3D ID label based on the overlapping rules (i.e., orphan 2D labels). In some implementations, orphan 2D cell labels which have no or limited overlap with any 3D ID labels from the 3D cell segmentation engine, are stitched using heuristic rules based on the cutoff on the area ratio of intersection of union and / or nuclear centroid-to-centroid distance for 2D cell labels in adjacent Z planes. Furthermore, cells that are identified in a limited number of Z planes (e.g., only one Z plane) are removed at this stage of the process to generate more accurate results in some implementations. The results of the updated cell labels after merging and removal of cells is used to generate a labeled 3D image 228. The labeled 3D image 228 is provided by the 3D cell segmentation system 208 for usage by a user. In an example, the labeled image 228 is transmitted by the 3D cell segmentation system 208 to a user device and / or imaging instrument for display, further processing and / or review by a user on another device.

[0047] FIG. 4A depicts an example of a comparison between results of prior art cell segmentation techniques and results of the 3D cell segmentation solution disclosed herein. The images in FIG. 4A are images of a biological sample, similar to the sample depicted in FIG. 1. The images 402, 404, 406. 408 and 410 are 2D unprocessed images of the same sample in different Z planes. For example, image 402 is a 2D image of the sample in a ZPATENT Docket No. 240613A-008PCTplane at coordinate Z = 1 micrometer, image 404 is the 2D image of the sample a Z plane at coordinate Z = 2 micrometer, image 406 is the 2D image of the sample a Z plane at coordinate Z = 3 micrometer, and so on. The images 412 represent 2D segmentation labeled versions of each of the images 402, 404, 406, 408 and 410 using an existing 2D segmentation technique which segments each 2D image and then stitches the identified cell labels based on heuristic rules (e.g. threshold on intersection of union between 2D labels in adjacent Z planes). As depicted, the images 412 identify some cells accurately in the 2D space (XY direction) within each z plane separately. However, identifying the cells correctly in the 3D dimension requires correlating the identified cells in the Z direction. As discussed above, linking the identified cell labels across the z planes is error prone, particularly for highly irregular cells. While the resulting image can display some correct cell segmentations in the XY direction, it suffers from a large number of cell splitting errors in the Z direction.

[0048] The images 414 (3rdrow) display images that have been labeled with an existing 3D segmentation algorithm at the same Z -stack image as images 402, 404, 406, 408 and 410. As discussed above, while use of existing 3D segmentation algorithms results in better continuity of cell identification in the Z direction, the algorithms also generate high cell splitting errors and false negative errors in the XY direction. The images 416 represent images that have been labeled with the 3D segmentation solution disclosed herein at the same Z-stack image as images 402, 404. 406, 408 and 410. As depicted, the images 416 display more accurate cell identification in both the XY and Z directions.

[0049] FIG. 4B depicts the final result in the 3D space. Image 418 represents one Z frame of the resulting labeled image in the XY view on the top left part of FIG.4B. The top right and bottom part of FIG.4B display side views for the YZ and XZ planes of the resulting labeled image, respectively. The image can be viewed in the Z direction by utilizing the side bar 420 which enables a cross-section view and allows a viewer to move through different volumetric spaces of the image. As depicted, the resulting labeled 3D image produces continuity of cells across the Z space, while correctly identifying cells in the XY direction.

[0050] FIG. 5 is a flow chart of an example process 500 for 3D cellular segmentation according to the techniques disclosed herein. The process 500 can be implemented by the system 200 of FIG. 1 discussed in the preceding examples.

[0051] The process 500 begins by providing a 3D input image to a 3D cellular segmentation engine, at step 502. As discussed above, the 3D input image is a stack of 2D images and as such includes multiple 2D images acquired at different z planes in the Z direction that together create a volumetric image. The 3D cellular segmentation engine mayPATENT Docket No. 240613A-008PCTinclude an existing model trained for identifying cells in a 2D image or a 3D image, or it may be a locally trained and / or finetuned machine learning model which is trained for identifying cells in 2D or 3D images. After providing the 3D input image to the engine, process 500 receives a plurality of 3D cell identification labels for objects identified in the input image, from the 3D cellular segmentation engine, at step 504. Each of the 3D cell ID labels correspond to a cell the 3D cellular segmentation engine identifies as a separate cell in the 3D input image.

[0052] In addition to providing the 3D input image to the 3D cellular segmentation engine, the 3D input image is also provided to a 2D cellular segmentation engine, at step 506. As discussed above, the 2D cellular segmentation engine may include an existing model trained for identifying cells in a 2D image, or it may include a locally trained and / or finetuned machine learning model which is trained for identifying cells in 2D images. After providing the 3D input image to the engine, process 500 receives one or more 2D cell identification labels from the 2D cellular segmentation engine for each of one or more Z planes of the 3D input image, at step 508. In some implementations, the 2D cell identification labels are stitched together in the Z direction using a stitching algorithm which may make use of heuristic rules. As discussed above, the 2D segmentation engine may also include a 2D nuclei identification model and a combination engine. Thus, the 2D segmentation engine may combine multiple sets of 2D cell labels from one or more models into one set, if applicable.

[0053] Once the 3D cell identification labels and 2D cell identification labels are received, process 500 proceeds to identify overlaps between the 3D cell identification labels and the 2D cell identification labels via an overlap identification engine, at step 510. This may be achieved by calculating a ratio of intersection of area between each 2D cell identification label of the one or more Z planes and each 3D cell identification label. Next, the process 500 proceeds to identify correlations between the 3D cell identification labels, via a correlation engine, based on the identified overlaps between the 3D cell identification labels and the 2D cell identification labels, at step 512. The identified correlations are then used to cluster the 3D cell identification labels into one or more cell groups based on a similarity between 3D cell labels' overlap profiles with respect to 2D cell identification labels across a plurality ofZ planes, at step 514.

[0054] Each of the one or more cell groups are then mapped wi th one or more 2D cell identification labels for one or more Z planes, via a mapping engine based on the identified overlap between the 3D cell labels and the 2D cell labels to generate a mapped list of 2D cell labels each having a corresponding group from among the one or more cell groups, at stepPATENT Docket No. 240613A-008PCT516. Once the mapped list of 2D cell labels is generated, the mapped list is used to update the 2D cell labels based on 3D cell labels associated with the corresponding group to which each 2D cell label is mapped, at step 518. The updated labels are used to generate a labeled 3D image for the 3D input image, at step 520. The labeled 3D image would contain 2D cell labels for each Z plane along the Z axis and the values of those 2D labels would be continuous across Z planes based on the best matched 3D cell segmentation groups. Label values in the labeled 3D image uniquely define the volume taken up by the identified cells in 3D. The image may then be provided as an output to a user or another device.

[0055] The detailed examples of systems, devices, and techniques described in connection with FIGS. 1-7 are presented herein for illustration of the disclosure and its benefits. Such examples of use should not be construed to be limitations on the logical process embodiments of the disclosure, nor should variations of user interface methods from those described herein be considered outside the scope of the present disclosure. It is understood that references to displaying or presenting an item (such as, but not limited to, presenting an image on a display device, presenting audio via one or more loudspeakers, and / or vibrating a device) include issuing instructions, commands, and / or signals causing, or reasonably expected to cause, a device or system to display or present the item. In some embodiments, various features described in FIGS. 1-7 are implemented in respective modules, which may also be referred to as, and / or include, logic, components, units, and / or mechanisms. Modules may constitute either software modules (for example, code embodied on a machine-readable medium) or hardware modules.

[0056] In some examples, a hardware module may be implemented mechanically, electronically, or with any suitable combination thereof. For example, a hardware module may include dedicated circuitry or logic that is configured to perform certain operations. For example, a hardware module may include a special-purpose processor, such as a field-programmable gate array (FPGA) or an Application Specific Integrated Circuit (ASIC). A hardware module may also include programmable logic or circuitry' that is temporarily configured by software to perform certain operations and may include a portion of machine-readable medium data and / or instructions for such configuration. For example, a hardware module may include software encompassed within a programmable processor configured to execute a set of software instructions. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (for example, configured by software) may be driven by cost, time, support, and engineering considerations.PATENT Docket No. 240613A-008PCT

[0057] Accordingly, the phrase “hardware module” should be understood to encompass a tangible entity capable of performing certain operations and may be configured or arranged in a certain physical manner, be that an entity that is physically constructed, permanently configured (for example, hardwired), and / or temporarily configured (for example, programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” refers to a hardware module. Considering examples in which hardware modules are temporarily configured (for example, programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where a hardware module includes a programmable processor configured by software to become a special-purpose processor, the programmable processor may be configured as respectively different special-purpose processors (for example, including different hardware modules) at different times. Software may accordingly configure a processor or processors, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time. A hardware module implemented using one or more processors may be referred to as being “processor implemented” or “computer implemented.”

[0058] Hardware modules can provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple hardware modules exist contemporaneously, communications may be achieved through signal transmission (for example, over appropriate circuits and buses) between or among two or more of the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory devices to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output in a memory device, and another hardware module may then access the memory' device to retrieve and process the stored output.

[0059] In some examples, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. Moreover, the one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by, and / or among, multiple computers (as examples of machines including processors), with these operations being accessible via a network (for example, the Internet) and / or via one or more software interfaces (for example, an applicationPATENT Docket No. 240613A-008PCTprogram interface (API)). The performance of certain of the operations may be distributed among the processors, not only residing within a single machine, but deployed across several machines. Processors or processor-implemented modules may be in a single geographic location (for example, within a home or office environment, or a server farm), or may be distributed across multiple geographic locations.

[0060] FIG. 6 is a block diagram 600 illustrating an example software architecture 602, various portions of which may be used in conjunction with various hardware architectures herein described, which may implement any of the above-described features. FIG. 6 is anon-limiting example of a software architecture, and it vvi 11 be appreciated that many other architectures may be implemented to facilitate the functionality described herein. The software architecture 602 may execute on hardware such as a machine 700 of FIG. 7 that includes, among other things, processors 710, memory 730, and input / output (I / O) components 750. A representative hardware layer 604 is illustrated and can represent, for example, the machine 700 of FIG. 7. The representative hardware layer 604 includes a processing unit 606 and associated executable instructions 608. The executable instructions 608 represent executable instructions of the software architecture 602, including implementation of the methods, modules and so forth described herein. The hardware layer 604 also includes a memory / storage 610, which also includes the executable instructions 608 and accompanying data. The hardware layer 604 may also include other hardware modules 612. Instructions 608 held by processing unit 606 may be portions of instructions 608 held by the memory / storage 610.

[0061] The example software architecture 602 may be conceptualized as layers, each providing various functionality. For example, the software architecture 602 may include layers and components such as an operating system (OS) 614, libraries 616, frameworks / middl eware 618, applications 620, and a presentation layer 644. Operationally, the applications 620 and / or other components within the layers may invoke API calls 624 to other layers and receive corresponding results 626. The layers illustrated are representative in nature and other software architectures may include additional or different layers. For example, some mobile or special purpose operating systems may not provide the frameworks / middl eware 618.

[0062] The OS 614 may manage hardware resources and provide common services. The OS 614 may include, for example, a kernel 628, services 630, and drivers 632. The kernel 628 may act as an abstraction layer between the hardware layer 604 and other software layers. For example, the kernel 628 may be responsible for memory management, processorPATENT Docket No. 240613A-008PCTmanagement (for example, scheduling), component management, networking, security settings, and so on. The services 630 may provide other common services for the other software layers. The drivers 632 may be responsible for controlling or interfacing with the underlying hardware layer 604. For instance, the drivers 632 may include display drivers, camera drivers, memory / storage drivers, peripheral device drivers (for example, via Universal Serial Bus (USB)), network and / or wireless communication drivers, audio drivers, and so forth depending on the hardware and / or software configuration.

[0063] The libraries 61 may provide a common infrastructure that may be used by the applications 620 and / or other components and / or layers. The libraries 616 typically provide functionality for use by other software modules to perform tasks, rather than interacting directly with the OS 614. The libraries 616 may include system libraries 634 (for example, C standard library) that may provide functions such as memory allocation, string manipulation, file operations. In addition, the libraries 616 may include API libraries 636 such as media libraries (for example, supporting presentation and manipulation of image, sound, and / or video data formats), graphics libraries (for example, an OpenGL library for rendering 2D and 3D graphics on a display), database libraries (for example, SQLite or other relational database functions), and web libraries (for example, WebKit that may provide web browsing functionality). The libraries 616 may also include a wide variety of other libraries 638 to provide many functions for applications 620 and other software modules.

[0064] The frameworks / middl eware 618 provide a higher-level common infrastructure that may be used by the applications 620 and / or other software modules. For example, the frameworks / middleware 618 may provide various graphic user interface (GUI) functions, high-level resource management, or high-level location services. The frameworks / middlew are 618 may provide a broad spectrum of other APIs for applications 620 and / or other software modules.

[0065] The applications 620 include built-in applications 640 and / or third-party applications 642. Examples of built-in applications 640 may include, but are not limited to, a contacts application, a browser application, a location application, a media application, a messaging application, and / or a game application. Third-party applications 642 may include any applications developed by an entity other than the vendor of the particular platform. The applications 620 may use functions available via OS 614, libraries 616, frameworks / middlew are 618, and presentation layer 644 to create user interfaces to interact with users.PATENT Docket No. 240613A-008PCT

[0066] Some software architectures use virtual machines, as illustrated by a virtual machine 648. The virtual machine 648 provides an execution environment where applications / modules can execute as if they were executing on a hardware machine (such as the machine 700 of FIG. 7, for example). The virtual machine 648 may be hosted by a host OS (for example, OS 614) or hypervisor, and may have a virtual machine monitor 646 which manages operation of the virtual machine 648 and interoperation with the host operating system. A software architecture, which may be different from software architecture 602 outside of the virtual machine, executes within the virtual machine 648 such as an OS 650, libraries 652, frameworks 654, applications 656, and / or a presentation layer 658.

[0067] FIG. 7 is a block diagram illustrating components of an example machine 700 configured to read instructions from a machine-readable medium (for example, a machine-readable storage medium) and perform any of the features described herein. The example machine 700 is in a form of a computer system, within which instructions 716 (for example, in the form of software components) for causing the machine 700 to perform any of the features described herein may be executed. As such, the instructions 716 may be used to implement modules or components described herein. The instructions 716 cause unprogrammed and / or unconfigured machine 700 to operate as a particular machine configured to carry' out the described features. The machine 700 may be configured to operate as a standalone device or may be coupled (for example, networked) to other machines. In a networked deployment, the machine 700 may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a node in a peer-to-peer or distributed network environment. Machine 700 may be embodied as, for example, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a gaming and / or entertainment system, a smart phone, a mobile device, a wearable device (for example, a smart watch), and an Internet of Things (loT) device. Further, although only a single machine 700 is illustrated, the term “machine"’ includes a collection of machines that individually or jointly execute the instructions 716.

[0068] The machine 700 may include processors 710. memory 730, and I / O components 750, which may be communicatively coupled via, for example, a bus 702. The bus 702 may include multiple buses coupling various elements of machine 700 via various bus technologies and protocols. In an example, the processors 710 (including, for example, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), an ASIC, or a suitable combination thereof) may include one or more processors 712aPATENT Docket No. 240613A-008PCTto 712n that may execute the instructions 716 and process data. In some examples, one or more processors 710 may execute instructions provided or identified by one or more other processors 710. The term “processor” includes a multi core processor including cores that may execute instructions contemporaneously. Although FIG. 7 shows multiple processors, the machine 700 may include a single processor with a single core, a single processor with multiple cores (for example, a multicore processor), multiple processors each with a single core, multiple processors each with multiple cores, or any combination thereof. In some examples, the machine 700 may include multiple processors distributed among multiple machines.

[0069] The memory / storage 730 may include a main memory 732, a static memory 734, or other memory, and a storage unit 736, both accessible to the processors 710 such as via the bus 702. The storage unit 736 and memory 732, 734 store instructions 716 embodying any one or more of the functions described herein. The memory / storage 730 may also store temporary, intermediate, and / or long-term data for processors 710. The instructions 716 may also reside, completely or partially, within the memory 732, 734, within the storage unit 736, within at least one of the processors 710 (for example, within a command buffer or cache memory), within memory at least one of I / O components 750, or any suitable combination thereof, during execution thereof. Accordingly, the memory 732, 734, the storage unit 736, memory' in processors 710, and memory in I / O components 750 are examples of machine-readable media.

[0070] As used herein, “machine-readable medium” refers to a device able to temporarily or permanently store instructions and data that cause machine 700 to operate in a specific fashion, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical storage media, magnetic storage media and devices, cache memory, network-accessible or cloud storage, other types of storage and / or any suitable combination thereof. The term “machine-readable medium” applies to a single medium, or combination of multiple media, used to store instructions (for example, instructions 716) for execution by a machine 700 such that the instructions, when executed by one or more processors 710 of the machine 700, cause the machine 700 to perform and one or more of the features described herein. Accordingly, a “machine-readable medium” may refer to a single storage device, as well as “cloud-based” storage systems or storage networks that include multiple storage apparatus or devices. The term “machine-readable medium” excludes signals per se.PATENT Docket No. 240613A-008PCT

[0071] The I / O components 750 may include a wide variety of hardware components adapted to receive input, provide output, produce output, transmit information, exchange information, capture measurements, and so on. The specific I / O components 750 included in a particular machine will depend on the type and / or function of the machine. For example, mobile devices such as mobile phones may include a touch input device, whereas a headless server or loT device may not include such a touch input device. The particular examples of I / O components illustrated in FIG. 7 are in no way limiting, and other types of components may be included in machine 700. The grouping of I / O components 750 are merely for simplifying this discussion, and the grouping is in no w ay limiting. In various examples, the I / O components 750 may include user output components 752 and user input components 754. User output components 752 may include, for example, display components for displaying information (for example, a liquid crystal display (LCD) or a projector), acoustic components (for example, speakers), haptic components (for example, a vibratory motor or force-feedback device), and / or other signal generators. User input components 754 may include, for example, alphanumeric input components (for example, a keyboard or a touch screen), pointing components (for example, a mouse device, a touchpad, or another pointing instrument), and / or tactile input components (for example, a physical button or a touch screen that provides location and / or force of touches or touch gestures) configured for receiving various user inputs, such as user commands and / or selections.

[0072] In some examples, the I / O components 750 may include biometric components 756, motion components 758, environmental components 760, and / or position components 762, among a wide array of other physical sensor components. The biometric components 756 may include, for example, components to detect body expressions (for example, facial expressions, vocal expressions, hand or body gestures, or eye tracking), measure biosignals (for example, heart rate or brain waves), and identify a person (for example, via voice-, retina-, fingerprint-, and / or facial-based identification). The motion components 758 may include, for example, acceleration sensors (for example, an accelerometer) and rotation sensors (for example, a gyroscope). The environmental components 760 may include, for example, illumination sensors, temperature sensors, humidity sensors, pressure sensors (for example, a barometer), acoustic sensors (for example, a microphone used to detect ambient noise), proximity sensors (for example, infrared sensing of nearby objects), and / or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment. The position components 762 may include, for example, location sensors (for example, a Global Position System (GPS) receiver), altitude sensors (forPATENT Docket No. 240613A-008PCTexample, an air pressure sensor from which altitude may be derived), and / or orientation sensors (for example, magnetometers).

[0073] The I / O components 750 may include communication components 764, implementing a wide variety7of technologies operable to couple the machine 700 to network(s) 770 and / or device(s) 780 via respective communicative couplings 772 and 782. The communication components 764 may include one or more network interface components or other suitable devices to interface with the network(s) 770. The communication components 764 may include, for example, components adapted to provide wired communication, wireless communication, cellular communication, Near Field Communication (NFC), Bluetooth communication, Wi-Fi, and / or communication via other modalities. The device(s) 780 may include other machines or various peripheral devices (for example, coupled via USB).

[0074] In some examples, the communication components 764 may detect identifiers or include components adapted to detect identifiers. For example, the communication components 764 may include Radio Frequency Identification (RFID) tag readers, NFC detectors, optical sensors (for example, one- or multi-dimensional bar codes, or other optical codes), and / or acoustic detectors (for example, microphones to identify tagged audio signals). In some examples, location information may be determined based on information from the communication components 764, such as, but not limited to, geo-location via Internet Protocol (IP) address, location via Wi-Fi, cellular. NFC, Bluetooth, or other wireless station identification and / or signal triangulation.

[0075] In the preceding detailed description, numerous specific details are set forth by way of examples in order to provide a thorough understanding of the relevant teachings. However, it should be apparent that the present teachings may be practiced without such details. In other instances, w ell known methods, procedures, components, and / or circuitry have been described at a relatively high-level, without detail, in order to avoid unnecessarily obscuring aspects of the present teachings.

[0076] While various embodiments have been described, the description is intended to be exemplary, rather than limiting, and it is understood that many more embodiments and implementations are possible that are within the scope of the embodiments. Although many possible combinations of features are shown in the accompanying figures and discussed in this detailed description, many other combinations of the disclosed features are possible. Any feature of any embodiment may be used in combination with or substituted for any other feature or element in any other embodiment unless specifically restricted. Therefore, it willPATENT Docket No. 240613A-008PCTbe understood that any of the features shown and / or discussed in the present disclosure may be implemented together in any suitable combination. Accordingly, the embodiments are not to be restricted except in light of the attached claims and their equivalents. Also, various modifications and changes may be made within the scope of the attached claims.

[0077] While the foregoing has described what are considered to be the best mode and / or other examples, it is understood that various modifications may be made therein and that the subject matter disclosed herein may be implemented in various forms and examples, and that the teachings may be applied in numerous applications, only some of which have been described herein. It is intended by the following claims to claim any and all applications, modifications and variations that fall within the true scope of the present teachings.

[0078] Unless otherwise stated, all measurements, values, ratings, positions, magnitudes, sizes, and other specifications that are set forth in this specification, including in the claims that follow', are approximate, not exact. They are intended to have a reasonable range that is consistent with the functions to which they relate and with what is customary' in the art to which they pertain.

[0079] The scope of protection is limited solely by the claims that now follow. That scope is intended and should be interpreted to be as broad as is consistent with the ordinary' meaning of the language that is used in the claims w hen interpreted in light of this specification and the prosecution history that follows and to encompass all structural and functional equivalents. Notwithstanding, none of the claims are intended to embrace subject matter that fails to satisfy the requirement of Sections 101, 102, or 103 of the Patent Act, nor should they be interpreted in such a way7. Any unintended embracement of such subject matter is hereby disclaimed.

[0080] Except as stated immediately above, nothing that has been stated or illustrated is intended or should be interpreted to cause a dedication of any component, step, feature, object, benefit, advantage, or equivalent to the public, regardless of whether it is or is not recited in the claims.

[0081] It will be understood that the terms and expressions used herein have the ordinary meaning as is accorded to such terms and expressions with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein. Relational terms such as first and second and the like may be used solely to distinguish one entity7or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions. The terms “comprises,” “comprising,” or any other variation thereof, are intended to cover a non-exclusive inclusion,PATENT Docket No. 240613A-008PCTsuch that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by “a” or "an" does not, without further constraints, preclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. Furthermore, subsequent limitations referring back to “said element” or “the element” performing certain functions signifies that “said element” or “the element” alone or in combination with additional identical elements in the process, method, article, or apparatus are capable of performing all of the recited functions.

[0082] The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, it can be seen that various features are grouped together in various examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter lies in less than all features of a single disclosed example. Thus, the following claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims

PATENT Docket No. 240613A-008PCTWHAT IS CLAIMED IS:

1. A system for three-dimensional (3D) cellular segmentation comprising:a processor; anda memory storing executable instructions that, when executed, cause the processor alone or in combination with other processors to perform operations of:providing a 3D input image to a 3D cellular segmentation engine, the 3D cellular segmentation engine being configured for identifying cells in 3D images; receiving a plurality of 3D cell identification labels for objects identified in the 3D input image as an output of the 3D cellular segmentation engine;providing the 3D input image to a 2D cellular segmentation engine, the 2D cellular segmentation engine being configured for identify ing cells in 2D images; receiving, from the 2D cellular segmentation engine, one or more 2D cell identification labels for objects identified in each of one or more Z planes of the 3D input image;identifying overlaps between the 3D cell identification labels and the 2D cell identifications labels via an overlap identification engine;identifying correlations between the 3D cell identification labels, via a correlation engine, based on the identified overlaps betw een the 3D cell identification labels and the 2D cell identification labels;clustering the 3D cell identification labels into one or more 3D cell groups based on a similarity between cell overlap profiles of the 3D cell identification labels with respect to 2D cell identification labels;mapping each of the one or more 3D cell groups with one or more 2D cell identification labels, via a mapping engine, based on the identified overlap between the 3D cell identification labels and the 2D cell identification labels to generate a mapped list of 2D cell identification labels, the mapped list including a list of 2D cell identification labels each being associated with a corresponding group from among the one or more 3D cell groups;updating label values of the 2D cell identification labels based on the corresponding 3D cell group to which each 2D cell label is mapped; and generating a labeled 3D image based on the updated labels.PATENT Docket No. 240613A-008PCT2. The system of claim 1, wherein the 3D input image is a user provided image or is received from an imaging instrument.

3. The system of claim 1, wherein the 3D input image is preprocessed for each of the 3D cellular segmentation engine and 2D cellular segmentation engine to provide an optimized version of the 3D input image to each of the 3D cellular segmentation engine and 2D cellular segmentation engine.

4. The system of claim 1, wherein a sampling rate for the 3D input image is adjusted across one or more of X, Y and Z axes to generate an anisotropic value that is within a predetermined range before the 3D input image is provided to the 3D cellular segmentation engine or the 2D cellular segmentation engine.

5. The system of claim 1, wherein 2D cellular segmentation engine includes at least one 2D cellular segmentation model, the at least one 2D cellular segmentation model being one of a nuclei segmentation model or a cell-body segmentation model, and the instructions further cause the processor to perform operations of:providing the 3D input image to each of the at least one 2D segmentation models and receiving from the at least one 2D segmentation model, one of one or more 2D nucleus labels for each Z plane of the 3D input image or one or more 2D cell-body identification labels, and when the 2D cellular segmentation engine includes the nuclei segmentation model and the cell-body segmentation model, , combining the one or more 2D nucleus labels for each Z plane of the 3D input image with the one or more 2D cell-body identification labels for each of one or more Z planes of the 3D input image, via a combination engine.

6. The system of claim 1, wherein identifying overlaps between the 3D cell identification labels and the 2D cell identification labels includes analyzing overlapping behavior between each of the one or more 2D cell labels in each Z plane and each of the plurality of cell labels.

7. The system of claim 1, wherein identifying correlations between the 3D cell identification labels includes determining how an identified 3D cell label interacts with an identified 2D cell label and comparing the interaction to other 3D cell labels to determine which other 3D cell labels have similar interaction profiles.PATENT Docket No. 240613A-008PCT8. The system of claim 1, wherein identifying correlations includes calculating a correlation between the 3D cell identification labels.

9. The system of claim 1, wherein generating a labeled 3D image based on the updated labels includes stitching the 2D cell identification labels for each Z plane along a Z axis based on a mapped 3D cell group label.

10. The system of claim 1 , wherein mapping each of the one or more 3D cell groups with one or more 2D cell labels includes ranking the one or more 3D cell groups based on a sum of an intersected area of any of the plurality of 3D cell labels that are in a cell group with a given 2D cell label.

11. The system of claim 10, wherein the labeled 3D image is provided as an output.

12. A method for three-dimensional (3D) cellular segmentation implemented in a system, the method comprising:providing a 3D input image to a 3D cellular segmentation engine, the 3D cellular segmentation engine being configured for identifying cells in 3D images; receiving a plurality of 3D cell identification labels as an output of the 3D cellular segmentation engine;providing the 3D input image to a 2D cellular segmentation engine, the 2D cellular segmentation engine being configured for objects identified in each of one or more Z planes of the 3D input image;receiving one or more 2D cell identification labels from a 2D cellular segmentation model for each of one or more Z planes of the 3D input image;identifying overlaps between the 3D cell identification labels and the 2D cell identification labels;identifying correlations between the 3D cell identification labels, via a correlation engine, based on the identified overlaps between the 3D cell identification labels and the 2D cell identification labels;clustering the 3D cell labels into one or more 3D cell groups based on a similarity betw een cell overlap profiles of the 3D cell identification labels with respect to 2D cell identification labels;PATENT Docket No. 240613A-008PCTmapping each of the one or more 3D cell groups with one or more 2D cell identification labels, via a mapping engine, based on the identified overlap between the 3D cell identification labels and the 2D cell identification labels to generate a mapped list of 2D cell identification labels, the mapped list including a list of 2D cell identification labels each being associated with a corresponding group from among the one or more 3D cell groups;updating label values of 2D cell identification labels based on the corresponding 3D cell group to which each 2D cell identification label is mapped; and generating a labeled 3D image based on the updated label values.

13. The method of claim 12, wherein generating a labeled 3D image based on the updated labels includes stitching the 2D cell identification labels for each Z plane along aZ axis based on a mapped 3D cell group label.

14. The method of claim 12, further comprising providing the labeled 3D image as an output.

15. The method of claim 12, wherein 2D cellular segmentation engine includes at least one of 2D cellular segmentation model, the at least one 2D cellular segmentation model being one of a nuclei segmentation model or a cell-body segmentation model and the method further comprises:providing the 3D input image to each of the at least one 2D segmentation models and receiving from the at least one 2D segmentation model, one of one or more 2D nucleus labels for each Z plane of the 3D input image or one or more 2D cell-body identification labels, and when the 2D cellular segmentation engine includes the nuclei segmentation model and the cell-body segmentation model, combining the one or more 2D nucleus labels for each Z plane of the 3D input image with the one or more 2D cell-body identification labels for each of one or more Z planes of the 3D input image, via a combination engine.

16. The method of claim 12, wherein identifying overlaps between the 3D cell identification labels and the 2D cell identification labels includes analyzing overlapping behavior betw een each of the one or more 2D cell labels in each Z plane and each of the plurality of cell labels.

17. The method of claim 12, wherein identifying correlations includes calculating a correlation between the 3D cell identification labels.PATENT Docket No. 240613A-008PCT18. Anon-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to perform functions ofproviding a 3D input image to a 3D cellular segmentation engine, the 3D cellular segmentation engine being configured for identifying cells in 3D images; receiving a plurality of 3D cell identification labels as an output of the 3D cellular segmentation engine;providing the 3D input image to a 2D cellular segmentation engine, the 2D cellular segmentation engine being configured for identify ing cells in 2D images; receiving one or more 2D cell identification labels from the 2D cellular segmentation engine for objects identified in each of one or more Z planes of the 3D input image;identifying overlaps between the 3D cell identification labels and the 2D cell identification labels;identifying correlations between the 3D cell identification labels, via a correlation engine, based on the identified overlaps between the 3D cell identification labels and the 2D cell identification labels;clustering the 3D cell labels into one or more 3D cell groups based on a similarity between cell overlap profiles of the 3D cell identification labels with respect to 2D cell identification labels;mapping each of the one or more 3D cell groups with one or more 2D cell identification labels, via a mapping engine, based on the identified overlap between the 3D cell identification labels and the 2D cell identification labels to generate a mapped list of 2D cell labels, the mapped list including a list of 2D cell identification labels each being associated with a corresponding group from among the one or more 3D cell groups;updating label values of 2D cell identification labels based on the corresponding 3D cell group to which each 2D cell identification label is mapped; and generating a labeled 3D image based on the updated labels.

19. The non-transitory computer readable medium of claim 18, wherein mapping each of the one or more 3D cell groups with one or more 2D cell identification labels includes ranking one or more 3D cell groups based on a sum of an intersected area of any of the plurality of 3D cell labels that are in a cell group with a given 2D cell label.PATENT Docket No. 240613A-008PCT20. The non-transitory computer readable medium of claim 18. wherein generating a labeled 3D image based on the updated labels includes stitching the 2D cell identification labels for each Z plane along aZ axis based on a mapped 3D cell group label.

Images