Method for computational multiplexing of nanoscale structures

Abstract

Computer-implemented method (200) configured for multiplexing of nanoscale structures, particularly cellular organelles (50) and / or semiconductor structures, comprising the steps of: - providing (230) at least one image, particularly super-resolution microscopy (SRM-)image (60), comprising at least two (identically) labeled nanoscale structures, particularly organelles (51, 52, 53), displayed in the at least one image and / or monochromatic representations of the at least two structures displayed in the at least one image to a machine-learning model (90) configured and trained for multiplexing of nanoscale structures, particularly cellular organelles (50) and / or semiconductor structures; - extracting (240) nanoscale-structure-specific nanotextures; and - generating (270) (probabilistic) multiplexed images (70) of the at least two (identically) labeled nanoscale structures displayed in the at least one image and / or the monochromatic representations of the at least two nanoscale structures displayed in the at least one image by (probabilistic) demixing (250) of the at least two nanoscale structures in the at least one image.

Description

[0001] 03 noventive Universitat Jena; 07745 Jena, Deutschland

[0002] Method for computational multiplexing of nanoscale structures

[0003] DESCRIPTION

[0004] The invention refers to a computer-implemented method configured for multiplexing of nanoscale structures, for example cellular organelles and / or semiconductor structures.

[0005] Super-resolution imaging (SRI) is a class of techniques that can enhance the resolution of an imaging system. In optical systems this may be achieved by exploiting the properties of light, allowing for higher resolution than would be possible with conventional optics due to the diffraction limit. This technique works within the limits set by physical laws and principles of information theory. By using these limits, SRI can increase the number of details or "resolution elements" that can be recorded in an image, beyond those achievable with traditional optical techniques such as standard light microscopy. Super-resolution imaging (SRI) has been widely applied in microscopy, where it enables high-resolution observation of biological and chemical systems at atomic and molecular scales. In super-resolution microscopy (SRM), fluorescence or other types of light are used to image a sample, which can be materials like semiconductors, semiconductor structures, cells or cellular organelles, and the image can be taken using an optical system that has been modified to allow for higher resolution than would otherwise be possible with traditional optical imaging and / or (light / electron) microscopy.

[0006] Fluorescence-based super-resolution microscopy is a specific application of this concept, which uses fluorescent probes attached to biological structures or chemical molecules for nanoscale resolution imaging. Influencing these probes can allow the manipulation of their positioning within a sample, enabling precise control over where light is focused and thus the location of detail that can be imaged. By focusing on specific regions of the sample rather than diffusing light across it, fluorescence-based super-resolution microscopy can resolve details at lengths inaccessible to conventional imaging methods.This technique has been widely used in research and industry for studying biology and chemistry, respectively. For instance, it is used extensively in the study of cellular structures, where fluorescent proteins can be placed on the surface of cells and super-resolved images can provide a detailed view of their internal structure. In chemical studies, fluorescence-based super-resolution microscopy (SRM) allows researchers to examine complex molecules at molecular levels, providing deep insight into how they function in biological systems.

[0007] In summary, SRI is a powerful tool that exploits (optical) physics and information theory principles to enhance the resolution of image acquisition. By using these limits, it can generate high-resolution images even with limited light sources or detectors. The specific application of fluorescence-based SRM offers a practical way to manipulate and study molecules at molecular levels in biology and chemistry, providing deep insights into biological processes and chemical reactions respectively. The same principles apply for inorganic structures such as semiconductors.

[0008] Nevertheless, as matter of example only, the following will show the respective SRI and SRM with respect to cellular organelles.

[0009] Nevertheless, SRI could potentially resolve structures on semiconductors at nanometer scale by illuminating the sample with light and to detect fluorescence of specific markers such that atomic and molecular structures can be imaged in analogy to imaging of cellular organelles. In traditional microscopy, this might not be possible due to absorption or scattering of light by semiconductor material but also due to the diffraction limit. With SRI, however, these details can be more likely be resolved than would otherwise be possible using conventional optics.

[0010] Super-resolution microscopy (SRM) can also resolve structures on the nanometer scale that would not be visible with traditional methods. For instance, it could enable the detection of structures on the nanometer scale below the diffraction limit. This can be applied in semiconductor research where understanding these details can lead to advancements in technologies like solar cells, lasers and chip development and manufacturing.

[0011] By using SRM techniques, researchers can measure dimensions and properties with greater precision than would be possible with traditional methods. This is crucial for fine-tuning semiconductor designs and manufacturing processes.SRI has the potential to revolutionize material science and engineering research by enabling detailed understanding of complex materials like semiconductors. For example, it could provide insights into the formation of dopants in solar cells or nanostructuring in advanced transistors.

[0012] SRI / SRM can rely on image reconstruction techniques to merge multiple separate images captured over time to form a single high-resolution image. These methods can be used, for example, to correct for differences in the illumination pattern or to combine multiple images taken at different stages of a sample's development process and / or to image dynamic processes.

[0013] In conclusion, SRI holds immense potential for studying and understanding semiconductors and other materials on the nanometer scale, opening up new possibilities in fields as diverse as material science, biology, and engineering.

[0014] Fluorescence-based super-resolution microscopy (SRM) is a cornerstone technique for visualizing biomolecular structures and their interactions with nanometer resolution in fixed and living cells. Classic immunofluorescence approaches are directly applicable to SRM and have been augmented by a diverse array of alternate labeling strategies, including genetically encoded fluorophores such as GFP and its derivatives [1][2], and chemical tags for live-cell compatible organic dyes [3][4].

[0015] These advancements have enabled the imaging of virtually any cellular component, often achieving single-molecule sensitivity [5][6].

[0016] A critical goal in SRM is the ability to simultaneously image multiple cellular structures in their native context. Traditional multicolor approaches rely on spectrally distinct fluorophores to differentiate signals, a method that requires dyes with minimal spectral overlap to simplify data acquisition and analysis [7]. Beyond spectral separation, techniques such as fluorescence lifetime imaging microscopy (FLIM)

[0017] [8][9] and fluorescence polarization

[0010] have been employed to enhance multiplexing capabilities by leveraging additional spectroscopic parameters. However, these approaches are typically limited to a small number of resolvable channels due to the need for distinct fluorophore properties.

[0018] State-of-the-art SRM methods like STED

[0011] ,

[0012] SMLM

[0012] , and MINFLUX

[0013] further constrain multicolor imaging due to their reliance on specialized dyes with delicate photophysical properties. These dyes often exhibit spectral limitations,photobleaching, and stringent excitation-emission requirements, making high-fidelity multicolor acquisitions challenging in complex biological contexts. In dynamic systems, such as live-cell imaging with structured illumination microscopy (SIM), multicolor imaging is fundamentally limited by the need to alternate between channels, precluding truly simultaneous fast acquisition

[0014] .

[0019] Conventional computational methods for multiplexing have primarily relied on image segmentation to delineate overlapping structures with ever-growing impressive performances

[0015] . However, all segmentation approaches inherently prioritize dominant structures within an image, often at the expense of less prominent features. This leads to the loss of critical spatial and structural information, particularly in highly crowded cellular environments where organelles are densely packed or overlapping, i.e. single pixels inherent signal contributions of multiple structures.

[0020] Prior art citations

[0021] [1] R. Y. Tsien, "The green fluorescent protein ' Annu Rev Biochem, vol. 67, pp. 509-544, 1998.

[0022] [2] K. Nienhaus and G. Ulrich Nienhaus, "Fluorescent proteins for live-cell imaging with superresolution," Chem Soc Rev, vol. 43, no. 4, pp. 1088-1106, 2014, doi: 10.1039 / C3CS60171D.

[0023] [3] D. Saimi and Z. Chen, "Chemical tags and beyond: Live-cell protein labeling technologies for modern optical imaging," Smart Molecules, vol. 1, no. 2, p. e20230002, Sep. 2023, doi: https: / / doi.org / 10.1002 / smo.20230002.

[0024] [4] J. B. Grimm and L. D. Lavis, "Caveat fluorophore: an insiders' guide to small-molecule fluorescent labels," Nat Methods, vol. 19, no. 2, pp. 149-158, 2022, doi: 10.1038 / s41592-021-01338-6.

[0025] [5] X. Chen, X. Wang, F. Huang, and D. Ma, "Multicolor single-molecule localization microscopy: review and prospect," PhotoniX, vol. 5, no. 1, p. 29, 2024, doi: 10.1186 / s43074-024-00147-2.

[0026] [6] C. Werner, M. Sauer, and C. Geis, "Super-resolving Microscopy in Neuroscience," Chem Rev, vol. 121, no. 19, pp. 11971-12015, Oct. 2021, doi: 10.1021 / acs.chemrev.0c01174.

[0027] [7] L. Nahidiazar and R. Harkes, "Multicolor Localization-Based Super Resolution Microscopy," in Multiplexed Imaging: Methods and Protocols, E. Zamir, Ed., New York, NY: Springer US, 2021, pp. 69-76. doi: 10.1007 / 978-l-0716-1593-5_5.

[0028] [8] T. Niehbrster et al., "Multi-target spectrally resolved fluorescence lifetime imaging microscopy," Nat Methods, vol. 13, no. 3, pp. 257-262, 2016, doi: 10.1038 / nmeth.3740.

[0029] [9] J. C. Thiele and others, "Confocal fluorescence-lifetime single-molecule localization microscopy," ACS Nano, vol. 14, pp. 14190-14200, 2020.

[0030]

[0010] L. Chen et al., "Advances of super-resolution fluorescence polarization microscopy and its applications in life sciences," Comput Struct Biotechnol J, vol. 18, pp. 2209-2216, 2020, doi:

[0031] https: / / doi.Org / 10.1016 / j.csbj.2020.06.038.

[0011] C. Spahn, F. Hurter, M. Glaesmann, C. Karathanasis, M. Lampe, and M. Heilemann, "Protein-Specific, Multicolor and 3D STED Imaging in Cells with DNA-Labeled Antibodies," Angewandte Chemie International Edition, vol. 58, no. 52, pp. 18835-18838, Dec. 2019, doi: https: / / doi.org / 10.1002 / anie.201910115.

[0032]

[0012] L. Andronov, R. Genthial, D. Hentsch, and B. P. Klaholz, "splitSMLM, a spectral demixing method for high-precision multi-color localization microscopy applied to nuclear pore complexes," Common Biol, vol. 5, no. 1, p. 1100, 2022, doi: 10.1038 / s42003-022-04040-l.

[0033]

[0013] J. K. Pape et al., "Multicolor 3D MINFLUX nanoscopy of mitochondrial MICOS proteins," Proceedings of the National Academy of Sciences, vol. 117, no. 34, pp. 20607-20614, Aug. 2020, doi: 10.1073 / pnas.2009364117.

[0034]

[0014] A. Markwirth et al., "Video-rate multi-color structured illumination microscopy with simultaneous real-time reconstruction," Nat Common, vol. 10, no. 1, p. 4315, 2019, doi:

[0035] 10.1038 / s41467-019-12165-x.

[0036]

[0015] A. Kirillov et al., "Segment Anything," Apr. 2023, [Online], Available: http: / / arxiv.org / abs / 2304.02643

[0037] It is an object of the invention to improve super-resolution imaging and / or superresolution microscopy (SRM).

[0038] The object of the invention is particularly solved by a method according to claim 1. The computer-implemented method can be configured for multiplexing of nanoscale structures. The nanoscale structures can particularly be at least one of cellular organelles, alternatively or additionally they can be semiconductor structures as described elsewhere herein or any other self-similar structure that can be, in principle, specifically labelled and imaged for the purpose of machine learning. The method may comprise the step of providing at least one image to a machine-learning model. The image may comprise at least two (identically) labeled nanoscale structures displayed in the at least one image. Alternatively or additionally, the image may comprise a monochromatic representations of the at least two structures displayed in the at least one image. The machine-learning model can be configured and trained for multiplexing of nanoscale structures.

[0039] The method may comprise the step of extracting nanoscale-structure-specific nanotextures.

[0040] The method may comprise the step of generating (probabilistic) multiplexed images of the at least two (identically) labeled nanoscale structures displayed in the at least one image and / or the monochromatic representations of the at least two nanoscalestructures displayed in the at least one image by (probabilistic) demixing of the at least two nanoscale structures in the at least one image.

[0041] The image may particularly be a super-resolution microscopy (SRM-)image or an electron microscopy image. The image may comprise at least two spectrally identically labeled nanostructures (in the case of SRM-images). Alternatively or additionally, the image may comprise at least two labeled nanostructures which were labeled to discriminate nanoscale structures, particularly in case of electron microscopy.

[0042] The respective method steps are described with more detail in the following as matter of example for cellular organelles as respective nanoscale structures. Nevertheless, the respective specificities apply to the more general aspect of nanoscale structures as well. Furthermore, the respective description with respect to the labeled images applies also to other types of images, for example to electron microscopy images or Raman microscopy which were labeled accordingly. Electron or Raman microscopy images may be labeled not by a fluorescence marker, but by a tag representing the pixel's belonging to a certain structure.

[0043] The respective structures to be studied and to be analyzed using the respective method particularly comprise a (certain degree) of self-similarity. The self-similarity can be comparable to that of cellular organelles. Therefore, semiconductor chip elements can also exhibit the respective amount of self-similarity. The method is particularly not limited to the use case of identifying cellular organelles, semiconductor structures or other elements and nanostructures exhibiting a similar degree of self-similarity. Training the network may rely on providing respective ground truth data of the components to be multiplexed or at least ground truth data similar "to some degree". This can be implemented using fluorescence microscopy by labeling the respective components - the nanoscale structures to be analyzed -specifically and / or individually. This allows identification of these nanoscale structures - also called nanostructures - to determine the structures also in more complex contexts and environments. In other imaging technology fields, such as electron microscopy or Raman microscopy, ground truth data can be generated by annotation for example. Alternatively, specific external labels may be utilized. This means that classification can be supervised to label the respective known structures individually and specifically. The limit of application of the method in imaging is mainly thegeneration of respective ground truth data. Where the respective ground truth data can be generated the method can be applied to. In principle, it is possible to apply the respective infrastructure to training and inferring a machine learning model when the respective ground truth can be generated. The respective hardware and method infrastructure can be used accordingly.

[0044] According to an aspect, the nanoscale structure is at least one of at least one cellular organelle or at least one semiconductor structure. Even though the following refers to the application of the method with cellular organelles as nanoscale structures the description applies to the aspect of semiconductor structures as well as to a mixture of organelle and semiconductor structures. This allows new applications in a lab on a chip setting. In most general terms, the respective method may be applied to so-called self-similar nanostructures.

[0045] The method can be applied to label-free microscopy, such as methods based on Raman spectroscopy, as well. Raman microscopy is an optical technique that particularly uses specific wavelengths of light (typically between 785 and 1200 nm, corresponding to Raman transitions) for imaging. Unlike other forms of spectroscopy or fluorescence-based super-resolution microscopy where probes are used to illuminate the sample, Raman microscopy does not in all embodiments require this additional step. Instead, it can use a system involving an excitation laser and a photodetector (a camera in some cases) at different wavelengths.

[0046] In Raman microscopy, molecules can be imaged by changing their Raman transitions, which occur when the vibrational energy levels of atoms are disrupted due to radiation. When the molecular structure is observed through an optical system that filters out these specific wavelengths, it allows for the imaging of complex structures without needing any special probes or markers.

[0047] Raman microscopy can be used as a label-free method in super-resolution imaging by using multiple overlapping images at different wavelengths and times. This approach is particularly useful in high-resolution studies where traditional methods might not yield sufficient resolution. By merging these overlapping images into one, it's possible to increase the signal-to-noise ratio of the final image and therefore obtain higher-resolution details.Moreover, Raman microscopy allows for imaging at extremely low concentrations, which can be challenging with other types of optical microscopy. For instance, in cell culture, most techniques require a colony to be visible, which is not always possible due to the high light levels emitted by cells themselves. With traditional fluorescence methods, these limitations would make it difficult to image complex biological samples at scale or with low concentrations.

[0048] In conclusion, Raman microscopy combined with label-free super-resolution imaging can offer a powerful tool for studying biological and chemical systems at the nanoscale, providing deep insights into molecular structures and processes. This method provides an efficient way of obtaining high-resolution images without needing any special probes or markers, making it widely applicable in various fields including biology, chemistry, materials science, and more.

[0049] The computer-implemented method can be configured for multiplexing of nanoscale cellular organelles. The method can comprise the step of providing at least one image, particularly a super-resolution microscopy (SRM-)image, comprising at least two (spectrally)(identically) labeled organelles displayed in the at least one (SRM-)image to a machine-learning model. Alternatively or additionally, the (SRM-)image can comprise monochromatic representations of the at least two organelles displayed in the at least one (SRM-)image. The image can be provided to a machine-learning model. The machine-learning model can be configured and trained for multiplexing of the displayed nanoscale cellular organelles.

[0050] The method can comprise the step of extracting organelle-specific nanotextures. The step of extracting organelle-specific nanotextures particularly refers to the identifying and isolating unique nanoscale features or patterns associated with specific cellular organelles present in a given microscopic image, particularly obtained through superresolution microscopy (SRM). Those textural features may be Haralick features. Haralick features particularly extract quantitative information about the texture of an image by measuring various aspects of its two-dimensional distribution of intensity values (pixels). These features capture the local characteristics of an image and provide a statistical description of its texture. They can be calculated from grayscale or colored images using mathematical operations such as co-occurrence matrices, histograms, and Gabor filters. Alternatively or additionally, the model is trained to generate certain patterns based on previous input. These patterns areapproximations of an optimal probability distribution of pixels, which are closest to the original partial structures. Therefore, the machine learning model can perform a numeric approximation method.

[0051] The computer-implemented method can comprise the step of generating probabilistic multiplexed images of the at least two (spectrally) (identically) labeled organelles displayed in the at least one (SRM-)image - alternatively or additionally, the monochromatic representations of the at least two organelles displayed in the at least one (SRM-)image - by probabilistic demixing of the at least two organelles in the at least one (SRM-)image. The images can be labeled spectrally identical. This refers to the same or similar dyes which show a comparable, particularly overlapping spectrum of emission, particularly in case of SRM-imaging. Just as matter of example may be mentioned here the spectrally identical pair of fluorescent dyes AlexaFluor647 and Cy5. Instead of spectrally identically one could also refer to spectrally overlapping, wherein the overlap is in particular at least not less than 50 %, particularly not less than 30 % of the convolutional integral.

[0052] The computer-implemented method described herein is particularly designed for multiplexing nanoscale cellular organelles using super-resolution microscopy (SRM) images. As described above, the method may also be applied for respectively labeled electron or Raman microscopy images. The same specificities apply to the latter, when compared to the SRM-images. The method particularly involves providing at least one SRM image that contains at least two (spectrally)(identically) labeled organelles to a machine-learning model. Alternatively, the SRM image can display monochromatic representations of those organelles.

[0053] The machine-learning model can be trained on these type of images to multiplex the displayed nanoscale cellular organelles. A training method particularly involves configuring and optimizing the model to accurately identify and distinguish between different organelles within the SRM and / or monochromatic images. Once the model has been trained, it can be applied to new SRM and / or monochromatic images to multiplex the organelles present in those images.

[0054] Overall, this method can leverage machine-learning techniques to enable a novel approach of nanoscale cellular organelle multiplexing in (SRM-)images, while improving the accuracy and efficiency of alternate, preexisting approaches.. By training a model on a dataset of images displaying respective individual organelles, itis possible to automatically identify and distinguish between different organelles, even when they are displayed in complex biological contexts. This can aid researchers in better understanding the structure and function of cellular organelles at the nanoscale level.

[0055] The described computer-implemented method for multiplexing nanoscale cellular organelles can include an additional step of extracting organelle-specific nanotextures from the (SRM-)images provided to the machine learning model. This step particularly involves identifying and isolating unique nanoscale features or patterns associated with each individual organelle present in the image, which can be used for accurate identification and demarcation of these structures during the multiplexing process.

[0056] The extraction of organelle-specific nanotextures allows for precise and reliable demixing and multiplexing of the various organelles within the image. It may involve the use of advanced image processing techniques, such as feature detection algorithms, to effectively isolate these unique nanoscale features from the surrounding biological context.

[0057] By incorporating this additional step of organelle-specific nanotexture extraction, the overall accuracy and efficiency of the multiplexing process can be significantly improved. It enables the machine learning model to better discern between individual organelles and accurately demix them for multiplexing them in the final output, even when they are densely packed or overlapping within the (SRM-)image. This can lead to more accurate and reliable results in studying the structure and function of these nanoscale cellular components.

[0058] The computer-implemented method can include a step of generating probabilistic multiplexed images of the at least two (spectrally)(identically) labeled (in the sense of dyed and / or colored, alternatively by supervised labeling) organelles displayed in the provided (SRM-) or monochromatic image(s). This step involves using probabilistic demixing techniques to separate and distinguish the overlapping signals from the two or more organelles within the image, resulting in a multiplexed representation that preserves the spatial information of each individual organelle.

[0059] Probabilistic demixing is particularly an advanced signal processing technique that allows to effectively disentangle mixed signals. In the context of nanoscale structureimaging, for example cellular organelle imaging, it allows for the separation of overlapping (fluorescence) signals from multiple nanoscale structures such as organelles in a single image, providing a more accurate representation of their individual spatial distributions (in several images, wherein each of the newly generated images represents one of the identified structures).

[0060] Alternatively or additonally, during this step, probabilistic demixing algorithms may employ various modeling approaches such as non-negative matrix factorization to effectively separate the mixed signals and generate a multiplexed image that preserves the spatial information of each organelle. The resulting multiplexed image can provide valuable insights into the structure and organization of nanoscale cellular organelles, facilitating advanced research in this field. To emphasize this further: In case of Haralick feature maps, different clustering techniques can be used to separate the respective feature maps and to get respective individualized and separated structures. As matter of example, k-means clustering can be used for separation resulting channels in respective number of k classes representing the respective number of individual and separated features.

[0061] In case of applying deep learning techniques using a machine learning model no separate method of disentangling the nanoscale structure representations in the images has to be performed as the model itself performs a mapping from at least one image comprising k structures to totally k images, wherein each image comprises one identified structure. Therefore, the demixing can function as a respective image generation from a single image to multiple images wherein in the latter the respective structures are mapped to the individual images, particularly a single structure for each image.

[0062] Overall, by incorporating probabilistic demixing techniques, the method enables accurate and reliable generation of multiplexed images that preserve the spatial information of individual structures such as organelles, even when they are densely packed or overlapping within the provided (SRM- or monochromatic) image(s). This can lead to more comprehensive understanding of the complex cellular structures and their interactions.

[0063] Multiplexing can refer to multicolor imaging of a biological sample. Herein, the term particularly refers to artificially dying different structures which belong to different organelles of a cell in different colors. In the context of nanoscale cellular organelleimaging, multiplexing refers to the process of simultaneously visualizing multiple organelles within a single microscopic image in these different colors.

[0064] According to an aspect the machine-learning model can be at least one of a convolutional neural network, a deep neural network and / or a U-Net.

[0065] The machine-learning model can be configured as at least one of a convolutional neural network (CNN), a deep neural network (DNN), or a U-Net for the purposes of multiplexing nanoscale cellular organelles in super-resolution microscopy images. A Convolutional Neural Network (CNN) is a type of machine learning model that uses convolutional layers to learn and extract features from data, such as images. It is particularly effective at processing and analyzing data with spatial structures, like images. CNNs particularly comprise any combination of an input layer, one or more convolutional layers, pooling layers, one or more (fully) connected layers and an output layer. CNNs particularly have the ability to capture local patterns and hierarchical features effectively.

[0066] A Deep Neural Network (DNN) is a particular type of machine learning model with multiple hidden layers between the input layer and output layer. It is particularly designed to learn complex representations of data by capturing higher-level abstractions through the depth of the network. DNNs can have many layers, such as convolutional layers, pooling layers, fully connected layers, and other specialized layers like recurrent or temporal convolutional layers. They can be used in tasks like image classification and object detection.

[0067] A U-Net is a specific type of DNN architecture particularly designed for feature detection and reconstruction tasks, such as those involving nanoscale cellular organelles in super-resolution microscopy (SRM) images. It particularly combines encoder (convolutional) and decoder (up-sampling) pathways to enable feature extraction and precise localization during the feature extraction process. The left arm of the U-Net particularly consists of down-sampling convolutional layers, while the right arm particularly comprises up-sampling transposed convolutional layers for precise reconstruction. U-Net can use information flow efficiently, can show high accuracy and fast training times.

[0068] The network architecture can be based on a U-Net architecture, particularly optimized for feature extraction and demixing.A U-Net can be a neural network architecture that can rely on data augmentation as described elsewhere herein. The name ”U-Net” particularly reflects its U-shaped structure, which consists of two main components:

[0069] A contracting path can function as an encoder. It can be responsible for extracting feature information by reducing the spatial dimensions step by step while increasing the depth of feature representations. It may also be referred to as a left-arm downsampling path.

[0070] A symmetric expanding path can function as a decoder. It can reconstruct the segmentation map with precise localization by restoring spatial resolution. It may also be referred to as a right-arm upsampling path. With respect to the following, an embodiment is shown giving respective dimensions and values which are chosen such that the respective method can be executed. Nevertheless, the method is not to be limited to these respective values as also other dimensions and values may be applied without leaving the respective scope of the method as described herein. The network can be adapted accordingly to allow other nanoscale structures to be identified and to be demixed.

[0071] The network may comprise the left-arm downsampling path. The network may comprise the right-arm upsampling path, particularly with transposed convolution. The network may comprise a left-arm downsampling path with 2 x 2 max-pooling layers and a right-arm upsampling path with transposed convolution. Each convolutional block particularly consists of two sub-blocks comprising zero-padded 3 x 3 convolutions, rectified linear unit (ReLU) activations, and batch normalization. The network can be configured to process 32 patches per input batch (32 x 256 x 256). The respective patches may be of dimension 256 x 256 pixels. Both arms can have the same convolutional passes, which can be configured as described.

[0072] Feature maps can be generated as projections which can be generated by modelspecific and model-internal transformations of the input data.

[0073] It is also to be mention that information can be passed between the same levels of the left and right arms as based on the definition of a UNet. Additionally the network can ultimately be configured as one sees fit. For the described embodiments only, the parameters mentioned here were used.A demixing block may be configured to generate the number of output channels corresponding to the predicted structures. A 1 x 1 convolutional layer in a demixing block can generate the number of output channels corresponding to the predicted structures, particularly three output channels for three predicted structures (3 x 256 x 256).

[0074] Linear regression particularly serves as a final activation function, ensuring pixel-level intensity predictions directly being mapped to single-structure outputs. Unlike semantic segmentation, the respective machine-learning model can reconstruct individual organelles without relying on classification, making it invariant to overlapping structures and capable of preserving gray-level shifts.

[0075] In an exemplary embodiment, the network can be trained for 4,000 epochs with augmented patches (ground truth: 32 x 3 x 256 x 256 per batch; allows input: 32 x 1 x 256 x 256). A so-called Adam optimizer can be used, particularly with the following parameters: y = 10A(-5),

[0076]

[0077] = 0.9, (32= 0.99, and e = 10A(-8). Loss can be calculated using L2 (mean squared error) as a cost function (also called loss function). Model weights can be saved based on the validation set performance, particularly evaluated at regular intervals.

[0078] According to an aspect a loss function (also called cost function), particularly mean squared error, can be used during training the machine-learning model.

[0079] During the training of a neural network, to measure the difference between the model’s prediction and true values (ground truth), at least one loss function can be used. This loss function may guide the optimization process by minimizing the error and improving the model’s predictions. Various loss functions can be tailored to specific applications.

[0080] Mean Squared Error (MSE) is particularly a loss functions that can be used in deep learning in regression tasks. The closer it is to zero, the closer are the predicted image (model output) and the true image intensities (ground truth). The MSE can be computed by averaging over the squared differences between predicted and true values:

[0081] 1 x-'n

[0082] MSE = - > (y, - y()2Here, n is the number of data, ytis the true value, ytis what the model has predicted and yt - ytis the error for each data point. By squaring this difference, it is made sure that negative and positive values can be treated equally and are not cancelling each other out. Then, by dividing the summation of all squared values by the number of data points, the average of squared error can be calculated. Herein, images can be compared to each other, rather than single valued data points. Therefore, the formula of MSE for grayscale images can be as follows:

[0083]

[0084] Here, is the true intensity value for pixel at position (i, j) and Ipred .i>isthe intensity value at the same position but in the predicted image. Squared differences are summed over height and width (m, n) of both images. Then an average is calculated to get MSE for the grayscale image.

[0085] According to an aspect the method comprises the step of synthesizing visual representations of underlying structures of the at least two organelles via linear regression.

[0086] The computer-implemented method for multiplexing nanoscale cellular organelles particularly involves a step where synthesized visual representations of the underlying structures of at least two organelles can be generated using linear regression techniques. In this process, the machine-learning model trained on SRM images or monochromatic representations of the organelles is employed to generate an output that predicts the visual appearance of the structures.

[0087] Linear regression, a simple and efficient statistical method, can be used to establish a relationship between the input features (organelle labels or other relevant data) and the corresponding visual representation of the underlying structures. By training the model on a dataset of labeled (in the sense of dyed) SRM images or monochromatic representations, it learns to map the input features to the desired output, i.e., synthesized visualizations of the organelles' structures.

[0088] During this step, the machine-learning model particularly uses its learned parameters to predict the pixel values in the output image based on the input features. This allows for the generation of realistic and accurate visual representations of the nanoscale cellular organelles, taking into account any spatial interactions or overlapsbetween them. These synthesized images can be used for further analysis, such as identifying specific structures or understanding the overall organization and function of these organelles within cells.

[0089] According to an aspect the machine-learning model was trained on isolated channels to generate feature-specific channels ("feature dreams") corresponding to each of the structures of the at least two organelles to be multiplexed.

[0090] The machine-learning model used in this method can be trained on isolated channels of SRM images and / or monochromatic representations of the organelles, enabling it to generate feature-specific channels ("feature dreams") corresponding to each structure of the at least two organelles being multiplexed.

[0091] Feature-specific channels refer to separate image channels or output components that represent specific features or characteristics of the underlying structures in the input data. In this context, they correspond to different nanoscale cellular organelles and their associated visual features. These channels are generated by the machinelearning model during the training process, where it learns to extract and isolate these distinct features from the original images or representations.

[0092] By generating feature-specific channels, the machine-learning model can effectively disentangle the mixed signals and capture the unique properties of each organelle. This allows for a more detailed and accurate understanding of the nanoscale cellular structures being analyzed. The generated feature-specific channels can be used for further analysis or visualization, such as synthesizing multiplexed images that preserve the spatial information of individual organelles while disentangling their overlapping signals.

[0093] The multiplexing can comprise a demixing as described elsewhere herein. As prerequisites high resolution images, particularly comprising 10 nanometers per pixel (nm / px) (for SMLM and MINFLUX images), can be provided. For training, images with separate structures in the respective images can be provided. For other imaging techniques (such as for example STED and SIM) bigger pixels with lower resolution may be provided.

[0094] Single-color-multi-structure images can be provided in a step for testing the training results. In this method, single-color-multi-structure images can be used as a testing step to evaluate the performance of the trained machine-learning model. Theseimages particularly consist of multiple structures representing different nanoscale cellular organelles labeled with the same color or dye. The aim is in particular to assess whether the model can accurately generate feature-specific channels corresponding to each structure, effectively demixing and thus disentangling the overlapping signals present in these images.

[0095] The single-color-multi-structure images are provided as input to the machine-learning model, which has been trained on isolated channels for features from (similar) SRM images or monochromatic representations of the organelles. The model then particularly processes these test images and generates the corresponding featurespecific channels, effectively predicting the underlying structure information for each labeled organelle in the image.

[0096] By testing the model on single-color-multi-structure images, it is possible to determine how well the trained machine-learning model generalizes to new, unseen data. If the model can accurately generate feature-specific channels that correspond to the individual structures present in these test images, it suggests that the model has learned robust and reliable features during training. This can be an important step in evaluating the performance of the model and assessing its suitability for various applications related to nanoscale cellular organelle analysis.

[0097] The training may comprise a step of data augmentation. Data augmentation particularly refers to a technique that can be used to artificially increase the size and diversity of training datasets by creating new, modified versions of existing samples. In the context of the described method for multiplexing nanoscale cellular organelles, a step of data augmentation can be included during the training phase of the machine-learning model. This involves generating additional training samples by applying various transformations or modifications to the existing SRM images or monochromatic representations of the organelles.

[0098] Data augmentation techniques may include (random) rotations, translations, and scaling of the images to simulate real-world variations in image orientation and size. Multiple images may be combined, or structures may be overlapped to create more complex samples that better reflect the variability and complexity of real-world data. Overlay images may be provided. The overlay images may comprise up to threeseparate structures in one exemplary embodiment. New image and / or overlap situations may be generated by permutating 90° rotations of 2 / 3 of the structures. Synthetic noise, blurring, or other perturbations may be introduced to increase the model's robustness and resistance to overfitting. Data preparation may comprise randomly apply blurring simulating worse imaging conditions and more overlap.

[0099] The augmented training dataset created through these techniques can provide the machine-learning model with a more diverse and realistic set of samples to learn from, improving its ability to generalize to new, unseen data during testing or deployment (inference). This can lead to better performance, increased accuracy, and more reliable feature extraction for nanoscale cellular organelle multiplexing tasks.

[0100] Data preparation can comprise a step of cutting randomly centered patches (256x256 px). Each patch may be randomly flipped horizontally and / or vertically.

[0101] Data preparation may comprise standardizing multi-structure-patches. This may be performed using zero-mean-unit-variance.

[0102] Data preparation may comprise normalizing ground truth single-structure-patches. In an embodiment the Al-Model as the machine-learning model can be a U-NET. Therein, a feature generation can be performed. The machine learning model can learn features on different scales of representation. Nevertheless, herein a multiscale U-NET is not used.

[0103] A demixing layer may be comprised by the machine-learning model. It can be a 2D-convolution layer. It may comprise a kernel, particularly a 1x1 kernel.

[0104] An identity activation may be provided, particularly of linear nature: f(x) = x. In this method for multiplexing nanoscale cellular organelles, an identity activation function can be employed as part of the machine-learning model architecture. An activation function is a component in a neural network that can decide whether to pass or block the flow of information through the network based on the output of a given neuron. In this case, an identity activation function of linear nature can be used. The identity activation function simply returns the input value x as the output f(x) = x. This means that the neural network can retain the original input values without any non-lineartransformations or modifications, effectively allowing the model to learn and represent the underlying features in their raw form.

[0105] Using an identity activation function can simplify the model's architecture and computation, reducing the risk of introducing unnecessary complexity or noise into the feature extraction process. It is particularly useful when the goal is to preserve the original structure and information present in the input data, such as in the case of feature-specific channel generation for nanoscale cellular organelle multiplexing. The machine-learning model may realize the following function (as matter of example for three organelle structures to be multiplexed):

[0106]

[0107] e ®

[0108] and 9 H [0,1] G IR

[0109] sizs(pXrow, pXcoiumn, channel)

[0110] According to an aspect the organelles displayed in the at least one SRM-image can be at least two selected from the list of microtubules, clathrin, actin filaments, endoplasmatic reticulum, at least partially depolymerized microtubules and / or endosomes.

[0111] In the method described for multiplexing nanoscale cellular organelles, the SRM images display at least two types of organelles, selected from a list including microtubules, clathrin, actin filaments, endoplasmic reticulum, at least partially depolymerized microtubules, and endosomes.

[0112] Microtubules are cylindrical protein structures that form part of the cytoskeleton in eukaryotic cells. They play a key role in maintaining cell shape, aiding cell division (mitosis), intracellular transport, and serving as tracks for motor proteins to move along. They can polymerize and depolymerize.

[0113] Clathrin is a type of protein found in the cytoplasm of eukaryotic cells that forms a lattice-like structure around the neck of small vesicles involved in endocytosis. It plays a crucial role in the formation and budding of vesicles from the cell membrane for the uptake and internalization of extracellular materials.Actin filaments are long, thin protein structures that form part of the cytoskeleton in eukaryotic cells. They contribute to various cellular processes such as muscle contraction, maintaining cell shape, cytokinesis (cell division), and providing a scaffold for the movement of organelles and proteins within the cell.

[0114] The endoplasmic reticulum is a type of cellular organelle that forms an interconnected network of flattened, membrane-enclosed sacs and tubules in eukaryotic cells. It plays important roles in various cellular functions, such as protein and lipid synthesis, modulation of calcium levels, and serving as a platform for the assembly and transport of vesicles involved in endocytosis and exocytosis.

[0115] At least partially depolymerized microtubules are microtubules that have undergone a partial breakdown or disassembly, resulting in shorter, less stable tubulin dimers. These structures can be observed during cellular processes like mitosis (cell division) or when cells experience stress or environmental changes.

[0116] Endosomes are membrane-bound organelles that form part of the cellular endocytic system. They play a key role in internalizing extracellular materials and molecules by engulfing them through the plasma membrane, then transporting them to other locations within the cell for processing or disposal.

[0117] According to an aspect the machine-learning model was trained using image data from a first S RM -technique and wherein the multiplexing is performed on at least one SRM-image obtained by a second SRM-technique.

[0118] The method may comprise data augmentation for the first type of SRM, for example SMLM. From rendering, grayscale images can be obtained each containing a single structure of interest. Prior to training the machine-learning model, particularly the deep neural network, respective training data from super resolution microscopy has to be provided in sufficient quantity. To increase the amount of respective training data a method of data augmentation can be used to increase the amount of training data and aid the network in developing invariance towards shifts, rotation, deformation, gray level inhomogeneities and overlap of multiple structures. All of which are effects particularly encountered in super resolution microscopy (SRM). First, rendered images can be evaluated for similarities in texture and shape of each structure before splitting them into train, test, and validation sub-sets. It can be taken care to include as much variety as possible to the sets before augmentation in orderto keep high information content. For each sub-set, several structures can be overlaid. In an embodiment up to three structures can be overlaid at a time. They can be rotated relative to each other by 90° in a clockwise fashion. Thus, from n images one can obtain 4noverlays.

[0119] At train-batch creation, randomly centered patches can be cropped from randomly selected overlays. These can get randomly flipped along each of its axis. They can get randomly blurred. In an embodiment, 32 randomly centered patches (32x256x256) can be cropped from 8 randomly selected overlays which can get randomly flipped along each of its axes before they can be randomly blurred to simulate bad staining. The intensity distribution of each patch can be standardized to zero-mean, unit-variance to increase network stability and ease later application. 256x256px can be chosen as patch size. It allows that 2x2 max pooling operations can be applied correctly. Further, it allows a field of view (FOV) sufficiently large to capture textural properties while keeping focus on smaller features.

[0120] To evaluate the pretrained weights from the first SRM technique on data from the second SRM technique, comprehensive preprocessing can be implemented. This allows to overcome inherent differences between images. The SRM techniques may lead to differences in the background. Some may have a non-zero background, while other have may have zero backgrounds. To counteract these differences a preprocessing pipeline can be implemented to prepare the data set from the second SRM technique for input into the neural network trained with the data from the first SRM technique. The pipeline may comprise reviewing and selecting high quality layers from z-stacks of the first SRM technique. The pipeline may comprise removing the background in order to achieve zero background images.

[0121] The term Z-stacks particularly refer to a series of two-dimensional images captured at different focal planes or distances from the sample, usually along the optical axis (Z-axis) in microscopy imaging techniques. In other words, they are a stack of images acquired by scanning the focus through the sample at various depths.

[0122] In confocal microscopy and other similar techniques, z-stacks are typically obtained by illuminating a thin slice of the sample with a focused laser beam while scanning the focal plane along the Z-axis. For each focal plane, a two-dimensional image is captured, resulting in a series of images that form a complete 3D representation of the sample.Z-stacks are particularly used in various microscopy techniques to obtain high-resolution images with improved depth resolution and minimized out-of-focus blurring. They allow researchers to capture detailed information about the sample's structure, particularly when dealing with thick or layered specimens. Z-stacks can be reconstructed into 3D models using software tools, enabling visualization and analysis of samples from different perspectives and depths.

[0123] In the context of super-resolution microscopy (SRM), z-stacks are particularly used to capture high-resolution images of nanoscale cellular organelles, as they provide a comprehensive view of the sample's structure and enable better visualization of individual structures in complex biological environments.

[0124] According to an aspect the first SRM-technique and the second SRM -technique can be any non-identical combination selected from the list of MINFLUX, multi-organelle SMLM imaging, STED, SIM.

[0125] Single-Molecule Localization Microscopy (SMLM) is a super-resolution imaging technique that allows visualization of cellular structures at nanometer-scale resolution, far beyond the diffraction limit of conventional optical microscopy, which is approximately 200 nm. In SMLM, the key principle particularly involves isolating and localizing individual fluorescent molecules through stochastic activation and deactivation of fluorophores. Techniques such as Photoactivated Localization Microscopy (PALM) and (direct) Stochastic Optical Reconstruction Microscopy ((d)STORM) allow sequential imaging of single fluorophores, achieving high spatial resolution by reconstructing the positions of each molecule to form a super-resolved image. SMLM achieves remarkable resolution, typically down to 10-20 nm, e.g. by fitting the point spread function (PSF) of each molecule to pinpoint its precise location. These are the most important SMLM microscopy approaches:

[0126] Photoactivated localization microscopy (PALM) particularly takes advantage of photoactivatable or photoswitchable fluorescent proteins to activate a distributed subset of molecules in each frame.

[0127] Stochastic Optical Reconstruction Microscopy (STORM) particularly employs photoswitchable fluorescent dyes. These dyes can be turned on and off at random intervals.The dSTORM (direct STORM) particularly utilizes photoswitchable properties of common fluorescent dyes in conventional microscopy, without requiring specific activation lasers or complex hardware.

[0128] (DNA) PAINT particularly leverages short, fluorophore-labeled DNA ‘imager’ strands that can bind transiently to complementary ‘docking’ strands on the sample. The repetitive binding and unbinding events can generate stochastic fluorescence signals that can be localized precisely, thus enabling super-resolution image reconstruction. Structured Illumination Microscopy (SIM) is a super-resolution microscopy technique that overcomes the diffraction limit of light. This goal can be achieved by leveraging Moire fringes. The sample can be illuminated with a spatially structured light, this creates Moire fringes in the observed image, which encodes information that is particularly not achievable in conventional microscopy methods. Structured illumination can be generated using a phase grating mounted on a precise piezoelectric translation stage, which enables the grating to be precisely mobile. By moving or rotating the grating, multiple images can be taken and can be computationally reconstructed to an image with higher resolution. This technique effectively doubles the resolution limit of optical microscopy, up to approximately 100 nm.

[0129] Stimulated Emission Depletion (STED) particularly employs a secondary, doughnutshaped depletion laser beam that can quench fluorescence everywhere except the central point of the excitation focus. By forcing surrounding fluorophores to revert to their ground state, STED can effectively narrow the fluorescence spot to well below the diffraction limit. This targeted depletion of excited fluorophores significantly enhances resolution and allows for imaging with nanoscale precision.

[0130] While SMLM almost exclusively performs well in fixed cells, SIM provides the capability to achieve sub-diffraction resolution in living cells. This is largely due to its relatively low phototoxic effects and the less stringent requirements on the photophysical properties of fluorescent dyes, particularly in comparison to SMLM or STED. Adapting the method described herein to SIM data can offer the opportunity to multiplex live-cell data without chromatic aberrations or time shifts between color channels in the future.According to an aspect at least one of the following performance evaluation measures can be obtained, a global Structural Similarity Index Measure (SSIM) value, a Multi-Scale SSIM (MSSSIM) value.

[0131] As the network is trained to synthesize single-structure images most similar to the ground truth from multi-structure inputs, one can straight-forwardly assess the prediction image quality by employing self-similarity metrices like the Structural-Self-Similarity-Measure (SSIM) and the Multi-Scale-Structural-Self-Similarity-Measure (MSSSIM). These measures can be suited for the purpose as they are based on extracting structural information and assume high interdependence of pixel intensities with respect to relative proximity.

[0132] In the context of SSIM and MS-SSIM values, the ranges can be interpreted, along with what values between 0.8-0.9 for SSIM and 0.9-0.94 for MS-SSIM can indicate: The value of 1.0 can be considered a perfect value. This can represent a theoretically perfect match between an Al-generated image and a ground truth. In practice, achieving exactly 1.0 is rare because even minimal noise, misalignment, or variations in pixel intensity can cause deviations.

[0133] The range of 0.8-0.9 in SSIM allows for the following interpretation: Values between 0.8 and 0.9 suggest that the generated image captures the overall structural features and major details well but may show some imperfections in finer details, such as texture, contrast, or subtle morphological nuances.

[0134] The range of 0.8-0.85 (upper border value not included in the range) can indicate good but not optimal similarity, possibly reflecting issues like minor misclassifications, reduced sharpness, or overlapping signals that the algorithm struggles to separate completely.

[0135] The range of 0.85-0.9 can represent very good structural similarity, where the majority of features are accurately captured but small deviations (e.g., noise, edge inconsistencies) can exist.

[0136] The range of 0.9-0.94 in MS-SSIM allows the following interpretation: MS-SSIM values in this range can indicate excellent multi-scale similarity, particularly in preserving global structural and textural details. Higher MS-SSIM values (e.g., closer to 0.94) reflect a more consistent representation across multiple scales, capturing fine-grained and coarse features alike.The range of 0.9-0.92 (upper border value not included in the range) can highlight strong similarity with slight imperfections in either finer textures or localized structural details.

[0137] The range of 0.92-0.94 can indicate near-perfect reconstruction, where only minimal differences remain, often imperceptible to the human eye.

[0138] In the context of evaluating Al-generated images with ground truth, values above 0.8 in SSIM and above 0.9 in MS-SSIM can be considered strong indicators of high-quality generation. Differences in these ranges may arise due to challenges in separating highly overlapping structures, noise, or the inherent limitations of the data used for training or validation.

[0139] Achieving 0.9 or higher in both metrics can indicate that the Al-generated images are highly reliable for downstream analysis, preserving both local features and global structural fidelity effectively.

[0140] After training the data, the trained model can be tested on the images that it has never seen. Testing can be done to assess the generalization capability of the model. The above-mentioned functions can be used to measure its performance on test images. One can also use MSE as it is described elsewhere herein during re-training of the neural network on SRM, particularly SMLM, images. MS-SSIM and SSIM can be utilized during testing the neural network in addition to MSE.

[0141] According to an aspect the demixing can be performed in a demixing block after the U-Net.

[0142] The network architecture can use a U-Net followed by a demixing block.

[0143] Convolutional blocks can consist of two successive sub-blocks, particularly made of a zero-padded 3x3 convolution. It can be followed by a rectified linear unit (ReLu) and batch normalization pass. The left arm downsampling can be done using 2x2 max pooling operations. The right arm upsampling can use transposed convolution. 32 feature maps (32x256x256) can be created by the U-Net per patch.

[0144] Demixing can be done by a 1x1 2D convolution followed by a linear identity function to regressively generate synthetic images, particularly three images for the respective structures to be demixed, e.g. 3x256x256, for each patch in a batch containing the predicted structures involved in the multiplexed input. In that way, human interpretable outputs can be gained without the need of segmentation. As allstructures can be predicted independently, the network can be indifferent to overlap, usually causing artifacts in semantic segmentation, and can compensate for gray level shifts as well as noise.

[0145] For each pixel in a patch possibly containing JV' biological structures, the demixing block can transfer an array of U-Net generated feature T (dimCF) = 32x1) into JV* arrays of real numbers (J\T e IR). The model can be trained to map each standardized patch into JV* normalized prediction patches

[0146]

[0147] : x e [0,1] a IR).

[0148] In an embodiment, the U-Net can be trained with batches of 32 augmented patches (32x32x256x256) per iteration e.g., for 4000 epochs. Each batch can be used to train the model to synthesize (32x^x256x256) predictions using linear regression, i.e. the identity, as final activation function. An Adam optimizer can be used with the following = 0.9, ?2= 0.99, e = 10-8and L2 loss (MSE =

[0149]

[0150] is the i-th ground truth and ytthe i-th prediction, as a cost function.

[0151] The network’s performance predicting the validation set can be tested in regular epoch intervals and the ultimately saved set of weight parameters can be determined according to that.

[0152] Due to the predictions being created by linear regression on a pixel-level, the image background can be non-zero. However, it can be advantageous for interpretability to have a homogeneous background which can be achieved by setting the most occurring intensity values in each predicted image to zero. The respective values can be extracted via histogram binning. This can be done as SMLM data particularly has no background.

[0153] The same reasoning can be used in a second filter in which all pixels knowingly belonging to the background can be set as obtained from the input, thus to zero. In doing so, potentially occurring artifactual pixels can be excluded.

[0154] A machine can be provided that can be linked to a storage unit. The storage unit may comprise a computer program product comprising machine-readable instructions which can be such that they may be used to execute a method of training and / or a method of inference as described elsewhere herein.As matter of example the following will refer to SIM and SMLM crossover application of the machine-learning model, particularly the neural network as described elsewhere herein. Nevertheless, the respective crossover application is not limited to the pair of SIM and SMLM data alone. The machine-learning model may also be trained by image data obtained by any of the other SRM methods as described elsewhere herein and / or the trained machine-learning model may be applied to any other SRM method as described herein. Even the role of SIM and SMLM may be swapped regarding training and inference.

[0155] The described method and the respective machine-learning model, particularly the deep neural network, particularly initially trained on image data from a first SRM technique, particularly Single Molecule Localization Microscopy (SMLM) images, allows to multiplex and demix organelle structures from images of a second SRM technique, particularly Structured Illumination Microscopy (SIM) images. The machine-learning model particularly has the ability to multiplex gray scale images into different channels based on nanotextures in the latter, particularly SIM images.

[0156] The neural network can be adapted to SIM images by preprocessing the SIM data to account for inherent differences from SMLM images. This adaptation particularly may involve layer selection and background removal from SIM z-stacks. The model can be initially trained on data, which can be derived from SMLM. Preprocessed SIM images can then be fed into the model to generate output channels. The most suitable output channel is selected for each ground truth channel. Finally, the performance of the trained model can be evaluated on the SIM data.

[0157] The results show that the machine-learning model, particularly the neural network, particularly trained on SMLM data, can effectively separate depolymerized microtubules and intact ones for example in SIM images. The performance of the model can be assessed by the Mean Squared Error (MSE), Structural Similarity Index (SSIM), and Multi-Scale SSIM (MS-SSIM). The respective modus operand! may allow two approaches: Training a new model for each imaging method independently. This allows to define the ground truth data for a respective analysis of images resulting from the same imaging technique. Additionally or alternatively, one may provide a pretrained machine learning model such as a neural network that was trained using imaging data from a first imaging technique and applying it to demix nanoscale structures from at least one image obtained with an imaging techniquethat is not identical to the imaging technique used to train the respective machine learning model.

[0158] Advantageous embodiments of the invention are the subject of the dependent claims, the description and the figures. Features, feature combinations, technical effects and advantages described in connection with the computer-implemented method configured for multiplexing of nanoscale cellular organelles, the machine, the storage unit and / or the computer program product also apply to the method of training the machine-learning model. This also applies the other way around, so that with regard to the disclosure of the individual aspects of the invention, reciprocal reference is or can always be made, particularly independent of the category described and / or claimed.

[0159] FIGURES

[0160] An embodiment of the invention is shown in the drawings and is explained in more detail below. It is shown in:

[0161] Fig. 1 a schematic drawing about an exemplary embodiment of computer-implemented methods configured for multiplexing of nanoscale cellular organelles; Fig. 2 a schematic drawing of a U-net architecture during training; and

[0162] Fig. 3 images of cellular organelles demixed and multiplexed by U-Net.

[0163] Some of the figures contain simplified, schematic representations. In some cases, identical reference signs are used for the same, but possibly not identical, elements. Different views of the same elements might be scaled differently. Directions such as "left", "right", "up" and "down" are to be understood in relation to the respective figure and may vary in the individual representations compared to the object depicted.

[0164] Fig. 1 shows a schematic drawing about an exemplary embodiment of computer-implemented methods 100, 200 configured for multiplexing of nanoscale cellular organelles.

[0165] A method 100 of training a machine-learning model 90 (see Fig. 2) can be trained using isolated channels 65 to generate feature-specific channels 75, so-called "feature dreams", corresponding to each of the structures of the at least two organelles 51 , 52, 53 to be multiplexed. To do so individual images displaying at least one of the organelle structures can be provided 110 to the machine-learningmodel 90 to be trained. A loss function, particularly mean squared error, can be used during improving 120 to enhance the training 100 of the machine-learning model 90. Training can be performed using SMLM datasets. SMLM datasets of actin 55, microtubules 54, and endosomes 56 - as matter of example - can be obtained from a combination of self-obtained data but can also be combined with publicly available resources for example on Shareloc. Localization data can be rendered into grayscale images using rapidSTORM, particularly ensuring each image containing a single structure.

[0166] The machine-learning model 90 can at least be one of a convolutional neural network, a deep neural network and / or a U-Net 95 as shown in Fig. 2.

[0167] The U-Net 95 shown in Fig. 2 (therein during training 100) particularly exhibits universal versatility across all major SRM modalities, i.e. SMLM, MINFLUX, STED, and SIM with very low demand on training data, as particularly <10 independent fields of view per trained organelle 51 , 52, 53, combined with a data augmentation approach, can be sufficient for high-fidelity multiplexing.

[0168] From rendering, grayscale images can be obtained each containing a single structure of interest. To enhance SRM data quantity data augmentation can be used to increase the amount of training data and aid the network in developing invariance towards shifts, rotation, deformation, gray level inhomogeneities and overlap of multiple structures. All of which are effects particularly encountered in SRM.

[0169] Rendered images can first be evaluated for similarities in texture and shape of each structure before splitting them into train, test, and validation sub-sets. As much variety as possible can be included to the sets before augmentation in order to keep high information content. For each sub-set, up to three structures can be overlaid at a time and they can be rotated relative to each other by 90° in a clockwise fashion. Thus, from n images one can obtain 4An overlays.

[0170] At train-batch creation, randomly centered patches, particularly 32 (32x256x256), can be cropped from randomly selected overlays, particularly 8, which can get randomly flipped along each of its axes before they are randomly blurred to simulate bad staining. The intensity distribution of each patch can be standardized to zero-mean, unit-variance to increase network stability and ease later application. A certain dimensionality of the patch size can be chosen to match the max pooling operationand to provide a sufficient field of view. As matter of example, 256x256px can be chosen as patch size since it not only ensures that 2x2 max pooling operations can be applied correctly but further allows a field of view (FOV) sufficiently large to capture textural properties while keeping focus on smaller features.

[0171] Following the method of training 100 a method of inference 200 can be applied. The computer-implemented method 200 can be configured for multiplexing of nanoscale cellular organelles 50.

[0172] As the U-Net 95 is particularly capable to be trained during a method of training 100 one can also use the U-Net 95 with any particular SRM technique or modality such as any chosen from MINFLUX, multi-organelle SMLM imaging, STED or SIM. The method of training 100 could be performed with any (non-identical) modality taken from the list.

[0173] To transfer the images into a respective format one can implement a background removal 220, particularly after a layer selection 210 of layers from a z-stack to ensure optimal data quality to be fed to the trained machine-learning model 90.

[0174] The method 200 may comprise the step of providing 230 at least one SRM-image 60 comprising at least two identically labeled organelles 51 , 52, 53 displayed in the at least one SRM-image 60 - alternatively or additionally monochromatic representations of the at least two organelles 51 , 52, 53 displayed in the at least one SRM-image 60 may be provided - to a machine-learning model 90 configured and trained for multiplexing of nanoscale cellular organelles 50.

[0175] Based on the training, the machine-learning model 90 can be capable of extracting 240 organelle-specific nanotextures. This can be based on demixing 250 particularly being performed in a demixing block 96 after the U-Net 95 (see Fig. 2).

[0176] The step of synthesizing 260 visual representations of underlying structures of the at least two organelles 51 , 52, 53 can be performed via linear regression as the final activation function. Linear regression particularly serves as the final activation function, enabling pixel-level intensity predictions directly mapping to single-structure outputs. Unlike semantic segmentation, the machine-learning model 90 can reconstruct individual organelles 51 , 52, 53 without relying on classification, making it invariant to overlapping structures and capable of preserving gray-level shifts.This allows generating 270 (probabilistic) multiplexed images 70 (see Fig. 3) of the at least two identically labeled organelles 51 , 52 displayed in the at least one SRM-image 60 - additionally or alternatively, the monochromatic representations of the at least two organelles 51 , 52 displayed in the at least one SRM-image 60 - by (probabilistic) demixing 250 of the at least two organelles 51 , 52, 53 in the at least one SRM-image 60.

[0177] In a step 300 of evaluation of the model performance at least one of the following performance evaluation measures can be obtained, a global Structural Similarity Index Measure (SSIM) value, a Multi-Scale SSIM (MSSSIM) value. Alternatively or additionally, a mean squared error (MSE) can be determined.

[0178] Fig. 2 shows a schematic drawing of a U-Net 95 architecture during training. It enables the regressive extraction of multiple complex structures (e.g., microtubules 54, clathrin, endosomes 56, endoplasmatic reticulum - short: ER -, and actin 55) for example from single-channel grayscale SMLM images as one SRM - image 60. This content-agnostic texture recognition pareticularly relies on probabilistic demixing 250 rather than classical image segmentation, accurately identifying multiple organelles 51 , 52, 53 even when heavily overlapped. This high-fidelity demixing 250 is shown in Fig. 3, by analyzing synthetically overlaid monochromatic SMLM images that comprise dense signals from actin 55, microtubules 54 and endosomes 56.

[0179] The U-Net architecture can be optimized for feature extraction and demixing 250. The network particularly includes a left-arm downsampling path 61 with 2 x 2 maxpooling layers and a right-arm upsampling path 62 with transposed convolution. Each convolutional block particularly comprises two sub-blocks comprising at least one of zero-padded 3 x 3 convolutions, rectified linear unit (ReLU) activations, and batch normalization. The network can process 32 feature maps (32 x 256 x 256) per input patch in an exemplary embodiment.

[0180] A 1 x 1 2D convolutional layer in the demixing block 96 can generate the number of output channels corresponding to the input structures, here three output channels (3 x 256 x 256) corresponding to the predicted structures of microtubule 54, actin filaments 55 and endosomes 56. Linear regression may serve as the final activation function, ensuring pixel-level intensity predictions directly map to single-structure outputs to synthesize 260 visual representations of the underlying structures of the organelles 51 , 52, 53. Unlike semantic segmentation, the U-Net 95 can reconstructindividual organelles 51 , 52, 53 without relying on classification, making it invariant to overlapping structures and capable of preserving gray-level shifts.

[0181] The network can be trained, for example for 4,000 epochs, with augmented patches (32 x 32 x 256 x 256 per batch). An Adam optimizer can be used with the following parameters: y = 10A(-5),

[0182]

[0183] = 0.9, (32= 0.99, and e = 10A(-8). Loss can be calculated using L2 (mean squared error) as the cost (or loss) function. Model weights can be saved based on the validation set performance, evaluated at regular intervals.

[0184] Instead of conventional semantic segmentation, which is a classification task, the model can be trained to synthesize 260 visual data of potentially underlying structures by means of linear regression (demixing 250). The network does not learn to take decisions based on the interdependence of intensity values but rather is made to understand the fundamental texture of involved biological structures and recreate them. This not only can eliminate artifacts created by structural overlap, but also allows to gain the ability to deconvolve the interplay of complex shapes.

[0185] Fig. 3 images of cellular organelles demixed 250 and multiplexed by the U-Net 95 as shown in Fig. 2. A synthetic multi-structure overlay image 68 was generated by combining three localization datasets and rendering a merged image (as ground truth images 68).

[0186] Machine-learning (Al-)enabled textural demixing 250 (see Fig. 1) with supervised U-Net 95 feature generation (see Fig. 2), successfully multiplexing monochromatic SMLM data with up to 3 simultaneously displayed organelles 51 , 52, 53 is shown. The machine-learning model 90, here the U-Net 95, enables the regressive extraction of multiple complex structures (e.g., microtubules 54, clathrin, endosomes 56, ER, and actin 55) from single-channel grayscale SMLM images 69 (as ground truth image 68). This content-agnostic texture recognition particularly relies on probabilistic demixing 250 rather than classical image segmentation, accurately identifying multiple organelles 51 , 52, 53 even when heavily overlapped. High-fidelity demixing 250 can be demonstrated by analyzing synthetically overlaid monochromatic SMLM images 69 that comprise dense signals from actin 55, microtubules 54 and endosomes 56 in Fig. 3. By assessing ground truth 68 (row b: for region of interest, short ROI) and feature dream channels 75 (row c: for ROI) with SSIM and MS-SSIM, exceptionally high similarities between both can be demonstrated (column d: comparing global images and ROI images).In particular, the terms "may" or “can” refer to optional features of the invention.

[0187] Consequently, there are also further aspects and / or embodiments of the invention which additionally or alternatively have the respective feature or features. All features in feature combinations are also disclosed independently thereof and may also be singled out from the combinations of features disclosed herein and used in combination with other features to specify the subject-matter of the any of the claims, dissolving any structural and / or functional relationship that may exist between the features. Terms like “first”, “second”, “third” may be used to refer to a list of elements but does not necessarily describe these features or elements according to their importance, their order of appearance or order of structure. Therefore, these elements or features may specify the different aspects in any other particular order, unless explicitly expressed otherwise.

[0188] LIST OF REFERENCE SIGNS

[0189] 50 nanoscale cellular organelles

[0190] 51 , 52, 53 individual organelles

[0191] 54 microtubule

[0192] 55 actin filaments

[0193] 56 endosomes

[0194] 60 SRM image

[0195] 65 isolated channels

[0196] 68 ground truth image

[0197] 69 SMLM image

[0198] 70 multiplexed images

[0199] 75 feature-specific channels

[0200] 90 machine-learning model

[0201] 95 U-Net

[0202] 100 training method

[0203] 110 providing imagesstep of improving

[0204] method configured for multiplexing of nanoscale cellular organelles layer selection from z-stack

[0205] background removal

[0206] providing at least one SRM-image

[0207] extracting organelle-specific nanotextures

[0208] demixing

[0209] synthesizing of representations

[0210] generation of multiplexed images

[0211] step of evaluation to obtain evaluation measure

Claims

CLAIMS1. Computer-implemented method (200) configured for multiplexing of nanoscale structures, particularly cellular organelles (50) and / or semiconductor structures, comprising the steps of:- providing (230) at least one image, particularly super-resolution microscopy (SRM-)image (60), comprising at least two (identically) labeled nanoscale structures, particularly organelles (51 , 52, 53), displayed in the at least one image and / or monochromatic representations of the at least two structures displayed in the at least one image to a machine-learning model (90) configured and trained for multiplexing of nanoscale structures, particularly cellular organelles (50) and / or semiconductor structures;- extracting (240) nanoscale-structure-specific nanotextures; and- generating (270) (probabilistic) multiplexed images (70) of the at least two (identically) labeled nanoscale structures displayed in the at least one image and / or the monochromatic representations of the at least two nanoscale structures displayed in the at least one image by (probabilistic) demixing (250) of the at least two nanoscale structures in the at least one image.

2. Method (200) according to claim 1 , wherein the nanoscale structure is at least one out of- at least one cellular organelle (51 , 52, 53); or- at least one semiconductor structure.

3. Method (200) according to at least one of claims 1 and 2, wherein the machinelearning model (90) is at least one of a convolutional neural network, a deep neural network and / or a U-Net (95).

4. Method (200) according to at least one of the previous claims, wherein a loss function, particularly mean squared error, was used during training (100) the machine-learning model (90).

5. Method (200) according to at least one of the previous claims comprising the step of synthesizing (260) visual representations of underlying structures of the at least two organelles (51 , 52, 53) via linear regression.

366. Method (200) according to at least one of the previous claims, particularly according to claim 4, wherein the machine-learning model (90) was trained (100) on isolated channels (65) to generate feature-specific channels ("feature dreams") (75) corresponding to each of the structures of the at least two organelles (51 , 52, 53) to be multiplexed.

7. Method (200) according to at least one of the previous claims, wherein the organelles (51 , 52, 53) displayed in the at least one SRM-image (60) are at least two selected from the list of microtubules (54), clathrin, actin filaments (55), endoplasmatic reticulum, at least partially depolymerized microtubules, endosomes (56).

8. Method (200) according to at least one of the previous claims, particularly according to claim 6, wherein the machine-learning model (90) was trained (100) using image data from a first SRM-technique and wherein the multiplexing is performed on at least one SRM-image obtained by a second SRM-technique.

9. Method (200) according to claim 8, wherein the combination of the first SRM- technique and the second SRM-technique is any non-identical combination selected from the list of MINFLUX, multi-organelle SMLM imaging, STED, SIM, Airy-Scan.

10. Method (200) according to at least one of the previous claims, wherein at least one of the following performance evaluation measures is obtained (300), a global Structural Similarity Index Measure (SSIM) value, a Multi-Scale SSIM (MSSSIM) value.

11. Method (200) according to at least one of the previous claims, wherein the demixing (230) is performed in a demixing block (96) after the U-Net (95).

Images