Training method for self-supervised microscopic image processing neural network

Abstract

Disclosed in the present application is a training method for a self-supervised microscopic image processing neural network. The method comprises: using an optical microscopic imaging system, which has a time-correlated single photon counting function, to acquire a fluorescence microscopic image data set, wherein the fluorescence microscopic image data set comprises a photon counting distribution set {Gmn(t)}MN for M×N pixels, M and N both being integers greater than 1, m ranging from 1 to M, n ranging from 1 to N, and t representing time; and by means of photon flux redistribution, constructing, from the photon counting distribution set for the M×N pixels, a training data set for a neural network comprising a fluorescence lifetime prediction convolution module and / or a denoising convolution module.

Description

Training methods for self-supervised neural networks for microscopic image processing Technical Field

[0001] This application generally relates to a training method for a self-supervised denoising neural network based on photon flow recombination of fluorescence images and a fluorescence lifetime prediction neural network, as well as the use of the trained neural network to denoise fluorescence images and predict fluorescence lifetime. Background Technology

[0002] Fluorescence microscopy is currently an important branch of microscopy, enabling the observation of biological cells, tissues, and organs under in vivo conditions. Typically, in fluorescence microscopy, specific structures of the sample are specifically labeled with fluorescent components, resulting in fluorescent molecules. When these fluorescent molecules are excited by laser irradiation, they produce fluorescence, which is then collected by a detector to obtain a fluorescence image of the sample. For in vivo observation, simultaneously achieving high temporal resolution and high image quality is challenging. Often, image quality is sacrificed for extremely high temporal resolution, causing useful information in the acquired fluorescence image to be submerged in background noise, severely impacting the performance of the fluorescence microscopy system.

[0003] To address the background denoising problem of microscopic images, various denoising algorithms exist, such as mean filtering, Gaussian filtering, median filtering, nonlocal mean denoising, wavelet transform denoising, and 3D block matching denoising (BM3D). However, these algorithms struggle to achieve satisfactory denoising results for images with low signal-to-noise ratios, such as dynamic fluorescence microscopy image sets. While deep learning network algorithms have made significant progress in fluorescence microscopy image denoising, convolutional neural networks (CNNs) such as Denoising Convolutional Neural Network (DnCNN), Fast and Flexible Denoising Network (FFDNet), and Convolutional Blind Denoising Network (CBDNet) all require supervised training based on noisy-clear microscopic images. However, in application scenarios such as live-body observation where obtaining temporally clear fluorescence images is difficult, such supervised training of CNNs is challenging. Training such supervised neural networks requires obtaining high-resolution images of biological samples beforehand. However, for scenarios involving live in vivo observation, it is difficult to obtain high-resolution fluorescence images that meet the requirements, limiting the application of supervised neural networks. Furthermore, while existing technologies include neural network models based on noisy image-noisy image or self-supervised training, such as Noise2Noise, Noise2Void, Noise2self, and Neighbor2Neighbor, the training of these models typically requires at least two noisy images with the same signal distribution or the addition of constraint terms to the loss function. This also limits the application of these neural network models in fluorescence microscopy denoising.

[0004] Furthermore, fluorescence lifetime imaging, a field within fluorescence microscopy, has garnered increasing attention in recent years. Molecules in biological samples specifically labeled with fluorescent components (such as biofluorescent dyes or antibodies) become fluorescent molecules. Fluorescence lifetime is a unique property of fluorescent molecules, used to distinguish different fluorescent molecules and even as a sensor dynamically reflecting changes in the microenvironment surrounding them. This allows for highly sensitive and quantitative descriptions of intracellular microenvironmental changes in biological samples. Fluorescence lifetime imaging systems characterize specific dynamic changes in cells by acquiring and recording the time it takes for fluorescent molecules to transition from excited states to the ground state (typically on the nanosecond scale). Utilizing the principle of fluorescence resonance energy transfer (FRET), changes in the fluorescence lifetime of fluorescent molecules are acquired / recorded. By analyzing these changes, the distance between two adjacent fluorescent molecules in a biological sample can be inferred, thereby determining / describing intracellular microstructural changes. However, for fluorescence lifetime imaging, fitting the fluorescence lifetime of each pixel in the final image often requires acquiring a sufficiently large number of photons to achieve a good fit. However, achieving a sufficient number of photons often necessitates extending the imaging time of the scanning system, making fluorescence lifetime imaging difficult to apply to rapid fluorescence imaging of live cells. Recent advancements in neural networks have offered a unique solution to this field. Neural network models such as the fluorescence lifetime imaging network (FLI-Net), few-photon fluorescence lifetime imaging (FPFLI), and fluorescence lifetime imaging based on generative adversarial network estimation (flimGANE) have provided solutions for fluorescence lifetime prediction with limited photons. However, these methods all train the network using simulated datasets, which can lead to domain shift issues with real samples. Summary of the Invention

[0005] To address the above issues, this application aims to propose a method for self-supervised training of a denoising network and a fluorescence lifetime prediction network based on fluorescence images and photon flow redistribution. This method enables the acquisition of a dataset for training neural networks (including denoising or fluorescence lifetime prediction neural networks) with a minimal number of fluorescence images collected from biological samples, especially live biological samples. Furthermore, the trained neural network is used to denoise and predict the fluorescence lifetime of biological samples.

[0006] According to one aspect of this application, a method for training a self-supervised microscopic image processing neural network is provided, comprising:

[0007] An optical microscopy imaging system with time-correlated single-photon counting capability was used to acquire a fluorescence microscopic image dataset of biological samples. The fluorescence microscopic image dataset includes a photon count distribution set {G} for M×N pixels. mn (t)} MN , where M and N are integers greater than 1, m ranges from 1 to M and n ranges from 1 to N, and t represents time;

[0008] A training dataset for a neural network, including a fluorescence lifetime prediction convolutional module and / or a denoising convolutional module, is constructed from the M×N pixel photon count distribution set using a photon flow redistribution method. Each pixel in the photon count distribution set has p time intervals Δt, where p is an integer greater than 1. In the photon flow redistribution, a pair of redistributed photon count distributions is generated for each pixel in the M×N pixel set. Each pair of redistributed photon count distributions also has p time intervals Δt, and the photon counts of each pixel within the p time intervals Δt are randomly assigned to the p time intervals Δt of the pair of redistributed photon count distributions. This photon flow redistribution is repeated Q times to generate Q pairs of redistributed photon count distribution sets, where Q is an integer greater than 1.

[0009] Specifically, from the Q pairs of photon count distribution sets that have undergone photon flow redistribution, a first group of Q photon count distribution sets that have undergone photon flow redistribution are selected. Each of these sets is then used to generate Q training fluorescence lifetime images using the maximum likelihood algorithm, followed by maximum value normalization, to form the input set for fluorescence lifetime prediction training. Furthermore, from the Q pairs of photon count distribution sets that have undergone photon flow redistribution, a second group of Q photon count distribution sets that have undergone photon flow redistribution are selected. Each of these sets is then used to generate Q training fluorescence lifetime images using the maximum likelihood algorithm, to form the target set or ground truth set for fluorescence lifetime prediction training; or...

[0010] From the Q pairs of photon count distribution sets that have undergone photon flow redistribution, a first group of Q photon count distribution sets with redistributed photon flows is selected. For each of the Q selected photon count distribution sets with redistributed photon flows, the photon count distribution of each pixel is summed over time and then normalized to its maximum value to generate Q training fluorescence intensity images, which form the input set for image denoising training. From the Q pairs of photon count distribution sets with redistributed photon flows, a second group of Q photon count distribution sets with redistributed photon flows is selected. For each of the Q selected photon count distribution sets with redistributed photon flows in the second group, the photon count distribution of each pixel is summed over time and then normalized to its maximum value to generate Q training fluorescence intensity images, which form the target set or ground truth set for image denoising training.

[0011] The training dataset includes the fluorescence lifetime prediction training input set and the fluorescence lifetime prediction training target set or ground truth set and / or the image denoising training input set and the image denoising training target set or ground truth set;

[0012] The method further includes training the neural network using the training dataset.

[0013] Optionally, in the photon flow redistribution, the Q photon count distributions of the first group and the Q photon count distributions of the second group selected from the Q pairs of photon flow redistribution photon count distribution sets are randomly selected.

[0014] Optionally, in the photon flow redistribution, the first group of Q photon flow redistribution photon count distribution sets selected from the Q pairs of photon flow redistribution photon count distribution sets includes one photon count distribution from each pair of photon flow redistribution photon count distributions, while the second group of Q photon flow redistribution photon count distribution sets selected from the Q pairs of photon flow redistribution photon count distribution sets includes another photon count distribution from one photon count distribution in each pair of photon flow redistribution photon count distributions.

[0015] Optionally, before using the maximum likelihood algorithm, the data of each pixel to be calculated is superimposed / merged with its neighboring pixels whose association degree (Bin) is 0, 1, or 2.

[0016] Optionally, the model used to build the neural network includes, but is not limited to, a U-shaped neural network model, a residual neural network model, a residual channel attention convolutional neural network model, or a Fourier channel attention convolutional neural network model.

[0017] Optionally, the loss function of the neural network is:

[0018] in, The table shows the overall loss function for building the entire neural network. The loss function representing the fluorescence lifetime prediction convolutional module. The loss function represents the denoising convolutional module, where μ is a number between 0 and 1. The model that constructs the loss function includes, but is not limited to, the mean squared error loss function (MSE), the cross-entropy loss function (CE), or the L2 loss function.

[0019] Optionally, the output features of the model used to build the neural network are independently passed through the fluorescence lifetime prediction convolutional module and the denoising convolutional module, respectively.

[0020] Optionally, the fluorescence microscopy image dataset acquired using an optical microscopy imaging system with time-correlated single-photon counting capability is a single fluorescence microscopy image dataset or a selected fluorescence microscopy image dataset from multiple acquired fluorescence microscopy image datasets.

[0021] According to another aspect of this application, a method for processing a fluorescence microscopic image dataset acquired using an optical microscopic imaging system with time-correlated single-photon counting capability is also provided, comprising:

[0022] Construct a neural network that includes a fluorescence lifetime prediction convolutional module and / or a denoising convolutional module;

[0023] The neural network is trained using the aforementioned training method; and

[0024] A trained neural network is used to process a dataset of fluorescence microscopic images acquired by an optical microscopy imaging system with time-correlated single-photon counting capabilities. The processing includes fluorescence lifetime prediction and / or noise reduction.

[0025] According to another aspect of this application, a computer program product is also provided, comprising a computer program / instructions, characterized in that the computer program / instructions, when executed by a processor, implement the steps of the aforementioned method.

[0026] By employing the aforementioned technical means of this application, a dataset sufficient for training denoising neural networks and fluorescence lifetime prediction neural networks can be obtained by simply redistributing a single fluorescence image containing photon counts for biological samples. This avoids the deficiency of traditional neural network training, which requires the acquisition of at least two fluorescence images with the same signal distribution. It enables truly fully self-supervised neural network training, while simplifying the acquisition process for neural network training and reducing the difficulty of acquiring training datasets. Furthermore, it provides a foundation for the easy implementation of denoising and fluorescence lifetime prediction of fluorescence micrographs of live biological samples. Attached Figure Description

[0027] A more comprehensive understanding of the principles and aspects of this application will be gained from the detailed description below, in conjunction with the accompanying drawings. It should be noted that the scale of the drawings may vary for clarity, but this will not affect the understanding of this application. In the drawings:

[0028] Figure 1A schematically illustrates the distribution of photons (or photon stream) recorded at a single pixel in an image captured by an optical microscopic imaging system with time-correlated single-photon counting capability.

[0029] Figures 1B and 1C schematically illustrate the results of photon flow redistribution for the photon distribution in Figure 1A;

[0030] Figure 2 schematically illustrates the results of photon flow redistribution for all pixels;

[0031] Figure 3 schematically illustrates the results of multiple photon flow redistribution for all pixels;

[0032] Figure 4 schematically illustrates a block diagram of an optical microscopic imaging system with time-correlated single-photon counting function according to an embodiment of this application;

[0033] Figure 5 schematically illustrates a flowchart of a method for denoising fluorescence images and / or predicting fluorescence lifetime of biological samples according to an embodiment of this application;

[0034] Figures 6A and 6B schematically illustrate the cases where the correlation between adjacent pixels is Bin=1 and 2, respectively, for overlay / merging.

[0035] Figures 7A and 7B schematically illustrate the image results of fluorescence lifetime prediction and denoising using a trained neural network, respectively. Detailed Implementation

[0036] In the accompanying drawings of this application, features with the same structure or similar function are indicated by the same reference numerals.

[0037] Figure 1A schematically illustrates the distribution of photons (or photon stream) recorded at a single pixel in an image captured by an optical microscopy system with time-correlated single-photon counting capability. It should be noted that, within the scope of this application, an optical microscopy system with time-correlated single-photon counting capability means that such a system can perform fluorescence microscopy imaging on biological samples (referred to as "fluorescent samples") labeled with biological fluorescent dyes or antibodies, particularly living biological samples such as biological cells, under excitation light, in a manner familiar to those skilled in the art. This allows for the detection and recording of fluorescent photons (or photon quantity) emitted due to energy level transitions caused by stimulated emission, while simultaneously recording the microscopic time (often on the nanosecond scale) required for each fluorescent photon to be emitted from excitation. An example of an optical microscopy system with time-correlated single-photon counting capability is the commercially available Luminosa time-correlated single-photon counting scanning confocal system from PicoQuant. For example, a fluorescence microscopic image (or fluorescence microscopic image dataset) of a biological sample labeled with a biofluorescent dye, containing photon counts at a microscopic time, can be represented by the optical microscopic imaging system as: {I(m,n)} MN |{G mn (t)} MN

[0038] Where m is any integer between 1 and M (M is an integer greater than 1), n ​​is any integer between 1 and N (N is an integer greater than 1), I represents the fluorescence image data captured by the optical microscopy imaging system of the fluorescence generated by the biological sample after being irradiated with excitation light, that is, the fluorescence intensity data of the (m,n)th pixel, and G mn (t) represents the photon count distribution over time for the (m,n)th pixel in the fluorescence image data. The symbol {} can, for example, represent the fluorescence intensity data for all pixels in a total M×N pixel count, or the photon count distribution over time for all pixels. The total time of the photon count distribution depends on the recording time of the photon count by the aforementioned optical microscopy imaging system. Therefore, {I(m,n)} MN This can be referred to as a fluorescence intensity dataset for the entire pixel M×N, {G mn (t)} MN This can be referred to as the M×N photon count distribution set for the entire pixel. Those skilled in the art will understand that there is a correlation between the fluorescence intensity dataset and the photon count distribution set. This is because the fluorescence intensity of each pixel is ultimately determined by the total number of photons generated by that pixel; it can also be considered that the fluorescence intensity of each pixel is the sum of the total number of photons generated by that pixel. The fluorescence intensity of a fluorescence image is {I(m,n)}. MNThe set of photon count distributions over time for all pixels in the fluorescence image is associated with it; for example, the fluorescence intensity of each pixel can be represented as the sum of the photon count distributions of that pixel over time.

[0039] For example, Figure 1A schematically illustrates the photon count distribution G of the (m,n)th pixel in a fluorescence microscopy image taken using an optical microscopy imaging system with time-correlated single-photon counting capability. mn (t). In Figure 1A, the horizontal axis represents time t, for example, the figure exemplifies the presence of photon recording time T. L The photon recording time is divided into p equal time intervals Δt, where p is an integer greater than or equal to 1. Specifically, in the illustrated embodiment, p = 8. It should be understood that in an alternative embodiment not shown, p = 2 × P, where P is an integer greater than or equal to 1. The photon recording time T... L The magnitude of G depends on the relevant parameter settings of the optical microscopy imaging system and is determined by the specific operation of the system. From a microscopic temporal perspective, within each time interval Δt, the cumulative number of photons represents the probability of a fluorescent molecule transitioning from the excited state back to the ground state. A higher number of accumulated photons indicates a greater probability of transitioning back to the ground state. Conversely, as the vertical axis increases, the time required for this transition increases, indicating a lower probability of the event occurring. Photons within the same time interval Δt can be approximated as having the same transition time from the excited state back to the ground state, facilitating statistical fitting. This probability distribution also intuitively reflects the properties of the fluorescent molecule itself. Therefore, the aforementioned G... mn (t) can also be called the photon distribution / photon flow distribution for the (m,n)th pixel.

[0040] The photon stream redistribution method according to embodiments of this application is described below with reference to Figures 1A, 1B, and 1C. It should be noted that the methods or method steps mentioned in the context of this application can be encoded and stored in the form of computer programs / instructions, and can be invoked and executed by a processor such as a computer chip when needed. For example, the computer program / instructions can be stored in a data storage device such as a computer-readable storage medium, a cloud server, etc., and can be invoked and executed by the cloud server via a corresponding computer interface or network interface.

[0041] In the technical solution of this application, it is assumed that when training a neural network, the theoretical true values ​​corresponding to the fluorescence imaging results and / or fluorescence lifetimes obtained for the same biological sample are the same. Based on this assumption, the basic idea of ​​the technical solution of this application is to use a photon flow redistribution method to obtain a set of fluorescence microscopic images (or fluorescence microscopic image datasets) for training the neural network by means of a single fluorescence microscopic image (or fluorescence microscopic image data) with photon count results obtained at a microscopic time.

[0042] Specifically, the photon flow redistribution refers to the photon count distribution G of a single pixel (the (m,n)-th pixel) involved in Figure 1A. mn The implementation process of (t) is as follows: First, for all photons with microscopic temporal information related to a pixel point (the (m,n)th pixel point) involved in Figure 1A, each photon is randomly assigned to the same pixel position in two images (of the same size) by generating random numbers, so as to generate the photon count distribution G after photon stream redistribution as shown in Figures 1B and 1C. mn ′(t) and G mn "(t). An example of random allocation is generating a random number between 0 and 1. When this random number is greater than a specific value between 0 and 1, a photon to be allocated is assigned to the photon counting distribution G." mn ′(t), and when the random number is less than or equal to a specific value between 0 and 1, one photon to be allocated is assigned to the photon counting distribution G. mn "(t). Then, the photon counting distribution G mn ′(t) and G mn "(t) in the time micro-acquisition period (e.g., photon recording time T)" LThe region is divided into p parts with a time interval of Δt to obtain the photon count distribution G_mn'(t) and G_mn"(t) after photon stream redistribution corresponding to the pixel. For example or alternatively, taking the first time interval Δt in Figure 1A as an example, four photons are recorded sequentially in this time interval Δt. Then, in the first time interval Δt shown in Figure 1B and the first time interval Δt shown in Figure 1C, the four photons are randomly distributed. In the context of this application, the random distribution can be implemented in a computer using any suitable known algorithm. Then, the other time intervals Δt in Figure 1A can also be randomly distributed to the other corresponding time intervals Δt shown in Figure 1B and Figure 1C in a similar manner. It should be clear that when performing photon stream redistribution, the photons to be distributed can be distributed in the order in which the photons were recorded (rather than in micro-temporal order); alternatively, they can also be distributed in micro-temporal order or in segments of micro-temporal order (e.g., in the order of the time intervals shown in Figure 1A).

[0043] The photon flow redistribution process described above for a single pixel can be extended to all pixels. Therefore, as shown in Figure 2, the time-varying photon count distribution for all pixels in the entire M×N pixel matrix (as shown on the left) {G mn (t)} MN It is possible to obtain, through the photon flow redistribution process, two (or a pair) photon count distribution sets {G} of all pixels in an M×N dataset for photon flow redistribution. mn ′(t)} MN and {G mn "(t)} MN (As shown in the right figure). The process of photon flow redistribution for the entire M×N pixel dataset is repeated Q times to finally obtain 2×Q photon count distribution sets, which can be respectively represented by {G} mnq ′(t)} MN and {G mnq "(t)} MN This is represented as follows: where Q is an integer greater than 1, and the subscript q ranges from 1 to Q, as shown in Figure 3.

[0044] In the embodiments of this application, considering the hardware limitations of the optical imaging system, the maximum photon count distribution over time for all pixels in the entire pixel M×N recorded by the optical imaging system is not high. Therefore, in the above-mentioned photon stream redistribution process, only one photon count distribution G is used for each pixel at a time. mn (t) Two photon counting distributions G are randomly generated. mn ′(t) and G mnThis is achieved in the manner of "(t)". ​​Of course, those skilled in the art should understand that, provided the maximum number of photons recorded by the optical imaging system for the photon count distribution over time for all pixels in the entire pixel is sufficient, then during the aforementioned photon stream redistribution process, one photon count distribution G can also be achieved for each pixel at a time. mn (t) can be achieved by randomly generating two or more photon counting distributions.

[0045] Therefore, through the above photon flow redistribution process, the photon count distribution {G} for the entire M×N pixel dataset is obtained. mn (t)} MN A set of 2×Q photon count distributions {G} can be obtained. mnq ′(t)} MN and {G mnq "(t)} MN Or it can also be called the Q-pair photon counting distribution set {G} mnq ′(t)} MN and {G mnq "(t)} MN Since each photon count distribution set is generated independently with random numbers as described above, it can be assumed that they represent the same theoretical truth value for fluorescence imaging results and / or fluorescence lifetimes obtained for the same biological sample. From this point, training sets can be generated from these 2×Q photon count distribution sets for training neural networks (e.g., neural networks for denoising fluorescence images and / or neural networks for predicting photon lifetimes).

[0046] According to one embodiment of this application, two photon count distribution sets, such as {G}, are selected from the qth photon count distribution set in the Q-pair photon count distribution set. mnq ′(t)} MM and {G mnq "(t)} MN Two fluorescence lifetime images are obtained by calculating using the maximum likelihood algorithm familiar to those skilled in the art; then, these two fluorescence lifetime images are subjected to maximum value normalization processing, well known to those skilled in the art, to generate two corresponding training fluorescence lifetime images, such as G′. q and G″ q One of the training images is a fluorescence lifetime image, such as G′. q The fluorescence lifetime prediction training input set is stored, while another training fluorescence lifetime image, such as G″, is used. q Store the data into the target set or ground truth set for fluorescence lifetime prediction training. Repeat the above process for all photon count distribution sets (e.g., Q pairs) to finally obtain the input set {G'} for fluorescence lifetime prediction training. q} Qand the target set or truth set {G” for fluorescence lifetime prediction training q} Q .

[0047] According to another embodiment of this application, the two photon count distributions in the qth pair of photon count distributions in the Q-pair photon count distribution set, for example G... mn ′(t) and G mn The fluorescence intensity images are summed over time (t) to obtain two fluorescence intensity images; then, these two training fluorescence lifetime images are subjected to maximum normalization, a process well known to those skilled in the art, to produce two corresponding training fluorescence intensity images, such as I′. q and I″ q One of the training images is a fluorescence intensity image, such as I′. q The input set for fluorescence denoising prediction training is stored, while another fluorescence intensity image, such as I″, is used for training. q Store the data into the target set or ground truth set for fluorescence denoising prediction training. Repeat the above process for all photon count distribution sets (e.g., Q pairs) to finally obtain the input set {I'} for denoising training. q} Q and the target set or truth set used for denoising training {I” q} Q .

[0048] Then, using the fluorescence lifetime prediction training input set {G' obtained as mentioned in the two embodiments above... q} Q and the target set or truth set {G” for fluorescence lifetime prediction training q} Q and the input set {I'} used for noise reduction training q} Q and the target set or truth set used for denoising training {I” q} Q The constructed denoising neural network and fluorescence lifetime prediction neural network can be trained separately.

[0049] Figure 4 schematically illustrates a block diagram of an optical microscopy imaging system with time-correlated single-photon counting functionality according to an embodiment of this application, including, for example, an imaging module 100 and a data processing module 200. The imaging module 100 includes a fluorescence imaging device and a single-photon counting device 110. For example, the single-photon counting device 110 may include a picosecond pulse laser, a single-photon detector, and a time-correlated single-photon counting acquisition device. When a biological sample 300 treated with a fluorescent reagent is placed on the stage 400, the picosecond pulse laser can emit a laser pulse toward the biological sample 300 as needed to excite a corresponding fluorescence signal in the biological sample 300. Simultaneously, the excited fluorescence signal can be detected by the single-photon detector, and the information of the recorded photons (e.g., the time taken for a photon to transition from the excited state to the ground state) is sent to the time-correlated single-photon counting acquisition device for storage as specific encoded information (e.g., as photon counting (distribution) data). This recording is repeated cyclically until the predetermined photon recording time is reached. The data processing module 200 is capable of processing fluorescence image data and photon count (distribution) data received from the imaging module 100 from the fluorescence imaging device. The photon count (distribution) data recorded in the manner of single photon counting described above has the characteristic of being independent and uncorrelated with respect to the data of each recorded photon.

[0050] Figure 5 schematically illustrates a flowchart of a method for denoising and / or predicting fluorescence lifetime of a biological sample (which may be a live biological sample) according to an embodiment of this application. First, in step S10, at least one fluorescence microscopic image dataset is acquired using an optical microscopy imaging system with time-correlated single-photon counting functionality. The fluorescence microscopic image dataset may include a fluorescence intensity dataset and a photon count distribution set, and the fluorescence microscopic image dataset can be represented as {I(m,n)}. MN |{G mn (t)} MN

[0051] Where m is any integer between 1 and M (M is an integer greater than 1), n ​​is any integer between 1 and N (N is an integer greater than 1), I represents the fluorescence image data obtained by the optical microscopy imaging system after the fluorescence generated by the biological sample after being irradiated with excitation light is received by the system, that is, the fluorescence intensity data of the (m,n)th pixel, and G mn (t) represents the photon count distribution of the (m,n)th pixel in the fluorescence image data over time, and {} represents the fluorescence intensity data of all pixels in the entire M×N pixel data or the photon count distribution of all pixels over time.

[0052] In step S20, a dataset for training the neural network (e.g., a denoising neural network or a fluorescence lifetime prediction neural network) is constructed. For example, the training dataset for the fluorescence lifetime prediction neural network and the training dataset for the denoising neural network can be constructed independently of each other or simultaneously.

[0053] Specifically, for the implementation of step S20 above, as a non-limiting example, a photon count distribution set {G} from the fluorescence microscopy image dataset obtained in step S10 is used. mn (t)} MN The photon flow is reconstructed using the photon flow redistribution method described above, resulting in 2×Q photon count distribution sets {G}. mnq ′(t)} MN and {G mnq "(t)} MN This serves as a training dataset for a neural network used to predict fluorescence lifetime. According to a non-restricted example, it can be obtained from a 2×Q photon count distribution set {G}. mnq ′(t)} MN and {G mnq "(t)} MN Q photon count distribution sets are randomly selected from the dataset. For each selected photon count distribution set, a fluorescence lifetime image is calculated using the maximum likelihood algorithm familiar to those skilled in the art, ultimately obtaining Q training fluorescence lifetime images to form the input set for fluorescence lifetime prediction training. Then, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq "(t)} MN The remaining Q photon count distribution sets are used to calculate the fluorescence lifetime image for each photon count distribution set, which is familiar to those skilled in the art. This results in Q training fluorescence lifetime images, which together form the target set or ground value set for fluorescence lifetime prediction training.

[0054] In another non-limiting example, the Q photon count distribution set {G} can be used. mnq ′(t)} MN Fluorescence lifetime images are calculated using the maximum likelihood algorithm familiar to those skilled in the art, ultimately yielding Q training fluorescence lifetime images to form the input set for fluorescence lifetime prediction training; then, the Q photon count distribution sets {G} are further processed... mnq "(t)} MN Fluorescence lifetime images are obtained by using the maximum likelihood algorithm familiar to those skilled in the art, so as to finally obtain Q training fluorescence lifetime images to form a target set or ground value set for fluorescence lifetime prediction training.

[0055] Those skilled in the art should understand that the fluorescence lifetime images in step S20 above can be subjected to maximum normalization before being used as the input set, target set, or ground truth set for fluorescence lifetime prediction training.

[0056] The aforementioned input set for fluorescence lifetime prediction training and the aforementioned target set or truth set for fluorescence lifetime prediction training constitute the training dataset for the fluorescence lifetime prediction neural network.

[0057] In the technical solution of this application, to improve accuracy from a spatial perspective, the data of the pixel to be calculated can be superimposed / merged with its neighboring pixels, and then the maximum likelihood algorithm can be used for calculation. The correlation between the superimposed / merged neighboring pixels can be represented by Bin. For example, Figures 6A and 6B show the cases where Bin = 1 or 2, respectively. As shown in Figure 6A, when Bin = 1, the black square in the figure represents the pixel to be calculated using the maximum likelihood algorithm, while its neighboring pixels with a step size of 1 are represented by hollow squares. The photon counting distribution G represented by these hollow squares... mn (t) are superimposed / merged into the photon count distribution G represented by the black squares respectively. mn In (t), the maximum likelihood algorithm is then used for calculation. As shown in Figure 6B, when Bin=2, the black squares in the figure represent the pixels that will be calculated using the maximum likelihood algorithm, and the adjacent pixels with a step size of 2 pixels are represented by hollow squares. Those skilled in the art should understand that when the pixels represented by the black squares are at the boundary of the image, the photon count distribution outside the corresponding boundary to be superimposed / merged can be regarded as zero. The superposition / merging process can be simply understood as targeting these photon count distributions G mn (t) Summing is performed on the photon counts recorded for each time interval Δt. Those skilled in the art will understand that for the case where Bin = 0, the above summation / merging is unnecessary; instead, the corresponding photon count distribution set can be directly calculated using the maximum likelihood algorithm.

[0058] Furthermore, according to another non-limiting example, in utilizing a photon count distribution set {G} from the fluorescence microscopy image dataset obtained in step S10... mn (t)} MN The photon flow is reconstructed using the photon flow redistribution method described above, resulting in 2×Q photon count distribution sets {G}. mnq ′(t)} MN and {G mnq "(t)} MN Then, the dataset can be created by summing the data over time to train the denoising neural network. According to a non-restricted example, it can be generated from a 2×Q photon count distribution set {G}. mnq′(t)} MN and {G mnq "(t)} MN Q photon count distribution sets are randomly selected. The photon count distribution of each selected photon count distribution set for each pixel is summed over time, and then subjected to maximum value normalization processing well-known to those skilled in the art to generate Q training fluorescence intensity images, which form the input set for image denoising training. Then, 2×Q photon count distribution sets {G mnq ′(t)} MN and {G mnq "(t)} MN The remaining Q photon count distribution sets are summed over time and subjected to maximum value normalization, as is well known to those skilled in the art, to generate Q training fluorescence intensity images, which together form the target set or ground value set for image denoising training. According to another non-limiting example, the Q photon count distribution sets {G} can be... mnq ′(t)} MN The fluorescence intensity images are summed over time and then normalized to their maximum values ​​using methods well-known to those skilled in the art to generate Q training fluorescence intensity images, which form the input set for image denoising training; and the Q photon count distribution sets {G} are then... mnq "(t)} MN The fluorescence intensity images are summed over time and then normalized to their maximum values ​​using methods well known to those skilled in the art to generate Q training fluorescence intensity images, which together form a target set or ground value set for image denoising training.

[0059] The above-mentioned input set for image denoising training and the above-mentioned target set or ground value set for image denoising training constitute the training dataset for the denoising neural network.

[0060] Those skilled in the art should understand that the fluorescence microscopic image dataset used to construct the dataset for training the neural network in step S20 can be a fluorescence microscopic image dataset obtained in step S10 using an optical microscopic imaging system with time-correlated single-photon counting function, or one of multiple fluorescence microscopic image datasets selected in step S10 using an optical microscopic imaging system with time-correlated single-photon counting function.

[0061] In step S30, a neural network for predicting fluorescence lifetime on the dataset acquired by the optical microscopy imaging system and / or a neural network for denoising fluorescence images are constructed. The neural network can be any network familiar to those skilled in the art, including but not limited to U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models. The selected neural network, such as the UNet neural network model or other neural network models, is configured to extract features from the input dataset containing time information. The extracted features can be fed into two convolutional modules (a fluorescence lifetime prediction convolutional module and a denoising convolutional module), and the two modules can output a denoised image and a fluorescence lifetime image, respectively. In this case, the loss function of the neural network is:

[0062] in, The table shows the overall loss function for building the entire neural network. The loss function representing the fluorescence lifetime prediction convolutional module. The loss function represents the denoising convolutional module. μ is a number between 0 and 1. For example, when μ = 0, the neural network can be considered to be dedicated to denoising, while when μ = 1, the neural network can be considered to be dedicated to fluorescence lifetime prediction. By adjusting the size of μ (e.g., adjusting it between 0 and 1), the processing weights between denoising and fluorescence lifetime in the neural network can be adjusted accordingly.

[0063] Those skilled in the art should understand that the constructed neural network may also include only a fluorescence lifetime prediction convolutional module or a denoising convolutional module.

[0064] In step S40, the neural network constructed in step S30 is trained using the training dataset for the fluorescence lifetime prediction neural network constructed in step S20 and / or the training dataset for the denoising neural network. For example, in training the neural network, the initial learning rate can be set to 1×10⁻⁶. -4 The training batch size is 1, and the Adam optimizer is used for backpropagation iterative optimization.

[0065] In step S50, the trained neural network is used to perform denoising and / or fluorescence lifetime prediction processing on the fluorescence microscopic image dataset obtained by the optical microscopic imaging system with time-correlated single-photon counting function.

[0066] According to a non-limiting example, when a neural network is used simultaneously for denoising and fluorescence lifetime prediction, the denoised image and the fluorescence lifetime image can be merged for pseudo-color display.

[0067] Figure 7A illustrates the use of a trained neural network (represented by S) to process a photon distribution set (represented by G) acquired using an optical microscopy system with time-correlated single-photon counting capabilities to obtain a fluorescence lifetime prediction image (represented by G′). Figure 7B illustrates the use of a trained neural network (represented by S) to process a fluorescence image (represented by I) acquired using an optical microscopy system with time-correlated single-photon counting capabilities to obtain a denoised fluorescence image (represented by I′). According to embodiments of this application, the neural network S may simultaneously include a fluorescence lifetime prediction convolutional module and a denoising convolutional module. However, depending on the settings of the neural network (e.g., the setting of μ mentioned above), the fluorescence lifetime prediction convolutional module and / or the denoising convolutional module may be selected separately, and the processing weights between the two may be changed accordingly when processing simultaneously.

[0068] Although specific embodiments of this application are described in detail herein, they are provided for illustrative purposes only and should not be construed as limiting the scope of this application. Furthermore, those skilled in the art will understand that the various embodiments described herein can be used in combination with each other. Various substitutions, modifications, and alterations can be conceived without departing from the spirit and scope of this application.

Claims

1. A training method for a self-supervised microscopic image processing neural network, comprising: A fluorescence microscopic image dataset for a biological sample is acquired using an optical microscopic imaging system with time-dependent single photon counting function, wherein the fluorescence microscopic image dataset comprises a set of photon counting distributions {G mn (t)} MN wherein M, N are integers greater than 1, respectively, and m takes from 1 to M and n takes from 1 to N, and t represents time. A training dataset for a neural network, including a fluorescence lifetime prediction convolutional module and / or a denoising convolutional module, is constructed from the M×N pixel photon count distribution set using a photon flow redistribution method. Each pixel in the photon count distribution set has p time intervals Δt, where p is an integer greater than 1. In the photon flow redistribution, a pair of redistributed photon count distributions is generated for each pixel in the M×N pixel set. Each pair of redistributed photon count distributions also has p time intervals Δt, and the photon counts of each pixel within the p time intervals Δt are randomly assigned to the p time intervals Δt of the pair of redistributed photon count distributions. This photon flow redistribution is repeated Q times to generate Q pairs of redistributed photon count distribution sets, where Q is an integer greater than 1. Specifically, from the Q pairs of photon count distribution sets that have undergone photon flow redistribution, a first group of Q photon count distribution sets that have undergone photon flow redistribution are selected. Each of these sets is then used to generate Q training fluorescence lifetime images using the maximum likelihood algorithm, followed by maximum value normalization, to form the input set for fluorescence lifetime prediction training. Furthermore, from the Q pairs of photon count distribution sets that have undergone photon flow redistribution, a second group of Q photon count distribution sets that have undergone photon flow redistribution are selected. Each of these sets is then used to generate Q training fluorescence lifetime images using the maximum likelihood algorithm, to form the target set or ground truth set for fluorescence lifetime prediction training; or... From the Q pairs of photon count distribution sets that have undergone photon flow redistribution, a first group of Q photon count distribution sets with redistributed photon flows is selected. For each of the Q selected photon count distribution sets with redistributed photon flows, the photon count distribution of each pixel is summed over time and then normalized to its maximum value to generate Q training fluorescence intensity images, which form the input set for image denoising training. From the Q pairs of photon count distribution sets with redistributed photon flows, a second group of Q photon count distribution sets with redistributed photon flows is selected. For each of the Q selected photon count distribution sets with redistributed photon flows in the second group, the photon count distribution of each pixel is summed over time and then normalized to its maximum value to generate Q training fluorescence intensity images, which form the target set or ground truth set for image denoising training. The training dataset includes the fluorescence lifetime prediction training input set and the fluorescence lifetime prediction training target set or ground truth set and / or the image denoising training input set and the image denoising training target set or ground truth set; The method further includes training the neural network using the training dataset.

2. The training method of claim 1, wherein, In the photon flow redistribution, the Q photon count distributions of the first group and the Q photon count distributions of the second group selected from the Q pairs of photon flow redistribution photon count distribution sets are randomly selected.

3. The training method of claim 1, wherein, In the photon flow redistribution, the first group of Q photon flow redistribution photon count distribution sets selected from the Q pairs of photon flow redistribution photon count distribution sets includes one photon count distribution from each pair of photon flow redistribution photon count distributions, while the second group of Q photon flow redistribution photon count distribution sets selected from the Q pairs of photon flow redistribution photon count distribution sets includes another photon count distribution from one photon count distribution in each pair of photon flow redistribution photon count distributions.

4. The training method according to any one of claims 1 to 3, characterized in that, Before using the maximum likelihood algorithm, the data of each pixel to be calculated is superimposed / merged with its neighboring pixels whose correlation (Bin) is 0, 1, or 2.

5. The training method according to claim 3 or 4, characterized in that, The models used to build the neural network include, but are not limited to, U-shaped neural network models, residual neural network models, residual channel attention convolutional neural network models, or Fourier channel attention convolutional neural network models.

6. The training method of claim 5, wherein, The loss function of the neural network is: wherein representing the overall loss function for the entire built neural network, a loss function representative of a fluorescent lifetime prediction convolutional module, The loss function represents the denoising convolutional module, where μ is a number between 0 and 1. The model that constructs the loss function includes, but is not limited to, the mean squared error loss function (MSE), the cross-entropy loss function (CE), or the L2 loss function.

7. The training method of claim 6, wherein, The output features of the model used to build the neural network are independently passed through the fluorescence lifetime prediction convolutional module and the denoising convolutional module, respectively.

8. The training method according to claim 1 or 2, characterized by, A fluorescence microscopy image dataset acquired using an optical microscopy imaging system with time-correlated single-photon counting capability is either a single fluorescence microscopy image dataset or a selected fluorescence microscopy image dataset from multiple acquired fluorescence microscopy image datasets.

9. A method for processing a dataset of fluorescence microscopic images acquired using an optical microscopy imaging system with time-correlated single-photon counting capability, comprising: Construct a neural network that includes a fluorescence lifetime prediction convolutional module and / or a denoising convolutional module; The neural network is trained using the training method according to any one of claims 1 to 8; as well as A trained neural network is used to process a dataset of fluorescence microscopic images acquired by an optical microscopy imaging system with time-correlated single-photon counting capabilities. The processing includes fluorescence lifetime prediction and / or noise reduction.

10. A computer program product comprising computer programs / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images