A module and method for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data

Abstract

This disclosure describes a computing module and method for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data. In particular, the computing module is configured to utilize a trained temporal-to-channel convolutional neural network (T2C CNN) to classify the received fluorescent microscopy video data in one of a plurality of predefined single-molecule binding types.

Description

A MODULE AND METHOD FOR CLASSIFYING SINGLE-MOLECULE BINDING EVENTS IN DIFFRACTION-LIMITED FLUORESCENT MICROSCOPY VIDEO DATA CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims the benefit of priority to Singapore patent application no.10202403434P which was filed on 5 November 2024, the contents of which are hereby incorporated by reference in its entirety for all purposes.TECHNICAL FIELD

[0002] This application relates to a computing module and method for classifying singlemolecule binding events in diffraction-limited fluorescent microscopy video data. In particular, the computing module is configured to utilize a trained temporal-to-channel convolutional neural network (T2C CNN) to classify the received fluorescent microscopy video data in one of a plurality of predefined single-molecule binding types.BACKGROUND

[0003] Single-molecule localization microscopy (SMLM) has transformed the study of biological systems by enabling nanoscale visualization beyond the diffraction limit of light. Landmark techniques such as photoactivated localization microscopy (PALM) and stochastic optical reconstruction microscopy (STORM) have driven this transformation, providing unprecedented insights into molecular structures and dynamics at high resolution.

[0004] However, current approaches for discriminating single-molecule binding types in SMLM are limited. Techniques adopted by those skilled in the art typically involve the use of multiple fluorescent dyes, temporal separation of fluorophores, and analysis of blinking kinetics such as binding on / off times. However, each of these approaches have their own unique drawbacks. For example, the use of different dyes increases experimental complexity and may introduce phototoxic effects or crosstalk. As for temporal separation strategies, such strategies can lead to asynchronization issues, while the analysis of blinking kinetics often requires prolonged observation times. These factors reduce the practicality of such methods in applications where high throughput, live-cell imaging, or real-time analysis is desired.

[0005] Recent advances in deep learning, particularly convolutional neural networks (CNNs), have demonstrated significant improvements in image classification and video analysis tasks. While deep learning has been applied to fluorescence microscopy and singlemolecule analysis, existing solutions do not fully exploit the capability of CNNs to jointly process both spatial and temporal features within a unified model. As a result, the classification of single-molecule binding events in diffraction-limited microscopy data remains a challenge, and there is a need for improved systems and methods that can achieve this with higher accuracy, lower computational burden, and greater suitability for real-time applications.

[0006] Those skilled in the art have proposed a lightweight temporal convolutional variant based on channel squeezing (hereinafter referred to as “SqueezeTime”). In this approach, SqueezeTime takes in short, fixed-length video clips (e.g., 16 frames) and aggregates stacked temporal image channels based on predetermined temporal-adaptive weights. While this design captures some temporal sequence dynamics, it relies on repeated channel-to-time and time-to-channel conversions and weighted convolutions, which leads to a loss of critical spatiotemporal detail, particularly in low signal-to-noise imaging contexts such as fluorescence microscopy.

[0007] Another approach adopted by those skilled in the art is the Encoder-Decoder Temporal Convolutional Network (ED-TCN) approach. The ED-TCN operates by aggregating pre-prepared image features through a hierarchy of temporal convolutions, using encoderdecoder layers with pooling and up-sampling operations While this architecture is useful for frame-level temporal sequence modeling, it is less suitable for use with single-molecule fluorescence data because fragmented associations between frames make it difficult to retain fine-grained spatiotemporal information. In particular, hierarchical temporal convolutions tend to suppress low-level detail features in deeper layers, and pooling / upsampling operations were found to impede the preservation of subtle fluorescence variations necessary for accurate binding-type classification.

[0008] Hence, in view of the issues faced above, those skilled in the art are constantly looking for improved systems and methods that enable more accurate, efficient, and reliable classification of single-molecule binding events in diffraction-limited fluorescent microscopy data. In particular, there is a need for solutions that reduce experimental complexity, minimizephototoxic effects, and overcome the limitations of existing approaches while remaining compatible with high-throughput, live-cell, and real-time imaging applications.SUMMARY

[0009] In one aspect, the present application discloses a computing module for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data. The disclosed computing module comprises a processing unit, and a non-transitory media readable by the processing unit, the media storing instructions that when executed by the processing unit causes the processing unit to receive, using an input module, the fluorescent microscopy video data comprising single-channel grayscale image frames representing diffraction-limited fluorescent spots, the video data characterized by low spatial resolution and a long temporal sequence, and partition the temporal sequence of the image frames of the video data into a plurality of temporal slices. The processing unit then classifies single-molecule binding events in the video data using a trained temporal -to-channel (T2C) convolutional neural network (CNN) module, the trained T2C CNN module comprising a T2C convolution block, a hidden layer transformation block, a multi-layer feature refinement block and a feature classification block. The T2C convolution block is configured to generate a two-dimensional (2D) feature map having a fixed number of channels by applying two-dimensional convolutions to the temporal slices received from the input module, wherein each temporal slice is represented as a channel in the two-dimensional convolutions, and to apply a first batch normalization to the 2D feature map and a first rectified linear unit (ReLU) activation function to the normalized 2D feature map to produce a normalized-activated 2D feature map comprising a plurality of channels. The T2C convolution block then generates a refined 2D feature map having higher-level feature representations by applying two-dimensional convolutions, a second batch normalization and a second ReLU activation function to the normalized-activated 2D feature map. The hidden layer transformation block is then configured to generate a normalized-activated hidden feature map based on the refined 2D feature map. The multi-layer feature refinement block is then configured to generate a normalized-activated intermediate multilayer feature map based on a combination of the feature maps generated by the hidden transformation block and the T2C convolution block and generate a normalized-activated refined multi-layer feature map based on the normalized-activated intermediate multi-layer feature map. The feature classification block then generates a normalized-activated feature vector based on a combination of the feature maps from the T2C convolution block, the hiddenlayer transformation block and the multi-layer feature refinement block and generates a classification output identifying one of a plurality of predefined single-molecule binding types based on the normalized-activated feature vector.

[0010] In embodiments of the one aspect, the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map are performed using kernels defined over spatial and temporal dimensions, and wherein the two-dimensional convolutions further applied in the T2C convolution block, the hidden transformation block, and the multi-layer feature refinement block are performed using kernels having only spatial dimensions.

[0011] In embodiments of the one aspect, the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map employ spatial padding to preserve spatial features and employ strides matched to a length of the temporal slice.

[0012] In embodiments of the one aspect, the input module is further configured to pre-process the fluorescent microscopy video data before partitioning the temporal sequence of the image frames, the pre-processing comprising the steps of correcting background intensity variations in the image frames of the video data to generate background-corrected fluorescent spots, and normalizing pixel intensity values of the image frames by scaling the pixel intensity values to a predetermined range.

[0013] In embodiments of the one aspect, the pre-processing further comprises aligning the background-corrected fluorescent spots over time by performing drift correction on the image frames of the video data to compensate for positional shifts occurring across the temporal sequence.

[0014] In embodiments of the one aspect, the T2C CNN module is trained by minimizing a cross-entropy loss function between inferred classification outputs and ground truth labels, and by updating learnable parameters of the T2C CNN module using stochastic gradient descent. In a further embodiment, the plurality of temporal slices comprises equal-length temporal segments of the video data

[0015] In embodiments of the one aspect, the generation of the normalized-activated hidden feature map based on the refined 2D feature map by the hidden layer transformation block comprises the hidden layer transformation block being configured to apply pointwise two-dimensional convolutions, a third batch normalization and a third ReLU activation function to the refined 2D feature map.

[0016] In embodiments of the one aspect, the generation of the normalized-activated intermediate multi-layer feature map by the multi-layer feature refinement block comprises the multi-layer feature refinement block being configured to apply two-dimensional convolutions, a fourth batch normalization and a fourth ReLU activation function to the feature maps generated by the hidden transformation block and the T2C convolution block

[0017] In embodiments of the one aspect, the generation of the normalized-activated refined multi-layer feature map by the multi-layer feature refinement block comprises the multi-layer feature refinement block being configured to apply two-dimensional convolutions, a fifth batch normalization and a fifth ReLU activation function to the normalized-activated intermediate multi-layer feature map.

[0018] In embodiments of the one aspect, the generation of the normalized-activated feature vector by the feature classification block comprises the feature classification block being configured to flatten the combined feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block to produce a feature vector, refine the feature vector using a first fully connected (FC) layer, and apply a sixth batch normalization and a sixth ReLU activation function to the refined feature vector.

[0019] In embodiments of the one aspect, the generation of the classification output identifying one of a plurality of predefined single-molecule binding types comprises the feature classification block being configured to apply a second FC layer to the normalized-activated feature vector.

[0020] In embodiments of this another aspect, a method for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data using a computing module is disclosed. The disclosed method comprises the steps of receiving, using an inputmodule, the fluorescent microscopy video data comprising single-channel grayscale image frames representing diffraction-limited fluorescent spots, the video data characterized by low spatial resolution and a long temporal sequence, and partitioning the temporal sequence of the image frames of the video data into a plurality of temporal slices; and classifying singlemolecule binding events in the video data using a trained temporal-to-channel (T2C) convolutional neural network (CNN) module. In embodiments of the disclosure, the trained T2C CNN module comprises a T2C convolution block configured to generate a two-dimensional (2D) feature map having a fixed number of channels by applying two-dimensional convolutions to the temporal slices received from the input module, wherein each temporal slice is represented as a channel in the two-dimensional convolutions, apply a first batch normalization to the 2D feature map and a first rectified linear unit (ReLU) activation function to the normalized 2D feature map to produce a normalized-activated 2D feature map comprising a plurality of channels; and generate a refined 2D feature map having higher-level feature representations by applying two-dimensional convolutions, a second batch normalization and a second ReLU activation function to the normalized-activated 2D feature map The T2C CNN module also comprises a hidden layer transformation block configured to generate a normalized-activated hidden feature map based on the refined 2D feature map and a multi-layer feature refinement block configured to generate a normalized-activated intermediate multi-layer feature map based on a combination of the feature maps generated by the hidden transformation block and the T2C convolution block, and generate a normalized-activated refined multi-layer feature map based on the normalized-activated intermediate multi-layer feature map. The T2C CNN module further comprises a feature classification block configured to generate a normalized-activated feature vector based on a combination of the feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block, and generate a classification output identifying one of a plurality of predefined single-molecule binding types based on the normalized-activated feature vector.BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Various embodiments of the present disclosure are described below with reference to the following drawings:Figure 1 illustrates a block diagram of components or modules that are provided within a computing module for classifying single-molecule binding events in diffraction-limitedfluorescent microscopy video data using a computing module in accordance with embodiments of the present disclosure;Figure 2a illustrates schematics of two DNA binding domains used to collect diffractionlimited fluorescent spots with exemplary frames from raw video data being shown next to the corresponding Domains 1 and 2;Figure 2b illustrates the exemplary plots of total intensity against frame index for binding events in Domain 1 whereby a length-based method is used to classify molecules based on the total binding event durations per molecule;Figure 2c illustrates the exemplary plots of total intensity against frame index for binding events in Domain 2 whereby an image-based method classifies individual binding events based on their diffraction-limited fluorescent spots;Figure 3a illustrates a conventional 3D convolutional neural network (CNN) model, Figure 3b illustrates a T2C CNN model in accordance with embodiments of the present disclosure;Figure 4 illustrates the architecture of the T2C CNN model in accordance with embodiments of the present disclosure;Figure 5a illustrates the plots of mean background intensity against the frame index for Experiments 1-4;Figure 5b illustrates an example of background intensity correction,Figure 5c illustrates an example of fluorescence blinking correction;Figure 5d illustrates the plots of drift trajectories in the dx and dy axes for Experiments 1-2; Figure 5e illustrates the plots of drift trajectories in the dx and dy axes for Experiments 3-4; Figure 5f illustrates the quantity and duration distribution of binding events in each of Experiments 1-4;Figure 6 illustrates a block diagram of a processing system for performing embodiments of the present disclosure,Figure 7 illustrates a flow chart showing the process for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data using a computing module in accordance with embodiments of the present disclosure;Figure 8a illustrates two exemplary types of DNA binding domains used to collect diffractionlimited fluorescent spots;Figure 8b illustrates the correlations between diffraction-limited fluorescent spot frames in binding events for Domain 1 and Domain 2;Figure 8c illustrates the correlation between the final classification output of the T2C CNN and common statistics;Figure 8d illustrates the scatter plot of the mean on-time and mean off-time for Domains 1 and 2;Figure 8e illustrates the UMAP dimensionality reduction scatter plot of the output embeddings by the convolutional layers of the proposed T2C CNN module;Figure 8f illustrates the correlation between diffraction-limited fluorescent spot frames in binding events for Domain 1;Figure 8g illustrates the correlation between diffraction-limited fluorescent spot frames in binding events for Domain 2;Figure 9a illustrates a comparison of average classification performance, expressed as accuracy versus measurement time, for different length-based and image-based methods, and the T2C CNN method as evaluated on 25,530 binding events across two domains;Figure 9b illustrates a comparison of confusion matrices for the Probability Density Function (PDF) method, the Video Transformer method, the 3D ResNet-18 method and the T2C CNN method;Figure 9c illustrates a comparison of fluorescence images whereby regions have been marked with dashed lines in all predicted and ground truth images so the same measurement time; Figure 9d illustrates a comparison of fluorescence images that were created by frame-wise summation of the two separated videos representing different binding types;Figure 9e illustrates a comparison of fluorescence super-resolution images reconstructed at 5 s, 1 min, 12 min, 24 min, and 33 min 20 s using image-based binding type classification models; Figure 10a illustrates three exemplary binding types that were used in the multi-class experiments,Figure 10b illustrates the data distributions from the experiments;Figure 10c illustrates the workflow and results of various types of image-based binding type classifications, including the T2C CNN model;Figure 11a illustrates the T2C CNN saliency maps on diffraction -limited fluorescent spots in different DNA binding domains;Figure 11b illustrates T-test of T2C CNN convolution layer outputs grouped by binding types; Figure 11c illustrates a comparison of T2C CNN binding type classification accuracies with different masked frame sections;Figure 11d illustrates the average classification accuracies of T2C CNN for binding events of varying lengths;Figure 11e illustrates feature heatmaps and classification probabilities of T2C CNN on single-frame diffraction-limited fluorescent spots;Figure 12a illustrates examples of generated noise that were added to the original signal, including Poisson light noise with Gaussian decay, Gaussian noise, and hot pixel noise; Figure 12b illustrates the average classification accuracies of image-based binding type classification models under different noise conditions;Figure 13a illustrates the comparison of classification accuracy, computational cost, and model sizes for variants of the proposed T2C CNN;Figure 13b illustrates the various variants of the T2C CNN architecture as evaluated in the ablation study;Figure 14a illustrates HER2-positive AU565 cells that were cultured, fixed, and immunolabeled with custom antibody -oligo conjugates (8nt or lOnt P3’ strands), followed by incubation with imager strands (lOnt P3 strands) in salt-buffered PBS for fluorescence imaging. Exemplary fluorescence images as shown on the corresponding sides of the two domains; Figure 14b illustrates the average binding durations for Domain 1 (8nt-10nt) and Domain 2 (10nt-10nt);Figure 14c illustrates the computed correlations between domain labels and several common statistical features of the fluorescence spots;Figure 15a illustrates bright-field and super-resolution microscopy images of two test HER2-positive AU565 cell samples, with HER2 proteins immunolabeled using 8nt and lOnt DNA strands;Figure 15b illustrates real-time classification of binding events from fluorescence spot videos on the two test samples; andFigure 16 illustrates a plot showing the comparison of classification accuracy, computational cost, and model sizes for single-frame discrimination of dye-labelled binding events.DETAILED DESCRIPTION

[0022] The following detailed description is made with reference to the accompanying drawings, showing details and embodiments of the present disclosure for the purposes of illustration. Features that are described in the context of an embodiment may correspondingly be applicable to the same or similar features in the other embodiments, even if not explicitly described in these other embodiments Additions and / or combinations and / or alternatives as described for a feature in the context of an embodiment may correspondingly be applicable to the same or similar feature in the other embodiments.

[0023] In the context of various embodiments, the articles “a”, “an” and “the” as used with regard to a feature or element include a reference to one or more of the features or elements.

[0024] In the context of various embodiments, the term “about” or “approximately” as applied to a numeric value encompasses the exact value and a reasonable variance as generally understood in the relevant technical field, e.g., within 10% of the specified value.

[0025] As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0026] As used herein, “comprising” means including, but not limited to, whatever follows the word “comprising”. Thus, use of the term “comprising” indicates that the listed elements are required or mandatory, but that other elements are optional and may or may not be present.

[0027] As used herein, “consisting of’ means including, and limited to, whatever follows the phrase “consisting of’. Thus, use of the phrase “consisting of’ indicates that the listed elements are required or mandatory, and that no other elements may be present.

[0028] As used herein, the term “single-molecule binding event” broadly refers to the interaction of an individual molecule with a target binding site that can be detected through fluorescence or other optical signatures in microscopy video data. In some embodiments, such events correspond to DNA binding events, such as those observed in DNA-PAINT (Points Accumulation for Imaging in Nanoscale Topography) experiments where complementary DNA strands transiently hybridize. In other embodiments, one skilled in the art will recognize that the term may encompass other types of molecular interactions, including but not limited to protein-ligand binding, protein-DNA binding, antibody-antigen binding, nucleic acidprotein interactions, or receptor-drug interactions. Accordingly, references to single-molecule binding events in this disclosure are not intended to be limited to DNA-based interactions but rather cover a range of molecular binding phenomena that can be classified using the disclosed computing module

[0029] One skilled in the art will recognize that when two-dimensional convolutions are applied, this refers to a computational operation in which a two-dimensional kernel is systematically shifted across the height and width dimensions of input data, such as an image or a frame of video data, to generate feature maps. At each position, the kernel performs an element-wise multiplication with a corresponding sub-region of the input, and the resulting values are summed to produce a single output value. Repeating this process across all positions across the input data extracts localized spatial features, such as edges, textures, or patterns, while preserving the relative spatial relationships within the input data. The detailed steps of performing two-dimensional convolutions are omitted for brevity in this disclosure as it is a process that is well known to one skilled in the art.

[0030] One skilled in the art will recognize that when three-dimensional convolutions are applied, this refers to a computational operation in which a three-dimensional kernel is systematically shifted across the height, width, and depth dimensions of input data, such as a sequence of video frames or volumetric image data, to generate feature maps. At each position, the kernel performs element-wise multiplication with a corresponding three-dimensional subregion of the input, and the results are summed to yield an output value. This process captures localized spatial and temporal (or volumetric) dependencies simultaneously, thereby enabling the extraction of features that represent both spatial structures and their variation across time or depth. The detailed steps of performing three-dimensional convolutions are omitted for brevity in this disclosure as it is a computational operation that is well known to one skilled in the art.

[0031] One skilled in the art will recognize that certain functional units in this description have been labelled as modules throughout the specification. The person skilled in the art will also recognize that a module may be implemented as circuits, logic chips or any sort of discrete component. Still further, one skilled in the art will also recognize that a module may be implemented in software which may then be executed by a variety of processor architectures In embodiments of the disclosure, a module may also comprise computer instructions or executable code that may instruct a computer processor to carry out a sequence of events based on instructions received. The choice of the implementation of the modules is left as a design choice for a person skilled in the art and does not limit the scope of the claimed subject matter in any way.

[0032] The temporal-to-channel convolutional neural network (T2C CNN) described in this disclosure utilizes previously overlooked discriminative spatiotemporal information within diffraction-limited fluorescent spots. By doing so, the T2C CNN enables synchronous classification of binding types at the single-event level using a single fluorescent dye. To achieve this, the T2C CNN transforms long temporal fluorescence signals by reshaping the temporal dimension into the channel axis, permitting long-term spatial convolutions to capture long-range temporal dependencies. In one embodiment, the T2C CNN incorporates three key concepts, namely long-term spatial convolutions, shallow cross-connected blocks, and a pooling-free design, which in combination enable effective capture of fine-grained temporal context and multi-scale features essential for binding-type classification. This approach provides robust and generalizable performance under noisy imaging conditions, as demonstrated by ablation studies and comparative evaluations against existing deep learning methods Accordingly, the T2C CNN provides an accurate, efficient, and generalizable solution for synchronous single-event classification in single-molecule localization microscopy (SMLM).

[0033] By reshaping the temporal dimension into the channel axis, the T2C CNN enables spatial convolutions to directly capture long-range temporal dependencies across entire video sequences. This approach, combined with the use of shallow cross-connected blocks, allows the T2C CNN to maintain multi-scale fluorescence features and overcome the limitations inherent in state-of-the-art solutions.

[0034] The T2C CNN approach described in this disclosure was validated using DNA-PAINT (Points Accumulation for Imaging in Nanoscale Topography), where it was found that the T2C CNN approach showed a substantial improvement in classification accuracy, increasing from approximately 75% using probability density function (PDF) estimation based on binding time to about 95%. This improvement is achieved together with a significant reduction in measurement time, from around 10 minutes to just 5 seconds, due to the model’s efficient spatiotemporal feature extraction. In comparative evaluations, the T2C CNN further outperformed representative deep learning methods, including 3D ResNet-18, Video Transformer, ED-TCN, and SqueezeTime, which achieved accuracies in the range of 70% to 80% on the same dataset. In addition to superior accuracy, the T2C CNN provides practicaladvantages in computational efficiency, with a model size 1.19 to 77 times smaller and computational requirements reduced by a factor of 4 to 534 relative to these alternatives. These features enable T2C CNN to be deployed in real-time, resource-constrained, or high-throughput single-molecule imaging and sensing applications, underscoring the potential of the T2C CNN to improve the analysis of SMLM and related fluorescence-based data for rapid and precise molecular investigations.

[0035] A block diagram of components or modules that are provided within a computing module for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data using a computing module in accordance with embodiments of the present disclosure is illustrated in Figure 1.

[0036] In general, computing module 100 receives video data 102 and provides it to input module 104. In embodiments of the disclosure, input module 104 may directly provide input video data 102 to T2C CNN module 105 or input module 104 may be configured to preprocess the data, e g., by performing background correction, normalization, or temporal slicing, before providing the preprocessed data to T2C CNN module 105. A T2C block 106 then receives the video data (i.e., the original data or preprocessed data) and proceeds to reshape temporal information contained within the received video data into channel dimensions for subsequent spatial convolutions.

[0037] The output from T2C block 106 is then processed by hidden transformation block 107, which may include convolutional, normalization, or activation layers to extract intermediate representations. Multilayer feature block 108 is then utilized to further process these representations from these blocks to capture higher-order spatial and temporal features. Finally, the extracted features are passed to feature classification block 109, which outputs a classification output identifying one of a plurality of predefined single-molecule binding types. In other words, the output obtained from T2C CNN module 105 is the classification of the single-molecule binding event into one of the predefined single-molecule binding types. It should be noted that this process does not require manually defined video features, which are difficult to discern with the naked eye.

[0038] In embodiments of the disclosure, video data 102 may comprise, but is not limited to diffraction-limited fluorescent microscopy video sequences captured by a microscope, in which conjugated dyes are excited during single-molecule binding events, such as DNA binding events Video data 102 records the appearance and disappearance of fluorescent spots corresponding to these such interactions. These sequences are typically characterized by low spatial resolution, due to the diffraction limit of light, and long temporal sequences, which reflects the binding and unbinding dynamics of the molecules over time, i.e. low spatial-temporal ratio objects. In other words, it can be said that video data 102 comprises singlechannel image frames (e.g., grayscale intensity images) representing diffraction-limited fluorescent spots where the video data is characterized by low spatial resolution and a long temporal sequence.

[0039] As an example, video data 102 may comprise raw video sequences that may be directly associated with binding domains designed to exhibit distinct binding kinetics. As shown in Figure 2a, two exemplary DNA binding domains (i.e., Domains 1 and 2) may be employed to generate diffraction-limited fluorescent spots. Both domains utilize the same dye label (e.g., ATTO532) and the same primary strand (e g., a 10-nucleotide P3 strand) but differ in their complementary sequences. Specifically, Domain 1 includes an 8-nucleotide partial complement of P3, whereas Domain 2 includes a fully complementary 10-nucleotide P3’ strand. As a result, the two domains exhibit different binding behaviors that can be captured in the video data. Further, exemplary frames from the raw video sequences corresponding to each domain are shown alongside the schematic representations of the binding domains.

[0040] It should be noted that existing analysis methods typically are configured to detect and localize fluorescent spots, perform drift correction, group spots and extract multiple binding events at each site to generate a sequence of on-off signals. Such an approach is illustrated in Figures 2b and 2c.

[0041] Binding time, which is also referred to as on-time, is defined as the duration during which an imager strand remains hybridized to a docking strand before dissociation. Binding time can be detected as a local intensity maximum with a gradient decrease toward surrounding pixels and is influenced by factors such as binding affinity and local imager strand concentration. Off-time, also referred to as dark time, is the interval between the dissociationof one imager strand and the subsequent binding of another imager strand to the same docking site, and is determined by parameters including strand concentration and diffusion kinetics.

[0042] Conventional length-based methods classify binding affinities by comparing average binding durations across multiple binding events. While such an approach may be suitable for differentiating interactions with large affinity differences, these methods are less effective when binding types exhibit similar affinities.

[0043] As illustrated in Figure 2b, length-based methods classify molecule types based on aggregated binding durations at a docking site, while as illustrated in Figure 2c, image-based methods classify individual binding events using diffraction-limited fluorescent spots as the input. By extracting meaningful features directly from video data, image-based classification methods are able to improve the resolution of binding type discrimination, reduce the number of repeated observations required, and enables real-time, high-throughput molecular detection. In the examples shown in Figures 2b and 2c, annotated time courses from Domain 1 and Domain 2 respectively both exhibit similar intensity jumps, but Domain 2 demonstrates longer binding durations. This shows that image-based methods are able to distinguish these differences at the single-event level, thereby providing more accurate classification than length-based approaches.

[0044] A conventional 3D convolutional neural network (CNN) model known in the art is illustrated in Figure 3a. The 3D CNN model comprises an input volume 302 having an input width Win, input height Hinand input depth Din. As shown, a convolution kernel of size Wkernel× Hkernel× Dkernelis applied across the three dimensions of the input volume to generate convolutional feature maps. Each kernel is convolved with localized sub-regions of the input data, and the resulting values are aggregated to form 3D convolution result 304 (or output volume 304) having output width Wout, output height Houtand output depth Dout. As shown in Figure 3a, result 306 for each kernel may be defined as:

[0045] Such a video classification model typically employs narrow 3D convolutional kernels or combines small-scale convolutions with sequential models. While 3D convolutions using strides smaller than the kernel size are able to capture fine-grained local features, they also produce an excessive number of transitional features along the temporal dimension, leading to redundancy and an increased risk of overfitting. Conversely, approaches based on small-scale convolutions combined with sequential models emphasize local token dependencies but often fail to capture the global temporal patterns that are critical for the accurate discrimination of binding events.

[0046] With reference to Figure 3a, the function f(x, y, z, c) is defined as the input video, where (x = X, y — Y,z — Z, c — C) denotes the pixel value at the three-dimensional coordinates (X, Y, Z) and the C-th channel in the input video under the assumption that the input video f(x, y, z, c) has a frame width of Win, a frame height of Hin, a number of frames Din, and a number of channels Cin. Therefore, x, y,z, c G No, x < Win, y < Hin, z < Din, and c < Cin■ For values of x,y, z, and c outside these ranges, f(x,y,z, c) is 0 to facilitate the compatibility of padding the expressions.

[0047] The convolution kernel m comprises a width of Wkernel, a height of Hkernel, a depth of Dkernel, and a channel size of Cin, with its value at the three-dimensional coordinates (u, v, w) and the channel index c given by m(u, v, w, c) under the assumption that the convolution strides in width, height, and depth are sw, sh, and sd, respectively, and the padding sizes on both sides for width, height, and depth are pw, ph, pd, respectively.

[0048] Wout, Hout, and Doutrepresent the output dimensions in width, height and depth, respectively. These values can then be calculated as follows:when the number of output channels is Cout, each corresponds to a unique kernel (x)d(u, v, w, c) where the value for d ranges from 1 to Cout. The convolution output of a typical 3D CNN may then be represented as:Conv<3r,,(f,6j) = g30'*...equation (4) wherewhere g3Di,j,k, d')' represents the output of the 3D convolution operation at position (i,j,k, d~) in the output volume. The indices i,j,k iterate over the output dimensions Wout, H0Ut, D0Ut, and Cout, respectively. The sums iterate over the kernel dimensions ^kernel’ kernei, Dkernel, Cin, and u, v, w, c index into the kernel. The function f(x,y, z, c) represents the input video, where values outside the video dimensions are assumed to be 0. The convolution kernel is denoted by <ud(u, v, w, c).

[0049] Figure 3b illustrates a general overview of the T2C CNN module in accordance with embodiments of the present disclosure as compared to the traditional 3D CNN model. Tn conventional approaches, different types of fluorescence signals are typically split into separate channels in fluorescence microscopy. However, distinguishing between different binding types within a single fluorescence channel can substantially increase labelling capacity and reduce the likelihood of non-specific binding events. In the disclosed T2C CNN module, the traditional definition of the channel dimension in CNNs is changed whereby the long temporal dimension from fluorescence microscopy is incorporated as a new channel dimension. This enhances the efficiency of information extraction and improves the accuracy of binding type prediction by the CNN.

[0050] In particular, as illustrated in Figure 3b, the T2C CNN module operates by dividing the input video data into a plurality of non-overlapping temporal slices 312a to 312n, with each slice having a temporal length Tslice. Two-dimensional convolutional kernels of size Wkernel× Hkernelare then applied across the spatial dimensions of each temporal slice. The predictions from all temporal slices are then aggregated at step 315 to form a combined featurerepresentation 314. As shown in Figure 3b, 2D convolution result 316 for each kernel may be defined as:

[0051] With reference to Figure 3b, the output of a T2C convolution (or also known as a 2D feature map) denoted by Conv(T2C), for each fluorescence channel can be represented as:...equation (6) where...qu(T2. Cwhere, gk(i,j, d) represents the output of the temporal -to-channel convolution operation at position (i,j, cT) in the output volume for the A-th temporal slice with length Tsliceand the number of slices is denoted by Nsiice. The indices i,j iterate over the output dimensions Woutand Hout, respectively. The sums iterate over the kernel dimensions Wkernel, Hkernel, Tslice, and u, v, a> index into the kernel. The function / Tx,y,z) represents the input single-channel video, where values outside the video dimensions are assumed to be 0. The convolution kernel is denoted by ωd(u, v, ω).

[0052] Figure 4 illustrates the architecture of the T2C CNN module in accordance with embodiments of the disclosure. Specifically, T2C CNN module 400 comprises T2C convolution block 106, hidden transformation block 107, multilayer feature refinement block 108 and feature classification block 109 Tn embodiments of the disclosure, T2C convolution block 106 comprises the following T2C two-dimensional convolution layer: Conv2D(512,64,3,3)|Stride2,padl - 106a, and two-dimensional convolution layer: Conv2D(512,64,5,5)|Stride5,pad0 - 106d; the following batch normalization (BN) layersBN2d(64) - 106b, and BN2d(64) - 106e; and the following rectified linear unit (ReLU) activation layers ReLU - 106c, and ReLU - 106f. Hidden transformation block 107 may comprise two-dimensional convolution layer Conv2D(64,64,l,l)|Stridel,padO - 107a, batch normalization (BN) layer BN2d(64) - 107b, and rectified linear unit (ReLU) activation layer ReLU - 107c. As for multilayer feature refinement block 108, it may comprise the following two-dimensional convolution layers Conv2D(128,64,l,l)|Stridel,pad0 - 108a, and Conv2D(64,64,l,l)|Stridel,padO - 108d; the following batch normalization (BN) layers BN2d(64) - 108b, and BN2d(64) - 108e; and the following rectified linear unit (ReLU) activation layers ReLU - 108c, and ReLU - 108f. Finally, feature classification block 109 may comprise the following fully connected (FC) layers FC1(192,64) - 109a, and FC2(2,64) -109d; batch normalization (BN) layer BN1d(64) - 109b and rectified linear unit (ReLU) activation layer ReLU - 109c. One skilled in the art will recognize that the dimensions of the various layers, i.e., the number of input and output channels, and the sizes of the kernels, may be varied without departing from this disclosure.

[0053] As illustrated in Figure 4, a batch of images 402 which has been processed by input module 104 (as previously shown in Figure 1), is provided to T2C convolution block 106. In embodiments of the disclosure, batch of images 402 may comprise single-channel grayscale image frames that represent diffraction-limited fluorescent spots and are characterized by low spatial resolution and a long temporal sequence. The temporal sequence of the image frames may then be partitioned by input module 104 into a plurality of temporal slices, and the resulting temporal slices are then provided to T2C convolution block 106 for further processing.

[0054] Upon receiving batch of images 402, which comprise the plurality of temporal slices, T2C convolution block 106 may then perform T2C convolution on these temporal slices, as defined in equations (6) - (8), to produce output

[0055] The outputthen undergoes batch normalization (BN) and rectified linear unit (ReLU) activation. Specifically, the batch normalization is applied to the 2D feature map on a per-channel basis using a first set of learnable parameters, and the rectified linear unit (ReLU) activation function is applied to the normalized convolved 2D feature map to produce a normalized activated 2D feature map comprising a plurality of channels.

[0056] It should be noted that BN is used to accelerate the training of the various blocks of T2C CNN module 400 by reducing internal covariate shifts, and may be defined as:...equation (9)where the mean E[x] and standard deviation Var[x] are calculated per-dimension over the mini-batch input x =represents the number of temporal slices for the b-th sample in a mini -batch with the size of Nbatch. The variables y and / ? are learnable parameter vectors of size (Wout, Hout, Tsi^g), matching the input size. The elements of y and ft are initialized to 1 and 0, respectively. The e is a small value added to avoid zeros under the square root. During training, BN normalizes the input using the current mini -batch statistics and updates the running statistics. During testing, BN normalizes the input using the running statistics accumulated during training.

[0057] The batch normalized convolution outputthen undergoes ReLU activation to accelerate the learning convergence, and may be defined as:ReLU(x) = -^-...equation (10)ReLU filters out the negative values of each element in the mini batch input x —Thus, the output of a T2C layer in T2C convolution block 106, i.e., a refined 2D feature map, may be defined as:...equation (11)where the symbolrepresents the composition of the functions. Hence, it can be said that after being processed by T2C convolution block 106, the original 3D input video segments have been converted into a refined 2D feature map with fixed channels.

[0058] 2D convolutions as defined in equation (12) below and fully connected (FC) layers as defined in equation (13) are then applied to the output from T2C convolution block 106 using hidden transformation block 107 and multi-layer feature refinement block 108, respectively.

[0059] In particular, hidden transformation block 107 is configured to generate a transformed hidden feature map by applying pointwise two-dimensional convolutions to the refined 2D feature map. The pointwise two-dimensional convolutions produce an equaldimensional output that represents element-wise transformed hidden layer features. Multi-layer feature refinement block 108 is then configured to generate an intermediate refined multi-layer feature map by applying two-dimensional convolutions to feature maps generated by hidden transformation block 107 and T2C convolution block 106.

[0060] Finally, feature classification block 109 is configured to generate an intermediate feature vector by refining features from the intermediate refined multi-layer feature map using a first fully connected (FC) layer, and to generate a classification output identifying one of a plurality of predefined single-molecule binding types by applying a second FC layer to the intermediate feature vector.

[0061] In embodiments of the disclosure, the 2D convolutions employed by the respective blocks in T2C CNN module 400 may be defined as:f(sM■ i + u - pM, sh• j + v - ph, c)... equation (12)

[0062] In embodiments of the disclosure, the FC function may be defined as:FC(l, a>, b) = I x u> + b...equation (13)For the equations above,a>i, yirand represent the output, weight, and two learnable parameters of batch normalization (BN) for the / -th layer, respectively.

[0063] First Embodiment for T2C CNN module

[0064] In a first embodiment of the disclosure, T2C convolution block 106 may be configured to integrate the spatiotemporal information of a video by transforming a temporal sequence of grayscale fluorescence images into a channel rich image representation and applying convolutions with large strides. This design effectively captures long-term spatiotemporal variations in fluorescence videos. In this embodiment, the layers in the T2C convolution block may be defined as follows:?! = T2C( / ,...equation (14) l2...equation (15)

[0065] where the Conv^2D^0 function is defined in equation (12) and the T2CQ function is defined in equation (11). From the equations above, it can be seen that the first two layers of T2C convolution block 106 use kernels with strides of 2 and 5, respectively, to abstract the spatial features of diffraction-limited fluorescent spots. The first convolutional layer, l1, abstracts the 512-frame time features of the same binding event into 64-dimensional channel features.

[0066] Hence, it can be said that T2C convolution block 106 generates a two-dimensional (2D) feature map having a fixed number of channels (i.e., as described in equation (6)) by applying two-dimensional convolutions to the temporal slices received from the input module, wherein each temporal slice is represented as an input channel in the two-dimensional convolutions. Batch normalization is then applied to the 2D feature map on a per-channel basis using a set of learnable parameters, and a rectified linear unit (ReLU) activation function is applied to the normalized 2D feature map to produce a normalized-activated 2D feature map Zt, comprising a plurality of channels (i.e., as described in equation (14)).

[0067] A refined 2D feature map l2, having higher-level feature representations is then generated by first applying two-dimensional convolutions, by next applying batch normalization using a set of learnable parameters and finally applying a ReLU activation function to the normalized-activated 2D feature map (i.e., as described in equation (15)).

[0068] Hidden transformation block 107 then generates a transformed hidden feature map based on the output from T2C convolution block 106. In this embodiment, hidden transformation block 107 is understood to comprise two layers as hidden transformation block 107 is repeated twice as shown in Figure 4 (i.e., as shown by the notation “x2”) to produce progressively refined hidden feature maps that capture higher-level spatiotemporal representations for subsequent processing. These two layers in hidden transformation block 107 may be defined as follows:l3Y3lP ))...equation (16) Z4= ReLU...equation (17)

[0069] The third, Z3, and / or fourth, Z4, convolutional layers of T2C CNN module 400 may use pointwise convolution (1 x 1 kernel) with equal dimensional output to transform the hidden layer features element wise. Hence, it can be said that hidden transformation block 107 generates a normalized-activated hidden feature map by applying pointwise two-dimensional convolutions, batch nonnalization and ReLU activation to the refined 2D feature map whereby the pointwise two-dimensional convolutions produce an equal-dimensional output that represents element-wise transformed hidden layer features (i.e., as described in equation (16)).

[0070] When repeated, this simply means that a further normalized-activated hidden feature map is generated by applying pointwise two-dimensional convolutions, followed by batch normalization and ReLU activation to the normalized-activated hidden feature map Z3(i.e., as described in equation (17)).

[0071] Multilayer feature refinement block 108 then utilizes a combination of feature maps generated by T2C convolution block 106 and hidden transformation block 107 to generate a normalized-activated intermediate multi-layer feature map. In this embodiment, the two layers in multilayer feature refinement block 108 may be defined as follows:Z5— ReLUY5,p5))...equation (18)...equation (19)where the operation “[•,•]” denotes the concatenation of channels.

[0072] In particular, the fifth convolutional layer, l5, of multilayer feature refinement block 108 simultaneously receives the feature maps from both T2C convolution block 106 and hidden transformation block 107, and applies two-dimensional convolutions, a batch normalization and a ReLU activation function to these combined feature maps to generate a normalized-activated intermediate multi-layer feature map.

[0073] The normalized-activated intermediate multi-layer feature map is then passed to the sixth convolutional layer, / 6, for one final transformation in preparation for classification. Hence, it can be said that multilayer feature refinement block 108 is configured to generate a normalized-activated intermediate multi-layer feature map by applying two-dimensional convolutions, a fourth batch normalization and a fourth ReLU activation function to the feature maps generated by the hidden transformation block 107 and the T2C convolution block 106 (i.e., as described in equation (18)). It then generates a normalized-activated refined multi-layer feature map by applying two-dimensional convolutions, batch normalization and a ReLU activation function to the normalized-activated intermediate multi-layer feature map (i.e., as described in equation (19)).

[0074] Feature classification block 109, which comprises the first fully connected layer of T2C CNN module 400 further refines the features that have been abstracted, transformed, and refined by all the above convolutional steps, before passing them to the second fully connected layer for classification. In this embodiment, the two layers in the feature classification block may be defined as follows:...equation (20) y = Softmax ° FC(l7, )Q,bQ)...equation (21) where... equation (22)...equation (23)where the output y of the last layer represents the predicted probabilities for each binding type. The function “ Flatten^-')” reshapes the input tensor into a one-dimensional vector per sample and the operation “[•>•]” denotes the concatenation of feature dimensions in fully connected layers. The function “FC( )” in equation (23) represents an index of the flattened layer output whereby j represents the index of the FC output, and the weight a>(i, f) is an index-to-weight mapping function of the FC weight matrix. Further, b7, b8and b(i) represent the learnable bias function in the FC layer, which maps each output index i to its associated bias value b(i).

[0075] Hence, it can be said that feature classification block 109 is configured to flatten the combined feature maps from T2C convolution block 106, hidden transformation block 107 and multilayer feature refinement block 108 to produce a feature vector, and refine the feature vector using a first fully connected (FC) layer. It then generates a normalized-activated feature vector by applying batch normalization and a ReLU activation function to the refined feature vector (i.e., as described in equation (20)). It then generates a classification output identifying one of a plurality of predefined single-molecule binding types by applying a second FC layer to the normalized-activated feature vector (i.e., as described in equation (21)).

[0076] Second Embodiment for T2C CNN module

[0077] In a second embodiment of the disclosure, the layers of T2C convolution block 106 are essentially the same as the first embodiment and may be defined as follows:= T2C f,a>1,y1,p1')...equation (24) Z2— ReLU...equation (25)

[0078] where the Conv ^Q function is defined in equation (12) and the T2CQ function is defined in equation (11). From the equations above, it can be seen that the first two layers of T2C convolution block 106 use kernels with strides of 2 and 5, respectively, to abstract the spatial features of diffraction-limited fluorescent spots. The first convolutional layer,abstracts the 512-frame time features of the same binding event into 64-dimensional channel features.

[0079] Hidden transformation block 107 then generates a transformed hidden feature map based on the output from T2C convolution block 106 In this embodiment, the two layers in the hidden layer transformation block may be defined as follows:l3= ReLU...equation (26)l4- Cat, Z2]...equation (27)

[0080] Hidden transformation block 107 transforms the hidden layer features elementwise by pointwise convolution (Hl kernel) with an equal -dimensional output. These transformations improve feature representation for binding type classification. The output of layer Z4is the concatenation (denoted by Cat[-]) of the convolution result from layer Z3with the output from Z2, which is the output from T2C convolution block 106. This operation provides multi-scale convolutional features for subsequent feature fusion in the next block.

[0081] Multilayer feature refinement block 108 then integrates features before and after transformations and prepares them for the final step of binding type classification. In this embodiment, the two layers in multilayer feature refinement block 108 may be defined as follows:Z5— ReLU (BN Conv(2D)(l4, ...equation (28)Z6= Flatten...equation (29)where the function '‘Flatten(-)" reshapes the input tensor into a one-dimensional vector for each sample, making it compatible with the subsequent fully connected (FC) layer.

[0082] Feature classification block 109 comprises of two fully connected (FC) layers. The first refines the features processed by the preceding blocks through fusion, transformation, and abstraction, while the second performs the final classification. They are defined as follows:...equation (30)The activated hidden features are then passed to the final fully connected layer for classification:y slice - Softmax o FC (Z7, a>P / , h8• • • equation (31) wherey = 77^— slice y slice ■ ■ ■ equation (32)where the output ysucefrom the final layer represents the predicted probabilities for each temporal slice, normalized using the SoftMax function. For multiple temporal slices from the same binding event, the average (y) of their predictions are then utilized. As mentioned in the previous section, b7, bsand b represent the learnable bias function in the FC layer, which maps each output index i to its associated bias value b(i).

[0083] Training of the T2C CNN module

[0084] In embodiments of the disclosure, a cross-entropy loss function is used to train T2C CNN module 400, and the cross-entropy loss function is defined as follows:...equation (33)where yij and y^ represent the predicted and actual probabilities of the z-th sample belonging to the j-th binding type, respectively. Stochastic gradient descent is employed on a single Tesla VI 00 32G GPU to update the parameters of T2C CNN module 400 and minimize the loss defined in equation (33). The training set is randomly partitioned into five equal subsets. For each experiment, one subset is designated for validation to evaluate the model’s classification performance, while the remaining four subsets are used for training. The training process is terminated when the validation performance does not improve for 100 consecutive epochs, and the model that exhibits the highest validation performance is retained for testing. In the described training environment (CPU: Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz, GPU: Tesla V100 32G), reading 3,000 binding events requires approximately five minutes, and training each epoch requires about three seconds. The optimal model is typically identifiedafter approximately 200 epochs, such that the complete training process for 3,000 binding events, including data reading, is completed in about 15 minutes.

[0085] Testing of the T2C CNN module

[0086] In embodiments of the disclosure, once the training of the T2C CNN module 400 is completed, the T2C CNN module 400 may be used to predict the binding type c of a binding event given a sequence of diffraction -limited fluorescent spots, as expressed by:c ~ arg...equation (34)where ycdenotes the c-th dimension of the model’s output y, representing the probability that the binding event belongs to the c-th binding type. In this testing environment, it was found that reading the data for all 31,479 binding events in a 33-minute and 20-second observation video takes approximately 25 minutes, and predicting the binding types takes about 18 seconds.

[0087] Data preprocessing

[0088] In embodiments of the disclosure, before the video data is partitioned by input module 104, input module 104 may be configured to preprocess the raw microscope images. Initially, due to continuous laser excitation during fluorescence imaging, the number of photons received by the microscope gradually decreases and fluctuates over time. This results in variations in the overall intensity of the raw microscope images, with noticeable differences observed between experiments depending on the specific laser and dye conditions.

[0089] To estimate the background intensity of each frame, an open-source software package used in SLMM called Picasso was employed to localize diffraction-limited fluorescent spots in the raw microscope images, and all detected spots were masked out so that the average intensity of the remaining pixels could be calculated. As shown in Figure 5a, substantial framelevel variations and significant differences between experiments were observed in the background intensity across four experiments from the two domains. To mitigate the effect of such background intensity fluctuations on fluorescence signal measurements, the background intensity was subtracted from the detected fluorescent spots, and a small constant value (e g.,100, to distinguish from zero-padding) was assigned to pixels below the background intensity threshold

[0090] An example of background intensity correction is shown in Figure 5b, where the background intensity changed significantly within only three frames (0.3 seconds), making the fluorescent spots appear brighter after correction. Following this correction, the fluorescence intensity remained stable beyond 0.3 seconds. Prior to inputting the background-corrected data into the model, all intensity values were further normalized by dividing by a scaling constant to reduce the numerical range to a suitable level (e.g., <10). In the training dataset, the raw intensities ranged from 87 to 4569, so a scaling constant of 500 was selected and consistently applied during both training and testing.

[0091] Fluorescence blinking and detection algorithm errors may cause intermittent losses of fluorescent spots within a single binding event, particularly at spatial edges or in regions of high overlap As illustrated in Figure 5c, two binding-on signals detected in close spatial and temporal proximity were considered to originate from the same binding event, with intervening frames padded with zeros.

[0092] To account for positional drift of the sample slide, drift correction was applied using Picasso Render, as shown in Figures 5d and 5e, thereby producing groups of binding events aligned to binding site locations. Subsequently, as illustrated in Figure 5f, the number of binding events per site (denoted as “#Binding events / group”) and the duration of each binding event were analysed over a 33-minute and 20-second observation window across four experiments in the two domains. The number of binding events per location was generally concentrated between 2 and 4, while binding durations were concentrated between 0 and 10 seconds. Compared with the partially matched 8nt-10nt binding events in Domain 1, the fully matched 10nt-10nt binding events in Domain 2 exhibited smoother distributions and longer binding durations.

[0093] In a cross-experiment binding type classification task, the data from the first two experiments were used to train the models, while the data from the last two experiments were used for testing. This approach demonstrated the robustness of the proposed model across experimental conditions with differing background intensities and binding kinetics.

[0094] In summary, the video data may be pre-processed by correcting background intensity variations in the image frames of the video data to generate background-corrected fluorescent spots, by normalizing pixel intensity values of the image frames by scaling the pixel intensity values to a predetermined range, and / or by aligning the background-corrected fluorescent spots over time by performing drift correction on the image frames of the video data to compensate for positional shifts occurring across the temporal sequence.

[0095] DNA sample preparation and fluorescence microscopy

[0096] In embodiments of the disclosure, the DNA-origami configuration used in this paper was obtained from the design module of the open-source software suit Picasso. Four docking strands per origami were used at the 4 comers with the sequence of docking and imager strands being adapted from other state of the art solutions. The origami is prepared by mixing the M13mpl8 single-stranded scaffold DNA (10 nM), core staples (100 nM), biotinylated staples (100 nM), and staples with docking strand (1 pM) and annealing in a thermocycler. First, the temperature is raised to 80°C, followed by a thermal gradient from 60°C to 4°C in 3 hours. The origami was then purified by centrifugal filtration using Amicon 0.5ml 50k MWCO, at 5000 Xg for 6 minutes followed by collection at 5000xg for five minutes. The substrate is prepared by cleaning a coverslip and sticking it with a CoverWell perfusion chamber. The origami is immobilized on the coverslip with BSA-biotin -streptavidin linkage, and the chamber is filled with 1 nM imager solution (imaging buffer consists of lxPBS, 500 mM NaCl, and saturated with Trolox). To limit fluorescence spots per frame and reduce overlap, 400 pM was used for the lOnt docking strand (with longer binding time) and 1 nM was used for the 8nt and 6nt strands (with shorter binding times). It should be noted that all experiments were conducted on separate samples.

[0097] With regard to the fluorescence microscopy setup, the transient binding of an imager strand and the resulting immobilization of the fluorophore on a docking strand of the DNA origami are captured using total internal reflection fluorescence (TIRF) imaging mode. An inverted microscope (IX-71, Olympus, Tokyo, lapan) with an oil immersion objective (PlanApo, 100x, NA 1.5, Olympus) was utilized. A 532 nm laser (Samba, Cobolt, AB, Sweden) is coupled with TIRF illuminator model IX2-RFAEVA-2 (Olympus) through an optical fiber.The laser is focused on the back-focal plane of the objective, away from the center to achieve TIRF. The laser power density is measured after the objective and determined to be 125 W / cm2. The emission is collected through the same objective, passed through a dichroic beamsplitter (Semrock, Di03-R405 / 488 / 532 / 635- tl-25x36) and emission filter (Semrock, NF03-405 / 488 / 532 / 635E-25), imaged with a scientific complementary metal-oxide semiconductor (sCMOS) camera (ORCA-Flash 4.0, Hamamatsu, Shizuoka, Japan). The image sequences are then collected at a frame rate of 10 Hz, and analysed without further processing.

[0098] Exemplary processing system

[0099] In accordance with embodiments of the present disclosure, a block diagram representative of components of processing system 600 that may be provided within computing module 100, and / or any of the modules shown in Figure 1 to carry out the computing and processing functions in accordance with embodiments of the disclosure is shown in Figure 6. One skilled in the art will recognize that the exact configuration of each processing system provided within these modules may be different and the exact configuration of processing system 600 may vary and the arrangement illustrated in Figure 6 is provided by way of example only.

[0100] In embodiments of the disclosure, processing system 600 may comprise controller 601 and user interface 602. User interface 602 is arranged to enable manual interactions between a user and the computing module as required and for this purpose includes the input / output components required for the user to enter instructions to provide updates to each of these modules. A person skilled in the art will recognize that components of user interface 602 may vary from embodiment to embodiment but will typically include one or more of display 640, keyboard 635 and optical device 636.

[0101] Controller 601 is in data communication with user interface 602 via bus 615 and includes memory 620, processing unit or processor 605 mounted on a circuit board that processes instructions and data for performing the method of this embodiment, an operating system 606, an input / output (VO) interface 630 for communicating with user interface 602 and a communications interface, in this embodiment in the form of a network card 650. Network card 650 may, for example, be utilized to send data from these modules via a wired or wirelessnetwork to other processing devices or to receive data via the wired or wireless network. Wireless networks that may be utilized by network card 650 include, but are not limited to, Wireless-Fidelity (Wi-Fi), Bluetooth, Near Field Communication (NFC), cellular networks, satellite networks, telecommunication networks, Wide Area Networks (WAN) etc.

[0102] Memory 620 and operating system 606 are in data communication with processor 605 via bus 610. The memory components include both volatile and non-volatile memory and more than one of each type of memory, including Random Access Memory (RAM) 623, Read Only Memory (ROM) 625 and a mass storage device 645, the last comprising one or more solid-state drives (SSDs). One skilled in the art will recognize that the memory components described above comprise non-transitory computer-readable media and shall be taken to comprise all computer-readable media except for a transitory, propagating signal. Typically, the instructions are stored as program code in the memory components but can also be hardwired. Memory 620 may include a kernel and / or programming modules such as a software application that may be stored in either volatile or non-volatile memory.

[0103] Herein the term “processor” or “processing unit” is used to refer generically to any device or component that can process such instructions and may include: a microprocessor, a processing unit, a microcontroller, a programmable logic device or other computational device. That is, processor 605 may be provided by any suitable logic circuitry for receiving inputs, processing them in accordance with instructions stored in memory and generating outputs (for example to the memory components or on display 640). In this embodiment, processor 605 may be a single core or multi-core processor with memory addressable space. In one example, processor 605 may be multi-core, comprising — for example — an 8 core CPU. In another example, it could be a cluster of CPU cores operating in parallel to accelerate computations.

[0104] A flowchart which sets out the process for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data using a computing module in accordance with embodiments of the present disclosure is illustrated in Figure 7. In embodiments of the disclosure, process 700 as illustrated in Figure 7 may be performed by computing module 100 or any combination of modules described in the sections above.

[0105] Process 700 begins at step 702 by receiving, using an input module, the fluorescent microscopy video data comprising single-channel grayscale image frames representing diffraction-limited fluorescent spots, the video data characterized by low spatial resolution and a long temporal sequence. Process 700 then partitions the temporal sequence of the image frames of the video data into a plurality of temporal slices. At step 704, process 700 then proceeds to classify single-molecule binding events in the video data using a trained temporal-to-channel convolutional neural network (T2C CNN) module.

[0106] In embodiments of the disclosure, the trained T2C CNN module comprises T2C convolution block 706 that is configured to generate a two-dimensional (2D) feature map having a fixed number of channels by applying two-dimensional convolutions to the temporal slices received from the input module, wherein each temporal slice is represented as a channel in the two-dimensional convolutions. T2C convolution block 706 then applies a first batch normalization to the 2D feature map and a first rectified linear unit (ReLU) activation function to the normalized 2D feature map to produce a normalized-activated 2D feature map comprising a plurality of channels before generating a refined 2D feature map having higher-level feature representations by applying two-dimensional convolutions, a second batch normalization and a second ReLU activation function to the normalized-activated 2D feature map.

[0107] The trained T2C CNN module also comprises hidden layer transformation block 708 that is configured to generate a normalized-activated hidden feature map based on the refined 2D feature map. Multi-layer feature refinement block 710 contained within the trained T2C CNN module is then configured to generate a normalized-activated intermediate multi-layer feature map based on a combination of the feature maps generated by the hidden transformation block and the T2C convolution block, and generate a normalized-activated refined multi-layer feature map based on the normalized-activated intermediate multi-layer feature map. Feature classification block 712 contained within the trained T2C CNN module is then configured to generate a normalized-activated feature vector based on a combination of the feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block, and to generate a classification output identifying one of a plurality of predefined single-molecule binding types based on the normalized-activated feature vector.

[0108] In embodiments of the disclosure, the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map are performed using kernels defined over spatial and temporal dimensions, and wherein the two-dimensional convolutions further applied in the T2C convolution block, the hidden transformation block, and the multi-layer feature refinement block are performed using kernels having only spatial dimensions.

[0109] A typical DNA binding design in DNA- Point Accumulation for Imaging in Nanoscale Topography (DNA-PAINT) is used in this disclosure and this is illustrated in Figure 8a, with different docking strands for the same lOnt imager strand 5’-GTAATGAAGA-3’: partially complementary 8nt 5’-TT-TCTTCATT-3’ (Domain 1) and fully complementary lOnt 5’ -TT -TCTTC ATT AC-3’ (Domain 2), where “-TT-” is a spacer between the docking strands and the DNA origami. Appropriate concentrations were chosen: 1 nM imager strands, 1 nM domain-1 docking strands, and 200 pM domain-2 docking strands. The concentration of domain-2 docking strands was reduced to decrease the probability of spatial and temporal overlap of binding events at different sites, due to its longer binding time. The fluorescence microscopy of 20,000 frames (33 minutes and 20 seconds duration, 10 fps frame rate) were measured for both domains, analyzed using Picasso to identify 4,977 and 1,183 binding sites with 19,457 and 6,073 binding events for domains 1 and 2, respectively.

[0110] In the subsequent experiments, 25,530 binding events from the two domains were analyzed by applying background and blinking corrections to the diffraction-limited fluorescence spots, computing inter-frame correlations and summary statistics, and evaluating their association with domain labels. Based on the interframe correlations of binding events within the two domains, it was determined that Domain 2 displays smooth correlations in the early frames of binding, suggesting stable and uniform interaction dynamics from the early stage of DNA binding. In contrast, it was found that Domain 1 exhibits dynamic and periodic changes in inter-frame correlations, indicating complex and varied interaction dynamics. This result aligns with conclusions from established studies on DNA hybridization and the impact of oligodeoxyribonucleotide structures on the fluorescent properties of conjugated dyes.

[0111] It is useful to note that frame correlation quantifies the similarity between frames based on pixel intensity patterns, revealing temporal and structural consistency. In embodiments of the disclosure, frame correlation may be defined as follows. Given a videorepresented as a sequence of frames, the Pearson correlation coefficient (PCC) is computed between all pairs of non-zero frames, and the resulting correlation matrix is accumulated across different video samples belonging to the same class. Let xtand xsdenote the vectorized pixel intensities of frames t and s, respectively. The Pearson correlation coefficient between two frames is defined as:...equation (35)where xt iis the intensity of pixel i in frame t. ptand psare the mean pixel intensities of frames t and s respectively. The denominator normalizes the covariance, ensuring the correlation values are within the range [-1; 1 ].

[0112] For each class label y, the accumulated frame correlation matrix is computed as:...equation (36)where pn(t, s) is the correlation coefficient between frames t and s in the / / -th video sample. Since different frame pairs appear with varying frequencies in the dataset, the final averaged correlation matrix is computed as:...equation (37)where Ny(t, s) is defined as the number of times the correlation between frames t and s were computed and e is defined as a small positive constant (e g., 10-6) to avoid division by zero. This ensures that correlation values are normalized by their respective occurrence counts, providing an unbiased estimate of frame-wise dependencies.

[0113] When the average inter-frame Pearson correlations for both domains were compared, it was found that initially, both Domain 1 (8nt-10nt) and Domain 2 (10nt-10nt) showstable correlations. However, Domain 2 maintains this stability longer, while Domain 1 rapidly develops alternating blocks of high and low correlation, mainly due to fewer long-duration events. These intermittent low-correlation segments in Domain 1 likely stem from peak shifts, spot shape changes, or transient fluctuations that are more prevalent due to its weaker binding affinity. Such frame-to-frame correlation patterns provide useful signals for distinguishing binding types in fluorescence videos.

[0114] In embodiments of the disclosure, the frame correlation patterns may be analyzed as follows. It was noted that a particularly notable deviation occurs around frame 320 in Domain 1, where frames exhibit high mutual correlation but lower correlation with subsequent frames. Upon examining the raw data, it was found that among the few binding events lasting over 300 frames, the longest event (655 frames) displayed two closely positioned spots between frames 308 and 334. This interference caused a fluorescence pattern distinct from the typical singlefluorescence pattern observed in other frames, leading to a drop in correlation with later frames, especially when fewer binding events followed.

[0115] To further highlight these fluorescence features, representative inter-frame correlation patterns were sampled from the data. Beyond the correlation variations, subtle differences in fluorescence images were observed. For example, Domain 1 (8nt-10nt) exhibits slightly greater fluctuations in peak positions and shapes compared to Domain 2 (10nt-10nt). Frames such as 198 and 620 in Domain 1 show a fluorescence shift toward the right, reducing their correlation with other frames. The shorter binding duration in Domain 1 also minimizes the impact of photobleaching. In contrast, the first long binding event in Domain 2 undergoes noticeable photobleaching in its later stages, reducing the contrast between the fluorescence spot and the background. However, peak positions and shapes remain relatively stable compared to Domain 1, providing a basis for differentiation. The final two blue lines in the Domain 2 inter-frame correlations result from transient blinking (signal loss) near the end of the longest binding event.

[0116] As shown in Figure 8b, besides the known binding time, common statistics of diffraction-limited fluorescent spots were found to reveal up to 0.5 Pearson correlation with binding type, beyond binding time alone. The four main types of novel information summarized by these common statistics are as follows.

[0117] 1) Sum features: “Sum”, “YZ sum”, “XZ sum”, “XY sum”: Compared to the fully matched lOnt- lOnt DNA strand binding, the total fluorescence intensity within a binding event of partially matched 8nt-10nt DNA strand binding is relatively lower, including fluorescence images projected onto width-time (YZ), height-time (XZ), and height-width (XY) planes.

[0118] 2) Non-zero features: “non-zero”, “YZ non-zero”, and “XZ non-zero”: Frames with transient disappearance or localization loss during the binding event are padded with zeros. Compared to the partially matched 8nt-l Ont DNA strand binding, the fully matched 10nt-1 Ont DNA strand binding exhibits less blinking / localization loss within a binding event, including fluorescence images projected onto width-time (YZ) and height-time (XZ) planes. However, for fluorescence images projected onto the height- width (XY) plane (“XY non-zero”), the time axis is overlapped and can no longer reflect the blinking / localization loss within a binding event.

[0119] 3) Non-masked features: “non-masked”, “YZ non-masked”, and “XZ non-masked”: Pixels around the fluorescent spot with intensity lower than the average background fluorescence intensity are masked. Compared to the fully matched 10nt-10nt DNA strand binding, the partially matched 8nt-10nt DNA strand binding exhibits weaker intensity at the edges of the fluorescent spots within a binding event, including fluorescence images projected onto width-time (YZ) and height-time (XZ) planes. For fluorescence images projected onto the height-width (XY) plane (“XY non-masked”), due to the overlapping time axis, there is almost no specific spatial location (xO, yO) that is masked in all time frames.

[0120] 4) XY statistics: “XY mean”, “XY std”, “XY min”, “XY max”, “XY median”, “XY range”: Compared to the fully matched 1 Ont-1 Ont DNA strand binding, the partially matched 8nt-10nt DNA strand binding shows weaker average intensity, smaller standard deviation (the fluorescence center’s position change over time more uniformly covers various spatial locations with greater randomness), and a smaller overall range of fluorescence intensity in the fluorescence images projected onto height-width (XY) plane.

[0121] By voting on binding types according to all the statistics in Figure 8b, weighted by their correlations with the binding type, it was found that the classification accuracy in bothdomains could still be improved by 4.05% and 37.75%, respectively, due to missing information The T2C CNN described in this disclosure captures additional information beyond these statistics, reducing the impact of missing information on accuracy to less than 4%, as shown in Figure 8c.

[0122] In this disclosure, the term “missing information" is defined as the gap between the current classification accuracy and an ideal scenario where all discriminative features are captured (i.e., 100% accuracy). This gap quantifies the amount of additional information needed to achieve perfect classification. By voting on binding types using all the statistics in Figure 8c weighted by their correlation with binding type, it was found that the classification accuracy could potentially be improved by 4.05% in Domain 1 and 37.75% in Domain 2. These differences indicate that the available statistics capture most of the necessary discriminative features in Domain 1, but much less so in Domain 2, where a substantial portion of key information is missing. The T2C CNN module described in this disclosure compensates for the missing information by extracting additional spatiotemporal features from fluorescence videos, recovering 0.25% in Domain 1 and 34.72% in Domain 2, ultimately improving classification accuracy to 96% in both domains. This highlights the effectiveness of the T2C CNN module in enhancing image-based binding type classification, where traditional image statistical features are insufficient.

[0123] To demonstrate the contribution of different information sources in distinguishing binding types, the mean on-time and mean off-time were plotted for all binding events observed at 4,977 binding sites in Domain 1 and 1,183 binding sites in Domain 2 over 33 minutes and 20 seconds (10 frames per second) and this is shown in Figure 8d. The Gaussian kernel density estimate of the probability distribution function (PDF) for the mean on-time is shown at the bottom of the figure, with the overlapping area between the two domains accounting for approximately 9.92% of their union area. The misclassified binding sites account for approximately 6.33% (396 out of 6,160) of the total binding sites if a classification threshold (about 4.9 s) at the intersection of the two PDFs is employed.

[0124] Figure 8e illustrates the Uniform Manifold Approximation and Projection (UMAP) dimensionality reduction scatter plot of the output embeddings by the convolutional layers of the T2C CNN module described in this disclosure. Each point represents the position of afeature vector extracted by the T2C CNN module from a binding event’s diffraction-limited fluorescent spots after dimensionality reduction. The misclustered binding events account for approximately 3.62% (923 out of 25,530) of the total binding events. By comparing the results, it can be seen that even when determining binding types based on all binding events at each site, the PDF method still has a 6.33% error rate. In contrast, the T2C CNN module was able to reduce the event-level classification error rate to a lower 3.62% without the need to track and combine binding events at a site. This result emphasizes that diffraction-limited fluorescent spots contain more information useful for distinguishing binding types, and the T2C CNN module was effective in extracting this information from the diffraction-limited fluorescent spots.

[0125] Figures 8f and 8g illustrate the interframe correlations of binding events within the Domain 1 and 2, respectively. These plots show that Domain 2 displays smooth correlations in the early frames of binding, suggesting stable and uniform interaction dynamics from the early stage of DNA binding. In contrast, Domain 1 exhibits dynamic and periodic changes in interframe correlations, indicating complex and varied interaction dynamics

[0126] Comparison of existing methods through cross-experiment studies for discriminating binding types.

[0127] Using the same DNA binding type design as described in the previous sections, these studies evaluate the classification performance of the models under conditions of increased docking strand concentration. Specifically, the concentration of the Domain-2 docking strand was doubled from 200 pM to 400 pM, and fluorescence microscopy images of equal duration were recollected for both domains. The resulting binding event data were analysed, and classification models were constructed to predict binding types under these new experimental conditions. The classification methods applied include: (1) PDF: Gaussian kernel density estimation is used during training to determine the probability density function of binding event durations. During testing, observed binding event durations at a site are compared against the likelihood distributions of different domains to infer the binding type. (2) MLP: Binding event durations within each domain are aggregated and segmented to train a multi-layer perceptron. During testing, binding event durations at a binding site are similarly segmented to produce binding type predictions. (3) 3D ResNet-18: A standard 3D CNN framework widely used in video classification tasks. (4) T2C CNN: The temporal -to-channel convolutional neuralnetwork described in this disclosure was specifically designed to learn and classify binding types directly from stacks of diffraction-limited fluorescent spots corresponding to single binding events.

[0128] Classification accuracy and measurement time (with standard deviations) for all methods are shown in Figure 9a. As illustrated, it can be seen that the PDF and MLP models, that rely on binding durations, reach only -75% accuracy per event and require at least 3 minutes of observation to exceed 85%. In contrast, image-based methods (dashed edges in the plot), such as 3D ResNet-18, achieve -80% accuracy per 5-second binding event. The T2C CNN was able to attain —95% accuracy with only one 5-second event by using cross-connected long-term spatial convolutions. Unlike traditional 3D CNNs (e.g., 3D ResNet-18) with shallow temporal strides, T2C CNN treats long temporal stacks as channels, enabling 2D convolutions equivalent to long-step 3D convolutions. This design captures frequency variations across spatial regions, effectively encoding DNA binding dynamics that can affect fluorescence intensity, angle, and density. For instance, hybridization-induced changes in fluorophore orientation can alter emission polarization or angular distribution. Leveraging these insights greatly improves the efficiency of binding type classification.

[0129] Figure 9b illustrates confusion matrices from the best-performing models across four cross-validation experiments. Duration-based (PDF) and standard image-based models (Video Transformer, 3D ResNet-18) often misclassify Domain 2 events as Domain 1. In contrast, T2C CNN was found to significantly reduces such errors by better distinguishing subtle binding type differences.

[0130] In a first study, Figure 9c illustrates pseudo-colored fluorescence microscopy images of predicted and ground truth binding types for each method. By comparing the pseudo-colored fluorescence microscopy images of each binding type prediction method at the same measurement time, it was found that T2C CNN more faithfully reproduces the ground truth binding types at each measurement time point. This increases the likelihood of deriving accurate experimental conclusions from multi-target fluorescence microscopy images, even with a measurement time as short as 5 seconds (50 frames). As the measurement time increases, ungrouped binding events gradually decrease, and the binding types of more events are identified.

[0131] In a second study, Figure 9d illustrates predicted binding types with pseudo-colored images across increasing measurement times. It can be seen that T2C CNN consistently reconstructs ground-truth binding site distributions with 95-100% accuracy, enabling reliable interpretation of multi-target fluorescence microscopy data.

[0132] In multi -class experiments, the proposed approach was extended to three-class classification by introducing an additional 6nt-6nt R1 strand binding type as shown in Figure 10a. The multi-class experiments may be described as follows.

[0133] Single-molecule experiments often involve two or three distinct binding types. To evaluate the method’s performance in a multi -class setting, an additional 6nt-6nt R1 strand binding type was introduced. In this configuration, as shown in Figure 10a, the target strand was shortened from 8nt to 6nt. However, because the binding duration of a 6nt strand with the original 10nt P3 imager was too short, a complementary 6nt R1 strand was instead employed as both the docking and imager strand. The 6nt R1 imager was labelled with the same ATTO532 dye, thereby enabling multiplexed detection of multiple binding types under uniform dye conditions.

[0134] Figure 10b illustrates the distribution of the experimental data that was corrected for background intensity fluctuations and blinking artifacts before the analysis was performed. Across six experiments, three binding types were examined under controlled concentrations so that each binding site exhibited a comparable number of binding events within a 20,000-frame acquisition (10 fps). Most binding sites exhibited between 1 and 9 binding events. The number of detected binding sites for the three binding types was 19,457, 6,073, and 10,888, respectively. The corresponding binding durations increased sequentially, with most events ranging from 0.1 to 10 seconds.

[0135] Analysis of integrated intensity distributions revealed that these distributions primarily reflected binding duration distributions, as per-frame intensities within each binding event remained relatively stable. The integrated intensity distribution primarily reflects the underlying distribution of binding durations, as expected given the relatively stable per-frame intensity within each binding event. Consequently, CNN models need to distinguish bindingtypes by learning from the full fluorescence video dynamics rather than relying on static intensity metrics This involves leveraging temporal features such as inter-frame correlations and other temporally correlated statistics. For the 6nt R1 imager, the average frame intensity was lower (approximately 50 and 100 for training and testing sets, respectively), likely due to short binding durations that sometimes result in incomplete fluorescence accumulation within a single frame

[0136] Figure 10c illustrates the workflow for image-based multi-class classification of binding types. When a binding event is detected (typically lasting 0.1-10 seconds), its fluorescence signals undergo background and blinking correction to eliminate background intensity variations and transient disappearances. The processed signals are then sliced, padded to a uniform length, and input into image-based binding classification models, including 3D ResNet-18, Video Transformer, ED-TCN, SqueezeTime, and the T2C CNN model described herein.

[0137] Following training, it can be seen from Figure 10c that the T2C CNN model achieved the highest test accuracy of 88.15 ± 1.86%, outperforming the other evaluated models, with the next best result being 68.88 ± 2.54% from Video Transformer. Each model classified detected binding events into one of three binding types (6nt-6nt, 8nt-10nt, or 10nt-10nt). Using the predictions of T2C CNN, targets labelled with distinct docking strands were differentiated in reconstructed super-resolution (SR) images, enabling accurate three-color SR imaging. T2C CNN consistently demonstrated superior classification performance across accumulated time points, with its advantage becoming more pronounced over longer durations. This indicates that T2C CNN provides a robust and generalizable framework for single-fluorophore, three-color SR imaging.

[0138] When multiple fluorophores are available, T2C CNN may be configured to first distinguish fluorophores on a frame-by-frame basis and subsequently apply its accurate single-fluorophore classification capability to increase the number of identifiable molecular targets. This substantially enhances both the analytical capacity and processing speed of fluorescencebased single-molecule imaging.

[0139] Interpretations of the T2C CNN module

[0140] The T2C CNN module receives as its input the video of diffraction-limited fluorescent spots, captured by a microscope, that result from the excitation of conjugated dyes in single DNA binding events. The output is the classification of the DNA binding event into predefined DNA binding types. This process does not require manually defined video features, which are difficult to discern with the naked eye. This section describes the additional information captured by the T2C CNN module to differentiate between different DNA binding types, building on the cross-experiment studies described above.

[0141] The part of the diffraction -limited fluorescent spot that best reflects the DNA binding type is firstly identified. By overlaying the model input with its saliency map, it was found that the edge region of the diffraction-limited fluorescent spot contributes most to the model’s prediction. Such an example is illustrated in Figure 1 la. This is likely because the edge plays a crucial role in forming the overall diffraction pattern. In diffraction-limited fluorescence imaging, the edges of the spots often exhibit unique interference patterns and intensity gradients that encode critical information about the underlying molecular interactions. The saliency map generated by T2C CNN shows that these edge regions are particularly influential in the model’s decision-making process. This suggests that the T2C CNN leverages the subtle variations along the edges, such as the differences in brightness, shape, and gradient that arise from diffraction effects to distinguish between binding types. By focusing on these edge features, the network can capture nuanced differences that may not be apparent in the central region of the spots, leading to more accurate classification outcomes.

[0142] The correlation between the model output and common statistics of the diffractionlimited fluorescent spot video were also analysed, which generally reflects the true domain correlations. Additionally, an F-test was performed on the features output by T2C CNN (as shown in Figure 11b), revealing that most features have a high F-statistic (F > 100). This indicates that the T2C CNN model has learned discriminative features from the diffractionlimited fluorescent spot videos to distinguish DNA binding types.

[0143] Next, the importance of different periods of the DNA binding events in determining the DNA binding type was studied. For each DNA binding event’s diffraction-limited fluorescent spot video, 10% of the frames (not masking fewer than 10 frames) weresequentially masked out. As shown in Figure lie, when the first 10% or between 10% and 20% of the frames were masked, the classification accuracy of Domain 1 DNA binding types dropped from around 96% to below 90%. It is hypothesized that this occurs because the early binding stages of a partially matched 8nt docking strand with a lOnt imager strand in Domain 1 are still unstable, compared to the more stable early binding stages of a fully matched lOnt docking strand in Domain 2.

[0144] Finally, the T2C CNN model was evaluated on DNA binding events of varying durations. As shown in Figure lid, the T2C CNN model was able to achieve > 87.5% accuracy for events shorter than 4.9 seconds (49 frames), which constitute 72.9% of all 25,530 events. The probability density function in Figure 8d shows substantial overlap between the two domains in this range, underscoring the challenge addressed by the proposed method. Occasionally, lower accuracies (<80%) arise in a small subset (1.6%) of longer events. Notably, only 1-6 events per binding length category (across 12 of 463 categories, or 2.6%) yielded 0% or 50% accuracy, remaining within acceptable limits for typical applications.

[0145] Furthermore, the T2C CNN model was able to achieve approximately 95% accuracy in distinguishing binding types on those single-frame events (see Figure lid when the x-axis is set to x 100ms). Accordingly, Figure 1 le illustrates a representative example of single-frame events from both domains, including the input image, a representative feature map, and the corresponding classification probability. Feature maps reveal differences in spot shape (e.g., round vs. square) and pixel intensity distribution (e g., concentrated vs. dispersed). The model consistently outputs high-confidence predictions (with the confidence >85%). These results explain how T2C CNN works at the single-frame level. For multi-frame spots, the model integrates temporal variations of these features. Given its ability to differentiate single-frame spots from the same dye, it should, in principle, also distinguish different dyes.

[0146] Robustness test of image-based binding-type classification models

[0147] In single-molecule fluorescence binding experiments, nonspecific binding, background intensity noise, and camera defects are common issues, making it crucial to evaluate the robustness of various methods under noisy conditions. The robustness of imagebased binding-type classification models (i.e., 3D ResNet, Video Transformer, ED-TCN,SqueezeTime, and the T2C CNN model) may be assessed by simulating certain types of noise interference.

[0148] Among all tested models, it was found that the T2C CNN model consistently demonstrated strong robustness, particularly against Poisson and Gaussian noise, due to its temporal-to-channel transformation and efficient parameter design. These results suggest that T2C CNN can reliably handle signal disturbances commonly encountered in fluorescence imaging, provided that interference spot intensity is kept below half that of the signal and the effective pixel ratio-the proportion of the signal area that remains unobstructed by camera defects-exceeds 93%.

[0149] In embodiments of the disclosure, the robustness of image-based binding-type classification models may be assessed as follows.

[0150] Robustness test settings - (a) Light interference with Poisson distributions: Nonspecific molecular binding events generate short, weak fluorescence signals, which can overlap with specific signals and disrupt accurate classification. Considering the discrete nature of photon detection and a radial intensity drop-off based on physical optics, interference signals are simulated using Poisson distributions with Gaussian decay. Figure 12a illustrates examples of generated noise that has added to the original signal, including Poisson light noise with Gaussian decay, Gaussian noise, and hot pixel noise. The symbols u and cr denote the mean and standard deviation (std.), respectively. As shown in images 1201, 1202 and 1203 of Figure 12a, an artificial light spot is overlaid on the original signal, to produce the third image with interference. Despite increasing the signal -to-noise ratio (SNR) (defined as the ratio of the image mean to its standard deviation), the added spot is not fluorescence from molecular binding To mimic experimental fluorescence images, the interference spot radius and Gaussian decay standard deviation were set to 5 and 1.8, respectively. The interference spot’s peak intensity is varied from 0.1 to 1.2 times the original signal’s peak intensity in increments of 0.1 or 0.2, with positions randomized near the original signal to test the robustness of the classification models.

[0151] (b) Gaussian noises: Fluorescence images often suffer from Gaussian noise caused by detector electronic noise, such as thermal and readout noise. Gaussian noise was simulatedwith a mean of 0 and variances ranging from 0.1 to 1.2 times the original signal’s mean intensity. Image 1204 in Figure 12a displays such an example with the Gaussian noise variance set to 0.1 times the original signal’s mean intensity, reducing the SNR from 3.94 to 3.79.

[0152] (c) Hot pixel noises: Prolonged camera usage can cause sensor defects, producing pixels with abnormally high intensity (hot pixels). To evaluate the robustness of classification models against hot pixel noise, 1, 3, 5, 7, 9, 10, and 12 hot pixels with intensities equal to the original signal’s peak intensity were randomly introduced. Image 1205 in Figure 12a shows an example with 5 hot pixels, which mask parts of the original signal.

[0153] Figure 12b illustrates the average classification accuracies of image-based binding type classification models (3D ResNet-18, Video Transformer, ED-TCN, SqueezeTime, and the T2C CNN model) under different noise conditions. The various models were found to exhibit stable performance across different random seeds (with minimal std., resulting in nearly invisible shading) but show performance degradation with increasing noise intensity. The robustness test results can then be summarized as follows. For SqueezeTime and 3D ResNet-18, the default 48- frame input was utilized, and it was found that these models achieved comparable accuracy to the 512-frame input but with significantly higher efficiency. The robustness test results are discussed as follows:

[0154] It is useful to note that Poisson noise has the most significant impact on classification performance, as it can result from specific or nonspecific binding events occurring in close proximity, causing signal overlap. When the interference spot’s peak intensity exceeds 0.5 times that of the signal, all models show a marked performance decline. Increasing the interference intensity from 0.5 to 0.7 times results in accuracy drops of 2.78% to 12.54%. SqueezeTime (2.78%) and T2C CNN (2.88%) exhibit the greatest resilience. Both models employ temporal-to-channel transformations, highlighting the effectiveness of this operation in enhancing robustness against interference spots. One possible reason is that the temporal-to-channel transformation captures long-term fluorescence variations within channels, helping the model focus on the primary signal.

[0155] The models display polarized performance under Gaussian noise. Video Transformer and ED-TCN, which extract image features before temporal operations, are themost vulnerable. In contrast, 3D ResNet-18 and T2C CNN, which feature residual connections and cross-block concatenation, show the strongest resistance Residual connections and crossblock concatenation pass low-level features from earlier layers directly to deeper layers, helping retain structural and spatiotemporal information and making it easier to denoise the signal. This is because Gaussian noise typically affects fine-grained details rather than structural patterns. Conversely, models that first extract image features propagate Gaussian noise into later temporal operations, exacerbating its effect.

[0156] 3D ResNet-18 and SqueezeTime, both with large model parameters (>100 MB), show the highest resilience to hot pixel noise. Smaller models (T2C CNN, ED-TCN, and Video Transformer, with parameters from 1.65 to 7.80 MB) are more susceptible, with T2C CNN being the least affected among them. Large models can memorize fine-grained patterns, making them less sensitive to small-scale disruptions such as hot pixels. In contrast, smaller models, with limited parameter capacity, are more susceptible to pixel losses Despite having the fewest parameters, T2C CNN stands out among small models due to its high parameter efficiency (long temporal kernel and equal -length stride), which enhances its resistance to hot pixel noise.

[0157] Based on the robustness test results, it was determined that optimal performance of the T2C CNN model was achieved when the intensity of interference spots was maintained at less than half of the signal spot intensity, and when imaging was performed using cameras having an effective pixel ratio greater than 93%, corresponding to a tolerance of up to seven hot pixels per one hundred pixels.

[0158] Hyperparameter analysis of the T2C CNN model

[0159] To evaluate the impact of hyperparameter choices in T2C CNN, its key hyperparameters were analysed, including network depth, width, and slice length, as shown in Figure 13a. The plots in Figure 13a illustrates the comparison of classification accuracy, computational cost, and model sizes for variants of the T2C CNN. Baseline methods shown in the plot of Figure 13a include 3D ResNet-18, Video Transformer, ED-TCN and SqueezeTime. In the plot, the abbreviation “w / o" represents “without" and Giga Floating Point Operations perSecond (GFLOPs) were used to quantify the computational power required for processing, indicating hardware demands.

[0160] Depth is a critical attribute of deep neural networks and has been found to significantly affect the performance of such networks. The T2C CNN model comprises eight hidden layers (convolutional or fully connected), evenly distributed across four blocks. To investigate the minimum required depth, each block was sequentially assigned 1 to 3 layers, increasing the total depth from 4 to 12 while maintaining the four-block structure. For the first block, which condenses spatial-temporal features, the convolutional kernel size was adjusted for configurations with one or three layers. Figure 13b illustrates variants of the T2C CNN Architecture as investigated in the Ablation Study below. The dashed boxes highlight the four blocks of T2C CNN, while solid rectangles represent a convolutional or fully connected layer followed by BN and ReLU layers (except for the final output layer, which omits BN and ReLU). The convolution kernel size is labeled as “N x N," and the rectangle length indicates the kernel size or the number of hidden units (i.e., width). Hidden dimension transformations are denoted as “NM" while rectangles without labelled kernel sizes or dimension transformations follow the default hidden layer configuration for an 8-layer depth.

[0161] Like depth, the network’s width (the number of convolutional kernels or hidden units) plays a critical role in the CNN’s performance. The default T2C CNN configuration uses 64 kernels or hidden units per layer. To explore the effect of width, variations ranging from 16 to 512 were tested, including commonly used widths such as 16, 32, 48, 100, 128, 256, and 512.

[0162] The slice length, which determines the interval at which fluorescence frames are input into T2C CNN, is critical for capturing long-term temporal patterns in fluorescence videos. It was found that most binding events occur within 50 seconds (500 frames), with the longest binding duration not exceeding 2000 frames before averaging. Therefore, the default slice length of 512 frames was varied by halving or doubling to explore alternatives of 64, 128, 256, 1024, and 2048 frames.

[0163] As shown in Figure 13a, among the tested depths ranging from 4 to 12 layers, the top-performing configurations were depths of 8 (94.76%), 4 (94.00%), and 12 (93.27%), andthey all comprise multiples of 4. These results indicate that performance is optimal when each block in T2C CNN contains an equal number of hidden layers, with two hidden layers per block yielding the best results, followed by one layer per block. Notably, at a depth of 4, the first block of T2C CNN required a larger convolution kernel (increased from "3* 3" to "10x10") to condense all spatial -temporal features within a single hidden layer. This adjustment increased the parameter count approximately 11 -fold, significantly contributing to the model size. Among the tested widths of 16, 32, 48, 100, 128, 256, and 512, a width of 128 slightly outperformed the default 64 (94.76% → 95.38%). However, this improvement came with a 2.62-fold increase in model size (1.65 MB4.32 MB) and a 2.04-fold increase in computational cost (0.48 GFLOPs0.98 GFLOPs). From an efficiency perspective, the default width of 64 strikes a good balance. If computational resources and latency are not constraints, a width of 128 is also recommended. Widths smaller than 64 or greater than 128 5consistently underperformed compared to the default, with performance generally declining as width became too wide.

[0164] Interestingly, even with an unusually small width of 16, T2C CNN maintained performance above 90%. This resilience can be attributed to the unique block design of T2C CNN, as breaking the balance of hidden layers within blocks or removing blocks led to performance degradation, even with an optimal width of 64. Remarkably, this suggests that when a fluorescence video spanning 512 frames is condensed into 16 channel dimensions, most discriminative information is retained. This highlights the substantial redundancy in fluorescence videos, much of which stems from identical optical properties of the same fluorescence type. The key discriminative signals arise from subtle changes in the fluorescence microenvironment caused by binding to different targets. These subtle variations are condensed into a small set of discriminative features by T2C CNN, which is fundamental to its functionality. This also explains why performance significantly declined at widths of 256 and 512. The increased number of hidden nodes likely led to overfitting on redundant information, impairing generalization to test data from different experiments. As the slice length input to T2C CNN increases up to 512 frames, the classification accuracy improves significantly. This is because the longer slice length enables T2C CNN to capture a more complete binding event, aiding in determining the binding type. However, beyond 512 frames, the performance plateaus, as only a small fraction of binding events extend beyond this duration. In practical applications, we recommend setting the slice length to encompass the majority of binding durations (e g., the maximum binding time excluding outliers) to achieve optimal efficiency.

[0165] In the following section, the performance of the T2C CNN model is compared with two closely related baseline methods: the classical temporal convolutional network ED-TCN and SqueezeTime, which compresses the temporal dimension into the channel dimension. For a fair comparison, the same 512-frame input length for SqueezeTime was utilized for the T2C CNN model as well (its default input is a 16-frame and 48-frame video clip). This adjustment significantly increased SqueezeTime’ s model size (28.7 MB113.3 MB) and computational cost (5.5 GFLOPs → 19.56 GFLOPs).

[0166] As can be seen from Figure 13a, SqueezeTime achieved classification accuracies of 71.72% with the default 16-frame input and 72.98% with the 512-frame input on fluorescence videos, indicating that increasing the number of input frames yields only marginal improvement. This is likely because the repeated transformations between channel and temporal dimensions, along with weighted convolutions in SqueezeTime, risk losing discriminative spatiotemporal details, regardless of the input length. For ED-TCN, both its default 64-frame input and a 512-frame input which matched T2C CNN’s input were tested. Its classification accuracies on fluorescence videos were 70.53% and 58.53%, respectively, demonstrating its unsuitability for long fluorescence video inputs. A key limitation is that ED-TCN’ s fragmented frame-wise spatial features hinder the capture of long-term spatiotemporal patterns in fluorescence videos.

[0167] In contrast, the T2C CNN model was specifically tailored for fluorescence video analysis. Its shallow cross-layer connections retain multi-scale spatiotemporal features, while long-term spatial convolutions are employed to capture extended spatial frequency patterns. Additionally, the integration of shallow cross-connected blocks with a pooling-free design facilitates efficient spatiotemporal information fusion. All spatiotemporal features are compressed into the channel dimension and jointly processed within the T2C layer. The individual and combined contributions of these three components are systematically evaluated in the Ablation Study section below. For reference, 3D ResNet-18 and Video Transformer are included in Figure 13a as well. These two models are widely adopted in computer vision for video classification and generally outperform lighter architectures such as SqueezeTime and ED-TCN due to their strong representational capacity. However, they are still suboptimal for fluorescence video classification, which demands a domain-specific architectural design.

[0168] In summary, it was found that a depth comprising 8 layers (two per block) achieved the highest accuracy (94.76%) and that configurations with evenly distributed layers (4, 8, and 12) consistently outperformed uneven ones (e.g., 5, 6, 7, 9, 10, 11), underscoring the importance of architectural symmetry. The default width of 64 provided the best trade-off between accuracy and efficiency, with 128 yielding only marginal gains (94.76% → 95.38%) at the cost of doubling model size and computation. Remarkably, even with a small width of 16, accuracy remained above 90%, indicating T2C CNN’s robust feature extraction and the redundancy in fluorescence video data Regarding input length, increasing the slice length improved performance up to 512 frames by capturing complete binding events, with performance plateauing beyond that The baseline models with their optimal depths and widths as also shown in Figure 13a, which were evaluated on randomly split training and validation datasets. Table 1 below sets out the hyperparameter evaluation summary for the various baseline models.TABLE 1'Model depths and widths were evaluated near their default configurations.2Class-wise accuracies for single-event classification were recorded.3For 3D ResNet and the ResNet-based method (SqueezeTime), a width of 512 was used when the depth was below 50; otherwise, a width of 2048 was used. ’'Bolded values indicate the optimal hyperparameters identified in the trials.

[0169] Ablation study of the T2C CNN

[0170] An ablation study focusing on the core building blocks and components of the T2C CNN are performed in this section. Three key components (long-term spatial convolutions, skip concatenations and pooling-free) were selected for a detailed evaluation of their individual and combined contributions, as presented in Figure 13a and Table 2 below.TABLE 2’Each component is removed or included to assess its individual and joint contribution to classification accuracy.2Class-wise accuracy at the single-event and single-molecule levels. Single-molecule predictions derived by confidence-based voting over associated single-event predictions.3Overall single-event accuracy is calculated across all events without class-wise averaging. Reported values are mean ± standard deviation over five cross-validation trials.

[0171] The T2C CNN model comprises of four blocks, with two being fundamental: the Temporal-to-Channel (T2C) block for condensing spatial -temporal features and the Feature Classification block. To assess the importance of the remaining blocks, the second (Hidden Transformation) and third (Multi-Scale Feature Fusion) were individually removed and the impact of the removal of these blocks were evaluated.

[0172] The T2C CNN is specifically designed for fluorescence video analysis. It employs long-term spatial convolutions to capture extended spatial frequency patterns. The combination of skip concatenations and a pooling-free design enables effective spatiotemporal information fusion across the network. For evaluation, the optimal configuration (slice length of 512, allskip concatenations enabled, no pooling) was compared with a baseline configuration (slice length of 64, no skip concatenation, average pooling), corresponding respectively to the presence and absence of the three proposed components.

[0173] As shown in Figure 13 a, by removing the second and third blocks from the T2C CNN architecture, the classification accuracy reduced from 94.76% to 88.80% and 90.33%, respectively. This significant performance drop indicates that both blocks are integral components of T2C CNN. Among the two, the second block (Hidden Transformation) plays a more critical role. It consists of two 1×1 convolutional layers that transform the condensed global spatiotemporal features into more discriminative representations, facilitating the differentiation of binding types. Notably, when the third block (Multi-Scale Feature Fusion) operates without the transformed features from the second block, its fusion-based features result in lower accuracy (88.80%) compared to using only the transformed features from the second block (90.33%). This demonstrates the importance of the second block in preparing features for effective multi-scale fusion.

[0174] Table 2 above presents a systematic evaluation of the key components of T2C CNN, long-term spatial convolutions, skip concatenations, and the pooling-free strategy, on both single-event and single-molecule classification accuracy, with all results averaged over five cross-validation trials.

[0175] Among the individual components, long-term spatial convolutions contribute the most significant performance gain, boosting class-wise single-event and single-molecule accuracies to 84.14% and 88.86%, respectively, compared to the baseline’s 72.01% and 79.87%. This highlights their pivotal role in capturing extended spatial-frequency patterns essential for distinguishing binding types in fluorescence videos. In contrast, skip concatenations (72.58% / 78.13%) and no-pooling (70.73% / 77.60%) result in slightly lower class-wise accuracies than the baseline, suggesting that their benefit emerges primarily when combined with other architectural elements to support multi-scale feature fusion and preserve spatiotemporal information. The overall single-event accuracy follows a similar trend, with long-term convolutions providing the most substantial standalone gain (89.07% vs. 72.95% baseline).

[0176] When components are combined, their effects are complementary. Integrating longterm convolutions with skip concatenations increases class-wise single-event accuracy to 86.33%, while pairing long-term convolutions with no-pooling pushes it to 87.34%. In contrast, combining skip concatenations with no-pooling without long-term convolutions yields only a modest 72.51%, reinforcing the central importance of long-term spatial convolutions The full T2C CNN, incorporating all three components, achieves the best performance: 94.76% classwise single-event accuracy, 96.99% single-molecule accuracy, and 96.78% overall accuracy. Notably, it also exhibits the lowest standard deviations across all metrics (±0.47%, ±0.46%, and ±0.50%, respectively), indicating stable and consistent predictions across cross validation folds. This demonstrates that the proposed architectural combination of existing elements enables not only optimal spatiotemporal information fusion and feature preservation, but also robust generalization, tailored specifically for high-precision fluorescence video analysis.

[0177] In summary, removing either the second (Hidden Transformation) or third (MultiScale Feature Fusion) block significantly degraded performance (94.76% -> 90.33% and 88.80%, respectively), confirming their critical roles. The second block proved particularly essential for transforming global features into discriminative representations that facilitate effective multi-scale fusion. Further analysis of individual components-long-term spatial convolutions, slap concatenations, and no-pooling-revealed that each contributed incrementally to performance, with their combined use achieving the highest accuracy. Notably, the default configuration also yielded the lowest standard deviation, reflecting stable generalization. The optimal configuration consistently outperformed the baseline across both single-event and single-molecule tasks. These results demonstrate that the full T2C CNN architecture forms a synergistic and domain-adapted design optimized for robust classification under noisy single-dye fluorescence conditions

[0178] Statistical significance analysis of model performance

[0179] A Wilcoxon signed-rank test conducted on single-event class-wise accuracies across five folds confirmed that T2C CNN significantly outperformed all of its ablated variants (p=0.031). To assess performance across a broader set of baseline models, including all seven ablated variants, four PDF variants (1 -event, 2-event, 3-event, and all-event) and four imagebased methods, a Friedman test was performed on the same single event accuracies (as shown in Table 3 below), which revealed a statistically significant difference among the 16 methods(χ2= 64.11, p = 4.89 × 10-8) All models were evaluated using five-fold cross-validation and tested on an independent sample. Although additional random seeds were not explored, the cross-validation results were consistent across folds, indicating stable performance and low sensitivity to model initialization. Post-hoc analysis using the Nemenyi test further showed that T2C CNN significantly outperformed the state-of-the-art baselines, including SqueezeTime (p=0.0030), EDTCN (p=0.0023), and Video Transformer (p=0.0145). These results underscore the necessity of domain-specific architectural design in fluorescence video classification.TABLE 3Average Methods Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 \rank (j1T2C CNN (ABC2) 94.76 94.28 94.36 95.60 94.78 1T2C CNN (AC) 90.91 86.68 86.85 87.01 85.26 2.8 PDF (All event) 83.88 84.42 84.26 84.42 84.42 4 T2C CNN (AB) 79.35 89.59 86.33 93.97 82.40 4.4 PDF (2 event) 83.80 83.90 83.92 83.90 83.90 5 T2C CNN (A) 81.84 80.70 88.11 89.38 80.69 5.2 PDF (3 event) 82.09 82.18 82.00 82.18 82.18 6.6 3D ResNet-18 80.17 72.58 82.79 80.45 83.72 7.4 PDF (1 event) 75.27 75.27 75.16 75.27 75.27 9.6 T2C CNN (B) 70.22 72.64 69.69 78.98 71.38 12 Video Transformer 74.67 70.85 69.76 72.44 70.85 12.4 T2C CNN (BC) 68.66 68.01 73.26 78.40 74.22 12.4 T2C CNN None 71.61 70.00 69.55 71.61 77.28 12.6 T2C CNN (C) 70.49 69.75 70.35 73.97 69.09 13.2 SqueezeTime 71.72 69.18 71.27 69.55 66.34 13.6 ED-TCN 62.19 63.63 76.01 67.43 71.84 13.8xRanks were computed within each fold using Friedman ranking and then averaged across folds. These ranks were used in the Friedman test and post-hoc analyses reported in the main text. All 16 methods and their variants were sorted in ascending order based on their average ranks.2The letters “A", “B", and “C" following “T2C CNN" indicate the inclusion of longterm spatial convolutions (A), skip concatenations (B), and no-pooling (C) components, respectively. Combinations of these letters represent variants that include the corresponding components. The suffix “None" denotes the variant in which none of these three components are included.

[0180] HER2 -targeted cell experiments

[0181] Figure 14a illustrates two groups of HER2-positive AU565 cell samples (Domain 1 and Domain 2) that were immobilized on separate glass slides. Herceptin antibodies conjugated with a 1 Ont P3 ’ docking strand (complementary to the 1 Ont P3 strand) were used to specifically bind to the HER2 proteins on the cell membranes. Identical fluorophores (ATTO532)-labelled imagers were introduced to both samples. The same truncated 8nt P3 strand and full-length lOnt P3 strand were used as in the previous experiments. TIRF microscopy was used to image a thin layer at the cell-glass interface, where imagers can access the contact region through gaps between the cell and the slide. A higher number of binding events is expected near the edges of the contact area, which are more exposed to the imaging buffer.

[0182] Details of the experimental preparation procedures are as follows. The antibody -oligo conjugate probes were prepared using the method described in the publication by Yan Shan Ang et. al, titled “Protein-DNA conjugates with a discrete number of oligonucleotide strands for highly reproducible protein quantification by the DNA proximity assay.” in Analytical Chemistry dated 2023. The 5’-amine-modified docking strands were conjugated with Herceptin monoclonal antibody (targeting HER2 protein on the cell membrane) (Trastuzumab, Roche, Graz Steiermark, Austria) using disuccinimidyl suberate (DSS) linker. The oligo (200 LIM in nuclease-free water) was first mixed with an equal volume of acetonitrile, DSS (25 mM, dissolved in dimethylformamide), and 1:800 (v / v) of triethylamine for 15 minutes at room temperature, and then purified via ethanol precipitation. Sodium acetate (0.3 M, pH 5.2) and magnesium chloride (10 mM) were added to the conjugation product and mixed with three times the volume of cold absolute ethanol. After 1 hour incubation at -20 °C, the slurry was centrifuged at 24,100 g for 15 minutes at 4 °C. The pellet was washed once with ice-cold 75% ethanol and reconstituted in nuclease-free water. The activated oligo was then incubated with a 3-fold molar excess of Herceptin antibody in 50 mM phosphate buffer (pH 7.2) for 12 hours at room temperature.

[0183] The antibody-oligo conjugates were purified by ion exchange chromatography (IEX) using an Agilent Bio SAXNP3 (4.6 * 50 mm) column on an Agilent 1260 Infinity HPLC system. Elution began with 100% buffer A (50 mM phosphate buffer, pH 7.2), followed by a step change to 70% buffer A and 30% buffer B (50 mM phosphate buffer, pH 7.2, supplemented with 1.0 M NaCl). A linear gradient was then applied, increasing the proportion of buffer B to 65% (corresponding to 35% buffer A) over 13 minutes to achieve progressivesalt-mediated elution. Each collected fraction, containing antibodies conjugated with discrete numbers of oligo strand(s), was concentrated using Amicon ultrafiltration columns with a 50 kDa cut-off. The final antibody-oligo conjugates were quantified using the Qubit ssDNA Assay Kit. The purified conjugates were stored at 4 °C in buffer containing phosphate saline (pH 7.4) and 1 mM EDTA.

[0184] HER2 receptor protein-expressing AU565 cells (CRL- 2351™, ATCC; Manassas, Virginia, USA) were cultivated in Roswell Park Memorial Institute 1640 Medium (RPMI / ATCC Modification #A1049101, Gibco, Thermo Fisher Scientific, Massachusetts, USA). The medium was supplemented with 1% penicillin-streptomycin (#15070063, Gibco) and 10% foetal bovine serum (FBS; #10270106, Gibco). The cells were seeded at a density of 50,000 cells per dish onto culture dishes (#P35G-1.5-20-C, MatTek, Massachusetts, USA). The cultures were maintained in a heated CO2 incubator at 37 °C and 5% (v / v) CO2 concentration (Forma Steri-Cycle CO2 incubator, Thermo Fisher Scientific) for 36 hours

[0185] Cells were then fixed with 4% paraformaldehyde (PF A, EMS) in PBS for 15 minutes, followed by quenching with 1 mg / mL sodium borohydride for 7 minutes. Cells were washed three times (5 minutes per wash) with PBS, and incubated with 1% BSA and 0.05% Tween-20 in PBS for 2 hours to block non-specific binding. The blocked cells were washed four times with 1% BSA and incubated with conjugated antibodies (1 nM in PBS containing 500 mM NaCl) for 1 hour at room temperature Cells were then washed four times with PBS supplemented with 500 mM NaCl. Imager solution (1-5 nM in PBS with 500 mM NaCl and 2 mM Trolox) was added immediately before imaging

[0186] With reference to Figure 14a, it can be seen that this figure illustrates raw fluorescence images from two hybridization types: 8nt-10nt and 1 Ont- 1 Ont, where lOnt imager strands bind to HER2 -targeted docking strands on AU565 cells. By carefully adjusting the concentration ratio between the docking and imager strands, spot overlap was minimized to accurately reveal the locations of individual HER2 proteins. After drift correction over 33 minutes 20 seconds (20,000 frames at 10 fps), super-resolution reconstruction as illustrated in Figure 14b reveals HER2 positions.

[0187] Binding events at these sites were tracked to compute average on / off times. As shown in Figure 14b - Domain 1 (8nt-10nt) shows shorter binding durations than Domain 2 (10nt-10nt), with similar off-times. Compared to the Origami setup, domain durations in cells overlap more, and this is likely due to environmental factors such as membrane interactions or local field instabilities. Kernel density estimation reveals a 45.76% intersection-over-union (IoU) between domains. Using the intersection point of the distributions (4.7s) as a threshold, 31.85% (402 / 1262) of events are misclassified, highlighting the limits of duration -based classification in cells.

[0188] To enhance feature richness, each event is converted into a video by stacking cropped regions across frames. Figure 14c shows feature-label correlations: duration correlates weakly (±0.13), while other features, e.g., min / max intensity and Y-Z median intensity, show stronger trends (up to ±0.17), offering improved discriminative power. Still, these cell-derived features are less informative than those from the Origami dataset (correlations ~ ±0.5), underscoring the increased complexity of cellular classification.

[0189] HER2 site-localized videos as shown in Figure 15a were background-corrected and normalized before being input to video classifiers. Labels (Domain 1 vs. Domain 2) guided model training via parameterized operations (convolution, matrix multiplication, etc ). During testing, models classified unseen videos. To evaluate multiplexing, super-resolution images from two samples (as shown in Figure 15b) were overlaid, and reconstructed at multiple durations (10s to 33min20s). Accuracy trends across time are summarized in the right-hand bar plot. Specifically, Figure 15b illustrates real-time classification of binding events from fluorescence spot videos on the two test samples. The models used in this figure were trained on another two separate HER2-positive samples prepared under the same experimental conditions.

[0190] Despite cellular complexity lowering overall performance, the results show that the T2C CNN consistently outperformed other models, reaching 78.48%-80.61% accuracy versus 40.19%-70.00% for other models. The performance gap widened with longer measurements, with T2C CNN surpassing the second-best model by 10.00%-20.18%. This enables more accurate reconstruction of binding types in real time, advancing high-accuracy classification for single-molecule fluorescence in cells.

[0191] Single-frame discrimination of different dye-labelled binding events

[0192] As a fluorescence classification model, T2C CNN can be used for multiplexing using the wavelength dependence of the emission PSF. This is an alternative to multi -fluorophore experiments using spectral separation.

[0193] In particular, T2C CNN can be used for multiplexing using the wavelength dependence of the emission point spread function (PSF). This is an alternative to multi-fluorophore experiments using spectral separation. Spectral isolation typically reduces the signal by avoiding spectrally overlapped regions, and increases noise due to cross-channel bleed-through. Some of these issues can be solved by sequential imaging but only at the expense of longer measurement times. Spectral unmixing strategies also require a sufficient signal to properly separate different fluorophores. The feasibility of this approach was demonstrated by applying T2C CNN to two fluorophore data sets (green and red) without spectrally isolating their emission, but using purely spatial features of the emission profiles as differentiating factors. The success of this strategy allows the classification of two or more spectrally different signals to be performed in any SMLM method such as PALM and STORM in addition to DNA-PAINT.

[0194] To discriminate different dye-labelled molecules from diffraction-limited images, those skilled in the art have made many efforts, including spectral imaging and unmixing, fluorescence lifetime imaging microscopy (FLZM), super-resolution fluorescence microscopy (SMFM), and machine learning methods. Spectral imaging and unmixing separate fluorophores based on their emission spectra, with the reliance on spectrally resolved microscopy. Fluorescence lifetime differences between dyes have been used to discriminate them in FLIM, but require prolonged observations. Super-resolution techniques like PALM and STORM are frequently used to distinguish fluorophores based on their blinking and emission properties. However, themselves generally cannot discriminate different dyes in a single frame because these techniques rely on temporal separation, where individual fluorophores are stochastically activated and imaged one at a time across many frames. Machine learning, or deep learning, methods have been employed to act as automatic SMFM trace selector, distinguish the true signal of fluorescently labelled molecules from backgroundfluorescence and noise, classify binding events based on blinking kinetic, spectrum, or statistical features, reconstruct a super-resolution image, de-overlap fibrils through fluorescence lifetime imaging

[0028] , among others. However, there has been limited work focused on distinguishing different dyes from single-frame diffraction-limited fluorescent spots. Therefore, the T2C CNN may be applied to classify the dyes corresponding to these fluorescent spots.

[0195] The point spread function (PSF) characteristics of emission wavelengths are leveraged to distinguish multi-color data at the single fluorophore level. In this experiment, the same 8-nucleotide docking and 10-nucleotide imager sequences are used, where the imager was alternately tagged with green (Atto532) and red (Atto647) dyes. 20,000 frames for each dye are then collected, using the binding events from the first 10,000 frames for training and the last 10,000 frames for testing. To avoid duplicated frames from the same binding site, only the first frame of each site was selected for classification by the model. For comparison, several commonly used CNNs were evaluated on the dataset, including (1) lightweight models: MobileNetV2, SqueezeNet, ShuffleNetV2, EfficientNetV2, and (2) standard-weight models: ResNet-18, ResNet-50, VGG16, and DenseNet-121. The classification accuracy, computational cost, and model sizes for T2C CNN and these commonly used CNNs across five cross-validation experiments were then measure. The results are presented in Figure 16.

[0196] As can be seen from Figure 16, the T2C CNN model demonstrated outstanding classification accuracy (92.88%), comparable to larger models like VGG16. This outcome confirms the ability to differentiate multi-color data at the single fluorophore level by analysing the PSF patterns of emission wavelengths. This finding lays the groundwork for future multicolor microscopy techniques that do not require wavelength -specific analysis. Notably, singleframe fluorophore classification is a relatively easier task compared to classifying binding types from single-fluorophore videos. This difference in difficulty stems from the fact that PSF characteristics differ more prominently across fluorophores with distinct emission wavelengths, whereas in the binding-type classification task, the same fluorophore is used, and the emission patterns are only subtly affected by different DNA strand bindings. As a result, as can be seen in Figure 16, models like ResNet-18 achieve higher performance in multi-color classification than in the more challenging single-fluorophore task. Importantly, the T2C CNN achieved this accuracy with a very small computational cost-only 0.02 Giga Floating PointOperations per Second (GFLOPs)-which is even lower than ShuffleNetV2 (0.07 GFLOPs). This efficiency stems from T2C CNN’s compact architecture: it has only six convolutional layers, one-third the number in ResNet- 18, with the last four being 1×1 convolutions-requiring one-ninth the operations of ResNet-18’s 3×3 convolutions. Additionally, its hidden transformations operate on a 64×64 width - 1 / 64 ofResNet-18’s 512×512 transformations, and its fully connected layers are only 192×64, approximately 1 / 40 of ResNet-18’s 512×1000 configuration. Consequently, the total computational cost of T2C CNN is less than one-thousandth of ResNet- 18, making it deployable on lightweight computing devices

[0197] The T2C CNN module described in this disclosure enables single-event level classification, in contrast to conventional methods that depend on different fluorescent dyes, temporal separation of fluorophores, or prolonged observation of blinking kinetics. By classifying individual binding events directly from their diffraction-limited fluorescent spots, the T2C CNN module supports synchronous classification of binding types at the single-event level while using the same fluorescent dye. The T2C CNN module also incorporates a temporal-to-channel (T2C) convolutional neural network (CNN), an architecture specifically designed for this application. This architecture transforms temporal information into channel representations, thereby enhancing the discriminative power of the network for distinguishing binding events.

[0198] In addition, the T2C CNN module achieves enhanced accuracy and efficiency, as demonstrated in DNA-PAINT experiments. In these studies, classification accuracy improved from approximately 75% to around 95%, while classification time was reduced from 10 minutes to about 5 seconds through advanced spatiotemporal pattern recognition. The T2C CNN module also reveals previously unexplored binding-type information contained within diffraction-limited fluorescent spots, expanding the analytical dimensions available to singlemolecule localization microscopy (SMLM). The approach demonstrates robust and interpretable performance across experiments, underscoring its potential for broad application in diverse research areas involving fluorescence microscopy. Collectively, these features enhance the analytical capabilities of SMLM, enabling more precise investigations of molecular interactions and contributing to a deeper understanding of complex biological processes.

[0199] Numerous other changes, substitutions, variations, and modifications may be ascertained by the skilled in the art and it is intended that the present application encompass all such changes, substitutions, variations, and modifications as falling within the scope of the appended claims

Claims

CLAIMS1. A computing module for classifying single-molecule binding events in diffractionlimited fluorescent microscopy video data, the module comprising:a processing unit; anda non-transitory media readable by the processing unit, the media storing instructions that when executed by the processing unit causes the processing unit to:receive, using an input module, the fluorescent microscopy video data comprising single-channel grayscale image frames representing diffraction-limited fluorescent spots, the video data characterized by low spatial resolution and a long temporal sequence, and partition the temporal sequence of the image frames of the video data into a plurality of temporal slices;classify single-molecule binding events in the video data using a trained temporal - to-channel (T2C) convolutional neural network (CNN) module, the trained T2C CNN module comprising:a T2C convolution block configured to:generate a two-dimensional (2D) feature map having a fixed number of channels by applying two-dimensional convolutions to the temporal slices received from the input module, wherein each temporal slice is represented as a channel in the two-dimensional convolutions;apply a first batch normalization to the 2D feature map and a first rectified linear unit (ReLU) activation function to the normalized 2D feature map to produce a normalized-activated 2D feature map comprising a plurality of channels;generate a refined 2D feature map having higher-level feature representations by applying two-dimensional convolutions, a second batch normalization and a second ReLU activation function to the normalized-activated 2D feature map;a hidden layer transformation block configured to generate a normalized-activated hidden feature map based on the refined 2D feature map;a multi-layer feature refinement block configured to:generate a normalized-activated intermediate multi-layer feature map based on a combination of the feature maps generated by the hidden transformation block and the T2C convolution block, andgenerate a normalized-activated refined multi-layer feature map based on the normalized-activated intermediate multi-layer feature map; and a feature classification block configured to:generate a normalized-activated feature vector based on a combination of the feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block; andgenerate a classification output identifying one of a plurality of predefined single-molecule binding types based on the normalized-activated feature vector.

2. The computing module according to claim 1, wherein the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map are performed using kernels defined over spatial and temporal dimensions, and wherein the two-dimensional convolutions further applied in the T2C convolution block, the hidden transformation block, and the multi-layer feature refinement block are performed using kernels having only spatial dimensions.

3. The computing module according to claim 2, wherein the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map employ spatial padding to preserve spatial features and employ strides matched to a length of the temporal slice.

4. The computing module according to claim 1, wherein the input module is further configured to pre-process the fluorescent microscopy video data before partitioning the temporal sequence of the image frames, the pre-processing comprising:correcting background intensity variations in the image frames of the video data to generate background-corrected fluorescent spots; andnormalizing pixel intensity values of the image frames by scaling the pixel intensity values to a predetermined range.

5. The computing module according to claim 4, wherein the pre-processing further comprises:aligning the background-corrected fluorescent spots over time by performing drift correction on the image frames of the video data to compensate for positional shifts occurring across the temporal sequence.

6. The computing module according to any one of claims 1 to 5, wherein the T2C CNN module is trained by minimizing a cross-entropy loss function between inferred classification outputs and ground truth labels, and by updating learnable parameters of the T2C CNN module using stochastic gradient descent.

7. The computing module according to claim 1, wherein the plurality of temporal slices comprises equal-length temporal segments of the video data.

8. The computing module according to claim 1, wherein the generation of the normalized-activated hidden feature map based on the refined 2D feature map by the hidden layer transformation block comprises the hidden layer transformation block being configured to:apply pointwise two-dimensional convolutions, a third batch normalization and a third ReLU activation function to the refined 2D feature map.

9. The computing module according to claims 1 or 8, wherein the generation of the normalized-activated intermediate multi-layer feature map by the multi-layer feature refinement block comprises the multi-layer feature refinement block being configured to: apply two-dimensional convolutions, a fourth batch normalization and a fourth ReLU activation function to the feature maps generated by the hidden transformation block and the T2C convolution block.

10. The computing module according to claim 9, wherein the generation of the normalized-activated refined multi-layer feature map by the multi-layer feature refinement block comprises the multi-layer feature refinement block being configured to:apply two-dimensional convolutions, a fifth batch normalization and a fifth ReLU activation function to the normalized-activated intermediate multi-layer feature map.

11. The computing module according to claims 1, 8, 9 or 10 wherein the generation of the normalized-activated feature vector by the feature classification block comprises the feature classification block being configured to:flatten the combined feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block to produce a feature vector;refine the feature vector using a first fully connected (FC) layer; andapply a sixth batch normalization and a sixth ReLU activation function to the refined feature vector.

12. The computing module according to claim 11, wherein the generation of the classification output identifying one of a plurality of predefined single-molecule binding types comprises the feature classification block being configured to:apply a second FC layer to the normalized-activated feature vector.

13. A method for classifying single-molecule binding events in diffraction-limited fluorescent microscopy video data using a computing module comprises:receiving, using an input module, the fluorescent microscopy video data comprising single-channel grayscale image frames representing diffraction-limited fluorescent spots, the video data characterized by low spatial resolution and a long temporal sequence, and partitioning the temporal sequence of the image frames of the video data into a plurality of temporal slices;classifying single-molecule binding events in the video data using a trained temporal-to-channel (T2C) convolutional neural network (CNN) module, the trained T2C CNN module comprising:a T2C convolution block configured to:generate a two-dimensional (2D) feature map having a fixed number of channels by applying two-dimensional convolutions to the temporal slices received from the input module, wherein each temporal slice is represented as a channel in the two-dimensional convolutions;apply a first batch normalization to the 2D feature map and a first rectified linear unit (ReLU) activation function to the normalized 2D feature map to produce a normalized-activated 2D feature map comprising a plurality of channels;generate a refined 2D feature map having higher-level feature representations by applying two-dimensional convolutions, a second batch normalization and a second ReLU activation function to the normalized-activated 2D feature map;a hidden layer transformation block configured to generate a normalized-activated hidden feature map based on the refined 2D feature map;a multi-layer feature refinement block configured to:generate a normalized-activated intermediate multi-layer feature map based on a combination of the feature maps generated by the hidden transformation block and the T2C convolution block, andgenerate a normalized-activated refined multi-layer feature map based on the normalized-activated intermediate multi-layer feature map; anda feature classification block configured to:generate a normalized-activated feature vector based on a combination of the feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block; andgenerate a classification output identifying one of a plurality of predefined singlemolecule binding types based on the normalized-activated feature vector.

14. The method according to claim 13, wherein the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map are performed using kernels defined over spatial and temporal dimensions, and wherein the two-dimensional convolutions further applied in the T2C convolution block, the hidden transformation block, and the multilayer feature refinement block are performed using kernels having only spatial dimensions.

15. The method according to claim 14, wherein the two-dimensional convolutions applied in the T2C convolution block to generate the 2D feature map employ spatial padding to preserve spatial features and employ strides matched to a length of the temporal slice.

16. The method according to claim 13, wherein before the step of partitioning the temporal sequence of the image frames using the input module, the method comprises the step of preprocessing the fluorescent microscopy video data by:correcting background intensity variations in the image frames of the video data to generate background-corrected fluorescent spots; andnormalizing pixel intensity values of the image frames by scaling the pixel intensity values to a predetermined range.

17. The method according to claim 16, further comprising the steps of:aligning the background-corrected fluorescent spots over time by performing drift correction on the image frames of the video data to compensate for positional shifts occurring across the temporal sequence.

18. The method according to any one of claims 13 to 17, wherein the T2C CNN module is trained by minimizing a cross-entropy loss function between inferred cl ssification outputs and ground truth labels, and by updating learnable parameters of the T2C CNN module using stochastic gradient descent.

19. The method according to claim 13, wherein the plurality of temporal slices comprises equal-length temporal segments of the video data.

20. The method according to claim 13, wherein the generation of the normalized-activated hidden feature map based on the refined 2D feature map by the hidden layer transformation block comprises the hidden layer transformation block being configured to:apply pointwise two-dimensional convolutions, a third batch normalization and a third ReLU activation function to the refined 2D feature map.

21. The method according to claims 13 or 20, wherein the generation of the normalized-activated intermediate multi-layer feature map by the multi-layer feature refinement block comprises the multi-layer feature refinement block being configured to:apply two-dimensional convolutions, a fourth batch normalization and a fourth ReLU activation function to the feature maps generated by the hidden transformation block and the T2C convolution block22. The method according to claim 21, wherein the generation of the normalized-activated refined multi-layer feature map by the multi-layer feature refinement block comprises the multi-layer feature refinement block being configured to:apply two-dimensional convolutions, a fifth batch normalization and a fifth ReLU activation function to the normalized-activated intermediate multi-layer feature map.

23. The method according to claims 13, 20, 21 or 22 wherein the generation of the normalized-activated feature vector by the feature classification block comprises the feature classification block being configured to:flatten the combined feature maps from the T2C convolution block, the hidden layer transformation block and the multi-layer feature refinement block to produce a feature vector;refine the feature vector using a first fully connected (FC) layer; andapply a sixth batch normalization and a sixth ReLU activation function to the refined feature vector.

24. The method according to claim 23, wherein the generation of the classification output identifying one of a plurality of predefined single-molecule binding types comprises the feature classification block being configured to:apply a second FC layer to the normalized-activated feature vector.

Images